Method and system for gesture-based control of device

Abstract

Methods and systems for gesture-based control of a device are described. A virtual gesture space is determined in a received input frame. The virtual gesture space is associated with a primary user in an ordered user list. The received input frame is processed only in the virtual gesture space to detect and track a hand. Gesture classification is performed to determine gesture inputs associated with a hand by detecting and tracking a hand bounding box generated by the hand. A command input associated with the determined gesture input is processed. The device can be a smart television, a smart phone, a tablet computer and the like.

Description

[0001] This application is a divisional application. The application number of the original application is 202080074426.5, and the original application date is March 20, 2020. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present invention relates to gesture detection and recognition, and using gestures to control devices such as televisions or smartphones. Background Art

[0003] There is a need to improve the user experience by detecting gestures in a sequence of frames of digital video captured by a camera of a device (e.g., a smartphone, tablet, television, etc.) through machine vision-based detection (commonly referred to in the art as computer vision) and using the gestures to remotely control such a device. Existing schemes for gesture detection and recognition in a sequence of frames of digital video have always been based on the classification of video actions (i.e., the classification of hand movements detected and tracked in multiple frames of the sequence of frames of the digital video). This usually requires the classification and recognition of gestures on the sequence of frames of the digital video. However, the classification and recognition of gestures on a sequence of digital images is usually computationally expensive and relatively low in accuracy.

[0004] In addition to the challenges of gesture classification and recognition for digital video frames, hand detection in real life also faces challenges. For example, gesture detection for digital video frames is often difficult in complex backgrounds (e.g., in a user's cluttered living space), when the distance between the user and the device is far (e.g., when the user controls a wall-mounted device), or in low-light environments (e.g., at night). Gesture detection and recognition for digital video frames under such conditions often results in unacceptably frequent false positives and false negatives. Another challenge is that when there are multiple people in the detection area, it may cause confusion in detecting and recognizing gestures.

[0005] Machine learning-based methods also face special challenges in classifying and recognizing sequences of frames from digital videos. For example, to achieve satisfactory accuracy, gesture detectors and classifiers for gesture detection and classification can be trained using large deep convolutional neural networks, which usually require a lot of computational resources. In addition, in order to detect smaller hands at greater distances, the frames of digital videos need to have high resolution, which greatly increases the computational cost. Another challenge is that the size of the bounding box generated by the gesture (or hand) detector may not match the expected input of the subsequent classifier.

[0006] Thus, more accurate and computationally less expensive machine learning-based methods and systems are provided for detecting and recognizing gestures in a sequence of frames of a digital video captured by a camera of the device, the gestures being used to remotely control the device, even in non-ideal real-life environments. Summary of the invention

[0007] In various examples, the present invention describes an end-to-end system for detecting and recognizing gestures. The disclosed systems and methods can detect and recognize gestures in real time. Various examples described herein can detect and recognize gestures with higher accuracy than existing methods in long-distance and / or low-light scenarios.

[0008] In some examples, the present invention describes a method for processing gesture input. The method includes: determining a virtual gesture space defined in a received input frame, the virtual gesture space being associated with a primary user in an ordered user list including one or more users; processing the received input frame only in the virtual gesture space to detect and track a hand; performing gesture classification to determine a gesture input associated with the hand by detecting and tracking a hand bounding box generated by the hand; and outputting the determined gesture input to process a command input associated with the determined gesture input.

[0009] In any of the above examples, determining the virtual gesture space may include: processing the input frame to detect the one or more users; generating the ordered user list based on the detected one or more users, the primary user being identified as the highest ranked user in the ordered user list; and generating the virtual gesture space based on the detected anatomical features of the primary user.

[0010] In any of the above examples, processing the input frame may include: selecting a region of interest (ROI) for processing the input frame, the ROI defining an area smaller than the total area of ​​the input frame; wherein the ROI is selected from a defined ROI sequence, the ROI sequence including multiple ROIs for processing a corresponding plurality of sequentially received input frames.

[0011] In any of the above examples, determining the gesture input may include: determining an invalid gesture input associated with a hand detected in the virtual gesture space associated with the primary user; selecting the second highest ranked user in the ordered user list as a new primary user; and repeating the method using the new primary user.

[0012] In any of the above examples, processing the input frame only in the virtual gesture space includes: determining a low-light condition associated with the received input frame; and automatically performing image adjustment to adjust the pixel value of the received input frame according to the low-light condition.

[0013] In any of the above examples, the input frames are processed using a trained joint neural network to detect and track hands, the trained joint neural network comprising a trained gesture classification convolutional neural network having an edge branch connection with a trained bounding box rectification convolutional neural network.

[0014] In any of the above examples, gesture classification may include: identifying a gesture category associated with a hand detected in the received input frame; determining a state transition from a previous gesture state associated with a previous input frame to a current gesture state, wherein the state transition is determined based on the identified gesture category; and determining a gesture input associated with the current gesture state.

[0015] In any of the above examples, the received input frames are received and processed at a frequency that is lower than a frame capture frequency of an image capture device used to capture the received input frames.

[0016] In some examples, the present invention describes an apparatus, comprising: a processing device coupled to a memory, the memory storing machine executable instructions. When the processing device executes the instructions, the apparatus performs any of the above methods.

[0017] In any of the above examples, the apparatus may be a gesture-controlled device.

[0018] In any of the above examples, the apparatus may include a camera for capturing the input frame.

[0019] In any of the above examples, the gesture control device may be one of the following devices: a television; a smart phone; a tablet computer; a vehicle-mounted device; an Internet of Things device; an artificial reality (AR) device; or a virtual reality (VR) device.

[0020] In some examples, the present invention describes a computer-readable medium storing machine-executable instructions. When a processing device of the apparatus executes the machine-executable instructions, the apparatus is caused to perform any of the above methods.

[0021] The examples disclosed herein can achieve more efficient and / or more accurate gesture input detection by detecting only gesture input from the primary user (from an ordered list of users). The present invention also describes an adaptive ROI technique that can improve performance through trained face or hand classifiers. The adaptive ROI also helps to reduce processor requirements. The present invention also describes an automatic image adjustment technique to improve hand detection by adjusting brightness when low light conditions are detected. In some examples, the present invention describes a joint neural network that can improve gesture classification by revising the bounding box of the detected hand. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Reference is now made by way of example to the accompanying drawings of example embodiments of the present application, in which:

[0023] Figure 1 A block diagram of a user interacting with an exemplary gesture control device;

[0024] Figure 2 is a block diagram of some components of an exemplary gesture control device;

[0025] 3A to 3H Some exemplary categories of gestures that may be detected and recognized by an exemplary gesture control device are shown;

[0026] FIG. 4A to FIG. 4C Some exemplary dynamic gestures that may be detected and recognized by an exemplary gesture control device are shown;

[0027] Figure 5 is a block diagram of some details of an exemplary adaptive gesture sensing system that may be implemented in an exemplary gesture control device;

[0028] Figure 6 is a flow chart of an exemplary method for detecting hands using a virtual gesture space;

[0029] Figure 7 An example of a virtual gesture space including a defined subspace for implementing a virtual mouse is shown;

[0030] Fig. 8A and Figure 8B Examples of 2D and 3D virtual gesture spaces are shown;

[0031] Fig. 9 A flowchart of an exemplary method for hand detection and redefining a virtual gesture space based on the detected hand;

[0032] Fig. 10A and Fig. 10B Shows Fig. 9 Exemplary implementations of the method;

[0033] Fig.11is a flow chart of an exemplary method for hand detection using a virtual gesture space and a user list;

[0034] Fig.12 is a flow chart of an exemplary method for detection using adaptive ROI;

[0035] Fig.13 Shows the available Fig.12 Example ROI sequence of methods;

[0036] Fig.14 is a block diagram of an example joint network that can be used for gesture recognition;

[0037] Fig.15 and Fig.16 It shows that it can be used for training Fig.14 Some exemplary training data samples of the joint network of

[0038] Fig.17 is a flow chart of an exemplary method for gesture recognition;

[0039] Fig.18 A state diagram for an example of state-based gesture recognition;

[0040] Fig.19 is a flow chart of an exemplary method for hand detection (including image adjustment);

[0041] Fig. 20 A schematic diagram of an example of image adjustment based on pixel intensity;

[0042] Fig.21A and Fig. 21B An example of image adjustment including adding an illumination portion to displayed content is shown.

[0043] Similar reference numerals may be used in different drawings to represent similar components. DETAILED DESCRIPTION

[0044] In various examples, the present invention describes methods and systems for implementing gesture-based device control. For example, a gesture-controlled device may be a television (e.g., a smart TV), a mobile communication device (e.g., a smartphone), a tablet device, a desktop device, a vehicle-based device (e.g., a dashboard device), or a smart speaker, etc. The gesture-based control described herein may enable a user to interact with any user interface provided by a gesture-controlled device, including a user interface with or without a display output. Examples of the present invention may also be used for augmented reality (AR), virtual reality (VR), or video game applications, etc.

[0045] For simplicity, the present invention describes an example in a gesture-controlled device with a display output (e.g., a smart TV, a smart phone, or a tablet), and describes gesture-based control for interacting with the playback of a video. However, it should be understood that the present application is not limited to these embodiments and can be used for gesture-based control of various devices in various application scenarios.

[0046] Figure 1 An example of a user 10 interacting with a gesture control device 100 is shown. In the simplified figure, the gesture control device 100 includes a camera 102 that captures a sequence of frames (e.g., digital images) in a field-of-view (FOV) 20, the sequence of frames including gesture input. As further described below, the FOV 20 may include at least a portion of the user 10, particularly the face and hands of the user 10. It is worth noting that the FOV 20 in actual use (e.g., outside of a laboratory setting) typically includes more than just the user 10. For example, the FOV 20 may also include other objects, background scenes, or other people that may be present. In addition to the camera 102, the gesture control device 100 may have another sensor that is capable of sensing the gesture input of the user 10, such as any image capture device / sensor (e.g., an infrared image sensor) that captures a sequence of frames (e.g., an infrared image). The sequence of frames includes the gesture input in the FOV 20 of the image capture device / sensor. The gesture control device 100 also includes a display 104 that provides an output (e.g., a video). Although in Figure 1 In the illustrated embodiment, the gesture control device 100 includes a camera 102. In other embodiments, the camera 102 may be a peripheral device that communicates with the gesture control device 100.

[0047] Figure 2 is a block diagram of some components of the gesture control device 100. Although an exemplary embodiment of the gesture control device 100 is shown and described below, other embodiments may be used to implement the examples disclosed herein and may include components different from those shown. Figure 2 A single instance of each component is shown, but there can be multiple instances of each component shown.

[0048] The gesture control device 100 includes one or more processing devices 202, such as a processor, a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated logic circuit, a dedicated artificial intelligence processor unit, or a combination thereof. The gesture control device 100 also includes one or more input / output (I / O) interfaces 204, which are interfaces between input devices such as the camera 102 and output devices such as the display 104. The gesture control device 100 may include other input devices (such as buttons, microphones, touch screens, keyboards, etc.) and other output devices (such as speakers, vibration units, etc.). The camera 102 (or other input devices) may have the ability to capture a sequence of frames including gesture input. The captured frames may be provided by the one or more I / O interfaces 204 to one or more memories 208 for storage (e.g., buffering in the memory) and provided to the one or more processing devices 202 for real-time or near real-time (e.g., within 10ms) processing.

[0049] The processing system 100 may include one or more optional network interfaces 206 for wired or wireless communication with a network (e.g., an intranet, the Internet, a P2P network, a WAN, and / or a LAN) or other nodes. The one or more network interfaces 206 may include wired links (e.g., Ethernet cables) and / or wireless links (e.g., one or more antennas) for intra-network and / or inter-network communication.

[0050] The gesture control device 100 includes one or more memories 208, wherein the one or more memories may include volatile or non-volatile memory (e.g., flash memory, random access memory (RAM) and / or read-only memory (ROM)). The one or more non-transitory memories 208 may store instructions executed by the one or more processing devices 202, for example, to perform the examples described in the present invention. For example, the one or more memories 208 may include instructions for executing the adaptive gesture sensing system 300. The one or more memories 208 may include other software instructions, such as for implementing an operating system and other applications / functionality. The one or more memories 208 may also include data 210, such as a video content file that may be provided as an output via the display 104.

[0051] In some examples, the gesture control device 100 may also include one or more electronic storage units (not shown), such as a solid state drive, a hard disk drive, a magnetic disk drive, and / or an optical disk drive. In some examples, one or more data sets and / or modules may be provided by an external memory (e.g., an external drive in wired or wireless communication with the gesture control device 100), or by a transient or non-transient computer-readable medium. Examples of non-transient computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable memory. For example, the components of the gesture control device 100 may communicate with each other via a bus.

[0052] To help understand the present invention, gestures are first described. In the present invention, gestures are generally defined as different hand shapes that can be recognized by the gesture control device 100 as specific command inputs. Gestures can have different shapes and movements. For example, the gesture control device 100 can recognize a gesture as belonging to 3A to 3H The image captured by the camera 102 is one of the following categories. 3A to 3H The gestures shown are called gesture inputs.

[0053] Figure 3A An "open hand" gesture 30 is shown. Figure 3B A "fist" (or "clenched hand") gesture 32 is shown. Figure 3C A “zoom” gesture 34 is shown. Figure 3D A "pinch" gesture 36 is shown. Figure 3E A “mute” (or “silence”) gesture 38 is shown. Figure 3F A "like" (or "agree") gesture 40 is shown. Figure 3G An "other" (or "next") gesture 42 is shown. Figure 3H A “touch” (or “select”) gesture 44 is shown. Other gesture categories may be recognized by the gesture control device 100 .

[0054] Based on these gesture categories, gestures can be classified as static gestures or dynamic gestures. A static gesture is defined as a single gesture category that is maintained in a generally fixed position (e.g., within a specific area that allows for certain margins of error) for at least a specific period of time (e.g., 1 second) or at least a specific number of consecutive frames (e.g., 100 frames) in a sequence of frames captured by the camera 102. For example, a static open hand gesture 30 can be recognized by the gesture control device 100 (e.g., the adaptive gesture sensing system 300) and can be interpreted as a pause command input in video playback. A static fist gesture 32 can be recognized by the gesture control device 100 and can be interpreted as a stop command input in video playback.

[0055] A dynamic gesture is defined as a combination of one or more gesture categories, positions, and / or movements. For example, a dynamic gesture may be a single gesture category that changes in position over time (e.g., detected at different positions over a sequence of frames captured by the camera 102). The position-changing open gesture 30 may be recognized by the gesture control device 100 (e.g., the adaptive gesture sensing system 300) and interpreted as a command to drag or move an interface object (e.g., a displayed icon).

[0056] Figure 4A Another type of dynamic gesture is shown, which can be a combination of gesture categories. Figure 4A In the example, the dynamic gesture includes a combination of a touch gesture 44 followed by a fist gesture 32. The dynamic gesture can be recognized by the gesture control device 100 (e.g., the adaptive gesture sensing system 300 running on the gesture control device 100) and interpreted as a command input for selecting and then clicking an interface object using a mouse device. In another example, a zoom gesture 34 and a subsequent pinch gesture 36 can be recognized together as a "zoom" dynamic gesture, which can be recognized by the gesture control device 100 and interpreted as a zoom-out command input.

[0057] More complex dynamic gestures may involve a combination of gesture categories as well as changes in position. Figure 4A In the embodiment of the present invention, if the position of the touch gesture 44 is detected to change over time (for example, detected in different positions in a sequence of frames captured by the camera device 102), followed by a fist gesture 32, the gesture control device 100 (for example, the adaptive gesture sensing system 300 running on the gesture control device 100) can interpret the complex dynamic gesture as a command input and move the displayed cursor in a manner that reflects the change in position of the touch gesture 44 (followed by a click command) when the fist gesture 32 is detected.

[0058] In another example, a zoom gesture 36, followed by a pinch gesture 38, followed by a change in position of the pinch gesture 38, and then the zoom gesture 36 can again be recognized by the gesture control device 100 (e.g., the adaptive gesture sensing system 300 running on the gesture control device 100) together as a dynamic "pinch-drag-release" gesture. It should be noted that dynamic gestures that include changes in position can be interpreted as different inputs depending on the specific change in position. For example, Figure 4B As shown, a pinch-drag-release gesture that vertically changes position 46 (or vertical "drag") of the pinch gesture 38 can be interpreted as a command input to modify the volume during video playback. Figure 4C As shown, the pinch-drag-release gesture that changes horizontally at the position 48 of the pinch gesture 38 can be interpreted as a command input to move forward or backward in the video during video playback. The pinch-drag-release gesture can provide an intuitive and convenient way for the user to interact with the gesture control device 100, and can also enable the gesture control device 100 (e.g., the adaptive gesture sensing system 300 running on the gesture control device 100) to detect and recognize gesture input with relatively high accuracy. Specifically, the pinch-drag-release gesture can be detected and recognized by decomposing a dynamic gesture into static gestures (e.g., a pinch gesture 36, followed by a pinch gesture 38, followed by another pinch gesture 36).

[0059] Some conventional existing computer vision techniques for gesture recognition perform hand detection, hand shape classification, and gesture recognition based on the hand shape and position in the frame sequence (e.g., digital image) captured by the camera 102. Gesture classification and recognition are usually computationally expensive. In addition, when the camera 102 of the gesture control device 100 is in an uncontrollable and potentially complex environment (e.g., a cluttered background, when the user is far away from the device, in a low-light environment, or when there are multiple people in the FOV), it is usually difficult to process the frame sequence captured by the camera 102 to detect and recognize gestures (whether static or dynamic).

[0060] In various examples, the present invention describes a more accurate and efficient gesture detection and recognition scheme in a sequence of frames captured by a camera of a gesture control device. In some examples, the present invention describes an adaptive gesture sensing system using a virtual gesture space to reduce the area of ​​each frame in a sequence of frames captured by a camera to be processed for gesture detection. In some examples, the present invention also describes a method for recognizing dynamic gestures based on state (rather than motion). It should be understood that the present invention includes other aspects and features.

[0061] The present invention describes the use of a virtual gesture space. The virtual gesture space is defined as at least the space around the user's face, which is used to process each frame in a sequence of frames captured by a camera to detect gestures in the sequence of frames. The virtual gesture space should be smaller than the FOV 20 captured by the camera 102. Only gestures (e.g., gesture inputs) detected and recognized within the virtual gesture space are considered valid gestures (e.g., gesture inputs). The use of a virtual gesture space can reduce false positives when detecting gestures in a sequence of frames (e.g., particularly in complex environments), make it easier to associate gestures with specific users, and can more efficiently process a sequence of frames to detect and recognize gestures (e.g., gesture inputs).

[0062] Figure 5 3 is a block diagram of some exemplary subsystems of the adaptive gesture sensing system 300. In this example, the adaptive gesture sensing system 300 can be implemented by a virtual gesture space subsystem 310 and a gesture parsing subsystem 320. In the following example, the adaptive gesture sensing system 300 is described as including two subsystems 310, 320 or providing the functions of two subsystems 310, 320. However, in other examples, the adaptive gesture sensing system 300 may include only one of the subsystems 310, 320 (or provide the functions of one of the subsystems). For example, the adaptive gesture sensing system 300 may (e.g., using the virtual gesture space subsystem 310) only provide adaptive generation of a virtual gesture space and gesture detection, wherein gesture detection is detection of a gesture within the virtual gesture space in a sequence of frames captured by the camera 102 of the gesture control device 100. Another component of the gesture control device 100 (e.g., using any suitable existing gesture recognition technology) may perform gesture recognition and parsing. In another example, the adaptive gesture sensing system 300 may (eg, using the gesture parsing subsystem 320 ) only provide gesture recognition and management for multiple users, and another component of the gesture control device 100 may perform gesture detection (eg, using any suitable existing gesture detection technology).

[0063] In some examples, the adaptive gesture sensing system 300 may not include the different subsystems 310, 320. Instead, sub-blocks of the subsystems 310, 320 may be considered sub-blocks of the adaptive gesture sensing system 300 itself. Therefore, implementation of the adaptive gesture sensing system 300 using the different subsystems 310, 320 is optional.

[0064] The adaptive gesture sensing system 300 includes a face detection and tracking subsystem 312, a virtual gesture space generation subsystem 314, a hand detection and tracking subsystem 316, a gesture recognition subsystem 322, and a user list 324. The face detection and tracking subsystem 312, the virtual gesture space generation subsystem 314, and the hand detection and tracking subsystem 316 may be part of the virtual gesture space subsystem 310. The gesture recognition subsystem 322 and the user list 324 may be part of the gesture parsing subsystem 320.

[0065] In some examples, instead of or in addition to the face detection and tracking subsystem 312, another subsystem (not shown) can be used to detect and track different anatomical features (e.g., a whole human body or a human torso) in the frame sequence captured by the camera 102. As further described below, instead of or in addition to the face, different anatomical features can be used as the basis for generating the virtual gesture space. For simplicity, the present invention focuses on the use of face detection and tracking in the frame sequence captured by the camera 102, but it should be understood that the present invention is not limited to this.

[0066] The frames in the frame sequence captured by the camera 1020 are received and used as input frames of the adaptive gesture sensing system 300. The face detection and tracking subsystem 312 performs face detection on the input frame. The face detection and tracking subsystem 312 can use any suitable face detection technology to detect the face in the input frame and generate a bounding box for the detected face. The bounding box can be a two-dimensional (2D) or three-dimensional (3D) box centered on and around the face detected in the input frame.

[0067] The virtual gesture space generation subsystem 314 defines a virtual gesture space using a bounding box generated for a face detected in the input frame. In the present invention, a virtual gesture space (or simply gesture space) refers to a 2D or 3D space defined in the input frame and mapped to a virtual space in the real environment of the user 10, in which gestures (e.g., gesture input) can be detected. That is, the user 10 can make gestures within a virtually defined 2D or 3D virtual gesture space to provide command input to the gesture control device 100. Gestures performed outside the virtual gesture space may not be detected and may not be recognized as command input by the gesture control device 100. The dimensions of the virtual gesture space may or may not match the dimensions of the bounding box of the face (e.g., the bounding box of the face may be 2D and the virtual gesture space may be 3D).

[0068] The virtual gesture space defined by the virtual gesture space generation subsystem 314 is used by the hand detection and tracking subsystem 316 to detect and track hands in a sequence of frames captured by a camera. Specifically, the hand detection and tracking subsystem 316 may analyze only the virtual gesture space defined within the input frame and each subsequent frame in the sequence of frames to detect the hand and track the hand in the sequence of frames. The hand detection and tracking subsystem 316 may use any suitable hand detection technique to detect the hand in the input frame and define a 2D or 3D bounding box for the detected hand.

[0069] In some examples, the face detection and tracking subsystem 312 may include a trained neural network for performing face detection. Similarly, the hand detection and tracking subsystem 316 may include another trained neural network for performing hand detection. For example, a suitable trained neural network for face or hand detection may be a trained object detector, such as YOLOv3 based on a residual neural network (ResNet) architecture (e.g., as described in "YOLO v3: An incremental improvement" (arXiv preprint arXiv:1804.02767, 2018) by Redmon et al. in the 2018 preprint document library 1804.02767). The ResNet architecture, such as ResNet34 (e.g., as described in "Deep residual learning for image recognition" (Deep residual learning for image recognition) by He Kaiming et al. in the 2016 IEEE Conference on Computer Vision and Pattern Recognition). Another example of a suitable trained neural network for face or hand detection can be a trained single shot detector (SSD) based on a convolutional neural network (CNN) architecture, such as a single shot multibox detector (e.g., as described in "Ssd: Singleshot multibox detector" by Liu et al. in the 2016 Springer Cham European International Conference on Computer Vision). The CNN architecture, such as MobileNetV2 (e.g., as described in "Mobilenetv2: Inverted residuals and linear bottlenecks" by Sandra et al. in the 2018 IEEE Conference on Computer Vision and Pattern Recognition). The face detection and tracking subsystem 312 and the hand detection and tracking subsystem 316 can use the Lucas-Kanad optical flow technique (as described in "Aniterative image registration technique with an application to stereo vision" proposed by Lucas et al. in the 1981 Imaging Understanding Workshop Conference Papers) to perform face and hand tracking respectively.

[0070] The gesture recognition subsystem 322 uses the bounding box defined for the detected hand to recognize the shape of the detected hand and classify it into a gesture category. The gesture recognition subsystem 322 can use any suitable classification technique to classify the shape of the detected hand into a specific gesture category. For example, the gesture recognition subsystem 322 may include a neural network (e.g., a CNN) that has been trained to classify gestures according to a predefined set of gesture categories. The neural network has been trained with a known machine learning algorithm to learn the parameters (e.g., weights) of the neural network. The trained neural network receives the bounding box of the detected gesture and predicts a specific gesture category from the set of predefined gesture categories corresponding to the bounding box. The gesture category predicted by the gesture recognition subsystem 322 can be output by the adaptive gesture sensing system 300 (e.g., as a label).

[0071] The software application (e.g., operating system) of the gesture control device 100 may convert the gesture category output by the adaptive gesture sensing system 300 into a command input. The conversion of the gesture category into a command input may depend on the application. For example, when a first application is active, a given gesture category may be converted into a first command input, but when a second application is active, it may be converted into a second command input (or may be considered invalid).

[0072] In some examples, the adaptive gesture sensing system 300 stores and maintains the user list 324 to track the primary user. For example, there may be multiple people in the FOV 20 of the camera 102. Therefore, the face detection and tracking subsystem 312 may detect and track multiple faces in the frame sequence captured by the camera 102 in the FOV 20. Each detected face may be a user providing input to the gesture control device 100. Therefore, each detected person may be considered a user (or potential user) even if the detected person is not currently providing a recognized gesture (e.g., gesture input). The user list 324 tracks all detected users and sorts the detected users according to some predefined criteria (e.g., according to the size of the user, the distance of the user from the gesture control device 100, whether the user is looking at the gesture control device 100, etc.). The user with the highest sorting in the user list 324 may be considered the primary user. A gesture (e.g., gesture input) in a frame sequence associated with the primary user has a higher priority than any other gesture (e.g., gesture input) detected in a frame sequence associated with other detected users. In some examples, the user list 324 may include only authorized or pre-registered users. For example, a user profile may be associated with an authorized or pre-registered user, and the user profile may include data (e.g., biometric data) that enables the authorized or pre-registered user to be identified by the gesture control device 100 (e.g., using suitable facial recognition technology). Such facial recognition of the authorized or pre-registered user may be performed by the adaptive gesture sensing system 300 or by a separate facial recognition system of the gesture control device 100. By limiting the user list 324 to only authorized or pre-registered users, unauthorized gesture-based control of the device 100 may be avoided. In addition, false positive detections of gesture inputs may be reduced.

[0073] Although the adaptive gesture sensing system 300 is shown as having different sub-blocks (or subsystems), it should be understood that the present invention is not limited thereto. For example, the adaptive gesture sensing system 300 may be implemented using a greater or lesser number of sub-blocks (or subsystems), or may not require any sub-blocks (or subsystems). In addition, the functions described herein as being performed by a particular sub-block (or subsystem) may be performed by another sub-block (or subsystem).

[0074] Generally, the functions of the adaptive gesture sensing system 300 can be implemented in various suitable ways, all of which fall within the scope of protection of the present invention.

[0075] An example of the operation of the adaptive gesture sensing system 300 is described below.

[0076] Figure 66 is a flow chart of an exemplary method 600 that may be performed by the adaptive gesture sensing system 300, for example using the virtual gesture space subsystem 310 (and subsystems 312, 314, and 316). The method 600 may be performed by a program or subroutine of software of the processing device 202 of the gesture control device 100. The coding of the software for executing these programs or subroutines of the present invention is within the scope of ordinary skill in the art. The method 600 may include additional or fewer steps or actions shown and described, and may be performed in a different order. For example, the computer readable code executed by the processing device 202 may be stored in a computer readable medium.

[0077] 602: Receive an input frame in a sequence of frames. Typically, the input frame and each subsequent frame of the input frame of the sequence of frames are received one at a time in real time or near real time. The input frame (and each subsequent frame) may be a raw, unprocessed digital image captured by the camera 102, or may be a digital image that has been minimally processed (e.g., normalized).

[0078] 604: The adaptive gesture sensing system 300 detects different anatomical features in the input frame. The entire input frame can be processed in step 604. The different anatomical features can be any part of the user's body that is easily detected and distinguished from the background. One example is face detection (e.g., using the face detection and tracking subsystem 312). In some cases, faces may be difficult to detect, in which case different anatomical features (e.g., an entire human body or a human torso) can be detected. As described above, the anatomical features can be detected using any suitable method, including using any suitable computer vision technology. Detecting the different anatomical features can include determining the location of the anatomical features (e.g., represented by a bounding box or coordinates).

[0079] 606: Generate a virtual gesture space based on the detected different anatomical features (e.g., detected faces) (e.g., using the virtual gesture space generation subsystem 314). In some examples, multiple instances of different anatomical features may be detected (e.g., if there are multiple people within the input frame in the FOV 20 captured by the camera 102, multiple faces may be detected). In this case, a virtual gesture space may be generated for each corresponding detected instance of the different anatomical feature. As further described below, in some examples, when multiple instances of different anatomical features are detected, only one virtual gesture space may be generated, or multiple virtual gesture spaces may be generated based on the order or priority of the detected different anatomical feature instances.

[0080] The virtual gesture space can be generated using predefined equations related to the positions of corresponding anatomical features detected in the input frame. For example, the virtual gesture space can be generated by calculating a rectangular space relative to a bounding box of a detected face. Some exemplary equations are further provided below.

[0081] 608: (Optional) Information about the generated virtual gesture space may be provided by the adaptive gesture sensing system 300 to enable the gesture control device 100 to provide feedback to the user 10 regarding the generated virtual gesture space. For example, the adaptive gesture sensing system 300 may provide information representing coordinates or other parameters of the virtual gesture space to enable the gesture control device 100 to render (e.g., overlaid on top of a real-time camera image) the virtual gesture space represented to the user 10 on the display 104. In another example, the virtual gesture space may be represented to the user 10 by having the gesture control device 100 render an inline window or auxiliary window on the display 104 to display only the FOV corresponding to the virtual gesture space. Other ways of providing feedback to the user 10 are also applicable.

[0082] 610: Detect a hand in an input frame in the generated virtual gesture space (e.g., using the hand detection and tracking subsystem 316). The detected hand may be associated (e.g., labeled) with the corresponding virtual gesture space in the input frame in which the hand was detected. If multiple virtual gesture spaces are generated, an attempt may be made to detect a hand in each generated virtual gesture space in the input frame. If a hand is not detected in a given virtual gesture space in the input frame, the given virtual gesture space may be ignored or discarded.

[0083] If a hand is not detected in any of the multiple generated virtual gesture spaces in the input frame, it can be determined that no gesture (e.g., gesture input) is found in the input frame, and the method 600 can return to step 602 to receive the next input frame in the frame sequence. Assuming that at least one hand is detected in at least one virtual gesture space, the method 600 performs optional step 612.

[0084] 612: (Optional) If more than one hand is detected in a given virtual gesture space in the input frame, a dominant hand may be identified in the given virtual gesture space. The dominant hand may be identified based on, for example, the largest hand in the given virtual gesture space in the input frame, the detected hand closest to a detected distinct anatomical feature (e.g., a face) in the given virtual gesture space in the input frame, or the detected hand closest in brightness and / or hue to a detected distinct anatomical feature (e.g., a face) in the given virtual gesture space in the input frame, etc. If only one hand is detected in the given virtual gesture space in the input frame, it may be assumed that the hand is the dominant hand.

[0085] 614: Track the detected hand (or master hand) in a corresponding virtual gesture space (e.g., using the hand detection and tracking subsystem 316) in subsequent frames of the input frame of the sequence of frames. Track the detected hand (or master hand) by processing each subsequent frame. Provide information about the hand (or master hand) detected and tracked in each subsequent frame for further analysis and parsing. For example, a bounding box and optional identifier can be generated to track the detected hand in subsequent frames. The bounding box (and optional identifier) ​​can then be provided (e.g., to the gesture recognition subsystem 322) for analysis and parsing.

[0086] In some examples, the method 600 may be implemented by the adaptive gesture sensing system 300 using only the virtual gesture space subsystem 310. The adaptive gesture sensing system 300 does not output gesture categories (e.g., Figure 5 ), but instead outputs the tracked hand (e.g., bounding box) information to a traditional gesture recognition system, which can perform hand classification and gesture recognition on the information (e.g., bounding box).

[0087] The above example does not detect the user's hand directly, but first detects different anatomical features in the input frame (such as the user's face) and generates a virtual gesture space (smaller than the FOV in the input frame) based on the detected features. Then, the hand in the input frame is detected only in the virtual gesture space. The user's face can be used as a different anatomical feature to generate the virtual gesture space, because face detection is generally more accurate and reliable than hand detection. By restricting hand detection to the virtual gesture space, the processing required for subsequent frames for detecting hands can be simplified, false positives can be reduced, and it is easier to identify the main hand in subsequent frames for gesture input.

[0088] In some examples, the method 600 may be used to process every frame captured by the camera 102. In other examples, the method 600 may only be used when gesture input is expected. For example, the method 600 may be initiated when input is received (e.g., via keyboard input, mouse input, or voice input). In some examples, the method 600 may be initiated when human attention is detected. For example, the gesture control device 100 may use attention detection technology (e.g., may execute eye tracking software) to determine whether a person is looking directly at the gesture control device 100, and may initiate the method 600 only when a person is detected looking directly at the gesture control device 100. Initiating the method 600 upon detected human attention helps avoid false positives or misinterpretations of gesture input.

[0089] Figure 7An exemplary implementation of the method 600 is shown. In this implementation, a virtual gesture space is generated based on a detected face. In this example, it is assumed that a face 12 of a primary user is detected at 604, and the face 12 is used as a different anatomical feature to generate a virtual gesture space at 606.

[0090] Any suitable face detection technique, including computer vision-based techniques as described above, may be used to generate a bounding box 702 for the face 12. In this example, the bounding box 702 is composed of a set of values ​​{x f ,y f ,w f ,h f} definition, where x f and f are the x and y coordinates of the anchor point (eg, center) of the bounding box 702 (in the reference frame defined by the adaptive gesture sensing system 300 ), respectively, and w f and h f are the width and height of the bounding box 702, respectively. A virtual gesture space 704 is generated based on the bounding box 702 (for example, in step 606), and the virtual gesture space 704 is composed of a set of values ​​{x g ,y g ,w g h g} definition, where x g and g are the x and y coordinates of the anchor point (eg, center) of the virtual gesture space 704 (in the reference frame defined by the adaptive gesture sensing system 300 ), respectively. g and h g are respectively the width and height of the virtual gesture space 704. For example, the following equation can be used to generate the virtual gesture space 704:

[0091]

[0092] h g =β·h f

[0093] Among them, (Δ x ,Δ y ) is a predefined relative position parameter, The parameter (Δ ) may be predefined (eg, by the user 10 or the manufacturer of the gesture control device 100) according to the desired size of the virtual gesture space 704 and the desired position of the bounding box 702 within the virtual gesture space 704. x ,Δ y )and It should be noted that in some examples, the virtual gesture space 704 may be generated in such a way that the bounding box 702 of the face 12 is partially or entirely outside the virtual gesture space 704. That is, although the face 12 may be used to generate the virtual gesture space 704, the virtual gesture space 704 need not include the face 12.

[0094] exist Figure 7 In the example of , the virtual gesture space 704 is generated as a 2D space having a rectangular shape. However, it should be understood that the virtual gesture space 704 can be generated as a 2D space or a 3D space, and can be generated to have any geometric shape (e.g., square, rectangle, circle, etc.), regular shape, or irregular shape.

[0095] In some examples, the virtual gesture space 704 may also define one or more subspaces 706, 708 for implementing specific input functions. The multiple subspaces 706, 708 may be defined based on the features of the detected face 12 (e.g., defined by the virtual gesture space generation subsystem 314). For example, a left subspace 706 and a right subspace 708 may be defined in the virtual gesture space corresponding to the lower left and lower right corners of the detected face 12 based on the detected eye and nose positions of the face 12. Detecting a touch (or point) gesture in the left subspace 706 may be interpreted as a left-button click input of a mouse. Similarly, detecting a touch (or point) gesture in the right subspace 708 may be interpreted as a right-button click input of a mouse. In this way, the virtual gesture space 704 including the defined subspaces 706, 708 may be used to implement a virtual mouse. In some examples, Figure 4A The dynamic gestures shown can be detected as gesture inputs to implement a virtual cursor. When the touch (or point) gesture 44 is tracked, the virtual cursor can be moved (and such movement can be represented by a visual overlay displayed to the user 10), and when the squeeze gesture 32 is detected, a mouse click input can be detected. In this way, a virtual mouse can be implemented.

[0096] Fig. 8A and Figure 8B Another example of a virtual gesture space 704 generated based on the face 12 of the user 10 detected in the input frame in the sequence of frames captured by the camera 102 is shown. Fig. 8A In the embodiment, the virtual gesture space 704 is a 2D space and has a rectangular shape (e.g., Figure 7 Similar examples). Figure 8BIn the embodiment, the virtual gesture space 704 is a 3D space including the face 12 and the hand 14 of the user 10. When depth information is available, the virtual gesture space 704 can be generated as a 3D space. For example, the depth information can be calculated based on the frame sequence using video analysis technology. For example, the gesture control device 100 can use two cameras 102 to calculate the depth information (for example, using computer stereo vision technology), or the camera 102 can be an RGB depth (RGB-depth, RGBD) camera capable of generating depth information (in addition to traditional RGB image information), or the gesture control device 100 can include a time of flight (time of flight, ToF) camera in addition to the traditional camera 102 to obtain RGB image information and other corresponding depth information. In some examples, in addition to the camera 102, a sensor capable of capturing depth information (such as an infrared sensor) can also be used to capture depth information. The use of the 3D virtual gesture space 704 can help detect and recognize depth-based gesture inputs (such as moving a hand near or away from the gesture control device 100) as command inputs.

[0097] In some examples, a virtual gesture space is initially generated based on different anatomical features (e.g., faces) detected in an input frame of a sequence of frames, and can then be redefined or updated based on a hand detected in a subsequent frame of the input frame in the sequence of frames. In this way, the virtual gesture space can follow the position of the detected hand even if the hand moves away from the different anatomical features.

[0098] Fig. 9 Flowchart of an exemplary method 900 that may be performed by the adaptive gesture sensing system 300, for example using the virtual gesture space subsystem 310 (and subsystems 312, 314, and 316). The method 900 may be performed by a program or subroutine of software of the processing device 202 of the gesture control device 100. The coding of the software for executing these programs or subroutines of the present invention is within the scope of ordinary skill in the art. The method 900 may include other or fewer steps or actions shown and described, and may be performed in a different order. For example, the computer readable code executed by the processing device 202 may be stored in a computer readable medium. The method 900 may include steps similar to those described in the method 600 above, and therefore will not be repeated.

[0099] 902: Receive an input frame of a frame sequence. This step may be similar to the above step 602.

[0100] 904: The adaptive gesture sensing system 300 detects different anatomical features in the input frame. This step may be similar to step 604 above.

[0101] 906: Generate a virtual gesture space based on different anatomical features detected in the input frame (e.g., a detected face) (e.g., using the virtual gesture space generation subsystem 314). The step can be similar to the above step 606. Optionally, information representing the virtual gesture space can be provided (e.g., providing information to provide feedback to the user 10). For simplicity, in the following description of the method 900, it is assumed that only one virtual gesture space is generated. However, it should be understood that the method 900 can be applicable to the case of generating multiple virtual gesture spaces (e.g., based on multiple detection instances of different anatomical features in the input frame).

[0102] 908: Detect a hand in the virtual gesture space of the input frame (e.g., using the hand detection and tracking subsystem 316), and associate with the virtual gesture space where the hand was detected. This step may be similar to step 610 above. Optionally, a master hand may be identified and associated with the virtual gesture space.

[0103] 910: Redefine the virtual gesture space based on the detected hand. Redefining the virtual gesture space may include recalculating the position and / or size of the virtual gesture space in subsequent frames of the input frame using predefined equations associated with the detected hand (rather than associated with the detected anatomical features). For example, the virtual gesture space may be redefined so that the virtual gesture space is centered around the bounding box of the detected hand. Some exemplary equations are further described below.

[0104] 912: Storing the redefined virtual gesture space. This enables the redefined virtual gesture space (redefined based on the detected hand) to be used for detecting and tracking the hand in subsequent frames of the input frame, instead of using the initially generated virtual gesture space (initially generated based on different anatomical features detected such as a face detected in the input frame).

[0105] 914: Track the hand (or master hand) detected in the input frame to the input frame in the redefined virtual gesture space in subsequent frames (e.g., using the hand detection and tracking subsystem 316). This step can be similar to step 614 above. The bounding box (and optional identifier) ​​can be provided (e.g., to the gesture recognition subsystem 322 or other hand classifier) ​​to analyze and interpret the gesture input.

[0106] The method 900 can redefine the virtual gesture space based on the hand detected in the input frame so that the hand can continue to be tracked and detected in each subsequent frame of the input frame even if the hand moves away from a different anatomical feature. After the virtual gesture space is redefined based on the detected hand, the subsequent frames can be processed using the redefined virtual gesture space. When the position of the hand in the space changes, the virtual gesture space can be continuously redefined in each subsequent frame so that when the hand moves, the virtual gesture space continues to be centered on the detected hand. For example, a similar method of the method 900 can be used to process subsequent frames of the input frame in the frame sequence without steps 904 and 906.

[0107] In some examples, if the hand is no longer detected in the redefined virtual gesture space of a subsequent frame, the virtual gesture space can be regenerated based on different anatomical features detected in the input frame. That is, different anatomical features can be used as anchors or default bases for defining the virtual gesture space. In some examples, the use of the different anatomical features as the default basis for defining the virtual gesture space is restored only when more than a predefined number of subsequent frames (e.g., at least 10 frames after the input frame) cannot be detected in the redefined virtual gesture space.

[0108] Fig. 10A and Fig. 10B 900. In this implementation, the virtual gesture space 704 is initially generated based on the face 12 of the user 10 detected in the input frame of the frame sequence, and then the virtual gesture space 704 is redefined based on the hand 14 of the user 10 detected in the input frame. Fig. 10A In the example, the virtual gesture space 704 is a 2D space having a rectangular shape (e.g., Figure 7 7). Detect the hand 14 in the virtual gesture space 704 of the input frame and define a bounding box 706 of the hand 14. Fig. 10B , the virtual gesture space 704 b is redefined based on the bounding box 706 of the hand 14 .

[0109] For example, the bounding box 706 of the hand 14 may be represented by a set of values ​​{x h ,y h ,w h ,h h} definition, where x h and h are the x and y coordinates of the anchor point (eg, center) of the bounding box 706 (in the reference frame defined by the adaptive gesture sensing system 300 ), respectively, and w h and h hare the width and height of the bounding box 706, respectively. The virtual gesture space 704b is redefined based on the bounding box 706 (e.g., at step 910). For example, the following equation can be used to redefine the virtual gesture space 704b:

[0110]

[0111] h g =β h ·h h

[0112] Among them, {x g ,y g ,w g , h g} is a parameter defining the virtual gesture space 704b as described above, (Δ xh ,Δ yh ) is a predefined relative position parameter, The parameter (Δ ) may be predefined (e.g., by the user 10 or the manufacturer of the gesture control device 100) according to the desired size of the redefined virtual gesture space 704b and the desired position of the bounding box 706 within the redefined virtual gesture space 704b. xh ,Δ yh )and It is worth noting that Fig. 10B As shown in the example of , the face 12 can be partially or completely removed from the redefined virtual gesture space 704b.

[0113] In some examples, when multiple persons are detected in the input frame (e.g., multiple persons are detected by the face detection and tracking subsystem 312), the adaptive gesture sensing system 300 can implement the user list 324 to sort the detected persons and identify one person as the primary user (or primary controller). The virtual gesture space can be generated only for the primary user.

[0114] Fig.11Flowchart of an exemplary method 1100 that may be performed by the adaptive gesture sensing system 300, for example using the virtual gesture space subsystem 310 and the gesture parsing subsystem 320 (and subsystems 312, 314, 316, 322, 324). The method 1100 may be performed by a program or subroutine of software of the processing device 202 of the gesture control device 100. The coding of the software for executing these programs or subroutines of the present invention is within the scope of ordinary skill in the art. The method 1100 may include other or fewer steps or actions shown and described, and may be performed in a different order. For example, the computer readable code executed by the processing device 202 may be stored in a computer readable medium. The method 1100 may include steps similar to those described in the method 600 above, and therefore will not be repeated.

[0115] 1102: Receive an input frame in a frame sequence. This step may be similar to the above step 602.

[0116] 1104: Determine whether the primary user has been identified and selected. For example, the primary user may have been identified and selected from the previous input frame of the analyzed frame sequence. If the primary user has been selected, the method 1100 performs step 1106; otherwise, the method 1100 performs step 1108.

[0117] 1106: Detect different anatomical features (e.g., a face) of the primary user in the input frame and track them to subsequent frames of the input frame in the frame sequence (e.g., using the face detection and tracking subsystem 312). If the primary user is identified in the previous input frame, a virtual gesture space may have been defined for the primary user. In this case, the anatomical features in the input frame and subsequent frames of the input frame can be detected and tracked by processing each subsequent frame only in the defined virtual gesture space. The anatomical features are tracked by processing each subsequent frame, but the detection can be performed at a lower frequency (e.g., skipping one or more subsequent frames). The anatomical features can be detected to correct tracking errors (e.g., tracking errors accumulate when tracking multiple subsequent frames). It should be noted that although the detection and tracking of different anatomical features of the primary user (e.g., a face) is described herein, the detection and tracking of different anatomical features of the primary user (e.g., a face) may also be performed based on the detection of a hand in the input frame (e.g., as described above). Fig. 10A and 10B In the case where the virtual gesture space is redefined as described above, the modification of step 1106 can be performed and the input frame and subsequent frames can be processed to detect and track the hand of the primary user.

[0118] 1108: If a primary user has not been previously selected, the entire input frame may be processed using the face detection and tracking subsystem 312, etc. to detect multiple instances of different anatomical features (e.g., all instances of a face). A bounding box and identifier may be generated for each detected anatomical feature. If multiple instances of different anatomical features are not detected in the input frame, the method 1100 may return to step 1102 to process the next input frame in the sequence of frames.

[0119] 1110: Generate an ordered list of users 324 based on multiple instances of the detected anatomical feature. Each detected instance can be associated with a corresponding detected user. The list of users 324 can be sorted according to a predefined sorting criteria by further analyzing the detected anatomical feature (e.g., using any suitable face analysis technology, such as any eye tracking technology). For example, if the anatomical feature is a face, the sorting criteria can specify that larger faces (assuming that the face is closer to the gesture control device 100) are ranked higher than smaller faces; faces facing the gesture control device 100 are ranked higher than faces away from the gesture control device 100; or faces with eyes looking at the gesture control device 100 are ranked higher than faces with eyes away from the gesture control device 100. Other sorting criteria can be used. If only one instance of a different anatomical feature is detected, then by default that instance may be ranked highest in the list.

[0120] 1112: Select the highest ranked user in the ordered user list 324 as the primary user. If there is only one user in the user list 324, the user may be selected as the primary user by default. Then, the method 1100 performs step 1106 to track the anatomical features of the primary user in the frame sequence captured by the camera 102.

[0121] 1114: Generate a virtual gesture space for the primary user. If the virtual gesture space has been previously defined (based on analysis of a previous frame captured by camera 102), the defined virtual gesture space may be used instead of generating the virtual gesture space. Otherwise, the virtual gesture space is generated based on different anatomical features (e.g., a face) of the primary user detected in the input frame of the frame sequence (e.g., similar to step 606 described above). Optionally, information representing the virtual gesture space may be provided (e.g., providing information to provide feedback to the primary user).

[0122] 1116: Detect (e.g., using the hand detection and tracking subsystem 316) a hand in the virtual gesture space of the input frame of the sequence of frames. This step may be similar to step 610 above. Optionally, if more than one hand is detected in the virtual gesture space in the input frame, a dominant hand may be identified (e.g., based on some predefined criteria such as hand size) and associated with the virtual gesture space. Optionally, the virtual gesture space may be redefined based on the hands detected in the input frame (as described above). Fig. 9 If the virtual gesture space has been redefined in a subsequent frame of the input frame based on the hand detected in the input frame, detection and tracking may have been performed in step 1106, and this step may not be performed.

[0123] 1118: Perform gesture recognition (e.g., using the gesture recognition subsystem 322) on the detected hand (e.g., defined by the bounding box) and track the hand in the input frame of the frame sequence to predict the gesture category of the detected hand. Output the predicted gesture category (if a valid gesture is recognized) for conversion into command input for a software application or the like. Any non-primary users in the user list 324 are discarded (i.e., only the primary user is retained), and the method 1100 returns to step 1102 to receive and process the next input frame in the frame sequence captured by the camera 102.

[0124] If gesture recognition cannot predict a valid gesture (e.g., cannot classify the hand shape into any predefined gesture category), or if the gesture is not a valid gesture for the currently activated software application (e.g., the software application reports invalid input to the adaptive gesture sensing system 300), the method 1100 executes step 1120.

[0125] 1120 : Discard the currently selected primary user from the user list 324 .

[0126] 1122: Determine whether there are any other users in the user list 324. If there is at least one remaining user in the user list 324, execute step 1112: select the highest ranked remaining user in the user list 324 as the primary user, and the method 1100 executes the above step 1106. If there are no remaining users in the user list 324, the method 1100 returns to step 1102 to receive and process the next input frame in the frame sequence.

[0127] The user list 324 enables the adaptive gesture sensing system 300 to analyze and process gesture input and reduce false positives even when there are multiple people within the FOV 20 of the camera 102. The user list 324 is a relatively effective approach for multiple people, and as described above, other techniques may be used to replace or supplement the user list 324.

[0128] In some of the above examples, different anatomical features (e.g., faces) may be detected by processing the entire input frame. In other examples described below, different anatomical features may be detected by processing only a region of interest (ROI) within the input frame. For example, in step 1108 above, an adaptive ROI approach may be used (e.g., the face detection and tracking subsystem 312 may detect faces).

[0129] In the present invention, an adaptive ROI can be considered "adaptive" because the size and / or position of the ROI can be adjusted according to the needs of the face detector (e.g., the face detection and tracking system 312, processing efficiency, or resolution of the input frame). For example, machine learning-based (e.g., computer vision-based) face and hand detectors are typically trained to process square input images. Therefore, in order to improve the performance of face and hand detection, the ROI for performing detection should be a square. Considering similar reasons, the virtual gesture space for hand detection can be defined as a square.

[0130] Fig.12 Flowchart of an exemplary method 1200 for detecting different anatomical features in an input frame of a sequence of frames captured by the camera 102 using an adaptive ROI. The method 1200 may be used as part of step 1108 described above. For example, the method 1200 may be implemented by the face detection and tracking subsystem 312. It should be understood that the method 1100 may be implemented using other techniques of step 1108, with or without the use of an adaptive ROI.

[0131] 1202: Check whether the anatomical feature was detected in the previous input frame. If yes, execute step 1204: select the ROI used to detect the anatomical feature (e.g., face) in the previous input frame to be reused in the current input frame. Generally, the ROI used to detect the anatomical feature should be smaller than the entire input frame and can be square in shape (based on how the detection algorithm is trained or designed).

[0132] If no anatomical feature was detected in the previous input frame (or there is no previous input frame), step 1206 is performed: a ROI is selected from the ROI sequence. If an ROI in the ROI sequence was used in the previous input frame, the next ROI in the ROI sequence can be selected for use in the current input frame. The ROI sequence can be predefined (e.g., predefined in the face detection and tracking subsystem 312). The ROI sequence defines a different sequence of ROIs for processing sequential input frames. For example, if the ROI sequence is a sequence of 8 different ROIs (e.g., with different positions and / or sizes), each ROI in the sequence is selected in turn for detecting anatomical features in the sequence of 8 input frames. The ROI sequence can then loop back to the first ROI in the sequence.

[0133] Fig.13 An exemplary ROI sequence is shown with eight different ROIs 1302, 1304, 1306, 1308, 1310, 1312, 1314, 1316. The eight ROIs 1302 to 1316 may be cycled over a sequence of eight different input frames and may be applied to input frames of different resolutions. For example, six ROIs 1302 to 1312 may be applied to the original resolution 1300 of the input frame and may be designed to be able to detect smaller anatomical features (e.g., if the user 10 is away from the gesture control device 100). Two ROIs 1314, 1316 may be applied to a scaled-down version 1350 of the input frame (having a lower resolution) and may be designed to be able to detect larger anatomical features (e.g., if the user 10 is close to the gesture control device 100). Using scaled-down versions of the input frame for some of the ROIs 1314, 1316 helps reduce computational costs for larger regions of the input frame.

[0134] It should be noted that the ROI sequence loops over the frame sequence captured by the camera 102 so that each input frame is processed using only one selected ROI (rather than processing the same input frame using two or more ROIs). Since the input frames captured by the camera 102 are usually captured at a high frequency, the time difference between adjacent frames can be small enough, so no information is lost (or very little information is lost) by processing the frame sequence using the ROI sequence in this way.

[0135] There may be predefined different ROI sequences (e.g. stored in the face detection and tracking subsystem 312). The ROI sequence used may be selected by the user 10, or there may be a predefined order in which different ROI sequences are cycled through (i.e., there may be a predefined order in which ROI sequences are to be used). Fig.13The example shows that each ROI in the ROI sequence is used once in the sequence. In some examples, the ROI sequence can be defined as using a particular ROI two or more times in the ROI sequence. Other types of approaches can also be used.

[0136] 1208: Detect different anatomical features using the selected ROI (the ROI selected in the previous input frame in step 1204, or the ROI selected from the ROI sequence in step 1206).

[0137] Using adaptive ROIs to detect different anatomical features (e.g., faces) can reduce computational costs and / or improve the performance of trained detectors.

[0138] In some examples, when gesture detection is activated (or when gesture detection is used by default), the adaptive ROI technique may be used to process every frame captured by camera 102. In other examples, the adaptive ROI technique may be used to process every N (where N>1) frames.

[0139] As described above, in some examples, the hand detection and tracking subsystem 316 can output a bounding box for gesture recognition by the gesture recognition subsystem 322. In some embodiments, the gesture recognition subsystem 322 can implement a model built using a machine learning algorithm. In some embodiments, the gesture recognition subsystem 322 can include a trained neural network for gesture classification (hereinafter referred to as a trained gesture classification network). The trained gesture classification network has been trained using a known machine learning algorithm to learn the parameters (e.g., weights) of the neural network. The trained gesture classification receives a bounding box of a detected gesture and predicts a specific gesture category from a set of predefined gesture categories corresponding to the bounding box.

[0140] Typically, the accuracy of gesture classification achieved by a trained neural network for gesture classification decreases as the hand image is cropped (e.g., when the bounding box has a large offset from the ground truth). Examples of bounding box adjustment are described in U.S. Patent Application No. 16 / 355,665, filed on March 15, 2019, entitled “ADAPTIVE IMAGE CROPPING FOR FACE RECOGNITION,” which is incorporated herein by reference in its entirety. A similar bounding box adjustment method is described herein to help improve gesture recognition.

[0141] Fig.141 is a block diagram of an exemplary implementation of a gesture classification network that may be used in the gesture recognition subsystem 322. The gesture classification network 1400 may be implemented with a side branch to a bounding box refinement network 1450. The gesture classification network 1400 performs gesture classification on a hand image defined by a bounding box in an input frame, and the bounding box refinement network 1450 refines the bounding box used by the gesture classification network 1400.

[0142] The gesture classification network 1400 receives an input frame as input data. The input data may be a cropped version of the input frame (e.g., according to a bounding box defined for a hand). In some embodiments, the input data may be a batch of images, e.g., for batch-based training of the networks 1400, 1450, or to enable gesture classification based on a sequence of frames. The gesture classification network 1400 includes a series of convolutional blocks 1402 (e.g., implemented using a ResNet design). For simplicity, three such convolutional blocks 1402 are shown, but there may be more or fewer convolutional blocks 1402 in the gesture classification network 1400. These convolutional blocks 1402 output to a gesture classification fully connected network (FCN) 1404, which outputs a determined gesture class. The gesture classification FCN 1404 receives as input a vector output from the last of these convolutional blocks 1402. The gesture classification FCN 1404 uses feature embeddings to determine the gesture class of the hand defined by the bounding box and outputs the determined gesture class as a label. In some examples, the gesture classification FCN 1404 outputs a vector including a probability distribution of possible gesture categories. That is, the output of the gesture classification network 1400 can be the probabilities of different gesture categories rather than a certain gesture category. In some examples, the gesture classification FCN 1404 includes a softmax function in the last output layer, which can be used to normalize the output probability distribution of possible gesture categories.

[0143] Each convolution block 1402 is also output to an edge branch 1452 belonging to the bounding box correction network 1450. Each edge branch 1452 is output to a bounding box correction FCN 1454. Each edge branch 1452 may individually include an optional maximum pooling layer, an optional resizing layer, and a convolution block. The outputs of the edge branches 1452 are concatenated into a combined output vector, which may be flattened by a 1x1 convolution block (not shown) before being input to the bounding box correction FCN 1454. The output of the bounding box correction FCN 1454 is information that adjusts or corrects the size and position of the bounding box defining the hand in the input frame (e.g., in the form of coordinate information of the bounding box).

[0144] The following describes the training of the joint network 1400, 1450. As described above, the gesture classification FCN 1404 may include a softmax layer. The gesture classification FCN 1404 may further calculate and output a cross entropy loss, which may be considered a measure of the difference between the output probability distribution and the original probability distribution in the model. The cross entropy loss may be used as a loss function for the softmax layer and may therefore also be referred to as a softmax loss. Similarly, a bounding box loss may be output by the bounding box correction FCN 1454. The softmax loss and the bounding box loss may be combined into a total loss function, which may be used to train (1456) the joint network 1400, 1450. The softmax loss, the bounding box loss, and the training (1456) using the total loss function may only be used during the training of the network 1400, 1450 and may not be required during inference.

[0145] During training of the networks 1400, 1450, training data samples may be generated using randomly cropped hand images based on ground truth hand bounding boxes.

[0146] Fig.15 Some examples are shown. The ground truth 1502 defines an optimized bounding box for the hand image, and other randomly cropped hand images are generated as training data samples 1504. It should be noted that the training data samples 1504 can not only move the position of the bounding box, but also change the size of the bounding box (making the hand image in the bounding box look larger or smaller). The bounding box offset of each training data sample 1504 relative to the ground truth 1502 is used as a label for training bounding box correction.

[0147] The joint network 1400, 1450 is trained by minimizing a total loss function, which in this example is a linear combination of a classification loss function (softmax loss) and a bounding box loss function. An example of a bounding box loss function is described below.

[0148] Fig.16 The simplified example in FIG. 1 shows a ground truth bounding box 1602 around an object and a cropped training data sample bounding box 1604. Let {(x1, y1), (x1, y2), (x2, y1), (x2, y2)} be the coordinates of the position (the four corners in this example) of the training data sample bounding box 1604, let is the coordinate of the position of the corresponding ground truth 1602 bounding box. The bounding box correction network 1450 estimates the relative rotation θ and relative displacement {z1, z2, z3, z4} between the training data sample bounding box 1604 and the ground truth bounding box 1602, where

[0149]

[0150] The bounding box loss function can be defined as:

[0151]

[0152] Where λ is the regularization parameter.

[0153] During inference, the bounding box defining the hand image in the input frame may be iteratively corrected until the offset predicted by the bounding box correction network 1450 is close to 0. The final gesture classification score may be calculated by combining all individual gesture classification scores obtained at each iteration, as follows:

[0154] where s m is the classification score of the mth iteration (e.g. softmax output), is the corresponding weight of the mth bounding box and the final corrected bounding box (e.g., detection evaluation function (intersection over union, IoU)).

[0155] During inference, iterative classification is applied to an input hand image (defined by a bounding box applied to the input frame) using the gesture classification network 1400. In each iteration, the previous input image is corrected according to the output bounding box correction parameters of the previous iteration to obtain the input image of the gesture classification network 1400.

[0156] Therefore, the bounding box correction network 1450 and the gesture classification network 1400 operate together to provide feedback, which is used to correct the bounding box defining the hand image in the input frame, which helps to improve the performance of the gesture classification network 1400.

[0157] Fig.17 Flowchart of an exemplary method 1700 for performing gesture recognition bounding box correction (e.g., using the joint network 1400, 1450 described above). The method 1700 may be used as part of step 1118 described above. For example, the method 1700 may be implemented by the gesture recognition subsystem 322. It should be understood that the method 1100 may be implemented using other techniques for step 1118, and may or may not use bounding box correction.

[0158] 1702: Receive an input frame and a bounding box defining a detected hand (eg, output by the hand detection and tracking subsystem 316).

[0159] 1704: The gesture classification network with the joint bounding box refinement network (described above) may be used to perform gesture classification with bounding box refinement.

[0160] 1706: (Optional) Gesture analysis may be performed on multiple input frames. For example, the gesture recognition subsystem 322 may store a buffer of a previous input frame and perform gesture analysis by considering the previous input frame.

[0161] A buffer (e.g., implemented in the adaptive gesture sensing system 300) may be used to store a predetermined number of previous input frames. The number of previous input frames stored in the buffer may be relatively small (e.g., the first 10 to 30 input frames) to more efficiently utilize memory resources. In some examples, the buffer may additionally or alternatively store gesture analysis results performed on a small number of previous input frames.

[0162] For a single static gesture to be recognized, the gesture recognition subsystem 322 may require a predefined minimum number (K) of detections of the same gesture category on a predefined number (N) of previous input frames, where K ≥ 1 and K ≤ N. This requirement may help improve detection accuracy and reduce false positives. In some examples, the same gesture category may be detected on K consecutive input frames of the first N input frames. The predefined minimum K may be selected to be a relatively small number (e.g., 10) to better detect and still achieve near real-time gesture recognition. Static gestures that may be detected using this approach may include, for example, a mute gesture (e.g., for a mute or unmute command) or an open gesture (e.g., for a play or pause command), etc. The first N input frames may be stored in a buffer and may be used as a sliding window for detecting gestures, helping to reduce false positives.

[0163] The gesture recognition subsystem 322 can recognize dynamic gestures based on a combination of two or more static gestures. For example, a complete dynamic gesture can be decoupled into different states, where the transitions between states are the detection results of static gestures. The gesture recognition subsystem 322 can implement a set of predefined state transition rules for state-based dynamic gesture recognition.

[0164] Fig.18 18 is a state diagram for one example of state-based dynamic gesture recognition that may be implemented by gesture recognition subsystem 322. Neutral state 1802 may be a default initial state when gesture input is first activated or when no current valid gesture is detected.

[0165] A sliding dynamic gesture may be detected when a fist gesture is first detected to transition to the slide-ready state 1804, followed by a spread gesture to transition to the slide state 1806. Upon reaching the slide state 1806, the gesture recognition subsystem 322 recognizes the gesture input as a dynamic slide gesture. Thus, recognition of a static fist gesture followed by recognition of a static spread gesture (in the appropriate order) results in recognition of a dynamic slide gesture. In addition, a change in position between a detected fist gesture and a detected spread gesture may be required to positively detect a dynamic slide gesture (e.g., to distinguish a slide gesture from another dynamic gesture that does not involve a change in position). The change in position of the detected gesture may be calculated based on the change in the hand bounding box coordinates. Upon reaching the slide state 1806, the gesture recognition subsystem 322 may generate an output representing a recognized gesture (e.g., a gesture category label), and may also provide an output representing a change in position between the slide-ready state 1804 and the slide state 1806 detection.

[0166] A vertical or horizontal dynamic drag gesture may be detected as a combination of a zoom gesture, a pinch gesture, and a zoom static gesture. For example, after detecting a zoom static gesture, a transition is made from the neutral state 1802 to the pinch ready state 1808. After detecting a pinch static gesture, a transition is made from the pinch ready state 1808 to the pinch activated state 1810. A change in vertical position (e.g., a change greater than a predefined threshold) may cause a transition from the pinch activated state 1810 and maintained in the pinch static gesture to the vertical drag state 1812. Similarly, a change in horizontal position (e.g., a change greater than a predefined threshold) may cause a transition from the pinch activated state 1810 and maintained in the pinch static gesture to the horizontal drag state 1814. If the change in position is a combination of vertical and horizontal changes (e.g., a diagonal change in position), a change with a larger magnitude may be used to determine a state transition. Alternatively, if the change in position is a combination of vertical and horizontal changes, a state transition may not be recognized. Upon reaching the vertical drag state 1812 or the horizontal drag state 1814, the gesture recognition subsystem 322 may generate an output representing a recognized gesture (e.g., a gesture category label), and may also provide an output representing a vertical or horizontal position change. For example, the gesture recognition subsystem 322 can calculate the distance (e.g., based on the coordinates defined in the input frame) and output the distance value. The distance value can be used to map the dynamic drag gesture to the drag command input. Detection of the zoom static gesture causes a transition from the vertical drag state 1812 or the horizontal drag state 1814 to the zoom ready state 1808. Returning to the zoom ready state 1808 can be recognized as the end of the dynamic drag gesture.

[0167] For dynamic gestures involving position changes, the gesture recognition subsystem 322 may implement predefined rules based on physical laws and / or expected human motion to further exclude possible false positives. For example, the predefined rule may be that the detected hand should not have a position change greater than a predefined threshold (e.g., a change of more than 100 pixels) between consecutive input frames.

[0168] Using state-based gesture recognition has advantages over motion-based gesture classification and recognition. For example, detecting and processing gesture motion requires more processing resources than detecting static gestures. Additionally, state-based gesture recognition may be less prone to false positives.

[0169] In some examples, the present invention describes a method for performing hand detection in an input frame of a sequence of frames captured by a camera 102. The method includes performing image adjustment to improve image quality. In general, gesture recognition performance tends to degrade in low-light scenes. For example, if the gesture control device 100 is a smart TV viewed in a dark room, the screen of the smart TV may be the only significant source of illumination when the sequence of frames is captured by the camera 102 of the gesture control device 100. The present invention proposes three methods to solve this problem. Some exemplary techniques for image adjustment are described below, each of which can be implemented by the hand detection and tracking subsystem 316, etc., and system combinations can be used.

[0170] Fig.19 1 is a flow chart of an exemplary method 1900 for performing hand detection and tracking, the method 1900 including performing image adjustments in a sequence of frames captured by the camera 102. The method 1900 may be used as part of step 1116 described above. For example, the method 1900 may be implemented by the hand detection and tracking subsystem 316. It should be understood that the method 1100 may be implemented using other techniques for step 1116, with or without the image adjustments described below.

[0171] The method 1900 begins after a virtual gesture space is defined (eg, by the virtual gesture space generation subsystem 314 at step 1114 ).

[0172] 1902: Determine whether a hand has been detected in a previous input frame of a frame sequence captured by the camera 102. If a hand has been detected, the method 1900 may execute step 1908. For example, if a hand has been successfully detected in a previous input frame, a bounding box generated by successfully detecting a hand in the previous input frame may be used as a starting point for tracking the hand in a current input frame in a frame sequence (e.g., assuming that the time between consecutive frames is short enough that the hand is at least partially still within the previous bounding box). In addition, if a hand has been successfully detected in a previous input frame of a frame sequence, it may be considered that the brightness in the environment is sufficient (e.g., assuming that the light level in the environment does not change significantly in the short time between consecutive frames) and that image adjustment is not required.

[0173] If a hand is not detected in a previous input frame in the sequence of frames captured by the camera 102 , the method 1900 performs step 1904 .

[0174] 1904: (Optional) Perform image adjustment. In some examples, image adjustment can be performed only if it is determined that image adjustment is required. For example, a light sensor of the gesture control device 100 can be used to detect ambient light levels and determine whether the light level is low enough to require image adjustment. In some examples, at least a portion of the input image (e.g., over the entire input image, only over the virtual gesture space, or only over the hand bounding box) can be analyzed (e.g., to determine overall pixel intensity levels) to determine whether image adjustment is required.

[0175] In some examples, image adjustments may be performed by default without first determining whether image adjustments are needed. In this case, if the image brightness is sufficient, performing image adjustments (e.g., gamma correction) may result in little or no change to the image.

[0176] Various techniques may be used for image adjustment, particularly for correcting for low light conditions.

[0177] In some examples, image adjustment may include adjusting the brightness of the image, using a face detected in a previous frame of the sequence of frames as a reference (in the case where the face is a distinct anatomical feature). Assuming a face has been detected in a previous frame (e.g., using the face detection and tracking subsystem 312), the detected face may be used as a reference to adjust the brightness on the incoming frame.

[0178] set up and denote the predefined (e.g., empirically determined by calibration or pre-encoded) mean and standard deviation of pixel intensities of face images captured under low-light and preferred brightness conditions, respectively.

[0179] Fig. 202002 is a schematic diagram of a representative histogram 2002 of pixel intensity values ​​when an image including a human face is captured under low light conditions, and a representative histogram 2004 of pixel intensity values ​​when an image including a human face is captured under optimal brightness conditions.

[0180] The mapping from low-light pixels to preferred brightness pixels can be calculated using the following formula:

[0181]

[0182] Using this equation, all pixels x in the virtual gesture space defined in the input frame i can be converted to the adjusted pixel y by using the above mapping formula i , to better detect the hand. Although the above describes brightness adjustment for correcting low light conditions, it should be understood that similar brightness adjustment can be used to correct for overly bright conditions.

[0183] In some examples, image adjustment may include performing gamma correction (which may be considered a form of brightness correction). Various gamma correction techniques may be used. For example, let x i is the pixel value of a single channel of the input frame. Then, the gamma transform can be calculated using the following formula:

[0184]

[0185] where y i is the adjusted pixel value, and γ is the standard parameter of the gamma transform.

[0186] In some examples, image adjustment may include changing the display of the gesture control device 100. This approach is advantageous when the screen of the gesture control device 100 is the source of brightness in the environment (e.g., the gesture control device 100 is a television, a smartphone, or a tablet). In this approach, a low-light condition may first be detected using any suitable technique. For example, a light sensor that is part of the gesture control device 100 may be used to detect a low-light condition. A low-light condition may also be detected by analyzing the pixel intensity of an input frame, and a low-light condition may be determined if a majority of the pixels are below a predefined intensity threshold.

[0187] After detecting a low light condition, the display content and / or brightness of the screen of the gesture control device 100 may be controlled so that the screen becomes a stronger light source to capture a better quality image. The output of the adaptive gesture sensing system 300 may be sent to indicate that the input frame has been analyzed as having poor brightness, and the processing device 202 may control the display 104 to change the display content and / or brightness of the screen accordingly. In some examples, the low light condition detection is not based on image analysis (e.g., using a light sensor of the gesture control device 100), and the processing device 202 may control the display 104 accordingly without the adaptive gesture sensing system 300 sending any information.

[0188] In some examples, controlling the display content and / or brightness in this manner may be performed after a low light condition is detected and after a gesture input has been initiated (e.g., the user 10 has manually provided input to activate a gesture-based command input, or a recognized gesture has been initially detected). In some examples, the screen of the gesture control device 100 may be controlled to change the screen backlight so that the screen is brighter. In some examples, the screen of the gesture control device 100 may be controlled to add a brightness portion to the display content. The brightness portion may be a brighter portion of the display (e.g., a full white portion).

[0189] For example, Fig.21A A conventional image 2102 is shown that can be displayed under normal light conditions, or when gesture input has not been initiated on the display 104 of a gesture-controlled device such as a television or smartphone. After low-light conditions are detected and gesture input is initiated, the display content can be adjusted to Fig. 21B In the content. Fig. 21B In the example, the size of image 2102 is reduced, and a brightness portion 2104 is added. In this example, the brightness portion 2104 is a border along all four sides of the display. In other parts, the brightness portion 2104 can be only along one side, two sides or three sides, or can include discontinuous parts along one or more sides of the display. The brightness, hue and / or size (e.g., thickness and length) of the brightness portion 2104 can be automatically adjusted according to the detected low light conditions. For example, different low light levels can exist under a predefined threshold (e.g., according to the detected pixel intensity, or based on the output of a light sensor), and different parameters of the brightness portion 2104 can be predefined under different low light levels. Typically, a weaker low light level may require a larger and / or brighter brightness portion 2104.

[0190] See again Fig.19, 1906: Perform hand detection in the virtual gesture space (e.g., using the hand detection and tracking subsystem 316). The hand detection and tracking subsystem 316 may include a trained neural network, which may be used as described above to detect a hand in the image 2104. The hand detection and tracking subsystem 316 may output a bounding box defined for the hand detected in the virtual gesture space using a hand successfully detected by the trained neural network.

[0191] 1908: Based on the bounding box defined for the detected hand, perform hand tracking using the hand detection and tracking subsystem 316. The bounding box may be updated based on the tracking.

[0192] The hand detection and tracking information (eg, as a bounding box defining the hand) may then be provided for gesture recognition (eg, by the gesture recognition subsystem 322).

[0193] In various examples, the present invention describes systems and methods that help improve the accuracy and efficiency of gesture input detection and recognition. The present invention can be used to detect and recognize gesture input in complex environments and / or for long-distance detection of gestures.

[0194] Using one or more of the above methods (e.g., adaptive ROI techniques, bounding box correction joint networks, virtual gesture spaces, image adjustments, state-based gesture recognition) can achieve more robust detection and recognition of gesture inputs, even in complex real-world scenarios. The improved gesture detection accuracy can more efficiently process captured input frames. In some examples, input frames can be processed at a frequency lower than the image capture rate. For example, instead of processing each input frame, every N (where N>1) frames are processed for detecting and tracking faces (or other different anatomical features) and hands. N can be a user-selected parameter, a pre-programmed parameter, or a parameter automatically selected by the gesture control device 100 (e.g., when images are captured at a rate lower than a predefined threshold, N can be 1; when images are captured above a predefined threshold, N can be 2 or more; when images are of poor quality or low resolution, N can be 1; when images are of high resolution, N can be 2 or more). By processing every N (N>1) frames, gesture detection and recognition can still be performed in near real time with high accuracy, and the processing resources required by the gesture control device can be reduced.

[0195] The present invention describes an exemplary implementation of using a neural network for face and hand detection. An exemplary joint neural network is described, which can correct the hand bounding box and help improve the accuracy of gesture classification and recognition.

[0196] In some examples, a virtual gesture space is described, which can be defined based on a detected face (or other different anatomical features). By using the defined virtual gesture space to detect hands, more accurate and / or more efficient gesture detection can be achieved. In some examples, the virtual gesture space can also have subspaces, and gesture inputs of specific subspaces can be mapped to mouse inputs. Therefore, the virtual gesture space can be used as a virtual mouse.

[0197] Although some examples have been described in the context of a gesture-controlled device with a display and a camera (e.g., a smart TV, a smartphone, or a tablet), the present invention may be relevant to other gesture-controlled devices that may or may not include a display and / or a camera. For example, the present invention may be relevant to smart speakers, smart appliances, Internet of Things (IoT) devices, dashboard devices (e.g., installed in a vehicle), or devices with low computing resources.

[0198] The examples described herein may be applicable to artificial reality (AR), virtual reality (VR), and / or video game applications.

[0199] Although the present invention uses steps in a certain order to describe methods and processes, one or more steps of the methods and processes may be omitted or modified as appropriate. One or more steps may be performed in an order different from the order of these steps as appropriate.

[0200] Although the present invention is at least partially described in terms of methods, it will be understood by those skilled in the art that the present invention also relates to various components for performing at least some aspects and features of the described methods, whether hardware components, software, or any combination of the two. Therefore, the technical solution of the present invention can be embodied in the form of a software product. Suitable software products can be stored in a pre-recorded storage device or other similar non-volatile or non-transient computer-readable medium, including DVDs, CD-ROMs, USB flash drives, mobile hard disks, or other storage media. The software product includes instructions tangibly stored thereon, which enable a processing device (such as a personal computer, a server, or a network device) to execute the method examples disclosed herein.

[0201] The present invention may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are considered to be illustrative and non-restrictive in all respects. Selected features in one or more of the above embodiments may be combined to create alternative embodiments not explicitly described, and features suitable for such combinations are understood to fall within the scope of the present invention.

[0202] In addition, all values ​​and subranges within the disclosed range are disclosed. In addition, although the disclosed and illustrated systems, equipment and processes herein may include a specific number of elements / assemblies, these systems, equipment and assemblies may be modified to include more or less such elements / assemblies. For example, although any disclosed element / assembly may be a single quantity, the embodiments disclosed herein may be modified to include a plurality of such elements / assemblies. The subject matter described herein is intended to encompass and include all suitable technical changes.

Claims

1. A method for processing a gesture, the method comprising: Determining a virtual gesture space defined within a received input frame, the virtual gesture space being associated with a primary user in an ordered user list of one or more users, and the virtual gesture space being smaller than a total field of view captured within the received input frame, wherein the input frame is a picture captured by a camera of a gesture control device; recognizing a gesture in the virtual gesture space; A command input associated with the gesture is determined.

2. The method according to claim 1, characterized in that The determining of a virtual gesture space defined within the received input frame comprises: processing the input frame to detect the one or more users; generating the ordered user list according to the detected one or more users, the primary user being identified as a user with the highest ranking in the ordered user list, wherein the ordered user list is generated by sorting the one or more users according to a predefined sorting criterion; The virtual gesture space is generated according to the detected anatomical features of the primary user.

3. The method according to claim 1 or 2, characterized in that: The input frame is located in a frame sequence, and the step of recognizing a gesture in the virtual gesture space includes: processing only the virtual gesture space within the input frame and one or more subsequent frames in the sequence of frames to detect and track a hand; Gesture classification is performed to determine the gesture associated with the hand by detecting and tracking a hand bounding box generated by the hand.

4. The method according to claim 2, characterized in that: The processing of the input frame comprises: A region of interest ROI is selected for processing the input frame, wherein the ROI defines a region smaller than the total area of ​​the input frame; wherein, The ROI is selected from a defined ROI sequence, wherein the ROI sequence includes a plurality of ROIs, and the plurality of ROIs are used for processing a corresponding plurality of sequentially received input frames.

5. The method according to claim 3 or 4, characterized in that: The determining the gesture associated with the hand comprises: determining, in the one or more subsequent frames, an invalid gesture associated with the hand detected in the virtual gesture space associated with the primary user; Selecting the user with the second highest ranking in the ordered user list as the new primary user; The method as claimed in claim 3 is repeated using the new primary user.

6. The method according to any one of claims 3 to 5, characterized in that: The processing of only the virtual gesture space within the input frame and one or more subsequent frames in the frame sequence comprises: determining a low-light condition associated with the input frame and at least one of the one or more subsequent frames; Image adjustments are automatically performed to adjust pixel values ​​of the at least one frame based on the low light condition.

7. The method according to any one of claims 3 to 6, characterized in that: The input frame and the one or more subsequent frames are processed using a trained joint neural network to detect and track the hand, wherein the trained joint neural network includes a trained gesture classification convolutional neural network.

8. The method according to any one of claims 3 to 7, characterized in that: The performing of the gesture classification comprises: identifying a gesture category associated with the detected hand; determining a state transition from a previous gesture state to a current gesture state, the state transition being determined based on the identified gesture category; The gesture associated with the current gesture state is determined.

9. The method according to any one of claims 3 to 8, characterized in that: The input frame and at least one of the one or more subsequent frames are received and processed at a frequency that is lower than a frame capture frequency of an image capture device used to capture the sequence of frames.

10. The method according to any one of claims 1 to 9, characterized in that The gesture control device is one of the following devices: television; Smartphones; Tablet computer; On-vehicle device; IoT devices; Augmented reality (AR) devices; or Virtual reality VR equipment.

11. A device for processing gestures, characterized in that include: A processing device is coupled to a memory, the memory storing machine executable instructions, wherein when the processing device executes the instructions, the apparatus causes the apparatus to perform the following operations: Determining a virtual gesture space defined within a received input frame, the virtual gesture space being associated with a primary user in an ordered user list of one or more users, and the virtual gesture space being smaller than a total field of view captured within the received input frame, wherein the input frame is a picture captured by a camera of the device; recognizing a gesture in the virtual gesture space; A command input associated with the gesture is determined.

12. The device according to claim 11, characterized in that The instructions further cause the device to perform the following operations to determine the virtual gesture space: processing the input frame to detect the one or more users; generating the ordered user list according to the detected one or more users, the primary user being identified as a user with the highest ranking in the ordered user list, wherein the ordered user list is generated by sorting the one or more users according to a predefined sorting criterion; The virtual gesture space is generated according to the detected anatomical features of the primary user.

13. The device according to claim 11 or 12, characterized in that The input frame is in a frame sequence, and the instructions further cause the device to perform the following operations to recognize a gesture in the virtual gesture space: processing only the virtual gesture space within the input frame and one or more subsequent frames in the sequence of frames to detect and track a hand; Gesture classification is performed to determine the gesture associated with the hand by detecting and tracking a hand bounding box generated by the hand.

14. The device according to claim 12, characterized in that The instructions further cause the device to perform the following operations to process the input frame: A region of interest ROI is selected for processing the input frame, wherein the ROI defines a region smaller than the total area of ​​the input frame; wherein, The ROI is selected from a defined ROI sequence, wherein the ROI sequence includes a plurality of ROIs, and the plurality of ROIs are used for processing a corresponding plurality of sequentially received input frames.

15. The device according to claim 13 or 14, characterized in that The instructions further cause the device to: determining, in the one or more subsequent frames, an invalid gesture associated with the hand detected in the virtual gesture space associated with the primary user; Selecting the user with the second highest ranking in the ordered user list as the new primary user; The operations performed by the apparatus of claim 13 are repeated using the new primary user.

16. The device according to any one of claims 13 to 15, characterized in that The instructions further cause the apparatus to perform the following operations to process only the virtual gesture space within the input frame and one or more subsequent frames in the sequence of frames: determining a low-light condition associated with the input frame and at least one of the one or more subsequent frames; Image adjustments are automatically performed to adjust pixel values ​​of the at least one frame based on the low light condition.

17. The device according to any one of claims 13 to 16, characterized in that The input frame and the one or more subsequent frames are processed using a trained joint neural network to detect and track the hand, wherein the trained joint neural network includes a trained gesture classification convolutional neural network.

18. The device according to any one of claims 13 to 17, characterized in that The instructions further cause the device to perform the following operations to perform the gesture classification: identifying a gesture category associated with the detected hand; determining a state transition from a previous gesture state to a current gesture state, the state transition being determined based on the identified gesture category; The gesture associated with the current gesture state is determined.

19. The device according to any one of claims 13 to 18, characterized in that The input frame and at least one of the one or more subsequent frames are received and processed at a frequency that is lower than a frame capture frequency of an image capture device used to capture the sequence of frames.

20. The device according to any one of claims 11 to 19, characterized in that The device is a gesture control device.

21. The device according to claim 20, characterized in that The gesture control device is one of the following devices: television; Smartphones; Tablet computer; On-vehicle device; IoT devices; Augmented reality (AR) devices; or Virtual reality VR equipment.

22. A computer readable medium, characterized in that The device stores machine executable instructions, and when a processing device of the device executes the instructions, the device executes the method according to any one of claims 1 to 10.

Images