Real-time on-device classification of finger motion patterns as gestures

Abstract

This application relates to classifying fingertip motion patterns on-device in real-time as hand gestures. Hand gestures form an intuitive means of interaction in augmented / mixed reality applications. However, accurate hand gesture recognition can be achieved through deep learning models or using expensive sensors. While these deep learning models are robust, they are often computationally expensive and achieving real-time performance remains a challenge. Embodiments of the present disclosure provide systems and methods for classifying fingertip motion patterns as different hand gestures. Red, green, blue images are fed as input to an object detector (MobileNetV2) to output hand candidate bounding boxes, which are then down-scaled to reduce processing time without compromising the quality of image features. The detected hand candidates are then fed to a fingertip regressor, which outputs the spatial locations of the fingertips representing the motion pattern, where the coordinates of the fingertips are fed to a bidirectional long short-term memory network for classifying the motion pattern as different hand gestures.

Description

[0001] Cross-reference of related applications and priorities

[0002] This patent application claims priority to Indian Patent Application 201921003256, filed on January 25, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This article generally deals with classification techniques, and more specifically, with classifying fingertip movement patterns into gestures on a device in real time. Background Technology

[0004] Over the past few decades, information technology has transitioned from desktop computing to mobile computing. Smartphones, tablets, smartwatches, and head-mounted displays (HMDs) are (or have) slowly replaced desktop-based computing. The shift from office and home office environments to computing on the go has been significant. Mobile phones constitute a large part of life: the percentage of internet traffic generated by them is surpassing that of their desktop counterparts. Naturally, with this transition, the way people interact with these devices has evolved from keyboard / mouse to gestures, voice, and brain-computer interfaces. Voice interfaces tend to be less accurate in noisy outdoor environments, and therefore most HCI researchers are interested in combinations of gesture interfaces and voice. Gesture recognition on real-time feeds or video is a form of activity recognition. Gestures form an intuitive means of interaction in mixed reality (MR) applications. However, accurate gesture recognition can only be achieved through deep learning models or the use of expensive sensors. While these deep learning models are robust, they are typically computationally expensive, and achieving real-time performance remains a challenge. Summary of the Invention

[0005] The embodiments of this disclosure propose technical improvements as solutions to one or more of the aforementioned technical problems in conventional systems recognized by the inventors. For example, one aspect is a processor-implemented method for classifying fingertip movement patterns as gestures in real time on a device. In a cascaded deep learning model (CDLM) executed via one or more hardware processors of a mobile communication device, multiple red, green, and blue (RGB) input images from an image acquisition device are received in real time, wherein each of the multiple RGB input images includes a gesture; multiple hand candidate bounding boxes from the received multiple RGB input images are detected in real time using an object detector included in the cascaded deep learning model (CDLM) executed via one or more hardware processors of the mobile communication device, wherein each hand candidate bounding box is specific to a corresponding RGB image from the received multiple RGB input images, and wherein each hand candidate bounding box includes a hand candidate; the process is then scaled down in real time. A reduced hand candidate set is obtained from each of the multiple hand candidate bounding boxes; the spatial location of the fingertip from each reduced hand candidate in the reduced hand candidate set is detected in real time using a fingertip regressor included in a cascaded deep learning model (CDLM) executed via one or more hardware processors on a mobile communication device, wherein the spatial location of the fingertip from the reduced hand candidate set represents a fingertip movement pattern; and the fingertip movement pattern is classified into one or more gestures in real time using a first coordinate and a second coordinate from the spatial location of the fingertip via a bidirectional long short-term memory (Bi-LSTM) network included in a cascaded deep learning model (CDLM) executed via one or more hardware processors on a mobile communication device.

[0006] In one embodiment, each hand candidate bounding box, which includes hand candidates, describes a pointing gesture for classification into one or more gestures.

[0007] In one embodiment, the step of classifying fingertip movement patterns into one or more gestures includes applying regression techniques to a first coordinate and a second coordinate of the fingertip.

[0008] In one embodiment, the spatial position of the fingertip is detected based on the presence of a pointing finger hand detection on a set of consecutive frames in a plurality of RGB input images, wherein the presence of the pointing finger hand detection indicates the start of the gesture.

[0009] In one embodiment, there is no pointing finger hand detection indication of the end of a gesture on a set of consecutive frames in a plurality of RGB input images.

[0010] On the other hand, a system is provided for classifying fingertip movement patterns into gestures in real time. The system includes a memory storing instructions; one or more communication interfaces; and one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by instructions to: receive in real time, within a cascaded deep learning model (CDLM), a plurality of red, green, and blue (RGB) input images from an image acquisition device, including those stored in the memory and executed via the one or more hardware processors of the system, wherein each of the plurality of RGB input images includes a gesture; and detect in real time, using an object detector included in the cascaded deep learning model (CDLM) executed via the one or more hardware processors of the system, a plurality of hand candidate bounding boxes from the received plurality of RGB input images, wherein each hand candidate bounding box is specific to the received plurality of RGB input images. The corresponding RGB image in the image, wherein each of the plurality of hand candidate bounding boxes includes a hand candidate; the hand candidates from each of the plurality of hand candidate bounding boxes are reduced in real time to obtain a reduced hand candidate set; the spatial position of the fingertip of each reduced hand candidate in the reduced hand candidate set is detected in real time using a fingertip regressor included in a cascaded deep learning model (CDLM) executed via one or more hardware processors on the system, wherein the spatial position of the fingertip from the reduced hand candidate set represents a fingertip movement pattern; and the fingertip movement pattern is classified into one or more gestures in real time via a bidirectional long short-term memory (Bi-LSTM) network included in a cascaded deep learning model (CDLM) executed via one or more hardware processors on the system using a first coordinate and a second coordinate from the spatial position of the fingertip.

[0011] In one embodiment, each hand candidate bounding box, which includes hand candidates, describes a pointing gesture for classification into one or more gestures.

[0012] In one embodiment, fingertip movement patterns are classified into one or more gestures by applying regression techniques to the first and second coordinates of the fingertip.

[0013] In one embodiment, the spatial position of the fingertip is detected based on the presence of a pointing finger hand detection on a set of consecutive frames in a plurality of RGB input images, wherein the presence of the pointing finger hand detection indicates the start of the gesture.

[0014] In one embodiment, there is no pointing finger hand detection indication of the end of a gesture on a set of consecutive frames in a plurality of RGB input images.

[0015] In another aspect, one or more non-transitory machine-readable information storage media are provided, comprising one or more instructions that, when executed by one or more hardware processors, cause: in a cascaded deep learning model (CDLM) executed via one or more hardware processors of a mobile communication device, to receive in real time a plurality of red, green, and blue (RGB) input images from an image acquisition device, wherein each of the plurality of RGB input images includes a gesture; and to detect in real time a plurality of hand candidate bounding boxes from the received plurality of RGB input images using an object detector included in the cascaded deep learning model (CDLM) executed via one or more hardware processors of a mobile communication device, wherein each of the plurality of hand candidate bounding boxes is specific to a corresponding RGB image from the received plurality of RGB input images. In the process, each of the multiple hand candidate bounding boxes includes a hand candidate; the hand candidates from each of the multiple hand candidate bounding boxes are narrowed down in real time to obtain a narrowed set of hand candidates; the spatial position of the fingertip of each narrowed hand candidate in the narrowed set of hand candidates is detected in real time using a fingertip regressor included in a cascaded deep learning model (CDLM) executed via one or more hardware processors on a mobile communication device, wherein the spatial position of the fingertip from the narrowed set of hand candidates represents a fingertip movement pattern; and the fingertip movement pattern is classified into one or more gestures in real time using a first coordinate and a second coordinate from the spatial position of the fingertip via a bidirectional long short-term memory (Bi-LSTM) network included in a cascaded deep learning model (CDLM) executed via one or more hardware processors on a mobile communication device.

[0016] In one embodiment, each hand candidate bounding box, which includes hand candidates, describes a pointing gesture for classification into one or more gestures.

[0017] In one embodiment, the step of classifying fingertip movement patterns into one or more gestures includes applying regression techniques to a first coordinate and a second coordinate of the fingertip.

[0018] In one embodiment, the spatial position of the fingertip is detected based on the presence of a pointing finger hand detection on a set of consecutive frames in a plurality of RGB input images, wherein the presence of the pointing finger hand detection indicates the start of the gesture.

[0019] In one embodiment, there is no pointing finger hand detection indication of the end of a gesture on a set of consecutive frames in a plurality of RGB input images.

[0020] It should be understood that the foregoing general description and the following detailed description are merely exemplary and illustrative, and not intended to limit the scope of the claimed invention. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and, together with the specification, serve to explain the principles disclosed:

[0022] Figure 1 An exemplary block diagram of a system for classifying fingertip movement patterns as gestures on a device in real time, according to an embodiment of the present disclosure, is shown.

[0023] Figure 2 An exemplary block diagram of a system for classifying fingertip movement patterns as gestures on a device in real time, according to an embodiment of the present disclosure, is shown.

[0024] Figure 3 The use of embodiments according to this disclosure is illustrated. Figure 1 An exemplary flowchart of a system that classifies fingertip movement patterns into gestures on a device in real time.

[0025] Figure 4 The exemplary embodiments described herein are provided by Figure 1 The system implements a fingertip regressor architecture for fingertip positioning.

[0026] Figure 5 A sequence of gestures is described in accordance with exemplary embodiments of this disclosure, prior to data collection.

[0027] Figure 6 The present disclosure describes an image comparison of the present disclosure with conventional methods according to exemplary embodiments thereof, the present disclosure and conventional methods respectively indicating the results of the detector (hand candidate bounding box) under different conditions such as lighting differences, rendering blur, indoor and outdoor environments.

[0028] Figures 7A to 7B A graphical representation illustrating an exemplary embodiment of the present disclosure and comparing finger positioning with conventional techniques is shown.

[0029] Figure 8 Exemplary embodiments according to this disclosure are described. Figure 3 The method uses smartphone-based Virtual cardboard box ( Overall performance across 240 egocentric videos captured by the Cardboard head-mounted device. Detailed Implementation

[0030] Exemplary embodiments are described with reference to the accompanying drawings. In the drawings, the leftmost numeral of the reference numeral identifies the drawing in which that numeral first appears. The same reference numerals are used in all drawings to denote the same or similar parts, whenever convenient. While examples and features of the disclosed principles have been described herein, modifications, adaptations, and other implementations are possible without departing from the spirit and scope of the disclosed embodiments. The following detailed description is to be considered exemplary only; the true scope and spirit are indicated by the appended claims.

[0031] Expensive augmented reality (AR) / mixed reality (MR) devices (e.g., HoloLens, Daqri, and Meta Glass offer rich user interfaces by utilizing recent hardware improvements. They are equipped with a variety of onboard sensors, including multiple cameras, depth sensors, and proprietary processors. This makes them expensive and difficult to adopt on a large scale.

[0032] To provide user-friendly interfaces via gestures, detecting hands in the user's field of view (FoV), locating key points on the hand, and understanding their motion patterns have been important tasks in the field of vision in recent years. Although robust deep learning models using existing object detectors and sequence tracking methods have been developed to address these problems, achieving real-time performance, especially on systems such as on-device systems, remains a challenge due to limitations in memory and processing resources.

[0033] In this disclosure, embodiments describe a computationally efficient gesture recognition framework that operates without requiring depth information and dedicated hardware, thereby providing a gesture interface with high accessibility to the most affordable video perspective HMDs. These devices provide virtual reality (VR) / MR experiences using stereoscopic rendering fed by smartphone cameras, but have limited user interaction capabilities.

[0034] Industrial inspection and maintenance, telepresence, and data visualization are some direct applications of the framework described in the embodiments of this disclosure, and it can operate in real time and has the benefit of being able to work in remote environments without requiring an internet connection. To illustrate the general properties of the framework implemented in this disclosure, the detection of 10 complex gestures using pointing gestures has been illustrated with an example Android application.

[0035] Therefore, embodiments of this disclosure provide systems and methods for implementing a gesture recognition framework that operates in a first-person view for wearable devices. The model is trained on a graphics processing unit (GPU) machine and ported to an Android smartphone for use with, for example... This disclosure implements a gesture recognition framework driven by cascaded deep learning models: MobileNetV2 for hand localization, followed by a fingertip regression architecture with a Bi-LSTM model for gesture classification.

[0036] Now refer to the accompanying drawings, and more specifically to... Figures 1 to 8 The reference numerals in the accompanying drawings always denote corresponding features, illustrate preferred embodiments, and these embodiments are described in the context of the following exemplary systems and / or methods.

[0037] Figure 1 An exemplary block diagram of a system 100 for classifying fingertip movement patterns into gestures in real time on an device according to embodiments of the present disclosure is shown. System 100 may also be referred to as a “classification system” or a “mobile communication device” or “video viewed through a head-mounted device,” and these terms are used interchangeably hereinafter. In one embodiment, system 100 includes one or more processors 104, a communication interface device or input / output (I / O) interface 106, and one or more data storage devices or memories 102 operatively coupled to one or more processors 104. The one or more processors 104 may be one or more software processing modules and / or hardware processors. In one embodiment, the hardware processor may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. In other functions, the processor is configured to acquire and execute computer-readable instructions stored in memory. In one embodiment, device 100 may be implemented in various computing systems, such as laptops, notebooks, handheld devices, workstations, mainframes, servers, network clouds, etc.

[0038] I / O interface device 106 may include various software and hardware interfaces, such as web interfaces, graphical user interfaces, etc., and may facilitate multiple communications within various network N / W and protocol types (including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, or satellite)). In one embodiment, the I / O interface device may include one or more ports for connecting multiple devices to each other or to another server.

[0039] Memory 102 may include any computer-readable medium known in the art, including, for example, volatile memory such as static random access memory (SRAM) and dynamic random access memory (DRAM); and / or non-volatile memory such as read-only memory (ROM), erasable programmable ROM, flash memory, hard disk, optical disk, and magnetic tape. In one embodiment, database 108 may be stored in memory 102, wherein database 108 may include, for example, red, green, and blue (RGB) input images obtained from one or more computing devices (e.g., video viewed through a head-mounted device), data involving bounding boxes (including hand candidates, scaled-down hand candidates, spatial locations of fingertips detected from scaled-down hand candidates, x and y coordinates derived from the spatial locations of the fingertips, and movement patterns of fingertips classified as one or more gestures), etc. In one embodiment, memory 102 may store one or more techniques (e.g., feature extractors or feature detectors, also known as MobileNetV2, image processing techniques (such as scaled-down), fingertip regression / regressors, bidirectional long short-term memory (Bi-LSTM) networks, etc.) that perform the methods described herein when executed by one or more hardware processors 104. The memory 102 further includes (or may further include) information about the inputs / outputs of each step performed by the systems and methods of this disclosure. In one embodiment, a MobileNetV2 (feature extractor or feature detector), image processing techniques, a fingertip regression / regressor, and a bidirectional long short-term memory (Bi-LSTM) network are coupled together to form a cascaded deep learning model (CDLM), which performs the methods described herein when executed by one or more hardware processors 104.

[0040] refer to Figure 1 , Figure 2 An exemplary block diagram of a system 100 for classifying fingertip movement patterns as gestures on a device in real time, according to an embodiment of the present disclosure, is shown. Optionally, Figure 2 An exemplary implementation of a system 100 for classifying fingertip movement patterns as gestures on a device in real time, according to embodiments of the present disclosure, is shown. Figure 2 The architecture shown is configured to recognize various gestures of an economical AR wearable device with a monocular RGB camera input that requires only a limited amount of labeled classification data to classify fingertip movement patterns into different gestures.

[0041] refer to Figures 1 to 2 , Figure 3 An embodiment of the present disclosure is shown for use. Figure 1An exemplary flowchart of a method by which system 100 classifies fingertip movement patterns into gestures on a device in real time. In one embodiment, system 100 includes one or more data storage devices or memories 102 operatively coupled to one or more hardware processors 104, and configured to store instructions for performing the steps of the method by the one or more processors 104. Reference will now be made to... Figure 1 System 100 components, Figure 2 and Figure 4 Block diagram and Figure 3 The flowchart described herein explains the steps of the method of this disclosure. In an embodiment of this disclosure, at step 302, one or more hardware processors 104 receive in real time multiple red, green, and blue (RGB) input images from an image acquisition device in a cascaded deep learning model (CDLM) executed via one or more hardware processors of the mobile communication device 100, wherein each of the multiple RGB input images includes a gesture. In other words, the mobile communication device 100 includes a cascaded deep learning model having a feature extractor / object detector (e.g., MobileNetV2 in this disclosure) that takes a single RGB image as input.

[0042] In embodiments of this disclosure, at step 304, one or more hardware processors 104 use an object detector included in a cascaded deep learning model (CDLM) executed on the mobile communication device 100 to detect in real time multiple hand candidate bounding boxes from multiple received RGB input images. In one embodiment, each hand candidate bounding box is specific to a corresponding RGB image from the multiple received RGB input images, and each hand candidate bounding box includes a hand candidate. In other words, the MobileNetV2 output includes hand candidate bounding boxes that include hand candidates. Each hand candidate bounding box that includes a hand candidate describes a pointing gesture pose for classification as one or more gestures. Figure 2 Described by In Figure 1 The hand candidates are output by the object detector of the cascaded deep learning model executed on System 100.

[0043] MobileNetV2 is a streamlined architecture that uses depthwise separable convolutions to build lightweight deep neural networks. Depthwise separable convolutions decompose standard convolutions into depthwise convolutions and 1×1 convolutions, also known as pointwise convolutions, thus reducing the number of parameters in the network. It builds on the ideas of MobileNetV1 (an earlier version of the object detector), but it incorporates two new features into the architecture: (i) linear bottlenecks between layers, and (ii) skip connections between bottlenecks. Bottlenecks encode the intermediate inputs and outputs of the model, while inner layers encapsulate the model's ability to translate from low-level concepts such as pixels to high-level descriptors such as image categories. Skip connections, similar to regular residual connections, enable faster training without any loss of accuracy.

[0044] In experiments conducted to detect hand candidates in RGB input images obtained from wearable devices, the systems and methods of this disclosure use conventional systems and methods / techniques (e.g., conventional technique 1 - SSDLite - object detection module) to evaluate the MobileNetV2 feature extractor. The Experiments and Results section highlights the results compared to existing techniques with a pre-trained VGG-16 model comprising 13 shared convolutional layers, as well as other compact models such as ZF (e.g., Zeiler and Fergus 2014) and VGG1024 (Chatfield et al., 2014) by modifying the last fully connected layer to detect hand categories (pointing gesture poses).

[0045] Return to reference Figure 3 In the embodiments of this disclosure, at step 306, one or more hardware processors 104 reduce the number of hand candidates from each of the plurality of hand candidate bounding boxes in real time to obtain a reduced set of hand candidates. In other words, the input image including hand candidates is first reduced to a specific resolution (e.g., 640×480 resolution for a specific use case scenario in this disclosure) to reduce processing time without compromising the quality of image features.

[0046] In embodiments of this disclosure, at step 308, one or more hardware processors 104 use a fingertip regressor included in a cascaded deep learning model (CDLM) executed on the mobile communication device 100 to detect in real time the spatial location of the fingertip of each reduced hand candidate from a reduced set of hand candidates. In one embodiment, the spatial location of the fingertip from the reduced set of hand candidates represents a fingertip movement pattern. In other words, the detected hand candidates are then fed to a system such as... Figure 2 The fingertip regressor shown outputs the spatial position of the fingertip movement pattern (or fingertip).

[0047] In this disclosure, system 100 implements a fingertip regressor based on a convolutional neural network (CNN) architecture to locate the (x, y) coordinates of the fingertip. The previously discussed hand candidate detection (pointing gesture pose) triggers the regression CNN for fingertip localization. The hand candidate bounding boxes are then fed into... Figure 4 Before the network is shown, the candidate bounding boxes are first cropped and resized to 99×99 resolution. More specifically, refer to... Figures 1 to 3 , Figure 4 An example embodiment of the present disclosure is shown. Figure 1 The system 100 implements a fingertip regressor architecture for fingertip positioning.

[0048] By system 100 and Figure 4 The CNN architecture implemented in this disclosure includes two convolutional blocks, each with three convolutional layers, followed by a max-pooling layer. Finally, three fully connected layers are used to regress the two coordinate values ​​of the fingertip point in the last layer. In this disclosure, Figure 4 The architecture of a fingertip regressor for fingertip localization is described. The input to the Bi-LSTM / LSTM classification network is a 3×99×99 RGB image. Each of the two convolutional blocks has three convolutional layers, followed by a max-pooling layer. The three fully connected layers regress on the fingertip's spatial location. Because the goal is to determine a continuous output corresponding to the fingertip location, the mean squared error (MSE) measure is used to calculate the loss of the last fully connected layer. The model is robustly trained for localization and compared with architectures proposed using conventional techniques.

[0049] In embodiments of this disclosure, at step 310, one or more hardware processors 104 use first and second coordinates from the spatial location of the fingertip to classify fingertip motion patterns into one or more gestures in real time via a bidirectional long short-term memory (Bi-LSTM) network included in a cascaded deep learning model (CDLM) executed on a mobile communication device. In other words, this set of coordinates (e.g., the spatial location – the x and y coordinates of the fingertip motion pattern) is then fed into the Bi-LSTM network for classifying the motion pattern into different gestures. More specifically, each fingertip motion pattern is classified into one or more gestures by applying regression techniques to the first coordinates (e.g., assumed "x" coordinate) and second coordinates (e.g., assumed "y" coordinate) of the fingertip. In one embodiment, for actions (e.g., gestures) performed by a user, such as… Figure 2 The 'x' and 'y' coordinates of the shown fingertip (or fingertip movement pattern) are 45 and 365, respectively. In another embodiment, for another action the user is performing, such as... Figure 2 The 'x' and 'y' coordinates of the fingertip shown are 290 and 340, respectively. In another embodiment, for another action the user is performing, such as... Figure 2 The 'x' and 'y' coordinates of the fingertip shown are 560 and 410, respectively. Additionally, in describing the Bi-LSTM / LSTM classification network... Figure 2 In part (c), this disclosure also describes classifying fingertip detections on subsequent frames into different gestures (e.g., checkmark, right, rectangle, X (or delete), etc.). Furthermore, for each of these gestures to which a particular fingertip motion pattern is classified, System 100 or the Bi-LSTM / LSTM classification network calculates (or provides) a probability score indicating the probability that the particular fingertip motion pattern is recognized / classified as a candidate gesture (e.g., the probability score can be calculated using techniques known in the prior art). For example, in one example embodiment, for the 'x' and 'y' coordinates of fingertips of 45 and 365 respectively, the Bi-LSTM / LSTM classification network has classified the fingertip motion pattern as a "checkmark gesture" and has calculated a probability score of 0.920 for the fingertip motion pattern as a checkmark gesture. In other words, in one example embodiment, a probability score of 0.920 indicates that the particular fingertip motion pattern is a possible checkmark gesture based on its associated spatial location (or x' and 'y' coordinates) and is classified. Similarly, as Figure 4 As shown, probability scores are calculated for other fingertip movement patterns to classify them as other gestures.

[0050] As described above, the fingertip localization network (or fingertip regressor) outputs the spatial position (x, y) of the fingertip, which is then fed as input to the gesture classification network (or Bi-LSTM network). To reduce computational cost, the input (x, y) coordinates are adjusted by System 100, rather than by the entire frame of the Bi-LSTM network, thus contributing to real-time performance. Experiments conducted according to this disclosure show that, for a specific classification task, the Bi-LSTM network implemented by System 100 outperforms the LSTM network because they process sequences in both forward and backward passes. The use of LSTM inherently implies that the entire framework is also suitable for video and live feeds with variable-length frame sequences. This is particularly important because the length of a gesture depends on the time it takes for the user to perform the gesture and the performance of the first two networks.

[0051] Feasibility studies have been conducted on conventional technologies for economical applications. Cardboard ranked available interaction modes and reported that frequent use of the magnetic trigger and conductive rod caused wear and tear on the device, resulting in a poor usability score. Therefore, this disclosure implements automatic and implicit triggering to indicate the start and end of a user input sequence. In the case of positive finger hand detection on five consecutive frames, the trigger frame is used to begin recording the spatial position of the fingertip. In other words, the spatial position of the fingertip is detected based on the presence of positive finger hand detection on a set of consecutive frames across multiple RGB input images, and the presence of positive finger hand detection indicates the start of a gesture.

[0052] Similarly, no hand detection represents the end of a gesture across (5) consecutive frames. In other words, no pointing finger hand detection represents the end of a gesture across a set of consecutive frames in multiple RGB input images. The recorded sequence is then fed as input into a Bi-LSTM layer consisting of 30 units. Forward and backward activations are doubled before being passed to the next flattened layer that makes the data one-dimensional. This is followed by a fully connected layer with 10 output scores corresponding to each of the 10 gestures. Since the task is to classify 10 gesture classes, the output scores are interpreted as unnormalized log probabilities using a softmax activation function, and the output scores are suppressed to between 0 and 1 using the following equation:

[0053]

[0054] Where K represents the number of classes, s is a K×1 score vector, the input to the softmax function, and j is an exponent from 1 to K. σ(s) is a K×1 output vector representing the posterior probability associated with each gesture. Cross-entropy loss is used during training to update the model during backpropagation.

[0055] Dataset

[0056] This disclosure uses the SCUT-Ego-Finger dataset (e.g., see Deepfinger: A Cascaded Convolutional Neural Network Approach for Finger Keypoint Detection in Autocentric Vision of Moving Cameras). It was used for training in the 2015 IEEE International Conference Journal of Systems, Humanity and Cybernetics (SMC), pp. 2944-2949, IEEE (see also Huang et al., 2015). Figure 2 The hand detection and fingertip localization module is shown. The dataset includes 93,729 pointing gesture frames, including candidate hand bounding boxes and index finger keypoint coordinates.

[0057] (EgoGestAR) dataset

[0058] In the task of recognizing temporal gestures, a major obstacle to deep learning is the lack of available large-scale datasets for training neural networks. Therefore, this disclosure uses an egocentric visual gesture dataset from an AR / MR wearable to train and evaluate the gesture classification network. The dataset includes 10 gesture patterns. To introduce variability into the data, the dataset was collected from 50 randomly selected (from a laboratory) subjects aged 21 to 50 years. The mean age of the subjects was 27.8 years. The dataset consists of 2500 gesture patterns, with 5 samples of each gesture recorded for each subject. Gestures were recorded by mounting a tablet PC to a wall. Patterns drawn by the user's index finger on a touch interface application with a position-sensing area are stored. Data was acquired at a resolution of 640×480. Figure 5 A standard input sequence is described, presented to the user prior to data collection. These gestures from subjects (or users) are primarily categorized into three types for effective utilization within the context of data visualization in the mixed reality (MR) applications of this disclosure. More specifically, Figure 5 Reference Figures 1 to 4 A sequence of gestures is depicted before data collection according to exemplary embodiments of this disclosure. These three categories should not be construed as limiting the scope of the invention, and are presented herein by way of example and for a better understanding of the embodiments described herein:

[0059] 1.4 swipe gesture modes (up, down, left, and right) to navigate graphical visualizations / lists.

[0060] 2.2 gesture modes (rectangle and circle) are used to highlight the region of interest (RoI) in the user's FoV and to perform zoom in and zoom out operations.

[0061] 3.4 gesture patterns (check mark: yes; caret: no; X: delete; star: bookmark) for answering contextual questions when interacting with applications such as industrial inspection (Ramakrishna et al., 2016).

[0062] In addition, to test the entire framework implemented by the system and method of this disclosure, 240 videos were recorded from a random subset of the aforementioned subjects performing each gesture 22 times. An additional 20 videos of random hand movements were also recorded. (The last sentence appears to be incomplete and possibly refers to a different method or technique.) Cardboard The device records video. It captures high-quality video at a resolution of 640×480 and 30 frames per second (FPS).

[0063] Experiments and Results

[0064] Since the framework implemented in System 100 of this disclosure includes three networks, the performance of each network is evaluated individually to achieve the optimal combination of networks for the application proposed in this disclosure. Experiments were conducted using 8 cores. Core TM i7-6820HQ CPU, 32GB storage and A machine with a Quadro M5000 GPU. It uses... 845 chipset smartphone, which uses a local area network hosted on a Linksys EA6350 802.11ac compatible wireless router to connect to the server (evaluate the method of running on the device if needed).

[0065] For all experiments involving hand detection and fingertip localization conducted in this disclosure, the hand dataset described above was used. Of the 24 subjects present in the dataset, data from 17 subjects were selected for training, validation was performed with a 70:30 split, and data from 7 subjects (24; 155 images) were used to test the network.

[0066] Hand inspection

[0067] Table 1 reports the percentage of mean absolute accuracy (mAP) and frame rate for hand candidate detection. More specifically, Table 1 describes the performance of various methods on the SCUT-Ego-Finger hand detection dataset. mAP scores, frame rates, and model sizes as a function of IoU are reported.

[0068] Table 1

[0069]

[0070] While MobileNetV2 achieves a higher frame rate than others, it produces a high false positive rate, resulting in poor classification performance. Existing techniques (e.g., YOLOv2, depicted by the dashed line) can also be observed to run on devices, although it outputs fewer frames compared to MobileNetV2. At an Intersection over Union (IoU) of 0.5, YOLOv2 (depicted by the dashed line) achieves 93.9% mAP on the SCUT-Ego-Finger hand dataset, while MobileNetV2 only achieves 89.1% mAP. However, it is also observed that existing techniques (e.g., YOLOv2, depicted by the dashed line) perform worse than MobileNetV2 in terms of the higher IoU required to locate hand candidates including fingertips. Figure 6 Reference Figures 1 to 5Image comparisons of the present disclosure and conventional methods according to exemplary embodiments of the present disclosure are depicted, with the present disclosure and conventional methods respectively indicating the results of detectors (hand candidate bounding boxes) under different conditions such as poor lighting, rendering blur, indoor and outdoor environments. It is noteworthy that even when neither detector is likely to predict false alarms in the background, prior art (e.g., YOLOv2 depicted by dashed lines) allows for more localization errors, demonstrating that MobileNetV2 is better suited to the use cases described in the present disclosure.

[0071] It is also worth noting that the MobileNetV2 model size is significantly smaller than the others. This allows the model to be ported to mobile devices and eliminates the framework's dependency on remote servers. This helps reduce latency introduced by the network and enables wider application of cost-effective devices in MR applications.

[0072] fingertip positioning

[0073] This disclosure evaluates a model for fingertip localization on a test set of 24,155 images. When compared with networks employing conventional techniques (e.g., see *Autocentric Interaction Systems Based on Pointing Gestures: Databases, Methods, and Applications*, IEEE Conference Proceedings on Computer Vision and Pattern Recognition, pp. 16-23, by Huang, Y., Liu, X., Zhang, X., and Jin, L., also Huang et al., 2016), the 2×1 continuous value output corresponding to the finger coordinates estimated in the last layer is compared with the ground truth to compute an error threshold (in pixels) and a synthetic map, such as... Figures 7A to 7B As shown. More specifically, refer to Figures 1 to 6 , Figures 7A to 7B A graphical representation illustrating a comparison between the finger positioning method of this disclosure and conventional techniques, according to exemplary embodiments of this disclosure.

[0074] This disclosure uses the Adam optimizer with a learning rate of 0:001. The model achieves 89.06% accuracy on an input image with a resolution of 99×99, with an error margin of 10 pixels. The mean absolute error (MAO) for the method of this disclosure is 2.72 pixels, while the MAO for networks proposed in conventional techniques is 3.59 pixels. Figures 7A to 7B The graphical representation clearly shows that the model implemented by this invention achieves a higher success rate at any given error threshold (see...). Figure 7B The method of this invention results in a higher proportion of images with low positioning errors.

[0075] Gesture Classification

[0076] This disclosure utilizes a proprietary dataset to train and test a gesture classification network. An attempt was made to use an LSTM network for classification in the same training and testing environment as Bi-LSTM. During training, 2000 gesture patterns from the training set were used. A total of 8230 network parameters were trained with batch size 64 and validation split 80:20. The Adam optimizer with a learning rate of 0:001 was used. After 900 epochs of training, the validation accuracies of LSTM and Bi-LSTM were 95.17% and 96.5%, respectively. LSTM and Bi-LSTM achieved classification accuracies of 92.5% and 94.3%, respectively, outperforming conventional methods (or techniques) for similar classification tasks. A comparison of the classification performance of the LSTM and Bi-LSTM methods of this system with conventional techniques is shown in Table 2 below.

[0077] Table 2

[0078] method accuracy recall F1 score Conventional Technology / Research X 0.741 0.76 0.734 Conventional technology / research Y 0.860 0.842 0.851 LSTM 0.975 0.920 0.947 Bi-LSTM (This Disclosure) 0.956 0.940 0.948

[0079] Conventional techniques / research include, for example, Conventional Techniques / Research X – a comparison involving two real-time gesture recognition systems using stereo cameras, depth cameras, and inertial sensors. In SPIE Photonics Europe, 91390C-91390C, International Society for Optics and Photonics, by Liu, K., Kehtarnavaz, N., and Carlsohn, M, 4 April 2014, and Conventional Techniques / Research Y – 'Liblinear: A Large Linear Classification Library', Journal of Machine Learning Research, August 9: 1871-1874, by Fan, R.-E., Chang, K.-W., Hsieh, C.-J., Wang, X.-R., and Lin, C.-J, 2008 – also authored by Fan et al. More specifically, Table 2 describes the performance of different classification methods on the proprietary dataset disclosed in this publication. The mean of precision and recall for all classes is calculated to obtain a single number.

[0080] Furthermore, the performance of conventional methods (or the conventional techniques shown in Table 2) was observed to deteriorate significantly in the absence of sufficient data points. Therefore, they rely on complex interpolation techniques (leading to additional processing time and memory consumption) to produce consistent results.

[0081] Framework assessment

[0082] Because the methods / procedures disclosed herein are implemented or performed using a series of different networks, the overall real-time classification accuracy can vary depending on the performance of each network used in the pipeline. Therefore, using smartphone-based... The framework was evaluated using 240 self-centered videos acquired by the Cardboard head-mounted device. The MobileNetV2 model was used in the experiments conducted in this publication because it achieves the best trade-off between accuracy and performance. Since the model can operate independently on smartphones using the TF-Lite engine, it eliminates the framework's dependence on remote servers and quality network connections.

[0083] The framework achieves an overall accuracy of 80.00% on a dataset of 240 self-centered videos obtained from FPV, such as... Figure 8 The matrix shown is also called the confusion matrix. More specifically, Figure 8 Reference Figures 1 to 7B An example embodiment according to the present invention is depicted. Figure 3 The method uses based Overall performance on 240 autocentric videos acquired by a smartphone for the Cardboard head-mounted device. Gestures were detected when the predicted probability was greater than 0.85. The accuracy of the method disclosed herein is 0.8 (excluding unclassified classes).

[0084] The MobileNetV2 network implemented by System 100 operates at 9 FPS on 640×480 resolution video, and the fingertip regressor implemented by System 100, configured to operate at 99×99 resolution, achieves a frame rate of up to 166 FPS. The gesture classification network implemented by System 100 processes a given data stream in less than 100 ms. As a result, in the... On smartphones powered by the 845 chipset, the average response time of the frame was found to be 0:12s. The entire model has a (very small) memory footprint of 16.3MB.

[0085] The system and method of this disclosure are further compared with conventional end-to-end training gesture classification techniques (TGCCAT), and the results are shown in Table 3. More specifically, Table 3 describes an analysis of gesture recognition accuracy and latency for various conventional models / techniques against the method of this disclosure. As can be seen from Table 3, the method of this disclosure works on the device and effectively achieves the highest accuracy and the lowest response time.

[0086] Table 3

[0087] method Accuracy time consuming On the equipment TGCCAT 1 32.27 0.76 no TGCCAT 2 58.18 0.69 no TGCCAT 3 66.36 1.19 no This disclosure 80.00 0.12 yes

[0088] The conventional technique TGCCAT 1 proposes a network that works with a difference image input to a convolutional LSTM to capture the motion of body parts involved in a gesture performed in a second-person view. Even after fine-tuning the model on the video dataset of this disclosure, it yielded only 32.14% accuracy because the data in this disclosure involves dynamic backgrounds and has no static reference to the camera.

[0089] The conventional technique TGCCAT 2 uses a 2D CNN to extract features from each frame. These frame-by-frame features are then encoded into a temporal depth video descriptor, which is fed into an LSTM network for classification. Similarly, the 3D CNN method (the conventional technique TGCCAT 3) uses a 3D CNN to extract features directly from the video clips. Table 3 shows that both conventional methods perform poorly. One possible intuitive reason for this is that the network may be learning noisy and bad features during training. Other conventional techniques, such as attention-based video classification, also perform poorly due to high inter-class similarity. Since only a small subset of features from the entire frame, namely the fingertip, is needed, and the fingertip location is known, this attention model seems redundant.

[0090] Other existing / conventional technologies and systems achieve this by using virtual buttons that appear in a stereoscopic view when a fingertip is placed on it, which is similar to user interaction based on air fingertips. This conventional technology employs a Fast Region Convolutional Neural Network (RCNN) to classify gestures and also implements powerful but underutilized networked GPU servers, which are more expensive. Conventional technologies and systems also rely on a high-bandwidth, low-latency network connection between the device and the aforementioned server. Unlike the conventional systems and methods / techniques described above, embodiments of this disclosure provide systems and methods for on-device finger-pointing gesture interfaces for devices (e.g., smartphones) and Video See-Through Headsets (VSTH) or VSTH devices. By using the VSTH device of this disclosure, the system 100 of this disclosure becomes a lightweight gesture interface for classifying pointing gestures, executed purely by the user on the device (particularly smartphones and VSTH devices). Furthermore, the system 100 of this disclosure implements and executes memory and computes an efficient MobileNetv2 architecture to locate hand candidates and different fingertip regressor frameworks to track the user's fingertips, and a Bidirectional Long Short-Term Memory (Bi-LSTM) model for gesture classification. The advantage of such an architecture or cascaded deep learning model (CDLM) implemented by System 100 of this disclosure is that System 100 does not rely on a powerful, networked GPU server. Since all computation is performed on the device itself, System 100 can be deployed in network-free environments and further opens up new avenues for applications in remote locations.

[0091] This written description sets forth the subject matter of this document to enable those skilled in the art to make and use embodiments. The scope of the subject matter embodiments is defined by the claims and may include other modifications that may occur to those skilled in the art. Such other modifications are intended to fall within the scope of the claims if they have similar elements that are not different from the literal language of the claims, or if they include equivalent elements that are not substantially different from the literal language of the claims.

[0092] It should be understood that the scope of protection extends to programs containing messages, and beyond computer-readable storage devices, such computer-readable storage devices contain program code means for implementing one or more steps of the method when the program is run on a server or mobile device or any suitable programmable device. The hardware device can be any type of programmable device, including any type of computer such as a server or personal computer, or any combination thereof. The device may also include means that can be, for example, hardware means such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination of hardware and software means such as an ASIC and an FPGA, or at least one microprocessor and at least one memory and a software module therein. Therefore, the device can include both hardware and software means. The method embodiments described herein can be implemented in both hardware and software. The device may also include software means. Optionally, embodiments may be implemented on different hardware devices, such as using multiple CPUs.

[0093] The embodiments described herein may include hardware and software elements. Software-implemented embodiments include, but are not limited to, firmware, resident software, microcode, etc. The functions performed by the various modules described herein may be implemented in other modules or combinations of other modules. For the purposes of this description, a computer-usable or computer-readable medium may be any means capable of including, storing, communicating, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0094] The steps described herein are intended to explain the exemplary embodiments shown, and it should be anticipated that ongoing technological developments will change the way particular functions are performed. These embodiments are provided herein for illustrative purposes and not for limitation. Furthermore, for ease of description, the boundaries of functional building blocks have been arbitrarily defined herein. The boundaries of substitutions can be defined as long as the specified functions and their relationships are properly performed. Based on the teachings contained herein, substitutions (including equivalents, extensions, variations, deviations, etc., of those described herein) will be apparent to those skilled in the art. These alternatives fall within the scope and spirit of the disclosed embodiments. Furthermore, the words “comprising,” “having,” “containing,” and “including,” and other similar forms, are intended to be equivalent in meaning and are open-ended, as one or more items following any of these words are not intended to be an exhaustive list of such one or more items, or to be limited to the one or more items listed. It must also be noted that, as used herein and in the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly indicates otherwise.

[0095] Furthermore, one or more computer-readable storage media may be used to implement embodiments according to this disclosure. A computer-readable storage medium refers to any type of physical memory on which processor-readable information or data can be stored. Therefore, a computer-readable storage medium may store instructions executable by one or more processors, including instructions for causing the processor to perform steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible items and exclude carrier waves and transient signals, i.e., non-transient. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard disk drives, CD-ROMs, DVDs, flash drives, magnetic disks, and any other known physical storage media.

[0096] This disclosure and examples are to be considered exemplary only, and the true scope and spirit of the disclosed embodiments are indicated by the appended claims.

Claims

1. A processor-implemented method for classifying fingertip movement patterns as gestures on a device in real time, the method comprising: In a cascaded deep learning model CDLM executed via one or more hardware processors of a mobile communication device, video from an image acquisition device or multiple red, green, and blue (RGB) input images from a live feed are received in real time (302), wherein the CDLM includes a MobileNetV2 as an object detector, a fingertip regressor, and a bidirectional long short-term memory Bi-LSTM network for accurate gesture recognition, and wherein the CDLM is ported to the mobile communication device and eliminates the gesture recognition framework's dependence on a remote server; Using the MobileNet V2 included in the CDLM executed via the one or more hardware processors on the mobile communication device, multiple hand candidate bounding boxes (304) from multiple received RGB input images are detected in real time, wherein each of the multiple hand candidate bounding boxes is specific to a corresponding RGB image from the multiple received RGB input images, and wherein each of the multiple hand candidate bounding boxes includes a hand candidate; The hand candidates are narrowed from each of the plurality of hand candidate bounding boxes to obtain a narrowed set of hand candidates (306). Using the fingertip regressor included in the CDLM executed via the one or more hardware processors on the mobile communication device, the spatial position (308) of the fingertip of each reduced hand candidate from the reduced hand candidate set is detected in real time, wherein a first coordinate and a second coordinate are derived from the spatial position of the fingertip, wherein the spatial position of the fingertip from the reduced hand candidate set represents a fingertip movement pattern, wherein the fingertip regressor is implemented based on a convolutional neural network (CNN) architecture to locate the first and second coordinates of the fingertip, wherein the CNN consists of two convolutional blocks and three fully connected layers for regressing the spatial position of the fingertip, wherein each of the two convolutional blocks has three convolutional layers followed by a max-pooling layer; and Using the first and second coordinates from the spatial location of the fingertip, the fingertip movement pattern is classified into one or more gestures in real time via the Bi-LSTM network included in the CDLM executed via the one or more hardware processors on the mobile communication device (310).

2. The processor-implemented method according to claim 1, wherein, Each hand candidate bounding box, which includes the hand candidates, describes a pointing gesture to be classified as one or more gestures.

3. The processor-implemented method according to claim 1, wherein, The step of classifying the fingertip movement pattern into one or more gestures includes applying regression techniques to the first and second coordinates of the fingertip.

4. The processor-implemented method according to claim 1, wherein, The spatial position of the fingertip is detected based on the presence of a positive pointing finger hand detection on a set of consecutive frames in the plurality of RGB input images, wherein the presence of the positive pointing finger hand detection indicates the start of the gesture.

5. The processor-implemented method according to claim 1, wherein, There is no positive pointing finger hand detection indicator indicating the end of the gesture in a set of consecutive frames of the plurality of RGB input images.

6. A system (100) for real-time classification of fingertip movement patterns into gestures, the system comprising: Memory (102), which stores instructions; One or more communication interfaces (106); as well as One or more hardware processors (104) are coupled to the memory (102) via one or more communication interfaces (106), wherein the one or more hardware processors (104) are configured by the instructions to: In a cascaded deep learning model CDLM executed via one or more hardware processors of the system, video from an image acquisition device or multiple red, green, and blue (RGB) input images from a live feed are received in real time. The CDLM includes a MobileNetV2 object detector, a fingertip regressor, and a bidirectional long short-term memory (Bi-LSTM) network for accurate gesture recognition. The CDLM is ported to mobile communication devices and eliminates the gesture recognition framework's dependence on remote servers. Using the MobileNet V2 included in the CDLM executed via the one or more hardware processors on the system, multiple hand candidate bounding boxes are detected in real time from multiple received RGB input images, wherein each of the multiple hand candidate bounding boxes is specific to a corresponding RGB image from the multiple received RGB input images, and wherein each of the multiple hand candidate bounding boxes includes a hand candidate; The hand candidates are narrowed down from each of the plurality of hand candidate bounding boxes to obtain a narrowed set of hand candidates; Using the fingertip regressor included in the cascaded deep learning model CDLM executed via the one or more hardware processors on the system, the spatial position of the fingertip from each of the reduced hand candidate sets is detected in real time, wherein a first coordinate and a second coordinate are derived from the spatial position of the fingertip, wherein the spatial position of the fingertip from the reduced hand candidate sets represents a fingertip movement pattern, wherein the fingertip regressor is implemented based on a convolutional neural network (CNN) architecture to locate the first and second coordinates of the fingertip, wherein the CNN consists of two convolutional blocks and three fully connected layers for regressing the spatial position of the fingertip, wherein each of the two convolutional blocks has three convolutional layers followed by a max-pooling layer; and Using the first and second coordinates from the spatial location of the fingertip, the fingertip movement pattern is classified into one or more gestures in real time via the Bi-LSTM network included in the CDLM executed by the one or more hardware processors on the system.

7. The system according to claim 6, wherein, Each hand candidate bounding box, which includes the hand candidates, describes a pointing gesture to be classified as one or more gestures.

8. The system according to claim 6, wherein, By applying regression techniques to the first and second coordinates of the fingertip, the fingertip movement pattern is classified into one or more gestures.

9. The system according to claim 6, wherein, The spatial position of the fingertip is detected based on the presence of a positive pointing finger hand detection on a set of consecutive frames in the plurality of RGB input images, wherein the presence of the positive pointing finger hand detection indicates the start of the gesture.

10. The system according to claim 6, wherein, There is no positive pointing finger hand detection indicator indicating the end of the gesture in a set of consecutive frames of the plurality of RGB input images.

11. One or more non-transitory machine-readable information storage media, the one or more non-transitory machine-readable information storage media comprising one or more instructions, which, when executed by one or more hardware processors, classify fingertip movement patterns into gestures in real time in the following manner: In a cascaded deep learning model CDLM executed via one or more hardware processors of a mobile communication device, video from an image acquisition device or multiple red, green, and blue (RGB) input images from a live feed are received in real time, wherein... The CDLM includes MobileNetV2 as an object detector, a fingertip regressor, and a bidirectional long short-term memory (Bi-LSTM) network for accurate gesture recognition, and wherein the CDLM is ported to the mobile communication device and eliminates the gesture recognition framework's dependence on a remote server. Using the MobileNet V2 included in the CDLM executed via the one or more hardware processors on the mobile communication device, multiple hand candidate bounding boxes from multiple received RGB input images are detected in real time, wherein each of the multiple hand candidate bounding boxes is specific to a corresponding RGB image from the multiple received RGB input images, and wherein each of the multiple hand candidate bounding boxes includes a hand candidate; The hand candidates are narrowed down from each of the plurality of hand candidate bounding boxes to obtain a narrowed set of hand candidates; Using the fingertip regressor included in the CDLM executed via one or more hardware processors on the mobile communication device, the spatial position of the fingertip of each reduced hand candidate from the reduced hand candidate set is detected in real time, wherein a first coordinate and a second coordinate are derived from the spatial position of the fingertip, wherein the spatial position of the fingertip from the reduced hand candidate set represents a fingertip movement pattern, wherein the fingertip regressor is implemented based on a convolutional neural network (CNN) architecture to locate the first and second coordinates of the fingertip, wherein the CNN consists of two convolutional blocks and three fully connected layers for regressing the spatial position of the fingertip, wherein each of the two convolutional blocks has three convolutional layers followed by a max-pooling layer; and Using the first and second coordinates from the spatial location of the fingertip, the fingertip movement pattern is classified into one or more gestures in real time via the Bi-LSTM network included in the CDLM executed via the one or more hardware processors on the mobile communication device.

12. One or more non-transitory machine-readable information storage media according to claim 11, wherein, Each hand candidate bounding box, which includes the hand candidates, describes a pointing gesture for classification as one or more gestures.

13. One or more non-transitory machine-readable information storage media according to claim 11, wherein, The step of classifying the fingertip movement pattern into one or more gestures includes applying regression techniques to the first and second coordinates of the fingertip.

14. One or more non-transitory machine-readable information storage media according to claim 11, wherein, The spatial position of the fingertip is detected based on the presence of a positive pointing finger hand detection on a set of consecutive frames in the plurality of RGB input images, wherein the presence of the positive pointing finger hand detection indicates the start of the gesture.

15. One or more non-transitory machine-readable information storage media according to claim 11, wherein, There is no positive pointing finger hand detection indicator indicating the end of the gesture in a set of consecutive frames of the plurality of RGB input images.

Images