Electronic device and method for performing gesture recognition

Abstract

A method and apparatus for performing gesture recognition in an electronic device are provided, including: tracking a visible trajectory of a user's hand from a plurality of frames captured by the electronic device; identifying, by the electronic device, a first frame in which the user's hand has left a field of view (FOV) of the electronic device; identifying that the user's hand has returned to a second frame in the FOV; obtaining, using an artificial intelligence (AI) model, a trajectory of the user's hand using one or more frames captured before the first frame and one or more frames captured after the second frame; and identifying at least one gesture performed during the visible trajectory of the user's hand and the obtained trajectory of the user's hand.

Description

Technical Field

[0001] The embodiments disclosed herein relate to human-computer interaction using gestures and electronic devices, and more particularly, to out-of-field hand tracking using spatiotemporal outward and inward convergence. Background Technology

[0002] Gestures are a primary mode of feedback and interaction with devices such as head-mounted displays (HMDs). Therefore, hand tracking is crucial for HMD interaction and augmented reality (AR) / virtual reality (VR) applications. A challenging aspect of hand tracking is that the user's hand is not always within the camera's field of view. Existing algorithms typically reset when the hand leaves the field of view and resume tracking once the hand returns. A drawback of using these algorithms is that gesture recognition systems generally require the hand to be present in all frames for accurate gesture recognition. Summary of the Invention

[0003] Technical solution According to various aspects of this disclosure, a method for performing gesture recognition in an electronic device is provided, the method comprising: tracking a visible trajectory of a user's hand from a plurality of frames captured by the electronic device; identifying, by the electronic device, a first frame in which the user's hand has left the field of view (FOV) of the electronic device; identifying, by the electronic device, a second frame in which the user's hand has returned to the FOV of the electronic device; obtaining, by the electronic device, the trajectory of the user's hand using an artificial intelligence (AI) model, using one or more frames captured before the first frame and one or more frames captured after the second frame; and identifying, by the electronic device, at least one gesture performed during the visible trajectory of the user's hand and the obtained trajectory of the user's hand.

[0004] The method may further include: selecting, by an electronic device, a frame containing the user's hand from a plurality of frames via a visible trajectory; and generating, by the electronic device, one or more references for hand markers based on the frame containing the user's hand.

[0005] The method may further include: estimating the position of one or more hand markers in one or more frames captured prior to the first frame by an electronic device based on a reference to one or more generated hand markers; calculating one or more dynamic parameters of each hand marker by the electronic device using the estimated positions of the one or more hand markers in consecutive frames, wherein the consecutive frames include one or more frames captured prior to the first frame; and obtaining a first trajectory of the user's hand corresponding to the one or more frames captured prior to the first frame by the electronic device using the one or more dynamic parameters of each of the calculated one or more hand markers.

[0006] The method may further include: reversing the order of multiple frames captured by the electronic device; estimating the position of one or more hand markers in one or more frames captured after the second frame by the electronic device based on a reference to one or more hand markers generated; calculating one or more dynamic parameters of each hand marker by the electronic device using the estimated positions of the one or more hand markers in consecutive frames, wherein the consecutive frames include one or more frames captured after the second frame; and obtaining a second trajectory of a user's hand using the one or more dynamic parameters of each of the one or more hand markers calculated by the electronic device in the one or more frames captured after the second frame.

[0007] One or more dynamic parameters of the user's hand include at least one of the velocity and acceleration of each of one or more hand signs.

[0008] The method may further include: after calculating one or more dynamic parameters of each of the one or more hand markers, verifying by the electronic device whether the user's hand is in the FOV of the electronic device; and if the user's hand is not in the FOV of the electronic device, using the calculated one or more dynamic parameters of each of the one or more hand markers to calculate the velocity and position of one or more hand markers in the next frame after one or more frames captured after the second frame.

[0009] The method may further include: after calculating the velocity and position of one or more hand markers in the next frame, verifying at least one parameter of the user's hand by an electronic device, wherein the at least one parameter includes at least one of whether the velocity becomes zero, whether the hand position exceeds a threshold, and whether one or more hand markers no longer conform to predetermined biomechanical constraints of the human hand; if at least one parameter of the user's hand is not satisfied, the electronic device repeats the verification of the user's hand in the FOV of the electronic device; and if the user's hand is stationary and at least one parameter of the user's hand is satisfied, the electronic device repeats the estimation of the position of one or more hand markers based on the previous frame before the next frame.

[0010] The method may further include: the electronic device checking the proximity of one or more hand markers for each frame of a first trajectory and a second trajectory of a user's hand in a plurality of frames; the electronic device achieving spatiotemporal convergence at a frame in the plurality of frames where the distance between two extrapolated hand markers is less than a specific threshold, wherein the trajectory prior to the spatiotemporally converged frame is considered the first trajectory, and the trajectory after the spatiotemporally converged frame is considered the second trajectory; the electronic device estimating a gesture by encoding the two extrapolated hand markers of the spatiotemporally converged gesture; and the electronic device recognizing at least one gesture based on a sequence of gesture information.

[0011] According to various aspects of this disclosure, an electronic device is provided, comprising: a memory storing at least one instruction; and at least one processor configured to execute at least one instruction stored in the memory, wherein the at least one processor is configured to execute at least one instruction to perform the following operations: tracking a visible trajectory of a user's hand from a plurality of frames captured by the electronic device, wherein the plurality of frames includes a first frame, a second frame, one or more frames captured before the first frame, and one or more frames captured after the second frame; identifying, in the plurality of frames, a first frame in which the user's hand has left the field of view (FOV) of the electronic device; identifying, in the plurality of frames, a second frame in which the user's hand has returned to the FOV of the electronic device; obtaining the trajectory of the user's hand using an artificial intelligence (AI) model, using one or more frames captured before the first frame and one or more frames captured after the second frame; and identifying at least one gesture performed during the visible trajectory of the user's hand and the obtained trajectory of the user's hand.

[0012] At least one processor is configured to execute at least one instruction to perform the following operations: select, from a plurality of frames, a frame in which a user’s hand is present by means of a visible trajectory; and generate a reference for one or more hand markers based on the frame in which a user’s hand is present.

[0013] At least one processor is configured to execute at least one instruction to perform the following operations: estimating the position of one or more hand markers in one or more frames captured prior to a first frame based on a reference to one or more generated hand markers; calculating one or more dynamic parameters for each hand marker using the estimated positions of the one or more hand markers in consecutive frames, wherein the consecutive frames include one or more frames captured prior to the first frame; and obtaining a first trajectory of a user's hand using the one or more dynamic parameters of each of the calculated one or more hand markers in the one or more frames captured prior to the first frame.

[0014] At least one processor is configured to execute at least one instruction to perform the following operations: reverse the order of a plurality of frames captured by an electronic device; estimate the position of one or more hand markers in one or more frames captured after a second frame based on a reference to one or more hand markers generated; calculate one or more dynamic parameters for each hand marker using the estimated positions of the one or more hand markers in consecutive frames, wherein the consecutive frames include one or more frames captured after the second frame; and obtain a second trajectory of the user's hand corresponding to the one or more frames captured after the second frame using the one or more dynamic parameters of each of the calculated one or more hand markers.

[0015] One or more dynamic parameters of the user's hand may include at least one of the velocity and acceleration of each of one or more hand signs.

[0016] At least one processor is configured to execute at least one instruction to: after calculating one or more dynamic parameters of each of the one or more hand markers, verify whether the user's hand is in the FOV of the electronic device; and if the user's hand is not in the FOV of the electronic device, use the calculated one or more dynamic parameters of each of the one or more hand markers to calculate the velocity and position of one or more hand markers in multiple frames and in the next frame after one or more frames captured after the second frame.

[0017] At least one processor is configured to execute at least one instruction to perform the following operations: after calculating the velocity and position of one or more hand markers in the next frame, verifying at least one parameter of the user's hand, wherein the at least one parameter includes at least one of whether the velocity becomes zero, whether the hand position exceeds a threshold, and whether one or more hand markers no longer conform to predetermined biomechanical constraints of the human hand; if at least one parameter of the user's hand is not satisfied, repeating the verification of the user's hand in the FOV of the electronic device; and if the user's hand is stationary and at least one parameter of the user's hand is satisfied, repeating the estimation of the position of one or more hand markers based on the previous frame before the next frame.

[0018] The processor can also be configured to: check the proximity of one or more hand markers for each frame of a first and second trajectory of a user's hand in a plurality of frames; achieve spatiotemporal convergence at frames where the distance between two extrapolated hand markers in the plurality of frames is less than a specific threshold, wherein the trajectory preceding the spatiotemporally converged frame is considered the first trajectory, and the trajectory following the spatiotemporally converged frame is considered the second trajectory; estimate a gesture by encoding the two extrapolated hand markers of the spatiotemporally converged frame; and identify at least one gesture based on a sequence of gesture information.

[0019] According to one aspect of this disclosure, a method for predicting gestures in a multi-camera device is provided, the method comprising: predicting an outward trajectory by the multi-camera device for one or more frames in which a user’s hand leaves the field of view (FOV) of the multi-camera device; predicting an inward trajectory by the multi-camera device for one or more frames in which the user’s hand returns to the FOV of the multi-camera device, wherein the inward trajectory is predicted by reversing the order in which the user’s hand returns to the FOV of the one or more frames; estimating convergence points of the outward and inward trajectories by the multi-camera device in a temporal and spatial domains; and recognizing at least one gesture by the multi-camera device based on a sequence of gesture information obtained from the estimated convergence points, wherein the gesture is estimated by encoding hand markings extrapolated from one or more convergence points.

[0020] According to one aspect of this disclosure, a multi-camera device is provided, comprising: a memory; and at least one processor configured to execute at least one instruction stored in the memory, wherein the at least one processor is configured to execute at least one instruction to perform the following operations: obtaining an outward trajectory for one or more frames in which a user's hand leaves the field of view (FOV) of the multi-camera device; obtaining an inward trajectory for one or more frames in which the user's hand returns to the FOV of the multi-camera device, wherein the inward trajectory is predicted by reversing the order in which the user's hand returns to the one or more frames in the FOV; estimating convergence points of the outward and inward trajectories in a temporal and spatial domain; and recognizing at least one gesture based on a sequence of gesture information obtained from the estimated convergence points, wherein the gesture is estimated by encoding hand markings extrapolated from one or more convergence points.

[0021] At least one processor is configured to execute at least one instruction to control actions on at least one of the video game and user interface windows based on the recognition of at least one gesture.

[0022] According to one aspect of this disclosure, electronic devices and methods disclose operations for performing out-of-field hand tracking using spatiotemporal outward and inward convergence.

[0023] According to one aspect of this disclosure, electronic devices and methods disclose operations for encoding hand trajectory information based on trajectories from one or more previous and subsequent frames, even when the hand is outside the field of view of the camera.

[0024] According to one aspect of this disclosure, electronic devices and methods disclose operations for tracking the acceleration and direction of motion of a hand from a visible frame to predict spatial and temporal convergence points.

[0025] These and other aspects of the exemplary embodiments herein will be better appreciated and understood when considered in conjunction with the following description and accompanying drawings. However, it should be understood that while the following description indicates exemplary embodiments and their many specific details, it is given by way of illustration rather than limitation. Many changes and modifications can be made within the scope of the exemplary embodiments herein without departing from the spirit of the embodiments herein, and the exemplary embodiments herein encompass all such modifications. Attached Figure Description

[0026] Embodiments described herein are illustrated in the accompanying drawings, throughout which similar reference numerals indicate corresponding portions in the various figures. The embodiments described herein will be better understood through the following description with reference to the illustrative drawings. Embodiments described herein are illustrated by way of example in the accompanying drawings, wherein: Figure 1 Existing hand tracking methods based on the prior art are described; Figure 2An example scene of a head-mounted display (HMD) with a multi-camera array and field of view (FOV) of the cameras, based on existing technology, is depicted. Figure 3 A block diagram representation of an electronic device for performing gesture recognition in an electronic device according to embodiments disclosed herein is depicted; Figure 4 A method for performing gesture recognition in an electronic device according to embodiments disclosed herein is described; Figure 5 A method for estimating spatiotemporal convergence to perform gesture recognition in an electronic device, according to embodiments disclosed herein, is described. Figure 6 A comparison is depicted between existing gesture recognition methods according to embodiments disclosed herein and the disclosed gesture recognition methods; Figure 7 A method for predicting the trajectory of a hand outside the field of view (FOV) is described according to embodiments disclosed herein; Figure 8 A process for performing gesture recognition according to embodiments disclosed herein is described; Figure 9A An example flow procedure for inferring the trajectory of a hand not in the field of view and predicting spatiotemporal convergence for gesture recognition, according to embodiments disclosed herein, is described. Figure 9B An example flow procedure for inferring the trajectory of a hand not in the field of view and predicting spatiotemporal convergence for gesture recognition, according to embodiments disclosed herein, is described. Figure 10 An example use case is described where a user performs a swipe-left gesture according to an embodiment disclosed herein; Figure 11 An example use case is described, according to embodiments disclosed herein, of a user grasping a VR object and moving their hand from one camera field of view (FOV) to another; and Figure 12 Example use cases of users playing VR games according to embodiments disclosed herein are described. Detailed Implementation

[0027] The embodiments herein, along with their various features and advantageous details, are explained more fully with reference to the non-limiting embodiments illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques have been omitted to avoid unnecessarily obscuring the embodiments herein. The examples used herein are intended only to facilitate understanding of how the embodiments herein can be practiced and to enable those skilled in the art to practice the embodiments herein. Therefore, these examples should not be construed as limiting the scope of the embodiments herein.

[0028] For the purposes of interpreting this specification, definitions (as defined herein) will be applied, and where appropriate, terms used in the singular will also include the plural forms, and vice versa. It should be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. Unless otherwise stated, the terms “comprising,” “having,” and “including” should be interpreted as open-ended terms.

[0029] The words / phrases “exemplary,” “example,” “illustration,” “in instance,” “etc.,” “for example,” and “i.e.” are used herein only to mean “used as an example, instance, or illustration.” Any embodiment or implementation of the subject matter described herein using the words / phrases “exemplary,” “example,” “illustration,” “in instance,” “etc.,” “for example,” and “i.e.” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0030] Embodiments herein are described and illustrated in terms of blocks that perform the described functions. These blocks (which may be referred to herein as managers, units, modules, hardware components, etc.) are physically implemented by analog and / or digital circuitry (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuitry, passive electronic components, active electronic components, optical components, hardwired circuitry, etc.) and may optionally be driven by firmware. The circuitry may be embodied, for example, in one or more semiconductor chips or on a substrate support (such as a printed circuit board). The circuitry constituting a block may be implemented by dedicated hardware, by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware (for performing some functions of the block) and a processor (for performing other functions of the block). Each block of an embodiment may be physically divided into two or more interactive and discrete blocks without departing from the scope of this disclosure. Similarly, the blocks of an embodiment may be physically combined into more complex blocks without departing from the scope of this disclosure.

[0031] It should be noted that the elements in the accompanying drawings are shown for the purposes of this specification and ease of understanding, and may not necessarily be drawn to scale. For example, flowcharts / sequence diagrams illustrate methods of steps required to understand aspects of the embodiments disclosed herein. Furthermore, in terms of device construction, one or more components of the device may be represented in the drawings using conventional symbols, and the drawings may show only specific details relevant to understanding the embodiments, so as not to obscure details that are readily understood by those skilled in the art from the description herein. Similarly, in terms of systems, one or more components / modules constituting the system may be represented in the drawings using conventional symbols, and the drawings may show only specific details relevant to understanding the embodiments, so as not to obscure details that are readily understood by those skilled in the art from the description herein.

[0032] The accompanying drawings are provided to aid in the easy understanding of the various technical features, and it should be understood that the embodiments presented herein are not limited to the drawings. Therefore, this disclosure should be construed as extending to any modifications, equivalents, and alternatives other than those specifically set forth in the drawings and corresponding descriptions. The use of terms such as first, second, third, etc., to describe components / elements / steps is for the purposes of this specification and should not be construed as a sequential order / placement / occurrence unless otherwise stated.

[0033] Figure 1 Existing hand tracking methods based on the prior art are described.

[0034] Figure 1 Existing hand tracking methods are illustrated. Initially, two hands, shown as a gray hand (110) and a black hand (120), are detected in the field of view (FOV, 100) of an electronic device such as a head-mounted display (HMD). When the black hand (120) leaves the field of view (100) and returns, it is treated as a new hand (130) as shown in frames N+3 and N+4 due to the lack of accurate tracking. With existing hand tracking methods, it can be difficult to track the hand when it leaves the field of view (100). The main problem with existing hand tracking methods is that gesture recognition becomes more difficult when the hand is not included in the intermediate frames. This leads to inaccurate gesture detection and the inability to detect any gesture completely. For example, a waving gesture may be misclassified as a swipe gesture to the right. Detecting gestures is a very important feature for receiving user feedback and for user interaction with the device's user interface (UI). Furthermore, for devices such as HMDs, it may not always be possible to keep the hand in the camera's field of view (100). Therefore, it is necessary to accurately track the hand that leaves the field of view (100).

[0035] Figure 2 An example scene of a head-mounted display (HMD, 200) with a multi-camera array and field of view (FOV) of the cameras, according to the prior art, is depicted.

[0036] like Figure 2 As depicted in the example scenario, in an HMD 200, multiple camera arrays may exist, and the FOVs of these cameras may not overlap. Furthermore, in some cases, even if overlap exists, blind spots may still exist. For example... Figure 2 As shown, the FOV of the forward-facing camera 210 and the FOV of the downward-facing camera 220 may not overlap. When performing gestures in the HMD 200 use case, it is necessary to understand hand movements even when the hand is outside the FOV. The hand trajectory 230 is shown using arrows.

[0037] Therefore, there is a need in the art for solutions that overcome the aforementioned drawbacks.

[0038] Embodiments herein disclose electronic devices and methods for predicting the trajectory of a user's hand even when the hand is outside the camera's field of view. Referring now to the accompanying drawings, and more specifically to… Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Figure 9A , Figure 9B , Figure 10 , Figure 11 and Figure 12 In the accompanying drawings, similar reference numerals are consistently used to denote corresponding features, illustrating embodiments.

[0039] Figure 3 A block diagram representation of an electronic device for performing gesture recognition in an electronic device according to embodiments disclosed herein is depicted. Figure 3 An electronic device 300 (e.g., a multi-camera device and a (Visual Transmission (VST) device)) is shown. The electronic device 300 includes a processor 302, a communication module 304, and a memory 306.

[0040] In the embodiments described herein, instructions, data structures, and program code that can be read by processor 302 may be stored in memory 306. In the embodiments described herein, operations performed by processor 302 can be implemented by executing the instructions or code of the program stored in memory 306. The instructions, algorithms, data structures, program code, and applications stored in memory 306 may be implemented using programming or scripting languages ​​such as, for example, C, C++, Java, and assembler.

[0041] In the embodiments herein, the frame recognition module 308, the trajectory prediction module 310, and the gesture recognition module 312 may be stored in the memory 306. In the embodiments herein, a “module” included in the memory 306 may refer to a unit for processing functions or operations performed by at least one processor 302. A “module” included in the memory 306 may be embodied in software (such as instructions, algorithms, data structures, or program code).

[0042] In embodiments herein, at least one processor 302 may execute instructions, program code, or algorithms of frame recognition module 308 to track the visible trajectory of a user's hand from multiple frames captured by electronic device 300. At least one processor 302 may execute instructions, program code, or algorithms of frame recognition module 308 to identify a first frame in which the user's hand has left the field of view (FOV) of electronic device 300. At least one processor 302 may execute instructions, program code, or algorithms of frame recognition module 308 to identify a second frame in which the user's hand has returned to the FOV of electronic device 300.

[0043] In embodiments herein, at least one processor 302 may execute instructions, program code, or algorithms of trajectory prediction module 310 to predict the trajectory of a user's hand using one or more frames captured before a first frame and one or more frames captured after a second frame. At least one processor 302 may execute instructions, program code, or algorithms of trajectory prediction module 310 to obtain the trajectory of a user's hand using an artificial intelligence (AI) model. In embodiments herein, the AI ​​model may be an AI model trained to predict the trajectory of a user's hand using one or more frames captured before a first frame and one or more frames captured after a second frame. In embodiments herein, the AI ​​model included in trajectory prediction model 310 may include a machine learning model or a deep learning model. The AI ​​model may be an AI model trained to predict trajectories based on a training dataset labeled with trajectories and captured frame images.

[0044] In embodiments herein, at least one processor 302 may execute instructions, program code, or algorithms of trajectory prediction module 310 to select frames containing a user's hand from a visible trajectory. At least one processor 302 may execute instructions, program code, or algorithms of trajectory prediction module 310 to generate references to one or more hand markers based on frames containing a user's hand. At least one processor 302 may execute instructions, program code, or algorithms of trajectory prediction module 310 to estimate the positions of hand markers in frames captured prior to the first frame based on the generated references to the hand markers. At least one processor 302 may execute instructions, program code, or algorithms of trajectory prediction module 310 to calculate one or more dynamic parameters for each hand marker using the estimated positions of the hand markers in consecutive frames. The dynamic parameters of the user's hand may include, but are not limited to, the velocity and acceleration of each hand marker. At least one processor 302 may execute instructions, program code, or algorithms of trajectory prediction module 310 to obtain a first trajectory of the user's hand corresponding to frames captured prior to the first frame using the calculated dynamic parameters of each hand marker. Hand markings can be any part of the hand, creases, shape, relative distance between fingers, finger spacing, marks, bruises, relative discoloration, tattoos, scars, hair, nails, etc.

[0045] In embodiments herein, at least one processor 302 may execute instructions, program code, or algorithms of trajectory prediction module 310 to reverse the order of frames captured by electronic device 300. At least one processor 302 may execute instructions, program code, or algorithms of trajectory prediction module 310 to estimate the position of hand markers in frames captured after the second frame based on references to generated hand markers. At least one processor 302 may execute instructions, program code, or algorithms of trajectory prediction module 310 to calculate one or more dynamic parameters for each hand marker using the estimated positions of hand markers in consecutive frames. At least one processor 302 may execute instructions, program code, or algorithms of trajectory prediction module 310 to obtain a second trajectory of the user's hand corresponding to frames captured after the second frame using the calculated dynamic parameters of each hand marker.

[0046] In the embodiments described herein, after calculating one or more dynamic parameters for each hand marker, at least one processor 302 may execute instructions, program code, or algorithms of the trajectory prediction module 310 to verify whether the user's hand is within the field of view (FOV) of the electronic device 300. If the user's hand is not within the FOV of the electronic device 300, at least one processor 302 may execute instructions, program code, or algorithms of the trajectory prediction module 310 to calculate the velocity and position of the hand marker in the next frame using the calculated dynamic parameters for each hand marker.

[0047] In the embodiments described herein, after calculating the velocity and position of the hand marker in the next frame, at least one processor 302 may execute instructions, program code, or algorithms of the trajectory prediction module 310 to verify at least one parameter of the user's hand. This parameter may be, but is not limited to, whether the velocity becomes zero, whether the hand position is outside the realistic range, or whether the hand marker no longer conforms to the biomechanical constraints of the human hand. If the user's hand parameters are not met, at least one processor 302 may execute instructions, program code, or algorithms of the trajectory prediction module 310 to repeatedly verify the user's hand in the field of view (FOV) of the electronic device 300. If the user's hand is stationary and the user's hand parameters are met, at least one processor 302 may execute instructions, program code, or algorithms of the trajectory prediction module 310 to repeatedly estimate the position of the hand marker based on the previous frame.

[0048] In embodiments herein, at least one processor 302 may execute instructions, program code, or algorithms of gesture recognition module 312 to recognize at least one gesture performed during the visible trajectory of a user's hand and the predicted trajectory of a user's hand.

[0049] In embodiments herein, at least one processor 302 may execute instructions, program code, or algorithms of gesture recognition module 312 to check the proximity of hand markers for each frame of a first and second trajectory from the user's hand. At least one processor 302 may execute instructions, program code, or algorithms of gesture recognition module 312 to achieve spatiotemporal convergence at frames where the distance between two extrapolated hand markers is less than a specific threshold. The trajectory preceding the spatiotemporally converged frame is considered the first trajectory (outward trajectory). The outward trajectory is predicted for one or more frames where the user's hand leaves the FOV of electronic device 300. The trajectory following the spatiotemporally converged frame is considered the second trajectory (inward trajectory). The inward trajectory is predicted based on one or more frames where the user's hand returns to the FOV of electronic device 300. The inward trajectory is predicted by reversing the order of the frames where the user's hand returns to the FOV. The convergence points of the outward and inward trajectories are estimated in the temporal and spatial domains. At least one processor 302 may execute instructions, program code, or algorithms of the gesture recognition module 312 to estimate a gesture by encoding two extrapolated hand signs of spatiotemporal convergence or convergence point. At least one processor 302 may execute instructions, program code, or algorithms of the gesture recognition module 312 to recognize a gesture based on a sequence of gesture information obtained from the estimated convergence point.

[0050] In embodiments herein, processor 302 may process and execute data from multiple modules of electronic device 300. Processor 302 may be configured to execute instructions stored in memory module 306. Processor 302 may include one or more of a microprocessor, circuitry, and other hardware configured for processing. Processor 302 may be at least one of a single processor, multiple processors, multiple homogeneous or heterogeneous cores, multiple different types of central processing units (CPUs), microcontrollers, special media, and other accelerators. Processor 302 may be an application processor (AP), a graphics-only unit (such as a graphics processing unit (GPU), a visual processing unit (VPU)), and / or an artificial intelligence (AI) dedicated processor (such as a neural processing unit (NPU)).

[0051] According to embodiments of this disclosure, at least one processor 302 may include circuitry such as a system-on-a-chip (SoC) or an integrated circuit (IC).

[0052] In the embodiments described herein, multiple modules of the processor 302 of the electronic device 300 can communicate via a communication module 304. The communication module 304 can be in the form of a wired network or a wireless communication network module. The wireless communication network may include, but is not limited to, Global Positioning System (GPS), Global System for Mobile Communications (GSM), Wi-Fi, Bluetooth Low Energy, Near Field Communication (NFC), etc. Depending on the usage environment, the wireless communication may also include one or more of Bluetooth, ZigBee, short-range wireless communication (such as Ultra Wideband (UWB)) and mid-range wireless communication (such as Wi-Fi) or long-range wireless communication (such as 3G / 4G / 5G / 6G and non-3GPP technologies or WiMAX).

[0053] In embodiments described herein, memory module 306 may include one or more volatile and non-volatile memory components capable of storing data and instructions for a module of electronic device 300 to be executed. Examples of memory module 306 may be, but are not limited to, NAND, embedded multimedia card (eMMC), secure digital card (SD), universal serial bus (USB), serial advanced technology accessory (SATA), solid-state drive (SSD), etc. Memory module 306 may also include one or more computer-readable storage media. Examples of non-volatile storage elements may include magnetic hard disks, optical disks, floppy disks, flash memory, or electrically programmable memory (EPROM) or electrically erasable programmable memory (EEPROM). Furthermore, in some examples, memory module 306 may be considered a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or propagating signal. However, the term "non-transitory" should not be construed as indicating that memory module 306 is non-removable. In certain examples, non-transitory storage media may store data that may change over time (e.g., in random access memory (RAM) or cache memory).

[0054] Figure 3 Example modules of electronic device 300 are shown, but it should be understood that other embodiments are not limited thereto. In other embodiments, electronic device 300 may include fewer or more modules. Furthermore, the labels or names of modules are for illustrative purposes only and do not limit the scope of the invention. One or more modules may be combined together to perform the same or substantially similar functions in electronic device 300.

[0055] Figure 4 A method for performing gesture recognition in an electronic device according to embodiments disclosed herein is described. Figure 4A method 400 is illustrated, comprising various steps for performing gesture recognition in electronic device 300. As described in step 402, method 400 includes tracking the visible trajectory of a user's hand from multiple frames captured by electronic device 300 by processor 302 of electronic device 300. As described in step 404, method 400 includes identifying, by processor 302, a first frame in which the user's hand has left the FOV of electronic device 300.

[0056] Subsequently, as described in step 406, method 400 includes the processor 302 recognizing a second frame in which the user's hand has returned to the field of view (FOV) of the electronic device 300. As described in step 408, method 400 includes the processor 302 using an AI model to predict the trajectory of the user's hand using one or more frames captured before the first frame and one or more frames captured after the second frame. As described in step 410, method 400 includes the processor 302 recognizing at least one gesture performed during the visible trajectory of the user's hand and the predicted trajectory of the user's hand.

[0057] Various steps in method 400 can be performed in the order they are presented, in different orders, or simultaneously. Furthermore, in some embodiments, steps may be omitted. Figure 4 Some of the steps are listed below.

[0058] Figure 5 A method 500 for estimating spatiotemporal convergence to perform gesture recognition in an electronic device, according to embodiments disclosed herein, is described. Figure 5 A method 500 is illustrated, comprising various steps for estimating spatiotemporal convergence to perform gesture recognition in electronic device 300. As described in step 502, method 500 includes selecting frames in which a user's hand is present via a visible trajectory by processor 302 of electronic device 300. As described in step 504, method 500 includes generating a reference for one or more hand markers by processor 302 based on the frames in which the user's hand is present. As described in step 506, method 500 includes estimating the position of hand markers in one or more frames captured prior to the first frame by processor 302 based on the generated reference to the hand markers.

[0059] Subsequently, as described in step 508, method 500 includes having processor 302 calculate one or more dynamic parameters for each hand sign using the estimated positions of the hand signs from consecutive frames. As described in step 510, method 500 includes having processor 302 use the calculated dynamic parameters for each hand sign to obtain a first trajectory of the user's hand corresponding to a frame captured prior to the first frame.

[0060] Subsequently, as described in step 512, method 500 includes the processor 302 reversing the order of frames captured by electronic device 300. As described in step 514, method 500 includes the processor 302 estimating the position of hand markers in frames captured after the second frame based on a reference to the generated hand markers. As described in step 516, method 500 includes the processor 302 calculating one or more dynamic parameters for each hand marker using the estimated positions of the hand markers in consecutive frames. As described in step 518, method 500 includes the processor 302 using the calculated dynamic parameters for each hand marker to obtain a second trajectory of the user's hand corresponding to the frames captured after the second frame.

[0061] As described in step 520, method 500 includes having processor 302 check the proximity of hand markers for each frame of the first and second trajectories from the user's hand. As described in step 522, method 500 includes having processor 302 achieve spatiotemporal convergence at frames where the distance between two extrapolated hand markers is below a specific threshold. Trajectories preceding the spatiotemporally converged frames are considered the first trajectory (outward trajectory). Trajectories following the spatiotemporally converged frames are considered the second trajectory (inward trajectory).

[0062] Subsequently, as described in step 524, method 500 includes the processor 302 estimating the gesture by encoding two extrapolated hand signs that converge in a spatiotemporal manner. As described in step 526, method 500 includes the processor 302 recognizing the gesture based on a sequence of gesture information.

[0063] Various steps in method 500 can be performed in the order they are presented, in different orders, or simultaneously. Furthermore, in some embodiments, steps may be omitted. Figure 5 Some of the steps are listed below.

[0064] Figure 6 A comparison is depicted between existing gesture recognition methods according to embodiments disclosed herein and the disclosed gesture recognition methods according to embodiments. Figure 6 A conventional gesture recognition method 600 and a disclosed gesture recognition method 610 are illustrated. In the conventional gesture recognition method 600, gesture estimation 601 provides no output when no hand is present in the field of view (FOV), and gesture recognition 602 is inaccurate if one or more frames in the sequence do not contain gesture information. When no hand is present in some frames of the sequence, the disclosed gesture recognition method according to the embodiment may encode the gesture 612 based on the hand trajectory 611 before and after these frames.

[0065] Figure 7 A method for predicting the trajectory of a hand outside the field of view (FOV) is described according to embodiments disclosed herein. Figure 7A method 700 for predicting the trajectory of a hand outside the field of view (FOV) is illustrated. Method 700 includes receiving a sequence of outward frames 700_1 and inward frames 700_2, and calculating dynamic parameters (such as, but not limited to, the velocity and acceleration of the hand) before the hand leaves the field of view in the outward frame 700_1. Embodiments herein use dynamics to extrapolate the trajectory to estimate the position and velocity of the hand when it is no longer in the field of view. Using the extrapolated trajectory of the hand in both outward frames 700_1 and inward frames 700_2, embodiments herein detect spatial and temporal convergence. Using the outward trajectory before convergence and the inward trajectory after convergence, embodiments herein predict the complete trajectory of the hand when it is outside the FOV.

[0066] As depicted in the figure, for frames where the hand is facing outward (outside the FOV), as described in step 702, the position of the hand marker 703 in the frame is estimated. As described in step 704, using the positions in consecutive frames, the velocity and acceleration of each hand marker 703 are calculated. After calculating the dynamic parameters (velocity and acceleration) of each hand marker, as described in step 706, a verification is performed to check whether the user's hand is within the FOV of the electronic device 300.

[0067] If the user's hand is not within the field of view (FOV) of the electronic device 300, then, as described in step 708, the calculated velocity and position of one or more hand markers 703 in the next frame are calculated using the calculated velocity and acceleration. If the user's hand is within the FOV, then, as described in step 702, the estimation of the position of the hand markers in the frame is repeated. After calculating the velocity and position of one or more hand markers 703 in the next frame, then, as described in step 710, at least one parameter of the user's hand is verified. Parameters may include, but are not limited to, whether the velocity becomes zero, whether the hand position exceeds a threshold, whether the hand marker no longer conforms to predetermined biomechanical constraints of the human hand, etc. The threshold may be set based on the physically possible distance between the electronic device 300 and the hand. The predetermined biomechanical constraints of the human hand may be set by the structure of the human body to restrict where the hand can be placed.

[0068] If at least one parameter of the user's hand is not met, then as described in step 706, the user's hand in the FOV of the electronic device 300 is repeatedly verified. If at least one parameter of the user's hand is met, then as described in step 712, the user's hand is assumed to be stationary, and the estimation of the position of one or more hand markers in the previous frame is repeated.

[0069] For frames with the hand facing inward (back into the FOV), as described in step 714, a verification is performed to check if the user's hand is within the FOV. If the user's hand is within the FOV, the frame order is reversed as described in step 716. If the user's hand is not within the FOV, at least one parameter of the user's hand is verified as described in step 710. Then, as described in step 718, the steps described above (702, 704, 706, 708, 710, 712, and 714) performed for frames with the hand facing outward are followed on the reversed frames.

[0070] Various steps in method 700 can be performed in the order they are presented, in different orders, or simultaneously. Furthermore, in some embodiments, steps may be omitted. Figure 7 Some of the steps are listed below.

[0071] Figure 8 A process 800 for performing gesture recognition according to embodiments disclosed herein is described. Figure 8 The flowchart 800 for performing gesture recognition is shown. For a received input frame, as described in step 802, verification is performed to check if a hand exists in the field of view (FOV). If a hand exists in the FOV, as described in step 804, verification is performed to check if the current frame is the first frame in which a hand exists. If the current frame is the first frame in which a hand exists, as described in step 806, the hand trajectory is initialized and a hand marker reference is generated. If the current frame is not the first frame in which a hand exists, as described in step 808, the hand marker is estimated and the hand trajectory is updated.

[0072] Next, as described in step 810, a verification is performed to check whether the hand is facing outwards or inwards in the updated hand trajectory. If the hand is facing inwards, the hand trajectory is extrapolated for frames containing hands outside the field of view (FOV), as described in step 812. If the hand is facing outwards, the hand trajectory is updated, as described in step 814.

[0073] As verified in step 802, if the hand is not present in the FOV, then as described in step 816, a verification is performed to check if a trajectory exists. If the trajectory does not exist, then as described in step 802, an input frame is received to check if the hand exists. If the trajectory exists, then as described in step 812, the hand trajectory is extrapolated from the frame for the hand outside the FOV.

[0074] After updating the hand trajectory as described in step 814, a verification process is performed as described in step 818 to check whether both the outward and inward trajectories have been estimated. If neither the outward nor inward trajectories have been estimated, the gesture cannot be detected. If the outward and inward trajectories have been estimated, spatiotemporal convergence is estimated as described in step 820. Then, the hand trajectory is encoded as described in step 822. The gesture is analyzed based on the encoded hand trajectory as described in step 824. The gesture is predicted based on the gesture and spatiotemporal convergence.

[0075] Various steps in method 800 can be performed in the order they are presented, in different orders, or simultaneously. Furthermore, in some embodiments, steps may be omitted. Figure 8 Some of the steps are listed below.

[0076] Figure 9A An example flow procedure is described for extrapolating the trajectory of a hand not in the field of view and predicting spatiotemporal convergence for gesture recognition, according to embodiments disclosed herein. Figure 9B An example flow procedure is described for extrapolating the trajectory of a hand not in the field of view and predicting spatiotemporal convergence for gesture recognition, according to embodiments disclosed herein.

[0077] Figure 9A and Figure 9B An example flow diagram is depicted for extrapolating the trajectory of a hand not in the field of view (FOV) and predicting spatiotemporal convergence for gesture recognition. In this scenario, six frames 900 are considered, and for two of those frames, the hand leaves the field of view. The trajectories of the hand before leaving the FOV (outward trajectories 910, 911) and after returning to the FOV (inward trajectories 920, 921) are considered. Frames N through N+3 show the hand before leaving the FOV. Outward trajectories 910 and 911 are shown in black, and the extrapolation trajectories are shown in gray. Frames N+5 through N+3 show the hand after entering the FOV. Inward trajectories 920 and 921 are shown in dark gray, and the extrapolation trajectories are shown in light gray.

[0078] The order of frames after the hand enters the FOV is reversed and extrapolated to the hand trajectory before entry. Then, the convergence 930, 931 of the outward trajectories 910, 911 and the inward trajectories 920, 921 are obtained in both the time (frame) domain and the spatial (position) domain.

[0079] For example, once both outward trajectories 910, 911 and inward trajectories 920, 921 of the hand markers are obtained, for each frame, the proximity of the hand markers from the two trajectories is checked. A frame where the distance between the two extrapolated hand markers is below a certain threshold is considered spatiotemporally convergent 930, 931. Before that frame, they are considered outward trajectories 910, 911, and after that frame, they are considered inward trajectories 920, 921. Using this information, the hand markers are encoded for processing in gesture recognition 940. Gesture recognition 940 can be performed on the sequence of gesture information encoded in the previous steps.

[0080] Figure 10 An example use case of a user performing a left swipe gesture according to embodiments disclosed herein is described. For frame 1010 captured by the upper left camera 1000 and frame 1030 captured by the lower left camera 1020, spatiotemporal convergence 1050 is estimated. Spatiotemporal convergence 1050 is estimated based on hand trajectory extrapolation for lost inward frames and hand trajectory extrapolation for lost outward frames. Subsequently, based on hand trajectory encoding 1050, a left swipe gesture prediction 1060 is performed.

[0081] Figure 11 An example use case is described, which describes a user grabbing a VR object and moving their hand from one camera field of view (FOV) to another, according to embodiments disclosed herein. Figure 12 Example use cases of users playing VR games according to embodiments disclosed herein are described.

[0082] If the hand is not tracked correctly, the VR object 1110 may not be rendered in the second view. The disclosed method accurately tracks the hand and renders the VR object 1110 in the second view. In the intense action game 1200 on a head-mounted display (HMD, 1100), the hand frequently leaves and returns to the camera's field of view.

[0083] Therefore, the embodiments herein are used for, for example, in Figure 11 and Figure 12 In a gaming or office / analysis environment, this controls the activities displayed to the user. For example, suppose in such an environment a user's hand is tracked to leave a frame and then return to the frame, and this is determined according to embodiments herein, then any specific gesture regarding leaving the frame and then returning to the frame is used to control or continue control in such an environment. For example, in Figure 12 Perform actions in the game 1200 shown (such as moving items or objects 1100 in the game environment), or similarly, in Figure 11 In this context, actions are performed (such as moving windows or otherwise interacting with any of the user interfaces shown in the environment).

[0084] Therefore, using the disclosed method, gesture recognition can be accurately performed even when the hand leaves the field of view. Gesture recognition can be accurately performed even in multi-camera setups where the hand moves from one camera's field of view (FOV) to another. Hand trajectory information can be encoded based on trajectories from previous and subsequent frames (even when the hand is not visible).

[0085] The embodiments disclosed herein can be implemented by at least one software program running on at least one hardware device. Figure 3 The modules shown include blocks that may be at least one of hardware devices or a combination of hardware devices and software modules.

[0086] The description of the specific embodiments above will fully reveal the general nature of the embodiments herein. Others can easily modify and / or adapt these specific embodiments for various applications by applying existing knowledge without departing from the general concept. Therefore, such modifications and adaptations should and are intended to be understood as being within the meaning and scope of equivalent forms of the disclosed embodiments. It should be understood that the wording or terminology used herein is for descriptive purposes and not for limitation. Therefore, although the embodiments herein have been described with reference to examples and examples, those skilled in the art will recognize that the embodiments and examples disclosed herein can be practiced with modifications within the scope of the embodiments described herein.

Claims

1. A method (400) for performing gesture recognition in an electronic device (300), comprising: The electronic device (300) tracks the visible trajectory of the user's hand from multiple frames captured by the electronic device (300); The electronic device (300) identifies in the plurality of frames the first frame in which the user's hand has left the field of view (FOV) of the electronic device; The electronic device (300) identifies in the plurality of frames that the user's hand has returned to the second frame in the FOV of the electronic device (300); The electronic device (300) uses an artificial intelligence (AI) model to obtain the trajectory of the user's hand using one or more frames captured before the first frame and one or more frames captured after the second frame from the plurality of frames. as well as The electronic device (300) identifies at least one gesture performed during the visible trajectory of the user's hand and the obtained trajectory of the user's hand.

2. The method (400) as claimed in claim 1, further comprising: The electronic device (300) selects the frame containing the user's hand from the plurality of frames by means of a visible trajectory; as well as The electronic device (300) generates one or more references to hand markers based on frames containing the user's hand.

3. The method (400) as described in claim 2, further comprising: The electronic device (300) estimates the position of the one or more hand signs captured in one or more frames prior to the first frame based on the reference of the generated one or more hand signs; The electronic device (300) calculates one or more dynamic parameters for each hand sign using the estimated positions of the one or more hand signs in consecutive frames, wherein the consecutive frames include one or more frames captured prior to the first frame; as well as The electronic device (300) uses the calculated dynamic parameters of each of the one or more hand markers to obtain a first trajectory of the user's hand corresponding to one or more frames captured prior to the first frame.

4. The method (400) as claimed in claim 2, further comprising: The electronic device (300) reverses the order of the plurality of frames captured by the electronic device (300); The electronic device (300) estimates the position of the one or more hand signs in one or more frames captured after the second frame based on the reference of the generated one or more hand signs; The electronic device (300) calculates one or more dynamic parameters for each hand sign using the estimated positions of the one or more hand signs in consecutive frames, wherein the consecutive frames include one or more frames captured after the second frame; as well as The electronic device (300) uses one or more dynamic parameters of each of the one or more hand markers calculated to obtain a second trajectory of the user's hand using one or more frames captured after the second frame.

5. The method (400) as claimed in claim 3 or claim 4, wherein, One or more dynamic parameters of the user's hand include at least one of the velocity and acceleration of each of the one or more hand markings.

6. The method (400) as claimed in claim 3 or claim 4, wherein, The method (400) includes: After calculating one or more dynamic parameters for each of the one or more hand markers, the electronic device (300) verifies whether the user's hand is within the field of view (FOV) of the electronic device (300); and If the user's hand is not in the FOV of the electronic device (300), the electronic device (300) uses one or more dynamic parameters of each of the one or more hand markers to calculate the velocity and position of the one or more hand markers in the next frame after the one or more frames captured after the second frame.

7. The method (400) of claim 6, further comprising: After calculating the velocity and position of the one or more hand markers in the next frame, the electronic device (300) verifies at least one parameter of the user's hand, wherein the at least one parameter includes at least one of whether the velocity becomes zero, whether the hand position exceeds a threshold, and whether the one or more hand markers no longer conform to a predetermined biomechanical constraint of the human hand. If at least one parameter of the user's hand is not met, the electronic device (300) shall repeatedly verify the user's hand in the field of view (FOV) of the electronic device (300); and If the user's hand is stationary and at least one parameter of the user's hand is satisfied, the electronic device (300) repeats the estimation of the position of the one or more hand markers based on the previous frame before the next frame.

8. The method (400) of claim 3 or claim 4, further comprising: The electronic device (300) checks the proximity of the one or more hand markers for each frame of the first and second trajectories of the user's hand in the plurality of frames; The electronic device (300) obtains spatiotemporal convergence at a frame in which the distance between two extrapolated hand signs in the plurality of frames is less than a specific threshold, wherein the trajectory before the spatiotemporally converged frame is considered the first trajectory, and the trajectory after the spatiotemporally converged frame is considered the second trajectory. The electronic device (300) estimates the gesture by encoding the two extrapolated hand signs of the spatiotemporal convergence; and The electronic device (300) identifies the at least one gesture based on a sequence of gesture information.

9. An electronic device (300), comprising: Memory (306) stores at least one instruction; as well as At least one processor (302) is configured to execute the at least one instruction stored in the memory (306). The at least one processor (302) is configured to execute the at least one instruction to perform the following operations: The visible trajectory of the user's hand is tracked from a plurality of frames captured by the electronic device (300), wherein the plurality of frames include a first frame, a second frame, one or more frames captured before the first frame, and one or more frames captured after the second frame; In the plurality of frames, the first frame in which the user's hand has left the field of view (FOV) of the electronic device (300) is identified; In the plurality of frames, it is identified that the user's hand has returned to the second frame in the FOV of the electronic device (300); Using an artificial intelligence (AI) model, the trajectory of the user's hand is obtained using one or more frames captured before the first frame and one or more frames captured after the second frame from the plurality of frames; and Identify at least one gesture performed during the visible trajectory of the user's hand and the obtained trajectory of the user's hand.

10. The electronic device (300) as claimed in claim 9, wherein, The at least one processor (302) is configured to execute the at least one instruction to perform the following operations: The frame containing the user's hand is selected from the plurality of frames by means of a visible trajectory; as well as One or more hand markers are generated based on frames containing the user's hand.

11. The electronic device (300) as claimed in claim 10, wherein, The at least one processor (302) is configured to execute the at least one instruction to perform the following operations: The positions of the one or more hand markers in one or more frames captured prior to the first frame are estimated based on the references to the generated one or more hand markers; One or more dynamic parameters for each hand sign are calculated using the estimated positions of the one or more hand signs in consecutive frames, wherein the consecutive frames include one or more frames captured prior to the first frame; as well as One or more dynamic parameters of each of the calculated one or more hand markers are used to obtain a first trajectory of the user's hand using one or more frames captured prior to the first frame.

12. The electronic device (300) as claimed in claim 10, wherein, The at least one processor (302) is configured to execute the at least one instruction to perform the following operations: Reverse the order of the plurality of frames captured by the electronic device (300); The position of the one or more hand markers in one or more frames captured after the second frame is estimated based on the reference of the generated one or more hand markers; One or more dynamic parameters for each hand sign are calculated using the estimated positions of the one or more hand signs in consecutive frames, wherein the consecutive frames include one or more frames captured after the second frame; as well as One or more dynamic parameters of each of the calculated one or more hand markers are used to obtain a second trajectory of the user's hand corresponding to one or more frames captured after the second frame.

13. The electronic device (300) as claimed in claim 11 or claim 12, wherein, One or more dynamic parameters of the user's hand include at least one of the velocity and acceleration of each of the one or more hand markings.

14. The electronic device (300) as claimed in claim 11 or claim 12, wherein, The at least one processor (302) is configured to execute the at least one instruction to perform the following operations: After calculating one or more dynamic parameters for each of the one or more hand markers, verify that the user's hand is within the field of view (FOV) of the electronic device; as well as If the user's hand is not in the FOV of the electronic device (300), the velocity and position of the one or more hand signs in the plurality of frames and in the next frame after one or more frames captured after the second frame are calculated using one or more dynamic parameters of each of the one or more hand signs.

15. The electronic device (300) as claimed in claim 14, wherein, The at least one processor (302) is configured to execute the at least one instruction to perform the following operations: After calculating the velocity and position of the one or more hand markers in the next frame, at least one parameter of the user's hand is verified, wherein the at least one parameter includes at least one of whether the velocity becomes zero, whether the hand position exceeds a threshold, and whether the one or more hand markers no longer conform to the predetermined biomechanical constraints of the human hand. If at least one parameter of the user's hand is not satisfied, the user's hand in the field of view (FOV) of the electronic device (300) is verified again; and If the user's hand is stationary and at least one parameter of the user's hand is satisfied, the estimation of the position of the one or more hand markers is repeated based on the previous frame before the next frame.

Images