Context-based gesture recognition

Abstract

The present application is directed to recognizing gestures in an image. The electronic device obtains an image including a hand region and detects the hand region in the image. A first gesture is determined from a hand region of the image and a second gesture is determined from the image. In accordance with a determination that the first gesture is not any of the plurality of contextual gestures, the electronic device determines that a final gesture of the image is the first gesture. Instead, in accordance with a determination that the first gesture is one of the plurality of contextual gestures, the electronic device determines a final gesture from the image based on a second gesture associated with the image and a corresponding confidence score.

Description

Technical Field

[0001] The present application generally relates to gesture recognition technology, including but not limited to methods, systems, and non-transitory computer-readable media for recognizing gestures from image data. Background Art

[0002] Existing gesture recognition solutions typically require that gestures (e.g., hand gestures, facial movements, etc.) be captured close to the camera (e.g., less than 1 meter away) in order to accurately detect the gesture. As the distance between the gesture and the imaging device increases, the gesture becomes smaller and blends in with many other objects captured simultaneously with the gesture. To overcome these challenges, electronic devices applied to gesture recognition are adjusted to focus on a small area containing the gesture, for example, by zooming in and limiting the field of view to a small area. However, some electronic devices are unable to focus on a specific gesture before capturing an image, and require cropping the image to obtain a small area containing the area. Other solutions rely on powerful deep learning models or a fusion of detection and classification processes, and therefore require a large amount of computing resources for gesture detection and recognition. Therefore, it would be beneficial to have a system and method that can accurately and efficiently detect gestures captured in an image that includes gestures that may be far from the camera and blended with the image background. Summary of the invention

[0003] Various embodiments of the present application are directed to gesture recognition technology that combines local gesture information and contextual gesture information to improve the accuracy and efficiency of gesture recognition. Local gesture information includes information about the gesture or the part of the body performing the gesture, and contextual gesture information includes information about the surrounding environment of the gesture such as the environment (e.g., an office), the position of the gesture relative to the user, and / or other factors. In addition, in some embodiments, initial gesture classification is applied to simplify the detection and classification process, thereby improving overall efficiency. In some embodiments, one or more gestures are recognized based on contextual information (e.g., moving a hand to the mouth to indicate silence). This contextual information in the image is used to improve the accuracy of gesture recognition and reduce the number of false positives. In the example, contextual information is used to distinguish between local gestures, contextual gestures, and / or non-gestures.

[0004] In one aspect, a method for gesture classification is provided. The method includes: obtaining an image including a hand region; detecting the hand region in the image; determining a first gesture from the hand region of the image; and determining a second gesture from the image (e.g., the entire image). The method also includes: determining that a final gesture of the image is the first gesture based on determining that the first gesture is not any of a plurality of contextual gestures; determining a final gesture based on a second gesture and a second confidence score based on determining that the first gesture is one of a plurality of contextual gestures, the second gesture and the second confidence score being associated with the image (e.g., the entire image).

[0005] In some embodiments, determining the first gesture from the hand region of the image further comprises: generating a first gesture vector from the hand region of the image. Each element of the first gesture vector corresponds to a corresponding gesture and represents a corresponding first confidence level of the hand region including the corresponding gesture. The method further comprises: determining the first gesture and the first gesture confidence score from the first gesture vector. In some embodiments, the method further comprises: associating the detection of the hand region in the image with the bounding box confidence score, combining the bounding box confidence score with the confidence score associated with the first gesture to generate a first gesture confidence score. In some embodiments, the first gesture comprises a corresponding gesture corresponding to the maximum first confidence level among the corresponding first confidence levels of each element of the first gesture vector, and the first gesture confidence score is equal to the maximum first confidence level among the corresponding first confidence levels of each element of the first gesture vector.

[0006] In some embodiments, determining the second gesture from the image further comprises: generating a second gesture vector from the image (e.g., the entire image). Each element of the second gesture vector corresponds to a corresponding gesture and represents a corresponding second confidence level of the image including the corresponding predefined gesture. The method further comprises: determining the second gesture and a second gesture confidence score from the second gesture vector. In some embodiments, the second gesture comprises a corresponding gesture corresponding to a maximum second confidence level among the corresponding second confidence levels of each element of the second gesture vector, and the second gesture confidence score is equal to the maximum second confidence level among the corresponding second confidence levels of each element of the second gesture vector.

[0007] In some embodiments, the method includes: before determining whether the first gesture is at least one of the multiple contextual gestures, determining whether the first gesture confidence score is greater than a second threshold value P2, and determining that the image is not associated with any gesture based on determining that the first gesture confidence score is less than the second threshold value P2. In some embodiments, the method also includes: before determining whether the first gesture is at least one of the multiple contextual gestures, determining whether the second gesture confidence score of the second gesture is greater than a first threshold value P1, and determining that the image is not associated with any gesture based on determining that the second gesture confidence score of the second gesture is less than the first threshold value P1.

[0008] In some embodiments, determining the final gesture based on the second gesture and the second confidence score further comprises: determining that the image is not associated with any gesture based on determining that the first gesture and the second gesture are different from each other. The method further comprises: based on determining that the first gesture and the second gesture are the same as each other, (1) determining that the final gesture is the second gesture based on determining that the third confidence score exceeds the comprehensive confidence threshold; and (2) determining that the image is not associated with any gesture based on determining that the third confidence score does not exceed the comprehensive confidence threshold.

[0009] In some embodiments, the method further comprises: filtering the final gesture using a filter function for identifying false positives with the aid of temporal information (i.e., results from previous images). In some embodiments, the filter function is one of a convolution function, a Fourier filter function, or a Kalman filter. In some embodiments, the filter function is a function of time.

[0010] In another aspect, some embodiments include an electronic device comprising one or more processors and a memory having stored thereon instructions that, when executed by the one or more processors, cause the processors to perform any of the methods described above.

[0011] In another aspect, some embodiments include a non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the processors to perform any of the methods described above.

[0012] These exemplary embodiments and implementations are not mentioned to limit or define the present disclosure, but to provide examples to help understand the present disclosure. More embodiments are discussed in the detailed description and further explanation is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] For a better understanding of the various embodiments described, reference should be made to the following detailed description taken in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout the figures.

[0014] Figure 1 is an example data processing environment having one or more servers communicatively coupled to one or more client devices in accordance with some embodiments.

[0015] Figure 2 is a block diagram illustrating an electronic device for processing content data (eg, image data) according to some embodiments.

[0016] Figure 3 is a flow chart of a gesture detection and classification process using image data in accordance with some embodiments.

[0017] Figure 4 is a flow chart of an example post-processing technique for determining a final gesture from two gestures determined from image data in accordance with some embodiments.

[0018] Figure 5 is a flow chart of a method of classifying one or more gestures in accordance with some embodiments.

[0019] Like reference numerals refer to corresponding parts throughout the several views of the drawings. DETAILED DESCRIPTION

[0020] Reference will now be made in detail to specific embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to one of ordinary skill in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter presented herein may be practiced without these specific details. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

[0021] Figure 1 1 is an example data processing environment 100 having one or more servers 102 communicatively coupled to one or more client devices 104 according to some embodiments. The one or more client devices 104 may be, for example, a desktop computer 104A, a tablet computer 104B, a mobile phone 104C, a head-mounted display (HMD) (also known as augmented reality (AR) glasses) 104D, or a smart multi-sensing networked home device (e.g., a surveillance camera 104E, a smart TV device, a drone). Each client device 104 may collect data or user input, execute user applications, and present output on its user interface. The collected data or user input may be processed locally on the client device 104 and / or remotely by the server 102. The one or more servers 102 provide system data (e.g., boot files, operating system images, and user applications) to the client device 104, and in some embodiments, process data and user input received from the client device 104 when the user application is executed on the client device 104. In some embodiments, data processing environment 100 also includes memory 106 for storing data related to server 102 , client device 104 , and applications executing on client device 104 .

[0022] One or more servers 102 are used to enable client devices 104 that are far away from each other or client devices 104 that are far away from one or more servers 102 to communicate real-time data. In addition, in some embodiments, one or more servers 102 are used to perform data processing tasks that cannot be or will not be completed locally by client devices 104. For example, client devices 104 include game consoles (e.g., HMD 104D) that execute interactive online game applications. The game console receives user instructions and sends them to the game server 102 together with user data. The game server 102 generates a video data stream based on the user instructions and user data, and provides the video data stream for display on the game console and other client devices participating in the same game session with the game console. In another example, the client device 104 includes a networked surveillance camera 104E and a mobile phone 104C. The networked surveillance camera 104E collects video data and transmits the video data to the surveillance camera server 102 in real time. Although the video data is optionally pre-processed at the surveillance camera 104E, the surveillance camera server 102 processes the video data to identify motion or audio events in the video data and shares information of these events with the mobile phone 104C, thereby allowing the user of the mobile phone 104 to remotely monitor events occurring near the networked surveillance camera 104E in real time.

[0023] One or more servers 102, one or more client devices 104, and storage 106 are communicatively coupled to each other via one or more communication networks 108, which are media for providing communication links between these devices and connected computers within data processing environment 100. One or more communication networks 108 may include connections such as wired connections, wireless communication links, or fiber optic cables. Examples of one or more communication networks 108 include a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination thereof. The one or more communication networks 108 are optionally implemented using any known network protocol, including various wired or wireless protocols, such as Ethernet, universal serial bus (USB), FireWire, long term evolution (LTE), global system for mobile communications (GSM), enhanced data GSM environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wi-Fi, voice over Internet protocol (VoIP), Wi-MAX, or any other suitable communication protocol. The connection to the one or more communication networks 108 can be established directly (e.g., using a 3G / 4G connection to a wireless carrier), or through a network interface 110 (e.g., a router, switch, gateway, hub, or smart dedicated whole-home control node), or through any combination thereof. Thus, one or more communication networks 108 may represent the Internet, a global collection of networks and gateways that communicate with each other using the Transmission Control Protocol / Internet Protocol (TCP / IP) suite of protocols. At the heart of the Internet lies a backbone of high-speed data communication lines between major nodes or host computers, including thousands of commercial, government, educational, and other computer systems that route data and messages.

[0024] In some embodiments, deep learning techniques are applied to the data processing environment 100 to process content data (e.g., video data, visual data, audio data) obtained by an application executed on the client device 104 to identify information contained in the content data, match the content data with other data, classify the content data, or synthesize related content data. The content data can broadly include inertial sensor data captured by an inertial sensor of the client device 104. In these deep learning techniques, a data processing model is created based on one or more neural networks to process the content data. These data processing models are trained with training data before being applied to process the content data. After the model is trained, the mobile phone 104C or HMD 104D obtains the content data (e.g., video data is captured by an internal camera) and processes the content data locally using the data processing model.

[0025] In some embodiments, model training and data processing are implemented locally on each individual client device 104 (e.g., mobile phone 104C and HMD 104D). The client device 104 obtains training data from one or more servers 102 or storage 106, and applies the training data to train the data processing model. Alternatively, in some embodiments, model training and data processing are implemented remotely on a server 102 (e.g., server 102A) associated with a client device 104 (e.g., client device 104A and HMD 104D). Server 102A obtains training data from itself, another server 102, or storage 106, and applies the training data to train the data processing model. The client device 104 obtains content data, sends the content data to server 102A (e.g., in an application) to perform data processing using the trained data processing model, receives data processing results (e.g., recognized gestures) from server 102A, presents the results on a user interface (e.g., associated with an application), renders a virtual object in the field of view based on the gesture, or implements some other functions based on the results. The client device 104 itself performs no or little data processing on the content data before sending the content data to the server 102A. In addition, in some embodiments, data processing is implemented locally on the client device 104 (e.g., the client device 104B and the HMD 104D), and model training is implemented remotely on the server 102 (e.g., the server 102B) associated with the client device 104. The server 102B obtains training data from itself, another server 102, or the memory 106, and applies the training data to train the data processing model. The trained data processing model is optionally stored in the server 102B or the memory 106. The client device 104 imports the trained data processing model from the server 102B or the memory 106, processes the content data using the data processing model, and generates a data processing result to be presented on the user interface or used to initiate some functions locally (e.g., rendering a virtual object based on a device gesture).

[0026] In some embodiments, a pair of AR glasses 104D (also referred to as HMD) are communicatively coupled in the data processing environment 100. The AR glasses 104D include a camera, a microphone, a speaker, one or more inertial sensors (e.g., a gyroscope, an accelerometer), and a display. The camera and microphone are used to capture video and audio data from the scene in which the AR glasses 104D are located, while one or more inertial sensors are used to capture inertial sensor data. In some cases, the camera captures the gestures of the user wearing the AR glasses 104D and uses a gesture recognition model to recognize the gestures locally in real time. In some cases, the microphone records ambient sounds, including the user's voice commands. In some cases, both the video or static visual data captured by the camera and the inertial sensor data measured by one or more inertial sensors are used to determine and predict the device posture. The video, static image, audio, or inertial sensor data captured by the AR glasses 104D is processed by the AR glasses 104D, the server 102, or both to identify the device posture. Optionally, the server 102 and the AR glasses 104D jointly apply deep learning techniques to identify and predict the device posture. The device gesture is used to control the AR glasses 104D itself, or to interact with an application (e.g., a game application) executed by the AR glasses 104D. In some embodiments, the display of the AR glasses 104D displays a user interface, and the recognized or predicted device gesture is used to render or interact with a user-selectable display item (e.g., an avatar) on the user interface.

[0027] As described above, in some embodiments, deep learning techniques are applied in the data processing environment 100 to process video data, static image data, or inertial sensor data captured by the AR glasses 104D. A first data processing model is used to identify and predict 2D or 3D device postures based on such video, static image, and / or inertial sensor data. A second data processing model is optionally used to generate visual content. The training of the first data processing model and the second data processing model is optionally implemented by the server 102 or the AR glasses 104D. The inference of device posture and visual content is implemented by the server 102 and the AR glasses 104D independently or jointly by the server 102 and the AR glasses 104D.

[0028] Figure 2 1 is a block diagram showing an electronic device 200 for processing content data (eg, image data) according to some embodiments. The electronic device 200 is a server 102, a client device 104 (eg, Figure 1 In the example, the electronic device 200 is a mobile device, including a gesture recognition module 230, which is applied end-to-end (e.g., Figure 3) neural network model to recognize gestures locally on the mobile device. The electronic device 200 typically includes one or more processing units (CPUs) 202, one or more network interfaces 204, a memory 206, and one or more communication buses 208 for interconnecting these components (sometimes referred to as chipsets). The electronic device 200 includes one or more input devices 210 for facilitating user input, such as a keyboard, a mouse, a voice command input unit or a microphone, a touch screen display, a touch-sensitive input pad, a gesture capture camera, or other input buttons or controls. In addition, in some embodiments, the electronic device 200 uses a microphone for voice recognition or a camera 260 for gesture recognition to supplement or replace the keyboard. In some embodiments, the electronic device 200 includes one or more optical cameras (e.g., RGB cameras 260), scanners, or light sensor units for capturing images of, for example, a graphic serial code printed on the electronic device. In some embodiments, the electronic device 200 also includes one or more output devices 212 for presenting a user interface and displaying content, including one or more speakers and / or one or more visual displays. Optionally, the electronic device 200 includes a location detection device, such as a GPS (Global Positioning System) or other geographic location receiver, for determining the location of the electronic device 200. Optionally, the electronic device 200 includes an inertial measurement unit (IMU) 280 that integrates sensor data captured by multi-axis inertial sensors to estimate the position and orientation of the electronic device 200 in space. Examples of one or more inertial sensors of the IMU 280 include, but are not limited to, gyroscopes, accelerometers, magnetometers, and inclinometers.

[0029] Alternatively or in addition, in some embodiments, the electronic device 200 is communicatively coupled to one or more devices (e.g., a server 102, a client device 104, a memory 106, or a combination thereof) via one or more network interfaces 204, which include one or more input devices 210, output devices 212, IMU 280, or other components described above, and provide data to the electronic device 200.

[0030] The memory 206 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and optionally includes non-volatile memory, such as one or more magnetic disk memories, one or more optical disk memories, one or more flash memory devices, or one or more other non-volatile solid-state memories. Optionally, the memory 206 includes one or more memories remote from the one or more processing units 202. The memory 206 or optionally the non-volatile memory within the memory 206 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 206 or the non-transitory computer-readable storage medium of the memory 206 stores the following programs, modules, and data structures or a subset or superset thereof: Operating system 214, including processes for handling various basic system services and for performing hardware-related tasks; A network communication module 216 for connecting each server 102 or client device 104 to other devices (e.g., server 102, client device 104, or storage 106) via one or more (wired or wireless) network interfaces 204 and one or more communication networks 108 such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.; A user interface module 218 for enabling presentation of information (e.g., graphical user interfaces of applications 224, widgets, websites and their web pages, and / or games, audio and / or video content, text, etc.) at each client device 104 via one or more output devices 212 (e.g., displays, speakers, etc.); An input processing module 220 for detecting one or more user inputs or interactions from one of the one or more input devices 210 and interpreting the detected inputs or interactions; A web browser module 222 for navigating, requesting (e.g., via HTTP), and displaying websites and web pages thereof, including a web interface for logging into a user account associated with the client device 104 or another electronic device, for controlling the client or electronic device when associated with a user account, and for editing and viewing settings and data associated with the user account; One or more user applications 224 executed by the electronic device 200 (e.g., a game, a social networking application, a smart home application, and / or other web-based or non-web applications for controlling another electronic device and viewing data captured by such a device); A model training module 226 for receiving training data and establishing a data processing model for processing content data (e.g., video, image, audio, or text data) to be collected or obtained by the client device 104; A data processing module 228 for processing the content data using the data processing model 250 to identify information contained in the content data, match the content data with other data, classify the content data, or synthesize related content data, wherein in some embodiments, the data processing module 228 is associated with one of the user applications 224 to process the content data in response to user instructions received from the user application 224. In an example, the data processing module 228 is used to implement Figure 3 The gesture detection and classification process 300; A gesture classification module 230 is used to classify one or more gestures in the image (see below). Figure 3 and Figure 4 ), wherein the gesture recognition module 230 further includes a detection module 232 for detecting one or more objects in the image and / or a classification module 234 for classifying one or more gestures in a region or portion of the image and / or the entire image, and the image data is processed jointly by the gesture recognition module 230 and the detection process 310 and the classification process 320 of the data processing module 228; One or more databases 240, used to store at least one or more of the following data: o Device settings 242, including common device settings of one or more servers 102 or client devices 104 (e.g., service tier, device model, storage capacity, processing power, communication capabilities, etc.); o User account information 244 for one or more user applications 224, such as user name, security questions, account history data, user preferences, and predefined account settings; o Network parameters 246 of one or more communication networks 108, such as IP address, subnet mask, default gateway, DNS server and host name; o Training data 248, used to train one or more data processing models 250; o A data processing model 250 for processing content data (e.g., video, image, audio, or text data) using deep learning techniques, wherein the data processing model 250 includes an image compression model for implementing an image compression process, a feature extraction model for implementing a multi-scale feature extraction process, and / or the following references Figure 3 and Figure 4 one or more classification models and networks; o Gesture database 252, for storing one or more gestures associated with an image (e.g., stored in a database (e.g., in memory 204); and ○ Content data and results 254 are obtained and output respectively by the electronic device 200 (or a device communicatively coupled to the electronic device 200 (e.g., the client device 104)), wherein the content data is processed locally on the client device 104 or remotely on the server 102 by the data processing model 250 to provide relevant results to be presented on the client device 104.

[0031] Optionally, one or more databases 240 are stored in one of the server 102, the client device 104, and the memory 106 of the electronic device 200. Optionally, one or more databases 240 are distributed in more than one of the server 102, the client device 104, and the memory 106 of the electronic device 200. In some embodiments, more than one copy of the above data is stored at different devices, for example, two copies of the data processing model 250 are stored at the server 102 and the memory 106, respectively.

[0032] Each element in the above-mentioned identified element can be stored in one or more of the previously mentioned memory devices, and corresponds to an instruction set for performing the above-mentioned functions. The above-mentioned identified modules or programs (i.e., instruction sets) do not need to be implemented as separate software programs, processes, modules, or data structures, and therefore each subset of these modules can be combined or otherwise rearranged in various embodiments. In some embodiments, memory 206 optionally stores a subset of the above-mentioned identified modules and data structures. In addition, memory 206 optionally stores additional modules and data structures not described above.

[0033] Figure 3 312 is a flow chart of a gesture detection and classification process 300 using image data 312 according to some embodiments. In some embodiments, the gesture detection and classification process 300 is used to detect and recognize gestures captured at least 0.5 meters to 2 meters from an imaging device that captured the image data 312. In some embodiments, the gesture detection and classification process 300 is used to detect and recognize gestures captured at more than 2 meters from an imaging device that captured the image data 312. The gesture detection and classification process 300 is optionally described by reference to Figure 1 and Figure 2 The described one or more client devices 104 , server 102 , and / or combinations thereof perform the gesture detection and classification process 300 . The gesture detection and classification process 300 includes a detection process 310 , followed by a classification process 320 , followed by a post-processing stage 330 to determine a final gesture 340 .

[0034] In some embodiments, the detection process 310 includes applying a first detection and classification model 314 to the received image data 312. The first detection and classification model 314 generates one or more feature maps 302 for object detection in the object detection stage 316 and gesture classification in the gesture classification stage 318. The detection process 310 is used to provide a first output through the object detection stage 316 and a second output through the gesture classification stage 318 to determine the final gesture 340. The first output of the object detection stage 316 includes information of a bounding box 303 and an associated box confidence score 304 for each first gesture 305. In some embodiments, the information of the bounding box 303 is used to generate a cropped image 322 that closely surrounds the gesture, and a second classification network 324 is applied to determine the first gesture 305 from the cropped image 322. The second output of the gesture classification stage 318 includes information of a second gesture 307 and an associated second confidence score 308.

[0035] In some embodiments, the image data 312 is provided by the electronic device 200 ( Figure 2 ) is captured by an input device 210 (e.g., an RGB camera 260) of the electronic device 200. The image data 312 is optionally processed locally on the electronic device 200. Alternatively, the image data 312 is uploaded to the server 102 or transmitted to a different electronic device 200. The different electronic devices 200 obtain the image 312 from the electronic device 200 with the camera 260, or download the image 312 from the server 102 through the web browser module 222 and / or one or more user applications 224. In some embodiments, the image data 312 includes one or more gestures. In some embodiments, the gestures in the image data 312 are at least 4 meters away from the electronic device 200 that captured the image data 312. Non-limiting examples of one or more gestures include one or more gestures, facial movements, and body postures. The image data 312 is received at an initial resolution (e.g., 1080p).

[0036] The image data 312 is passed through a first detection and classification model 314 to compress the image data 312 and / or generate one or more feature maps 302 from the image data 312. In some embodiments, the image data 312 is processed (e.g., reduced using one or more neural networks (e.g., one or more convolutional neural networks)) before passing through the first detection and classification model 314. The first detection and classification model 314 includes one or more machine learning models. For example, in some embodiments, the first detection and classification model 314 includes one or more convolutional neural networks (CNNs) known in the art. In some embodiments, the one or more machine learning models are used to identify and enrich one or more features in the image data 312 (e.g., extract details therefrom), and reduce the feature resolution of one or more features (relative to the original resolution of the image data 312), and / or generate a sequence of (scaled) feature maps based on the image data 312. In some embodiments, one or more feature maps are provided as an output 302 of the first detection and classification model 314. Alternatively, in some embodiments, the sequence of feature maps is combined into a comprehensive feature map, which is provided as an output 302 of the first detection and classification model 314. The output 302 of the first detection and classification model 314 is used by at least an object detection stage 316 and a gesture classification stage 318 .

[0037] The gesture classification stage 318 identifies the second gesture 307 based on the output 302 of the first detection and classification model 314. Specifically, the gesture classification stage 318 is used to determine the information of the second gesture 307 and the associated second confidence score 308 (i.e., DetCls Conf Score), the second confidence score indicating the confidence level of the second gesture 307 detected from the one or more feature maps 302 corresponding to the entire image data 312. In some embodiments, the information of the second gesture 307 generated by the gesture classification stage 318 includes a second gesture vector. Each element of the second gesture vector corresponds to a corresponding gesture and represents a corresponding first probability or confidence level of the second gesture 307 corresponding to the corresponding gesture. In some embodiments, the second gesture 307 and the second confidence score 308 are determined based on the second gesture vector. In some embodiments, the second gesture vector is normalized. In some embodiments, the gesture classification stage 318 is performed in parallel with the object detection stage 316 and the classification process 320.

[0038] In some embodiments, the gesture classification stage 318 is used to identify gestures for a specific application and / or system. In some embodiments, the second gesture 307 determined by the gesture classification stage 318 includes only local information that can be used to further determine the final gesture 340. The local information of the gesture includes information specific to the body performing the gesture, information specific to the gesture (e.g., the exact category), and / or information specific to a specific application and / or system. More specifically, the local information is information based only on the hand or part of the body performing the gesture. For example, Figure 3 As shown, the local information can be "scissors".

[0039] The object detection stage 316 is used to detect one or more gestures in the output of the first detection and classification model 314. Specifically, the object detection stage 316 is used to generate one or more bounding boxes 303 around the one or more gestures detected in the image data 312. In some embodiments, each bounding box 303 corresponds to a corresponding first gesture 305 and is determined using a box confidence score 304 (i.e., BBox ConfScore) indicating a confidence level of the corresponding bounding box 303 associated with the first gesture 305. In addition, in some embodiments, the gesture area 322 is cropped and resized from the image data 312 for each first gesture 305 based on the information of the bounding box 303 (326).

[0040] Classification process 320 applies second classification network 324 to each gesture region 322 (i.e., cropped image 322) to determine first gesture 305. In some embodiments, second classification network 324 is one or more of neural networks known in the art (e.g., mobilenet v1, mobilenet v2, ShuffleNet). In some embodiments, second classification network 324 is selected based on the expected gesture task (e.g., the expected gesture for a particular application and / or system) and / or based on the number of categories classified (e.g., different types of gestures that can be classified by a particular application and / or system).

[0041] For each cropped image 322, the second classification network 324 determines a corresponding first gesture 305 by determining a first gesture vector for each received cropped image 322. Each element of the first gesture vector determined by the second classification network 324 corresponds to a corresponding gesture and represents a corresponding first probability or confidence level of the gesture area 322 that includes the corresponding gesture. In some embodiments, the classification process 320 determines the first gesture 305 and the first gesture confidence score 306 based on the first gesture vector. In some embodiments, the first gesture confidence score 306 for the first gesture 305 is combined with the box confidence score 304 determined by the object detection stage 316 to generate a first confidence score for the first gesture 305. The first gesture vector is normalized, which means that the sum of the probability values ​​corresponding to all gestures is equal to 1.

[0042] The information of the first gesture 305 provided by the object detection stage 316 is used to determine whether the first gesture 305 is associated with contextual information. Such contextual information is used to determine whether to select the first gesture 305 or the second gesture 307 to determine the final gesture 340 associated with the image data 312. If the first gesture 305 is not associated with the contextual information, the first gesture 305 is used to determine the final gesture 340. Conversely, if the first gesture 305 is associated with the contextual information, the second gesture 307 is used to determine the final gesture 340. In the example, the first gesture 305 or the second gesture 307 is used to distinguish between gestures performed near the user's face (e.g., raising a finger to the lips to indicate silence) and gestures performed in space (e.g., raising a finger). Examples of contextual information include, but are not limited to, gestures performed on and / or near a particular part of the body, gestures performed in a particular environment (e.g., at home, at the office, on a bus, in a library, etc.), previous gestures performed (e.g., an “answer call” gesture performed before a “hang up call” gesture), and / or movements around the performance of a gesture (e.g., a pinch gesture and / or an expand gesture to zoom in and / or zoom out).

[0043] A first gesture 305 and a second gesture 307 are determined from a gesture area 322 and an entire image 312, respectively, and are both used to determine a final gesture 340 by a post-processing stage 330. In the post-processing stage 330, the outputs of the detection process 310 and the classification process 320 are used together to determine the final gesture 340. The post-processing stage 330 includes one or more filters applied to the first gesture 305 and the second gesture 307 and the associated confidence scores 304, 306, and 308. The filter function is optionally used to identify false positives using time information from a previous state. In some embodiments, the filter is represented as a function of time (e.g., ), as follows: Wherein, F is a filter function. In some embodiments, in the post-processing stage 330 , a selected one of the first gesture 305 and the second gesture 307 needs to be stable for at least a predefined number of consecutive frames before being selected as the final gesture 340 .

[0044] Any number of methods may be used to construct the filter function. In some embodiments, the filter function is used to smooth the output to avoid jitter and provide a smooth user experience. Non-limiting examples of filters and filtering techniques may be moving / weighted average smoothing (or convolution), Fourier filtering, Kalman filtering, and variations thereof. Although the above filtering process is specific to gestures (e.g., K cls and K det ), but similar filters and filtering techniques can be applied to detection box smoothing.

[0045] refer to Figure 4 Further details of determining the final gesture 340 based on the first gesture 305 and the second gesture 307 and the associated confidence scores 304, 306, and 308 are discussed. Note that Figure 3 An example process 300 of determining gestures is shown, and the gesture detection and classification process 300 can be similarly applied to detect one or more of facial movements, arm gestures, body postures, and / or other classifiable gestures performed by a user.

[0046] Figure 4 is a flow chart of an example post-processing technique 400 for determining a final gesture 340 based on first gesture 305 and second gesture 307 determined from image data 312, according to some embodiments. Post-processing technique 400 is a flowchart of an example post-processing technique 400 described above with reference to Figure 3 An embodiment of one or more processes performed by the post-processing stage 330 is described. The post-processing technique 400 shows two branches for determining the final gesture 340 - the left branch is based on the second gesture 307 (Kdet), the second gesture confidence score 308 (DetClsi) associated with the second gesture 307, and the box confidence score 304 (Pbox) associated with the bounding box 303 of the first gesture 305; the right branch is based on the first gesture 305 (Kcls) and the first gesture confidence score 306 (Clsi), as described above with reference to Figure 3 described.

[0047] Beginning at operation 410, the post-processing technique 400 obtains the first gesture 305 (Kcls) and the first gesture confidence score 306 (Clsi) using the object detection stage 316. At operation 420, the processing technique 400 determines whether the first gesture confidence score 306 (Clsi) of the first gesture 305 (Kcls) is greater than or equal to a second threshold probability (P2). In some embodiments, the second threshold probability P2 is at least 0.10. In some embodiments, the second threshold probability P2 is any probability (e.g., at least 0.15) defined by a user and / or a system implementing the gesture detection and classification process 300. In some embodiments, the second threshold probability P2 is adjusted for optimal performance, and its value is highly dependent on the accuracy of the detection process 310 (e.g., including the classification stage 318).

[0048] If the first gesture confidence score 306 (Clsi) is lower than the second threshold probability P2 (“No” at operation 420), the corresponding first gesture 305 (Kcls) is determined to be an invalid gesture for the image (i.e., gesture 480 is not detected). On the contrary, if the first gesture confidence score 306 (Clsi) is greater than or equal to the second threshold probability P2 (“Yes” at operation 420), the corresponding first gesture 305 (Kcls) is maintained as a candidate gesture for the final gesture 340 and is used at operation 430. At operation 430, the post-processing technique 400 determines whether the first gesture 305 (KCls) is context-related (i.e., whether the first gesture 305 is associated with context information). If the first gesture 305 (KCls) is not context-related (“No” at operation 430), the post-processing technique 400 determines that the first gesture 305 (KCls) is a gesture category 490 (i.e., a final gesture 340). Alternatively, if the first gesture 305 (KCls) is context-dependent ("yes" at operation 430), the post-processing technique 400 proceeds to operation 460 and utilizes the first gesture 305 (KCls) in combination with the second gesture 307 to determine the final gesture 340 (wherein the first gesture 305 is centered on the gesture area 322 and the second gesture 307 is based on the entire image).

[0049] Transitioning to operation 440, the post-processing technique 400 obtains the second gesture 307 (Kdet), the second gesture confidence score 308 (DetClsi) associated with the second gesture 307, and the box confidence score 304 (Pbox). At operation 450, it is determined whether the second gesture confidence score 308 (DetClsi) of the second gesture 307 is greater than or equal to a first threshold probability (P1). In some embodiments, the first threshold probability P1 is at least 0.10. In some embodiments, the first threshold probability P1 is any probability defined by a user and / or a system implementing the gesture detection and classification process 300 (e.g., at least 0.15). In some embodiments, the first threshold probability P1 is adjusted for optimal performance, and its value is highly dependent on the accuracy of the detection process 310 and the classification process 320.

[0050] If the second gesture confidence score 308 (DetClsi) of the second gesture 307 (Kdet) is lower than the first threshold probability P1 (“No” at operation 450), it is determined that the second gesture 307 (Kdet) is an invalid gesture for the image (i.e., gesture 480 is not detected). On the contrary, if the second gesture confidence score 308 (DetClsi) of the second gesture 307 (Kdet) is greater than or equal to the first threshold probability P1 (“Yes” at operation 460), the second gesture 307 (Kdet) remains as a candidate gesture for the final gesture 340 and is used at operation 460.

[0051] At operation 460, the post-processing technique 400 determines whether the second gesture 307 (Kdet) is the same as the first gesture 305. If the second gesture 307 (Kdet) and the first gesture 305 (KCls) are different ("No" at operation 460), the post-processing technique 400 determines that the potential gesture is invalid (i.e., gesture 480 is not detected). Alternatively, if the second gesture 307 (Kdet) and the first gesture 305 (KCls) are the same ("Yes" at operation 460), the post-processing technique 400 proceeds to operation 470 and determines whether the third confidence score 402 is greater than a third threshold probability (P3). The third confidence score 402 is equal to the second gesture confidence score 308 (DetClsi) of the second gesture 307 (Kdet) multiplied by the box confidence score 304 (Pbox). Figure 4 The third confidence score 402 is given by DetCls Kdet *Pbox indicates. In some embodiments, the third threshold probability P3 is at least 0.10. In some embodiments, the third threshold probability P3 is any probability (e.g., at least 0.15) defined by the user and / or the system implementing the gesture detection and classification process 300. In some embodiments, the third threshold probability P3 is adjusted to obtain optimal performance, and its value is highly dependent on the accuracy of the detection process 310 and the classification process 320.

[0052] If the third confidence score 402 is less than the third threshold probability P3 ("No" at operation 470), the second gesture 307 (Kdet) is determined to be invalid (i.e., gesture 480 is not detected). If the third confidence score 402 is greater than the third threshold probability P3 ("Yes" at operation 470), the post-processing technique 400 determines that the second gesture 307 (Kdet) is a gesture category 490 (i.e., the final gesture 340). By combining the information provided by the detection process 310 and the classification process 320, it can be determined holistically whether the gesture is context-related (rather than relying solely on the classification process). Compared to existing solutions, this holistic approach in the above technology improves the ability of electronic devices to detect and recognize gestures.

[0053] Figure 5 is a flow chart of a method 500 for classifying one or more gestures according to some embodiments. The method 500 includes the method described above with reference to Figure 3 and Figure 4 Method 500 provides a method for performing one or more operations across different electronic devices and / or systems (e.g., as described above with reference to Figure 1 and Figure 2 The present invention relates to a solution for gesture recognition (e.g., hand gestures, facial movements, arm gestures, etc.) based on the existing solutions. The gesture determination method 500 improves the accuracy of local gesture classification (e.g., gesture classification that does not depend on context) and contextual gesture classification (e.g., situational gesture classification based on context) relative to existing solutions. For example, in some embodiments, the gesture determination process 500 shows that the accuracy of local gesture classification is improved by at least 4-10% compared with existing solutions, and the accuracy of contextual gesture classification is improved by at least 43-120% compared with existing solutions.

[0054] The operations (eg, steps) of method 500 are performed by one or more processors (eg, CPU 202 ; FIG. 2 ) of an electronic device (eg, at server 102 and / or client device 104 ); Figure 2 )implement. Figure 5 At least some of the operations shown in correspond to operations stored in a computer memory or computer-readable storage medium (e.g., memory 206; Figure 2). Operations 502-512 may also be performed in part using one or more processors and / or using instructions stored in a memory or computer-readable medium of one or more devices communicatively coupled together, such as a laptop, AR glasses, or other head-mounted displays, servers, tablets, security cameras, drones, smart TVs, smart speakers, toys, smart watches, smart appliances, or other computing devices that can perform operations 502-512 alone or in conjunction with corresponding processors of the communicatively coupled electronic device 200.

[0055] Method 500 includes obtaining (502) an image 312 including a hand region 322, detecting (504) the hand region 322 in the image 312, and determining (506) a first gesture 305 from the hand region of the image. For example, as described above with reference to Figure 3 As shown, image data 312 is obtained, and a detection process 310 (which can be used for cropping an image) and a classification process 320 are applied to the image data 312 (after the image data 312 is processed by a first detection and classification model 314) to determine a classified gesture vector for a cropped image 322 (i.e., a hand region 322) fused with contextual information.

[0056] In some embodiments, determining first gesture 305 from hand region 322 of image 312 includes: generating a first gesture vector from hand region 322 of the image, each element of the first gesture vector corresponding to a respective gesture and indicating a respective first confidence level that hand region 322 includes the respective gesture, determining first gesture 305 and first gesture confidence score 306 from the first gesture vector. For example, as described above with reference to Figure 3 As shown, the object detection stage 316 is applied to detect one or more gestures of the image data 312 (after passing through the first detection and classification model 314), which are cropped and used to determine the first gesture vector. In some embodiments, the method 500 also includes associating the detection of the hand region 322 in the image 312 with the bounding box confidence score 304, and combining the bounding box confidence score 304 with the first gesture confidence score 306 (Clsi) of the first gesture 305. For example, as described above with reference to Figure 3 As shown, the output of the classification process 320 may be combined with the output of the object detection stage 316 (eg, bounding box confidence scores).

[0057] In some embodiments, first gesture 305 includes a corresponding gesture corresponding to the maximum first confidence level among the corresponding first confidence levels of each element of the first gesture vector, and the gesture confidence score is equal to the maximum first confidence level among the corresponding first confidence levels of each element of the first gesture vector. In other words, first gesture 305 may have a greater confidence score than a corresponding set of one or more gestures (e.g., as described above with reference to Figure 3 The other gestures in the first gesture 305) have a greater confidence score (e.g., the first confidence score 306).

[0058] Method 500 includes determining (508) a second gesture 307 from the image (e.g., the entire image). In some embodiments, determining the second gesture 307 from the image includes: generating a second gesture vector from the image (e.g., the entire image), each element of the second gesture vector corresponding to a respective gesture and representing a respective second confidence level that the image includes the respective gesture; and determining the second gesture 307 and a second gesture confidence score 308 from the second gesture vector.

[0059] In some embodiments, second gesture 307 includes a corresponding gesture corresponding to the maximum second confidence level among the corresponding second confidence levels of each element of the second gesture vector, and second gesture confidence score 308 is equal to the maximum second confidence level among the corresponding second confidence levels of each element of the second gesture vector. In other words, second gesture 307 may have a greater confidence level than a corresponding set of one or more gestures (e.g., as described above with reference to Figure 3 The second group of one or more gestures) may be assigned a greater confidence score (e.g., the second confidence score 308) than the other gestures in the second group of one or more gestures.

[0060] Method 500 includes determining (510) that the final gesture of the image is the first gesture 305 based on determining that the first gesture 305 is not any of the plurality of contextual gestures. Figure 4 As shown, according to determining that the gesture Kcls (e.g., the gesture of the first gesture 305) is not context-related ("No" at operation 430), the gesture Kcls is determined as the final gesture 340. In some embodiments, the method 500 includes: before determining whether the first gesture 305 is at least one of the multiple context gestures, determining whether the first gesture confidence score 306 is greater than a second threshold value P2, and according to determining that the first gesture confidence score 306 is less than the second threshold value P2, determining that the image is not associated with any gesture. For example, as described above with reference to Figure 4 As shown, according to the confidence score of the gesture Kcls (e.g., Cls Kcls ) is less than the second threshold probability P2 ("No" at operation 420), method 500 determines that there is no gesture.

[0061] In some embodiments, method 500 includes: before determining whether first gesture 305 is at least one of the plurality of contextual gestures, determining whether second gesture confidence score 308 (DetClsi) of second gesture 307 is greater than first threshold value P1, and determining that the image is not associated with any gesture based on determining that second confidence score 308 (DetClsi) of second gesture 307 is less than first threshold value P1. For example, as described above with reference to Figure 4 As shown, according to the confidence score of the gesture Kdet (e.g., DetCls Kdet ) is less than the first threshold probability P1 (“No” at operation 450 ), method 500 determines that there is no gesture.

[0062] Method 500 also includes: determining (512) a final gesture based on second gesture 307 and second gesture confidence score 308, based on determining that first gesture 305 is one of the plurality of contextual gestures, the second gesture 307 and second gesture confidence score 308 being associated with an image (e.g., the entire image). In some embodiments, determining final gesture 340 based on second gesture 307 and second gesture confidence score 308 also includes: determining that image 312 is not associated with any gesture, based on determining that first gesture 305 and second gesture 307 are different from each other. For example, as described above with reference to Figure 4 As shown, based on determining that the gesture Kdet is different from the gesture Kcls (“Yes” at operation 460 ), the method 500 determines that the gesture does not exist.

[0063] Alternatively, in some embodiments, based on determining that first gesture 305 and second gesture 307 are identical to each other and based on determining that third confidence score 402 does not exceed a combined confidence threshold (eg, Figure 4 P3 in the method 500 includes determining that the image is not associated with any gesture. For example, as described above with reference to Figure 4 As shown, according to determining the third confidence score 402 (eg, DetCls Kdet *Pbox - the confidence score of the gesture Kdet multiplied by the box confidence score 304 (Pbox) (output of the object detection stage 316; Figure 3 )) is less than the third threshold probability P3 ("No" at operation 450), method 500 determines that there is no gesture. In some embodiments, based on determining that first gesture 305 and second gesture 307 are identical to each other and based on determining that third confidence score 402 exceeds comprehensive confidence threshold P3, method 500 includes determining that final gesture 340 is second gesture 307. For example, as described above with reference to Figure 4 As shown, based on determining that the third confidence score 402 is greater than or equal to the third threshold probability P3 (“yes” at operation 470 ), the method 500 determines that the second gesture (Kdet) is the final gesture 340 .

[0064] In some embodiments, the first threshold P1 , the second threshold P2 , and the comprehensive confidence threshold P3 are adjusted to obtain optimal performance and are highly dependent on the accuracy of processes 310 and 320 .

[0065] In some embodiments, method 500 also includes filtering the final gesture using a filter function. The filter function is used to identify false positives. In some embodiments, the filter function is one of a convolution function (or a moving and / or weighted average smoothing function), a Fourier filter function, or a Kalman filter. In some embodiments, the filter function is a function of time that allows the determination of the first gesture and / or the second gesture to be stable (e.g., stable for at least 5 consecutive frames). In some embodiments, the filter function helps to smooth the detection boxes and recognized categories, helps to avoid jitter and detection loss in the process, and helps to make it easy to engineer (e.g., implementing gesture control for volume adjustment). The above Figure 3 Provides additional information about filters.

[0066] It should be understood that the Figure 5 The specific order of operations in is merely exemplary and is not intended to indicate that the described order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize various ways of retrieving candidate images or determining camera poses as described herein. In addition, it should be noted that the above description of Figure 3 and Figure 4 The details of the other processes described above also apply in a similar manner to Figure 5 The method 500 is described. For the sake of brevity, these details are not repeated here.

[0067] The terms used in the description of various embodiments described herein are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various embodiments described and the appended claims, the singular forms "one", "one", and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It should also be understood that when used in this specification, the terms "include" and / or "comprising" indicate the existence of stated features, integers, steps, operations, elements and / or components, but do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. In addition, it should be understood that although the terms "first", "second", etc. can be used in this article to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish an element from another element.

[0068] As used herein, the term “if” is optionally interpreted to mean “when” or “at” or “in response to determining” or “in response to detecting” or “according to a determination of…”, depending on the context. Similarly, the phrase “if it is determined” or “if [stated condition or event] is detected” is optionally interpreted to mean “upon the determination” or “in response to determining” or “upon detecting [stated condition or event]” or “in response to detecting [stated condition or event]” or “according to a determination of detecting [stated condition or event]”, depending on the context.

[0069] For the purpose of illustration, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the claims to the exact forms disclosed. In view of the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to best explain the operating principles and practical applications, thereby enabling those skilled in the art to practice.

[0070] Although the various figures show multiple logical stages in a particular order, stages that are not dependent on the order may be reordered, and other stages may be combined or removed. Although some reordering or other groupings are specifically mentioned, other orderings and groupings will be obvious to those of ordinary skill in the art, and thus the orderings and groupings presented herein are not an exhaustive list of alternatives. In addition, it should be appreciated that the stages may be implemented in hardware, firmware, software, or any combination thereof.

Claims

1. A gesture classification method, implemented by an electronic device, comprising: obtaining an image including a hand region; detecting the hand region in the image; determining a second gesture from the image; determining a first gesture from the hand region of the image; determining that a final gesture of the image is the first gesture based on determining that the first gesture is not any of a plurality of contextual gestures; as well as Based on determining that the first gesture is one of the plurality of contextual gestures, the final gesture is determined based on the second gesture and a second gesture confidence score, the second gesture and the second gesture confidence score being associated with the image.

2. The method according to claim 1, wherein: Determining the second gesture from the image further comprises: generating a second gesture vector from the image, each element of the second gesture vector corresponding to a respective gesture and representing a respective second confidence level that the image includes the respective gesture; and The second gesture and the second gesture confidence score are determined from the second gesture vector.

3. The method according to claim 2, wherein: The second gesture includes a corresponding gesture corresponding to a maximum second confidence level among the corresponding second confidence levels of each element of the second gesture vector, and the second gesture confidence score is equal to the maximum second confidence level among the corresponding second confidence levels of each element of the second gesture vector.

4. The method according to any one of claims 1 to 3, further comprising: Before determining whether the first gesture is at least one of the plurality of contextual gestures, determining whether the first gesture confidence score is greater than a second threshold; Based on determining that the first gesture confidence score is less than the second threshold, it is determined that the image is not associated with any gesture.

5. A method according to any one of the preceding claims, wherein: Determining the first gesture from the hand region of the image further comprises: generating a first gesture vector from the hand region of the image, each element of the first gesture vector corresponding to a respective gesture and representing a respective first confidence level that the hand region includes the respective gesture; The first gesture and a first gesture confidence score are determined from the first gesture vector.

6. The method according to claim 5, further comprising: associating a detection of the hand region in the image with a bounding box confidence score; as well as The bounding box confidence score is combined with a confidence score associated with the first gesture to generate the first gesture confidence score.

7. The method according to claim 5 or 6, wherein: The first gesture includes a corresponding gesture corresponding to a maximum first confidence level among the corresponding first confidence levels of each element of the first gesture vector, and the first gesture confidence score is equal to the maximum first confidence level among the corresponding first confidence levels of each element of the first gesture vector.

8. The method according to any one of claims 5 to 7, further comprising: Before determining whether the first gesture is at least one of the plurality of contextual gestures, determining whether the second gesture confidence score of the second gesture is greater than a first threshold; Based on determining that the second gesture confidence score of the second gesture is less than the first threshold, determining that the image is not associated with any gesture.

9. A method according to any one of the preceding claims, wherein: Determining the final gesture based on the second gesture and the second gesture confidence score further includes: Determining that the image is not associated with any gesture based on determining that the first gesture and the second gesture are different from each other; and According to determining that the first gesture and the second gesture are identical to each other: Determining that the final gesture is the second gesture based on determining that the third confidence score exceeds the combined confidence threshold; and Based on determining that the third confidence score does not exceed the combined confidence threshold, it is determined that the image is not associated with any gesture.

10. The method according to any of the preceding claims, further comprising filtering the final gesture using a filter function, wherein: The filter function is used to identify false positives.

11. The method according to claim 10, wherein: The filter function is one of a convolution function, a Fourier filter function, or a Kalman filter.

12. The method according to claim 10 or 11, wherein: The filter function is a function of time.

13. An electronic device comprising: one or more processors; as well as A memory having instructions stored thereon, which, when executed by the one or more processors, cause the processor to perform the method according to any one of claims 1-12.

14. A non-transitory computer readable medium having instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform the method according to any one of claims 1-12.

Images