Real-time on-device remote gesture recognition using lightweight deep learning models
Through lightweight hand detection models and compressed image data, the computing complexity problem of long-distance posture recognition on IoT devices is solved, and efficient and accurate posture detection is achieved, which is suitable for real-time deployment on IoT devices.
Patent Information
- Application Number
- CN202280095354.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-28
- Publication Date
- 2024-12-06
AI Technical Summary
Existing posture recognition solutions are less reliable and have high false alarm rates when capturing postures at long distances (such as 1 meter away), and deep learning solutions require powerful processors and high-resolution inputs, making it difficult to calculate in limited quantities. Resources are implemented on IoT devices.
Using a lightweight hand detection model, using scaled and rich compressed image data, combined with lightweight backbone and pyramid detection structure, reduces the computational complexity and achieves accuracy at different distances (1 meter to 4 meters or more) Detect postures and reduce inference time on IoT devices.
It realizes efficient and accurate posture detection under limited computing resources, and the inference time is reduced by several orders of magnitude, which hardly affects the accuracy of the identification results, and is suitable for real-time deployment on IoT devices.
Smart Images

Figure CN119096265A_ABST
Abstract
Description
Technical Field
[0001] The present application generally relates to deep learning-based gesture recognition, including but not limited to methods, systems, and non-transitory computer-readable media for long-distance gesture recognition using compressed image data and multi-scale detection. Background Art
[0002] Existing gesture recognition schemes typically require that gestures (e.g., hand gestures, face gestures, etc.) be captured at a location close to the camera (e.g., less than 1 meter away) to accurately detect the gesture. When the gesture is captured at a distance beyond 1 meter, the gesture is detected by shape recognition using continuous images, but this scheme has low reliability and increases the false positive rate. Deep learning algorithms are used to increase the distance at which gestures can be recognized from remotely captured images. However, existing deep learning solutions require powerful processors and / or models with high-resolution input (e.g., image data), which is not practical for Internet of Things (IoT) devices or smaller electronic devices such as smartphones. Therefore, it would be beneficial to have a system and method that can accurately and efficiently detect gestures captured in image data that can be implemented in IoT devices and / or smaller electronic devices with limited computing and / or storage capabilities. Summary of the invention
[0003] Embodiments of the present application relate to gesture recognition techniques that use scaled and enriched compressed image data to detect one or more gestures in a captured image. Embodiments provide a lightweight hand detection model that can be run on a mobile device or edge device with limited computing resources. The lightweight hand detection model has a lightweight backbone (backbone, skeleton) and a pyramid detection structure. The pyramid detection structure is capable of detecting gestures of different sizes in images captured at different distances (e.g., from 1 meter to 4 meters or more) while having ultra-low computational complexity. In these ways, the disclosed system and method provide a long-distance gesture recognition solution that operates at an inference time similar to existing solutions known in the art, but at the same time uses substantially less computing resources (e.g., compared to 5.6-780 GFLOPS (giga floating-point operations per second) used by existing solutions, the solution uses 3.5 GFLOPS per second).
[0004] In other words, the present solutions provide a lightweight end-to-end deep learning structure that reduces the computational complexity of hand recognition and reduces the inference time on IoT devices by several orders of magnitude, while having little or no impact on the accuracy of the recognition results. These solutions enable real-time on-device deployment at the expense of a slight decrease in accuracy that does not significantly affect performance. In some embodiments, the end-to-end model for recognizing and detecting gestures is combined with an additional classification model to improve the reliability of gesture recognition and reduce false positives.
[0005] In one aspect, a method for detecting a gesture is provided. The method includes: acquiring image data of an image, wherein the image captures a hand; and generating a sequence of feature maps based on the image data. The sequence of feature maps has a continuously downscaled feature resolution. The method also includes combining the sequence of feature maps to generate a composite feature map, and determining a gesture of the hand captured in the image based on the composite feature map.
[0006] In some embodiments, generating a feature map sequence based on image data includes: using a machine learning model to identify one or more features from the image data and enriching the one or more features, and reducing the feature resolution of the feature map sequence relative to the resolution of the image data.
[0007] In some embodiments, generating a feature map sequence based on image data includes: using a backbone network to generate a feature map sequence based on the image data, the backbone network including at least two convolutional layers and multiple bottleneck structures.
[0008] Further, in some embodiments, each of the at least two convolutional layers is a 3×3 kernel with a stride of 2. The first convolutional layer in the at least two convolutional layers is followed by the second convolutional layer in the at least two convolutional layers. In addition, in some embodiments, the multiple bottleneck structures include a first bottleneck structure and a second bottleneck structure. The first bottleneck structure is configured to extract one or more features from the output of the at least two convolutional layers. The first bottleneck structure includes a first 1×1 point-by-point convolutional layer, a first 3×3 depth convolutional layer after the first 1×1 point-by-point convolutional layer, and a second 1×1 point-by-point convolutional layer after the first 3×3 depth convolutional layer and an addition operation including a jump connection structure. The second bottleneck structure is configured to reduce the output of the first bottleneck structure. The second bottleneck structure includes a third 1×1 point-by-point convolutional layer, a second 3×3 depth convolutional layer with a stride of 2 after the third 1×1 point-by-point convolutional layer, and a fourth 1×1 point-by-point convolutional layer after the second 3×3 depth convolutional layer. The output of the second bottleneck structure is a plurality of feature maps. Furthermore, in some embodiments, each of the plurality of bottleneck structures has 3 layers and channels that are a multiple of 3. In some embodiments, each convolutional layer is followed by an activation function.
[0009] In some embodiments, generating a sequence of feature maps includes: generating a first feature map having a first feature resolution; reducing the first feature map to a second feature map according to a scaling factor, wherein the second feature map immediately follows the first feature map in the sequence of feature maps; and reducing the second feature map to a third feature map according to the scaling factor, wherein the third feature map immediately follows the second feature map in the sequence of feature maps.
[0010] In some embodiments, combining a sequence of feature maps to generate a comprehensive feature map includes: decoding one or more bounding boxes from the sequence of feature maps; identifying a subset of bounding boxes in the one or more bounding boxes that meet a confidence criterion; and formatting (formatting) the comprehensive feature map based on the subset of bounding boxes in the one or more bounding boxes.
[0011] In some embodiments, the hand is captured in the image by a camera device at least 4 meters away from the hand.
[0012] In some embodiments, the electronic device includes one of a security camera, a smart television, a smart speaker, a toy, a smart watch, a smart appliance, or a head-mounted display.
[0013] In some embodiments, the resolution of the image data is at least 1080p, and the method is implemented using 3.5 GFLOPS or less of computing resources of the electronic device.
[0014] In some embodiments, determining a gesture in an image based on a comprehensive feature map further includes: detecting and extracting a hand region in image data based on the comprehensive feature map, determining a first posture vector based on the hand region extracted from the comprehensive feature map, determining a second posture vector from the comprehensive feature map corresponding to the image data of the image, combining the first posture vector and the second posture vector to generate a final posture vector, and determining the gesture based on the final posture vector. The image includes a hand region and one or more remaining regions different from the hand region.
[0015] In another aspect, some embodiments include an electronic system including one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the processors to implement any of the above methods.
[0016] In yet another aspect, some embodiments include a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to implement any of the methods described above.
[0017] These exemplary embodiments and implementations are not mentioned to limit or define the present disclosure, but to provide examples to aid understanding of the present disclosure. Other embodiments are discussed in the detailed description section, and further description is provided. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] For a better understanding of the various embodiments described, the following detailed description should be viewed in conjunction with the following drawings, wherein like reference numerals refer to corresponding parts throughout the drawings.
[0019] Figure 1 An example data processing environment with one or more servers communicatively coupled to one or more client devices is shown in accordance with some embodiments.
[0020] Figure 2 A block diagram of an electronic system configured to process content data (eg, image data) is shown in accordance with some embodiments.
[0021] Figure 3 A flow chart is shown of a gesture recognition process for using compressed image data to detect and recognize one or more gestures in accordance with some embodiments.
[0022] Figure 4 A flow chart illustrating the image processing phase of a gesture recognition process in accordance with some embodiments.
[0023] Figure 5 A flow chart of a gesture detection and recognition method according to some embodiments is shown.
[0024] Like reference numerals designate corresponding parts throughout the several views of the drawings. DETAILED DESCRIPTION
[0025] The specific embodiments of the invention will now be described in detail, examples of which are shown in the accompanying drawings. In the following specific embodiments, many non-limiting specific details are set forth to aid in understanding the subject matter presented herein. It will be apparent to those skilled in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter may be implemented without these specific details. For example, it will be apparent to those skilled in the art that the subject matter presented herein may be implemented on various types of electronic devices having digital video capabilities.
[0026] Embodiments of the present application relate to gesture recognition techniques that use scaled and enriched compressed image data to detect one or more gestures in a captured image. Embodiments provide a lightweight hand detection model that can run on a mobile device or edge device with limited computing resources. The lightweight hand detection model has a lightweight backbone and a pyramid detection structure. The pyramid detection structure is capable of detecting gestures of different sizes in images captured at different distances (e.g., from 1 meter to 4 meters or more) while having ultra-low computational complexity. In these ways, the disclosed systems and methods provide a long-distance gesture recognition scheme that operates at an inference time similar to existing schemes known in the art, but at the same time uses substantially less computing resources (e.g., the scheme uses 3.5 GFLOPS per second compared to 5.6-780 GFLOPS used by existing schemes).
[0027] In other words, the present solutions provide a lightweight end-to-end deep learning structure that reduces the computational complexity of hand recognition and reduces the inference time on IoT devices by several orders of magnitude, while having little or no impact on the accuracy of the recognition results. These solutions enable real-time on-device deployment at the expense of a slight decrease in accuracy that does not significantly affect performance. In some embodiments, the end-to-end model for recognizing and detecting gestures is combined with an additional classification model to improve the reliability of gesture recognition and reduce false positives.
[0028] Figure 11 is an example data processing environment 100 according to some embodiments, which has one or more servers 102 communicatively coupled to one or more client devices 104. One or more client devices 104 may be, for example, a desktop computer 104A, a tablet computer 104B, a mobile phone 104C, a head-mounted display (HMD) (also known as augmented reality (AR) glasses) 104D, or a smart multi-sensing networked home device (e.g., a surveillance camera 104E, a smart TV device, a drone, etc.). Each client device 104 may collect data or user input, execute a user application, and present output on its user interface. The collected data or user input may be processed locally on the client device 104 and / or remotely by the server 102. One or more servers 102 provide system data (e.g., boot files, operating system images, and user applications) to the client device 104, and in some embodiments, process data and user input received from (one or more) client devices 104 when executing a user application on the client device 104. In some embodiments, data processing environment 100 also includes memory 106 for storing data related to server 102 , client device 104 , and applications executing on client device 104 .
[0029] One or more servers 102 are configured to communicate real-time data with client devices 104 that are remote from each other, or to communicate real-time data with client devices 104 that are remote from one or more servers 102. In addition, in some embodiments, one or more servers 102 are configured to perform data processing tasks that cannot be or will not be performed locally by client devices 104. For example, client devices 104 include game consoles (e.g., HMD 104D) that execute interactive online game applications. The game console receives user instructions and sends them to the game server 102 together with user data. The game server 102 generates a video data stream based on the user instructions and user data, and provides the video data stream for display on the game console and other client devices that are engaged in the same game session with the game console. For another example, the client device 104 includes a networked surveillance camera 104E and a mobile phone 104C. The networked surveillance camera 104E collects video data and transmits the video data to the surveillance camera server 102 in real time. Although the video data is optionally pre-processed at the surveillance camera 104E, the surveillance camera server 102 processes the video data to identify motion or audio events in the video data and shares information about these events with the mobile phone 104C, thereby enabling the user of the mobile phone 104 to remotely monitor events occurring near the networked surveillance camera 104E in real time.
[0030] The one or more servers 102, the one or more client devices 104, and the storage 106 are communicatively coupled to each other via one or more communication networks 108, which are media for providing communication links between these devices and connected computers within the data processing environment 100. The one or more communication networks 108 may include connections such as wires, wireless communication links, or fiber optic cables. Examples of the one or more communication networks 108 include a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination thereof. One or more communication networks 108 are optionally implemented using any known network protocol, including various wired or wireless protocols, such as Ethernet, Universal Serial Bus (USB), FireWire, Long Term Evolution (LTE), Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol. Connections to one or more communication networks 108 can be established directly (e.g., using 3G / 4G to connect to a wireless carrier), or through a network interface 110 (e.g., a router, switch, gateway, hub, or smart dedicated whole-home control node), or through any combination thereof. Thus, one or more communication networks 108 may represent the Internet, a global collection of networks and gateways that communicate with each other using the Transmission Control Protocol / Internet Protocol (TCP / IP) suite. At its core, the Internet is a backbone of high-speed data communication lines between major nodes or host computers, consisting of thousands of commercial, government, educational, and other computer systems for routing data and messages.
[0031] In some embodiments, deep learning techniques are applied in the data processing environment 100 to process content data (e.g., video data, visual data, audio data) acquired by an application executed on the client device 104, thereby identifying information contained in the content data, matching the content data with other data, classifying the content data, or synthesizing related content data. The content data can broadly include inertial sensor data captured by an inertial sensor of the client device 104. In these deep learning techniques, data processing models are created based on one or more neural networks to process the content data. Before applying these data processing models to process the content data, these data processing models are trained using training data. After the model is trained, the mobile phone 104C or HMD 104D acquires the content data (e.g., video data is captured by an internal camera) and uses the data processing model locally to process the content data.
[0032] In some embodiments, model training and data processing are implemented locally on each individual client device 104 (e.g., mobile phone 104C and HMD 104D). Client device 104C obtains training data from one or more servers 102 or storage 106 and applies the training data to train the data processing model. Alternatively, in some embodiments, model training and data processing are implemented remotely at a server 102 (e.g., server 102A) associated with a client device 104 (e.g., client device 104A and HMD 104D). Server 102A obtains training data from itself, another server 102, or storage 106, and applies the training data to train the data processing model. Client device 104A obtains content data, sends the content data to server 102A (e.g., in an application) for data processing using the trained data processing model, receives data processing results (e.g., recognized gestures) from server 102A, presents the results on a user interface (e.g., associated with an application), presents a virtual object in the field of view based on the gesture, or implements some other functions based on the results. The client device 104A itself does not perform data processing or performs only minimal data processing on the content data before sending the content data to the server 102A. In addition, in some embodiments, data processing is implemented locally on the client device 104 (e.g., the client device 104b and the HMD 104D), and model training is implemented remotely on the server 102 associated with the client device 104 (e.g., the server 102B). The server 102B obtains training data from itself, another server 102, or the memory 106, and applies the training data to train the data processing model. The trained data processing model is optionally stored in the server 102B or the memory 106. The client device 104 imports the trained data processing model from the server 102B or the storage device 106, processes the content data using the data processing model, and generates a data processing result presented on the user interface or uses the data processing result to initiate some functions locally (e.g., rendering a virtual object based on the device posture).
[0033] In some embodiments, a pair of AR glasses 104D (also referred to as an HMD) is communicatively coupled to the data processing environment 100. The AR glasses 104D include a camera, a microphone, a speaker, one or more inertial sensors (e.g., a gyroscope, an accelerometer), and a display. The camera and microphone are configured to capture video and audio data from a scene of the AR glasses 104D, while the one or more inertial sensors are configured to capture inertial sensor data. In some cases, the camera captures gestures of a user wearing the AR glasses 104D and utilizes a gesture recognition process (e.g., Figure 3The process 300 in ) recognizes gestures locally and in real time. In some cases, the microphone records ambient sounds including the user's voice commands. In some cases, video data or static visual data captured by the camera and inertial sensor data measured by one or more inertial sensors are all used to determine and predict the device posture. The video, static image, audio or inertial sensor data captured by the AR glasses 104D is processed by the AR glasses 104D, (one or more) servers 102, or both to identify the device posture. Optionally, deep learning techniques are jointly applied by (one or more) servers 102 and AR glasses 104D to identify and predict the device posture. The device posture is used to control the AR glasses 104D itself or to interact with an application (e.g., a game application) executed by the AR glasses 104D. In some embodiments, the display of the AR glasses 104D displays a user interface, and the recognized or predicted device posture is used to present a user-selectable display item (e.g., a virtual image) on the user interface or to interact with it.
[0034] As described above, in some embodiments, deep learning techniques are applied to the data processing environment 100 to process video data, static image data, or inertial sensor data captured by the AR glasses 104D. Using a first data processing model, a 2D or 3D device posture is identified and predicted based on such video, static image, and / or inertial sensor data. Visual content is optionally generated using a second data processing model. Optionally, training of the first data processing model and the second data processing model is implemented by the server 102 or the AR glasses 104D. Reasoning of device posture and visual content is independently implemented by each of the server 102 and the AR glasses 104D, or the server 102 and the AR glasses 104D jointly implement reasoning of device posture and visual content.
[0035] Figure 2 1 is a block diagram of an electronic device 200 configured to process content data (eg, image data) according to some embodiments. The electronic device 200 is a server 102, a client device 104 (eg, Figure 1 In one example, the electronic device 200 is a mobile device including a gesture recognition module 230, and the gesture recognition module 230 is applied end-to-end, for example Figure 4The neural network model in the mobile device is used to recognize gestures locally. The electronic device 200 generally includes one or more processing units 202 (such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a neural processing unit (NPU), an AI processing unit (APU)), one or more network interfaces 204, a memory 206, and one or more communication buses 208 for interconnecting these components (sometimes referred to as a chipset). The electronic device 200 includes one or more input devices 210 for facilitating user input, such as a keyboard, a mouse, a voice command input unit or a microphone, a touch screen display, a touch-sensitive input board, a gesture capture camera, or other input buttons or controls. In addition, in some embodiments, the electronic device 200 uses a microphone for voice recognition or a camera 260 for gesture recognition to supplement or replace the keyboard. In some embodiments, the electronic device 200 includes one or more optical cameras (e.g., RGB cameras 260), scanners, or light sensor units for capturing images of, for example, a graphic serial code printed on the electronic device. In some embodiments, the electronic device 200 also includes one or more output devices 212 capable of presenting a user interface and displaying content, including one or more speakers and / or one or more visual displays. Optionally, the electronic device 200 includes a position detection device for determining the position of the electronic device 200, such as a global positioning system (GPS) or other geographic location receiver. Optionally, the electronic device 200 includes an inertial measurement unit (IMU) 280 that integrates sensor data captured by a multi-axis inertial sensor to provide an estimate of the position and orientation of the electronic device 200 in space. Examples of one or more inertial sensors of the IMU 280 include, but are not limited to, gyroscopes, accelerometers, magnetometers, and inclinometers.
[0036] Alternatively or additionally, in some embodiments, the electronic device 200 is communicatively coupled to one or more devices (e.g., a server 102, a client device 104, a memory 106, or a combination thereof) via one or more network interfaces 204, which include one or more input devices 210, output devices 212, multiple IMUs 280, or other components described above, and provides data to the electronic system 200.
[0037] The memory 206 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices; optionally includes a non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Optionally, the memory 206 may include one or more storage devices remote from the one or more processing units 202. The memory 206 or the non-volatile memory within the memory 206 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 206 or the non-transitory computer-readable storage medium of the memory 206 stores the following programs, modules and data structures, or a subset or superset thereof:
[0038] Operating system 214, including programs for handling various basic system services and for performing hardware-related tasks:
[0039] A network communication module 216 for connecting each server 102 or client device 104 to other devices (e.g., a server 102, a client device 104, or a storage device 106) via one or more (wired or wireless) network interfaces 204 and one or more communication networks 108 (such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.);
[0040] A user interface module 218 for enabling presentation of information (e.g., a graphical user interface of application(s) 224, widgets, websites and their web pages, and / or games, audio and / or video content, text, etc.) on each client device 104 via one or more output devices 212 (such as a display, speakers, etc.).
[0041] An input processing module 220 for detecting one or more user inputs or interactions from one of the one or more input devices 210 and interpreting the detected inputs or interactions;
[0042] a web browser module 222 for navigating, requesting (e.g., via HTTP) and displaying websites and their pages,
[0043] including a web interface for logging into a user account associated with the client device 104 or another electronic device, for controlling the client or electronic device when associated with the user account, and for editing and viewing settings and data associated with the user account;
[0044] One or more user applications 224 executed by the electronic system 200 (e.g., games, social networking applications, smart home applications, and / or other web-based or non-web-based applications for controlling other electronic devices and viewing data captured by such devices);
[0045] A model training module 226 for receiving training data and building a data processing model for processing content data (e.g., video, image, audio, or text data) to be collected or acquired by the client device 104;
[0046] A data processing module 228 for processing the content data using the data processing model 250 to identify information contained in the content data, match the content data with other data, classify the content data, or synthesize related content data, wherein in some embodiments, the data processing module 228 is associated with one of the plurality of user applications 224 to process the content data in response to user instructions received from the user application 224, and in one example, the data processing module 228 is used to implement Figure 3 The gesture recognition process 300;
[0047] A gesture recognition module 230 for compressing image data and generating a multi-scale feature map based on the compressed image data (e.g., Figure 3 and Figure 4 ), wherein in some embodiments, the gesture recognition module 230 and the data processing module 228 jointly compress the image data and generate a multi-scale feature map, and the multi-scale feature map is upscaled and fused to generate a comprehensive feature map for gesture recognition;
[0048] One or more databases 240, used to store at least one or more of the following data:
[0049] o Device settings 242, including general device settings of one or more servers 102 or client devices 104 (such as service tier, device model, storage capacity, processing power, communication capabilities, etc.);
[0050] o User account information 244 of one or more user applications 224, such as user name, security questions, account history data, user preferences, and predefined account settings;
[0051] o Network parameters 246 of one or more communication networks 108, such as IP addresses, subnet masks, default gateways, DNS servers, and host names;
[0052] o training data 248 for training one or more data processing models 250;
[0053] o Data processing model 250, for processing content data (such as video, image, audio or text data) using deep learning technology, wherein the data processing model 250 includes the following reference Figure 3 and Figure 4The image compression model for implementing the image compression process, the feature extraction model for implementing the multi-scale feature extraction process and / or one or more classification models and networks;
[0054] o a gesture database 252 for storing one or more gestures associated with the candidate images (eg, stored in database 240); and
[0055] o Content data and results 254, which are acquired and outputted respectively by the electronic system 200 (or a device communicatively coupled to the electronic system 200, such as the client device 104), wherein the content data is processed locally on the client device 104 or remotely on the server 102 by the data processing model 250 to provide associated results to be presented on the client device 104, the content data including candidate images.
[0056] In some embodiments, the gesture recognition module 230 also includes an image processing module 232 configured to reduce the image data, extract one or more features, and scale the one or more features to provide a comprehensive feature vector from which a gesture is generated. Figure 3 and Figure 4 Describes more details of gesture recognition using an end-to-end lightweight neural network that can be implemented in resource-limited electronic devices.
[0057] Optionally, one or more databases 240 are stored in one of the server 102, client device 104, and memory 106 of the electronic system 200. Optionally, one or more databases 240 are distributed in more than one of the server 102, client device 104, and memory 106 of the electronic system 200. In some embodiments, more than one copy of the above data is stored at different devices, for example, two copies of the data processing model 250 are stored at the server 102 and the memory 106, respectively.
[0058] Each of the above identified elements can be stored in one or more aforementioned storage devices, and corresponds to a group of instructions for performing the above functions. The above identified modules or programs (i.e., instruction sets) do not need to be implemented as independent software programs, processes, modules or data structures, so the various subsets of these modules can be combined or otherwise rearranged in various embodiments. In certain embodiments, memory 206 optionally stores the subset of the above modules and data structures. In addition, memory 206 optionally stores additional modules and data structures not described above.
[0059] Figure 3A flow chart of a gesture recognition process 300 according to some embodiments is shown, which is used to detect and recognize one or more gestures using image data. The gesture recognition process 300 is configured to detect gestures captured by the camera 260 from image data taken at a long distance. In some embodiments, the long distance between the gesture and the camera 260 is greater than a predetermined distance. For example, the gesture recognition process 300 is configured to detect and recognize gestures captured at least 4 meters away from the camera 260. The gesture recognition process 300 uses limited computing resources, such as 3.5 GFLOPS or less computing resources of the electronic device 200. In some embodiments, the gesture recognition process 300 is configured to detect and recognize gestures at a distance greater than a predetermined distance from the capturing imaging device. Although the gesture recognition process 300 is configured to detect gestures located at a long distance from the capturing imaging device, the gesture recognition process 300 is also configured to detect gestures captured at a close distance relative to the capturing imaging device (e.g., 0.5-1 meters from the camera 260 that captures the image data).
[0060] The gesture recognition process 300 is optionally referred to above. Figure 1 and Figure 2 The gesture recognition process 300 is performed by one or more client devices 104, servers 102, and / or a combination thereof as described above. The gesture recognition process 300 includes an image processing stage 310, a feature map upsampling process 320, a feature map decoding process 330, a bounding box selection process 340, an object detection process 350, and / or a classification process 370. The object detection process 350 is followed by a cropping process and applying a classification network 360 to identify one or more object detection gestures 365. The classification process 370 identifies one or more classification gestures 375 based on the bounding box selected by the bounding box selection process 340. Each object detection gesture 365 is identified based on a corresponding hand region 355 in the entire image, i.e., based on a corresponding portion of the entire image. In contrast, each classification gesture 375 is determined directly from the comprehensive feature map 332 of the entire image. For the same gesture of the same hand, the object detection gesture 365 may be more or less accurate than the classification gesture 375, depending on a number of factors (e.g., the distance of the hand from the camera). The gesture recognition process 300 determines a final gesture 380 based on the one or more object detection gestures 365 and / or the one or more classification gestures 375. Each process of the gesture recognition process 300 is discussed in detail below.
[0061] In the image processing stage 310, one or more machine learning modules are applied to receive image data 312. Before applying the one or more machine learning modules to the image data 312, the image data 312 may be pre-processed, as shown in FIG. Figure 4 In some embodiments, the image data 312 is provided by the electronic device 200 ( Figure 2) is captured by an input device 210 (e.g., an RGB camera 260) of the electronic device 200. Alternatively, in some embodiments, the image data 312 is acquired or downloaded via a web browser module 222 and / or one or more user applications 224 included on the electronic device 200. Alternatively or additionally, in some embodiments, the image data 312 is received from another device (e.g., a laptop, a smart phone, AR glasses, a server, etc.) that is communicatively coupled to the electronic device 200. The image data 312 includes one or more gestures. In some cases, the gestures within the image data 312 are at least 4 meters away from the electronic device 200 that captured the image data. Non-limiting examples of one or more gestures recognized from the image data 312 include one or more hand gestures, facial gestures, and body gestures. The image data 312 is received at an initial resolution (e.g., 1080p).
[0062] The image data 312 passes through one or more machine learning modules in the image processing stage 310 to compress the image data 312 and / or generate one or more feature maps based on the image data 312. In some embodiments, the image data 312 is downscaled before passing through the one or more machine learning modules. In some embodiments, the one or more machine learning modules include a first machine learning module 314 and a second machine learning module 316, the first machine learning module 314 being configured to identify and enrich (e.g., extract details from) one or more features from the image data 312, and to reduce (relative to the initial resolution of the image data 312) the feature map feature resolution of a plurality of feature maps (generated by the first machine learning module 314), and the second machine learning module 316 being configured to generate a sequence of (scaled) feature maps (e.g., first feature maps, second feature maps, and third feature maps 318a-318c) based on the (compressed) image data 312. In some embodiments, the sequence of feature maps is combined into a comprehensive feature map, which is used for gesture detection and gesture recognition as discussed in detail below.
[0063] In some embodiments, the first machine learning module 314 is a backbone network, for example, including a feature extraction network for extracting features from the image data 312. The first machine learning module 314 is configured to compress the image data 312 while enriching and / or exploiting details in the image data 312. In some embodiments, enriching and / or exploiting details in the image data 312 means improving and / or preserving as many details as possible in the image data 312 so that the information in the image data 312 is not lost and can be recovered during compression. The first machine learning module 314 generates one or more features from which one or more gestures are detected and recognized.
[0064] In some embodiments, the second machine learning module 316 includes a multi-scale feature extraction model that generates a feature map sequence 318. The feature map sequence is generated based on the output of the first machine learning module 314 (e.g., the first feature map generated by the first machine learning module 314). In some embodiments, the feature map sequence 318 includes a first feature map 318-a, a second feature map 318-b, and a third scaled feature map 318-c. In some embodiments, each feature map 318 of the feature map sequence each has a feature resolution that is continuously downscaled in the sequence, and therefore, the feature map sequence 318 corresponds to feature maps of multiple different scales. For example, the first feature map 318-a has a first feature resolution, the second feature map 318-b has a second feature resolution that is smaller than the first feature resolution, and the third feature map 318-c has a third feature resolution that is smaller than the second feature resolution. The feature maps generated by the first machine learning module 314 and the second machine learning module 316 are referred to later. Figure 4 Discuss in detail.
[0065] The feature map upsampling process 320 upsamples the output of the second machine learning model 316. Each feature map subset is upsampled to the higher feature resolution of the respective previous feature map and combined with the respective previous feature map. In some embodiments, one of the subsets of feature maps (e.g., 318-b and 318-c) is generated by the machine learning module 316. The output third feature map 318-c is upsampled to the feature resolution of the second feature map 318-b and combined with the second feature map 318-b to form a combined second feature map 318-b'. The second feature map 318-b can be upsampled to the feature resolution of the first feature map 318-a and combined with the first feature map 318-a to form a combined first feature map 318a'. Alternatively, in some embodiments, one of the subsets of feature maps (e.g., 318-b') is combined from the upsampled feature map 318. For example, the combined second feature map 318-b' is upsampled to the same feature resolution as the first feature map 318-a, and is combined with the first feature map 318-a to form the combined first feature map 318-a'. In this way, the feature map sequence 318 is continuously downscaled via the machine learning module 316, and the feature map subsets 318-c and 318-b are both upsampled to the same feature resolution as the respective previous feature maps 318-b and 318-a, and are respectively fused into the respective previous feature maps 318-b and 318-a.
[0066] In some embodiments, the feature map decoding process 330 is configured to combine the updated sequence of feature maps of different scales into a comprehensive feature map 332, and determine one or more bounding boxes 334 based on the comprehensive feature map 332. The updated sequence of feature maps includes feature maps 318-a', 318-b', and 318-c output via the feature map upsampling process 320. Alternatively, in some embodiments, the feature map decoding process 330 is configured to combine the sequence of feature maps 318 of different scales into a comprehensive feature map 332, and determine one or more bounding boxes 334 based on the comprehensive feature map 332. The updated sequence of feature maps includes feature maps 318-a, 318-b, and 318-c output by the second machine learning module 316. Each bounding box corresponds to an object captured in the image data 312.
[0067] The bounding box selection process 340 identifies a subset 338 of the one or more bounding boxes 334 that meet a confidence criterion. In some embodiments, the bounding box selection process 340 identifies the subset 338 of the one or more bounding boxes based on the corresponding probabilities or confidence levels 336 that the detected bounding boxes 334 are gestures according to the confidence criterion. For example, after the bounding boxes 334 are detected, the bounding boxes 334 are sorted in a list based on the corresponding probabilities or confidence levels 336, and a predetermined number of boxes sorted at the top of the list are selected as the bounding box subset 338. In another example, the bounding box subset 338 is selected according to the confidence criterion so that the probability or confidence level 336 of each bounding box in the bounding box subset 338 corresponding to a corresponding predefined gesture is greater than a threshold. In other words, the one or more most reliable bounding boxes 338 are selected to determine the final gesture 380 in the image data 312.
[0068] The output of the bounding box selection process 340 is used to determine a final pose 380 based on an object detection process 350 and / or a classification process 370. The object detection process 350 detects one or more potential objects in the bounding box, as shown in the image 352, and the image data 312 is cropped to the hand region 355. That is, the cropping process crops the image data 312 to the hand region 355 centered on the pose. The classification network 360 is used on the hand region 355 to determine a first pose vector 365. More specifically, based on the extracted hand region 355 determined from the comprehensive feature map 332, the classification network 360 determines at least the first pose vector 365. In some embodiments, the classification network 360 is one or more networks known in the art (e.g., such as a mobilenet v1 network, a mobilenet v2 network, a ShuffleNet network, etc.). In some embodiments, classification network 360 is selected based on an intended gesture task (e.g., gestures expected by a particular application and / or system) and / or based on multiple categories for classification (e.g., different types of gestures that may be classified by a particular application and / or system).
[0069] Optionally, in some embodiments, the gesture recognition process 300 does not include the classification network 360 (instead, only the end-to-end framework is used to detect and recognize gestures). By removing the classification network 360, the gesture recognition process 300 can detect and recognize gestures with less computation but lower accuracy (e.g., a trade-off between accuracy and response time). In some embodiments, the classification network 360 is removed based on the requirements of a particular application and / or system. For example, in some embodiments, when one or more processors 202 ( Figure 2 ; For example, when (one or more) CPUs, GPUs, DSPs, NPUs, APUs) are less powerful (and / or use cheaper processors), remove the classification network 360. In some embodiments, the removal of the classification network 360 reduces the inference time of the gesture recognition process 300 by at least 30%.
[0070] In some embodiments, based on the selected bounding box 338 generated by the bounding box selection process 340, the classification process 370 identifies one or more classified gestures represented by the second gesture vector 375. The classification process 370 determines the second gesture vector 375 from the integrated feature map 332 in a relatively more direct manner. That is, the second gesture vector 375 is generated directly from the image data 312, rather than generating the second gesture vector 375 from the hand region 355 cropped from the image data 312.
[0071] In some embodiments, the classification process 370 is performed in parallel with the object detection process 350, the cropping process, and the classification network 360. A final pose 380 of the image data 312 is generated by combining one or more object detection poses 365 and one or more classification poses 375. Both poses 365 and poses 375 are determined based on the integrated feature map 332 (e.g., the output of the bounding box selection process 340). In some embodiments, the poses 365 and poses 375 are represented by a first pose vector 365 and a second pose vector 375, respectively. The final pose 380 is determined based on the integrated feature map 322 by determining a first pose vector 365 centered on the hand region 355 cropped from the image data 312, and determining a second pose vector 375 using the classification process 370.
[0072] In some embodiments, the first posture vector and the second posture vector (e.g., posture 365 and posture 375, respectively) are combined into a final posture vector 380. For example, in some embodiments, the final posture 380 is determined based on the final posture vector 380 (e.g., a posture vector including multiple elements, each element representing the probability of a respective predefined gesture). In some embodiments, one or more object detection postures 365 and / or one or more classification postures 375 with the highest probability (or satisfying a predetermined threshold and / or a predetermined confidence level) are used to determine the final posture vector. In some embodiments, the first posture vector 365 and the second posture vector 375 are combined in a weighted manner. For example, the first posture vector 365 and the second posture vector 375 have an equal weight of 0.5. In one example, the first posture vector 365 and the second posture vector 375 have weights of 1 and 0, respectively. Without applying the classification process 370, the final posture 380 is obtained from the hand region 355. In contrast, in another example, the first posture vector 365 and the second posture vector 375 have weights of 0 and 1, respectively. The classification process 370 is applied to determine the final pose 380 without applying the object detection process 350 , the cropping process, and the classification process 360 .
[0073] In some embodiments, each of first posture vector 365, second posture vector 375, and final posture vector 380 has a predefined number of elements corresponding to the same number of different postures. Each element of each posture vector 365, 375, or 380 indicates the probability that the detected posture corresponds to the respective different posture. For each posture vector 365, 375, or 380, the predefined number of elements is normalized. In some embodiments, for the final posture vector 380, the predefined gesture corresponding to the maximum probability is selected as the final posture 380.
[0074] although Figure 3An example of determining a gesture is shown, but in some embodiments, the gesture recognition process 300 is configured to detect one or more of a facial gesture, an arm gesture, a body gesture, a hand gesture, and / or other classifiable gestures performed by a user.
[0075] Figure 4 400 is a flowchart illustrating an example image processing stage 400 of a gesture recognition process according to some embodiments. The image processing stage 400 illustrates the above reference Figure 3 One or more components of the image processing stage 310 described. The image processing stage 400 illustrates reducing the resolution of received image data 312 using a first machine learning module 314 (e.g., including a backbone network 410), using a second machine learning module 316 (e.g., including a multi-scale feature extraction network 420). The multi-scale feature extraction network 420 includes a network 420A, a network 420B, a network 420C, and a network 420D, which are configured to generate a sequence of feature maps 318-aa, feature maps 318-a, feature maps 318-b, and feature maps 318-c based on the image data 312, each feature map 318 having a corresponding feature resolution that is successively downscaled in the sequence of feature maps 318.
[0076] In some embodiments, the received image data 312 has a resolution of at least 1080p. In some embodiments, the received image data 312 is downscaled (e.g., by the first neural network 410) to an input size of 288×512×3. The image data 312 can be downscaled to any resolution or input size. The output of the first neural network 410 is further downscaled using a network 420A including an input network 430 and a bottleneck network 440. In some embodiments, the neural network 410 and the neural network 430 are convolutional layers with a 3×3 kernel and a stride of 2 (conv-k3s2). In some embodiments, the convolutional layers of the networks 410 and 430 are followed by an activation function (e.g., a rectified linear activation function (ReLU)). In some embodiments, the activation function is an identity function. The downscaling step size of the neural network 410 and the neural network 430 is low, so that Figure 3 The gesture recognition process 300 starts with a low level of information loss and preserves as much local information as possible within the small convolution kernel size of each corresponding output.
[0077] The first machine learning module 314 includes a first bottleneck structure (bottleneck-1) and a second bottleneck structure (bottleneck-2). In some embodiments, the first machine learning module 314 is referred to as a backbone structure. Figure 3As described above, the bottleneck structure (eg, the first machine learning module 314) is used to utilize as many details as possible and compress large networks (ie, the outputs of the first neural network 410, the second neural network 420, and the third neural network 430).
[0078] The first bottleneck structure includes a first neural network structure 440 and a skip connection structure 445. In some embodiments, the first neural network structure 440 includes: a 1×1 point-by-point convolution layer 440a, a depth convolution layer 440b with a stride of 1 and a 3×3 kernel after the convolution layer 440a, and a 1×1 point-by-point convolution layer 440c after the convolution layer 440b. In some embodiments, each convolution layer (e.g., convolution layer 440a, convolution layer 440b, and convolution layer 440c) of the first neural network structure 440 includes an activation function (or an identity function). In some embodiments, the skip connection structure 445 is applied to the output of the third neural network 430 and connected to the output of the first neural network structure 440 via an addition operation 447. The skip connection structure 445 is configured to help convergence by retaining features generated by the first neural network structure 440. In some embodiments, the skip connection structure 445 is a residual block (ResBlock) structure.
[0079] The second bottleneck structure includes a second neural network structure 450. In some embodiments, the second neural network structure 450 includes: a 1×1 point-by-point convolution layer 450a, a depthwise convolution layer 440b with a 3×3 kernel with a stride of 2 after the convolution layer 450a (which reduces the spatial dimension by half), and a 1×1 point-by-point convolution layer 450c after the convolution layer 450b. In some embodiments, each convolution layer of the second neural network structure 450 (e.g., convolution layer 450a, convolution layer 450b, and convolution layer 450c) includes an activation function (or identity function).
[0080] In some embodiments, in order to further compress the outputs (e.g., reduced image data) of the first neural network 410, the second neural network 420, and the third neural network 430 while maintaining performance, the number of channels of the first bottleneck structure 440 and the second bottleneck structure 450 is a multiple of 3 instead of a multiple of 6. Based on fine-tuning one or more parameters of each bottleneck structure so that the overall accuracy and training efficiency are not reduced, the first bottleneck structure 440 and the second bottleneck structure 450 with a channel number of multiple of 3 are selected.
[0081] In some embodiments, the second machine learning module 316 includes a first bottleneck structure (bottleneck-1), a second bottleneck structure (bottleneck-2), and a third neural network structure 460. The third neural network structure 460 includes a plurality of convolutional layers (a first convolutional layer 460a, a second convolutional layer 460b, and a third convolutional layer 460c). In some embodiments, the first convolutional layer 460a of the plurality of convolutional layers has a step size of 2 (conv-k3s2). In some embodiments, the second convolutional layer 460b and the third convolutional layer 460c of the plurality of convolutional layers have a step size of 1 (conv-k3s1). In some embodiments, each convolutional layer of the third neural network structure 460 (e.g., convolutional layer 460a, convolutional layer 460b, and convolutional layer 460c) includes an activation function (or identity function).
[0082] In some embodiments, the second machine learning module 316 is referred to as a multi-scale feature extraction model. The second machine learning module 316 is configured to gradually downscale the output (e.g., feature map) of the first machine learning module 314, thereby reducing the overall complexity of the output of the first machine learning module 314 (compared to the output with a larger kernel size) with minimal performance loss. For example, in some embodiments, the second machine learning module 316 uses the output of the first bottleneck structure to generate a first feature map 318a, uses the second bottleneck structure to generate a second feature map 318b, and uses the third neural network structure 460 to generate a third feature map 318c (e.g., as described above with reference to Figure 3 The second machine learning module 316 (i.e., multi-scale feature pyramid) plays a key role because it outputs feature maps after each reduction (downscaling). Feature maps of different sizes can represent different levels of detail. For example, Figure 3 and Figure 4 The second machine learning module 316 is shown outputting feature maps of at least three different sizes. Feature maps with larger feature sizes (e.g., first feature map 318a) allow for the recognition of smaller gestures (e.g., gestures made by smaller hands, or gestures that are farther away from the image device that captured the image data 312), while feature maps with smaller feature sizes (e.g., third feature map 318c) allow for the detection of larger bounding boxes (e.g., decoded from the feature map decoding process 330; Figure 3 ). This ensures that the same gesture captured at a distance from the imaging device that captured the image data and captured near the imaging device will be consistently recognized as the same gesture, which will help increase the accuracy of the gesture recognition process 300, thereby improving the performance of the system.
[0083] An advantage of the image processing stage 400 of the gesture recognition process is the ability of the first machine learning module 314, the second machine learning module 316, and the third neural network structure 460 to maintain ultra-low computational complexity of the feature map, which enables real-time gesture recognition on electronic devices with limited computing resources (e.g., smartphones, smart watches, smart glasses, and / or other electronic devices with weaker processors). In addition, using the image processing stage 400 of the gesture recognition process reduces the power consumption of the electronic device and increases the overall battery life of the electronic device.
[0084] Figure 5 1 is a flowchart of a method for gesture detection and recognition according to some embodiments. The method 500 includes executing the above reference Figure 3 and 4 The gesture recognition process 300 and image processing stage 400 described above. As described above, the gesture recognition process 300 and image processing stage 400 are configured to detect and recognize one or more gestures in image data while using a limited amount of computing resources (e.g., using 3.5 GLOPS or less). The limited computing resources are measured in GLOPS and are controlled within a predefined resource threshold so that gesture recognition can be implemented on many electronic devices with computing resources lower than the limited computing resources. The operations (e.g., steps) of the gesture detection and recognition process 500 are performed by one or more processors 202 ( Figure 2 ; for example, multiple CPUs, GPUs, DSPs, NPUs, APUs). Figure 5 At least some of the operations shown correspond to data stored in a computer memory or computer-readable storage medium (e.g., memory 206; Figure 2 ). Operations 502 to 508 may also be performed in part using one or more processors and / or using instructions stored in a memory or computer-readable medium of one or more devices that are communicatively coupled together, such as a laptop, AR glasses (or other head-mounted displays), a server, a tablet, a security camera, a smart TV, a smart speaker, a toy, a smart watch, a smart appliance, or other computing devices that can perform operations 502-508 alone or in conjunction with a corresponding processor of the communicatively coupled electronic device 200.
[0085] Method 500 includes acquiring (502) image data of an image. The image captures a hand. In some embodiments, the hand is making a gesture. For example, as described above with reference to Figure 3As shown, image data 312 includes gestures made by a user's hand. In some embodiments, the image data includes gestures at least 4 meters away from the electronic device that captured the image data. That is, the hand is captured in the image by a camera device at least 4 meters away from the hand. In some embodiments, the image data is captured at a resolution of at least 1080p.
[0086] The method 500 also includes generating (504) a sequence of feature maps based on the image data. The sequence of feature maps has a continuously downscaled feature resolution. For example, as described above with reference to Figure 3 As shown, by the image processing stage (as mentioned above Figure 3 and 4 In some embodiments, a first machine learning module (e.g., the first feature map, the second feature map, and the third feature map, 318a-318c) is used to generate a feature map sequence (e.g., a first feature map, a second feature map, and a third feature map, 318a-318c). Figure 3 and Figure 4 The first machine learning module 314) generates a feature map sequence based on the image data. The first machine learning module is configured to identify and enrich (e.g., extract and / or generate more significant details) one or more features from the image data and reduce the feature resolution of the feature map sequence relative to the resolution of the image data.
[0087] In some embodiments, a feature map sequence is generated based on applying a backbone network to image data. The backbone network includes at least two convolutional layers and multiple bottleneck structures. For example, Figure 4 As shown, the backbone network may include a second neural network 420 and a third neural network 430 (e.g., as at least two convolutional layers) and a first machine learning module 314, which Figure 4In some embodiments, each of the at least two convolutional layers of the backbone network has a 3×3 kernel with a stride of 2, and the first convolutional layer of the at least two convolutional layers is followed by the second convolutional layer of the at least two convolutional layers. For example, the second neural network 420 is followed by the third neural network 430. In some embodiments, the first bottleneck structure is configured to extract one or more features from the outputs of the at least two convolutional layers, the first bottleneck structure includes a first 1×1 point-by-point convolutional layer, a first 3×3 depth convolutional layer after the first 1×1 point-by-point convolutional layer, and a second 1×1 point-by-point convolutional layer after the first 3×3 depth convolutional layer and an addition operation including a skip connection structure; the second bottleneck structure is configured to reduce the output of the first bottleneck structure, the second bottleneck structure includes a third 1×1 point-by-point convolutional layer, a second 3×3 depth convolutional layer with a stride of 2 after the third 1×1 point-by-point convolutional layer, and a fourth 1×1 point-by-point convolutional layer after the second 3×3 depth convolutional layer. In some embodiments, the output of the second bottleneck structure is a plurality of feature maps. In some embodiments, each of the plurality of bottleneck structures has 3 layers and a multiple of 3 channels. Figure 4 Additional information about at least two convolutional layers and multiple bottleneck structures is provided above.
[0088] In some embodiments, generating a sequence of feature maps includes: generating a first feature map having a first feature resolution; reducing the first feature map to a second feature map according to a scaling factor, the second feature map immediately following the first feature map in the sequence of feature maps; and reducing the second feature map to a third feature map according to the scaling factor, the third feature map immediately following the second feature map in the sequence of feature maps. Figure 4 As shown, in some embodiments, the second machine learning module 316 includes a first bottleneck structure (bottleneck-1), a second bottleneck structure (bottleneck-2), and a third neural network structure 460, which is configured to gradually downscale the output of the first machine learning module 314, thereby reducing the overall complexity of the output of the first machine learning module 314 with minimal performance loss. Specifically, the second machine learning module 316 generates a first feature map 318a, a second feature map 318b, and a third feature map 318c. In some embodiments, each convolution layer (reference numerator, reference numerator, and convolution layer) is configured to gradually downscale the output of the first machine learning module 314, thereby reducing the overall complexity of the output of the first machine learning module 314 with minimal performance loss. Figure 4 The activation function (or identity function) is followed by the activation function.
[0089] The method 500 includes combining (506) the sequence of feature maps to generate a composite feature map. More specifically, in some embodiments, the first feature map, the second feature map, and the third feature map 318a-318c are fused together. In some embodiments, each subsequent feature map is upsampled to the same spatial size as the previous feature map before being fused. For example, as described above with reference to Figure 3 As shown, the third feature map 318c is upsampled and fused with the second feature map 318b, and the second feature map 318b is upsampled and fused with the first feature map 318a.
[0090] In some embodiments, the comprehensive feature map (e.g., the above reference Figure 3 The fused feature map shown and described above is decoded to identify one or more bounding boxes from the sequence of fused feature maps. In some embodiments, method 500 includes: identifying a subset of bounding boxes in one or more bounding boxes that meet a confidence criterion; and formatting a comprehensive feature map based on the subset of bounding boxes in one or more bounding boxes.
[0091] Method 500 includes determining (508) a gesture in the image based on the integrated feature map. In some embodiments, determining the gesture based on the integrated feature map includes: detecting and extracting a hand region (e.g., Figure 3 The first posture vector (e.g., corresponding to the hand region 355 in FIG. 1 ) is determined based on the hand region 355 extracted from the comprehensive feature map. Figure 3 365). Determine a second posture vector (e.g., corresponding to Figure 3 The image includes a hand region and one or more other regions different from the hand region. The first posture vector and the second posture vector are combined to generate a final posture vector based on which the gesture is generated. It should be noted that the first gesture is determined based on the hand region in the entire image, that is, based on the corresponding part of the entire image. In contrast, the second gesture is directly determined based on the comprehensive feature map of the entire image.
[0092] In some embodiments, method 500 is configured to detect a gesture in image data that is at least 4 meters away from the electronic device that captured the image data. Figure 3 Provides additional information about the detection and identification process.
[0093] It should be understood that Figure 5 The specific order in which the operations are described in is merely exemplary and is not intended to represent that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize various ways of retrieving candidate images or determining camera poses as described herein. In addition, it should be noted that the above reference Figure 3 and Figure 4 The other details of the process are similarly applicable to the above references. Figure 5 The method 500 is described. For the sake of brevity, these details are not repeated here.
[0094] The terms used in the description of various embodiments described herein are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various embodiments described and in the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to also include plural forms. It should also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the items listed in association. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping. It will also be understood that although the terms "first", "second", etc. are used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element.
[0095] As used herein, the term “if” is optionally interpreted to mean “when” or “upon” or “in response to determining” or “in response to detecting” or “in accordance with a determination of…”, depending on the context. Similarly, the phrases “if it is determined that…” or “if [a stated condition or event] is detected” are optionally interpreted to mean “upon determining that…” or “in response to determining that…” or “upon detecting [a stated condition or event]” or “in response to detecting [a stated condition or event]” or “in accordance with determining detecting [a stated condition or event]”, depending on the context.
[0096] For the purpose of explanation, the foregoing description is described by reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the invention to the disclosed precise form. According to the above teachings, many modifications and variations are possible. These embodiments are selected and described in order to best explain the principles of these technologies and their practical applications, so that other technical personnel in the field can implement them.
[0097] Although the various figures illustrate several logical stages in a particular order, these order-independent stages may be reordered, and other stages may be combined or decomposed. Although some reordering or other groupings are specifically mentioned, other ways will be obvious to those of ordinary skill in the art, and thus the ordering and groupings presented herein are not exhaustive alternatives. In addition, it should be recognized that these stages can be implemented in hardware, firmware, software, or any combination thereof.
Claims
1. A method for detecting a gesture, applied to an electronic device, the method comprising: acquiring image data of an image, wherein the image captures a hand; generating a sequence of feature maps based on the image data, wherein the sequence of feature maps has a continuously downscaled feature resolution; combining the feature map sequences to generate a comprehensive feature map; and A gesture of the hand captured in the image is determined based on the comprehensive feature map.
2. The method according to claim 1, wherein: The generating of the feature map sequence based on the image data comprises: using a machine learning model to identify one or more features from the image data and enrich the one or more features, and reducing the feature resolution of the feature map sequence relative to the resolution of the image data.
3. The method according to claim 2, wherein: The generating the feature map sequence based on the image data comprises: using a backbone network to generate the feature map sequence based on the image data, wherein the backbone network comprises at least two convolutional layers and multiple bottleneck structures.
4. The method according to claim 3, wherein: Each of the at least two convolutional layers is a 3×3 kernel with a stride of 2; and A first convolutional layer of the at least two convolutional layers is followed by a second convolutional layer of the at least two convolutional layers.
5. The method according to claim 3, wherein: The multiple bottleneck structures include: a first bottleneck structure configured to extract one or more features from outputs of the at least two convolutional layers, wherein the first bottleneck structure comprises: a first 1×1 point-by-point convolutional layer, a first 3×3 depth convolutional layer after the first 1×1 point-by-point convolutional layer, and a second 1×1 point-by-point convolutional layer after the first 3×3 depth convolutional layer and an addition operation including a skip connection structure; and A second bottleneck structure is configured to reduce the output of the first bottleneck structure, wherein the second bottleneck structure includes: a third 1×1 point-by-point convolutional layer, a second 3×3 depth convolutional layer with a stride of 2 after the third 1×1 point-by-point convolutional layer, and a fourth 1×1 point-by-point convolutional layer after the second 3×3 depth convolutional layer; The output of the second bottleneck structure is a plurality of feature maps.
6. The method according to claim 5, wherein: Each of the first bottleneck structure and the second bottleneck structure has 3 layers and channels which are multiples of 3.
7. The method according to claim 3, wherein: Each of the convolutional layers is followed by an activation function.
8. A method according to any one of the preceding claims, wherein: The generating of the feature graph sequence comprises: generating a first feature map having a first feature resolution; reducing the first feature map to a second feature map according to a scaling factor, the second feature map immediately following the first feature map in the feature map sequence; and The second feature map is reduced to a third feature map according to the scaling factor, and the third feature map immediately follows the second feature map in the feature map sequence.
9. A method according to any one of the preceding claims, wherein: The step of combining the feature map sequences to generate a comprehensive feature map comprises: Decoding one or more bounding boxes from the sequence of feature maps; identifying a subset of the one or more bounding boxes that satisfy a confidence criterion; and The synthetic feature map is formatted based on the bounding box subset of the one or more bounding boxes.
10. A method according to any one of the preceding claims, wherein: The hand is captured in the image by a camera device at least 4 meters away from the hand.
11. A method according to any one of the preceding claims, wherein: The electronic device includes one of a security camera, a smart TV, a smart speaker, a toy, a smart watch, a smart appliance or a head-mounted display.
12. A method according to any one of the preceding claims, wherein: The image data has a resolution of at least 1080p, and the method is implemented using computing resources of the electronic device of 3.5 billion floating point operations per second (GFLOPS) or less.
13. A method according to any one of the preceding claims, wherein: Determining the gesture in the image according to the comprehensive feature map further includes: Based on the comprehensive feature map, detecting and extracting a hand region in the image data; Determining a first posture vector based on the hand region extracted from the comprehensive feature map; determining a second posture vector from the integrated feature map corresponding to the image data of the image, the image including the hand region and one or more remaining regions different from the hand region; combining the first posture vector and the second posture vector to generate a final posture vector; and The gesture is determined based on the final pose vector.
14. An electronic device comprising: one or more processors; as well as A memory storing instructions which, when executed by the one or more processors, cause the processors to implement the method according to any one of claims 1-13.
15. A non-transitory computer-readable medium storing instructions, which when executed by one or more processors cause the one or more processors to implement the method according to any one of claims 1-13.
