A gesture recognition method and apparatus

Abstract

The application provides a gesture recognition method and device, to solve the problem that the operation mode of an acceleration processor is limited and cannot run a complex target detection model (YOLO-V4 model). The method comprises: obtaining a to-be-recognized picture; obtaining a hand image from the to-be-recognized picture; obtaining a hand feature image by continuously obtaining a hand feature in a high-dimensional space and performing down-sampling processing on the hand feature for a plurality of times for the hand image, wherein the hand feature image comprises hand feature information in different dimensions and different resolutions in the hand image; performing multi-scale maximum pooling processing on the hand feature image, and performing fusion processing on the feature image after the multi-scale maximum pooling processing to obtain a first multi-scale feature image; obtaining gesture category probabilities corresponding to a plurality of prediction boxes respectively according to the first multi-scale feature image; and recognizing a gesture of a hand in the to-be-recognized picture according to the gesture category probabilities corresponding to the plurality of prediction boxes respectively.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a gesture recognition method and apparatus. Background Technology

[0002] Currently used gesture recognition algorithms generally employ the YOLO-V4 model. The YOLO-V4 model requires a high-performance processor, such as a Central Processing Unit (CPU) or a Graphics Processing Unit (GPU). Smart TVs and other display devices typically incorporate accelerator processors (APUs) to run deep learning recognition algorithms, handle artificial intelligence (AI) calculations, and implement AI applications. Common APUs include embedded neural network processing units (NPUs), accelerated processing units (APUs), deep learning processing units (DPUs), and tensor processing units (TPUs). However, due to the limitations of APUs' computational capabilities, they cannot run complex YOLO-V4 models and therefore cannot meet the gesture recognition requirements of display devices. Summary of the Invention

[0003] This application provides a gesture recognition method and apparatus to solve the problem that the limited computing power of accelerator processors makes it impossible to run complex YOLO-V4 models and meet the gesture recognition requirements of display devices.

[0004] In a first aspect, embodiments of this application provide a display device, including:

[0005] An accelerator processor is configured to acquire an image to be identified and perform the following processing on the image to be identified using a neural network model:

[0006] A hand image is obtained from the image to be identified. The hand image is then subjected to multiple consecutive high-dimensional space acquisitions and downsampling processes to obtain a hand feature map. This hand feature map includes hand feature information at different dimensions and resolutions. Multi-scale max pooling is performed on the hand feature map, and the resulting feature maps are then fused to obtain a first multi-scale feature map. This first multi-scale feature map includes spatial information from multiple scales of the hand image. Based on the first multi-scale feature map, multiple prediction boxes are obtained, each corresponding to a specific gesture category probability. These multiple prediction boxes are obtained by segmenting the image to be identified.

[0007] The central processing unit is configured to identify hand gestures in the image to be identified based on the probability of the gesture category corresponding to the plurality of prediction boxes.

[0008] Based on the above solution, this application proposes a neural network model deployed on an accelerator processor. By repeatedly extracting features in high-dimensional space, performing downsampling, and max pooling, it can acquire hand feature information at different dimensions, resolutions, and scales. The algorithm for extracting features in high-dimensional space, performing downsampling, and max pooling is simple and consumes fewer processing resources, thus making it suitable for accelerator processors with limited processing resources. Furthermore, compared to existing methods that use processors for gesture recognition, this approach improves efficiency and reduces resource consumption.

[0009] In some embodiments, the neural network model includes multiple network layers, each network layer including one or more cascaded identity mapping network layers and downsampling network layers;

[0010] The first network layer is an identity mapping network layer, used to obtain hand features in high-dimensional space based on the input image;

[0011] The downsampling network layer of the first network layer is used to downsample the hand features obtained in the high-dimensional space to obtain a hand feature map while reducing the resolution of the image input to the downsampling network layer, and input it into the identity mapping network layer of the second network layer.

[0012] The first network layer and the second network layer are any two adjacent network layers among the multiple network layers included in the neural network model.

[0013] Based on the above scheme, this application proposes to process the input hand image by iterating through multiple identity mapping network layers and downsampling network layers, which can better obtain the hand features in the hand image and improve the accuracy of gesture recognition.

[0014] In some embodiments, the identity mapping network layer includes a first convolutional linear correction part, a first depthwise separable convolutional correction part, and a first convolutional part; the input of the first convolutional linear correction part is skip-connected to the first convolutional part.

[0015] The first convolutional linear correction part consists of a first convolutional layer and a first correction linear unit (ReLU), and the first depth-separable convolutional correction part consists of a first depth-separable convolutional layer and a second correction linear unit (ReLU). The first convolutional part includes a second convolutional layer.

[0016] Based on the above scheme, the identity mapping network layer in the neural network only contains simple operations such as simple convolution operations, depthwise separable convolution operations, and ReLU correction linear units. Accelerators can support these simple operations, so they can run the neural network model proposed in this application very well.

[0017] In some embodiments, the downsampling network layer includes:

[0018] The second convolution linear correction part, the second depthwise separable convolution correction part, and the second convolution part;

[0019] The second convolutional linear correction part consists of a third convolutional layer and a third correction linear unit (ReLU), and the second depthwise separable convolutional correction part consists of a second depthwise separable convolutional layer and a fourth correction linear unit (ReLU). The second convolutional part includes a fourth convolutional layer.

[0020] Based on the above scheme, the downsampling network layer in the neural network only contains simple operations such as simple convolution operations, depthwise separable convolution operations, and ReLU correction linear units. The accelerator processor can support these simple operations, so the accelerator processor can run the neural network model proposed in this application very well.

[0021] In some embodiments, the last network layer of the neural network model further includes a direct sampling network layer;

[0022] The direct sampling network layer is used to adjust the number of feature channels of the image input to the direct sampling network layer without changing the resolution of the image input to the direct sampling network layer.

[0023] Based on the above scheme, the neural network model proposed in this application also includes a direct sampling network layer, which adjusts the number of feature channels of the image while keeping the resolution of the hand image unchanged, including obtaining more hand features.

[0024] In some embodiments, the direct sampling network layer includes:

[0025] The third convolutional linear correction part, the third depthwise separable convolutional correction part, and the third convolutional part;

[0026] The third convolutional linear correction part consists of a fifth convolutional layer and a fifth correction linear unit (ReLU). The third depthwise separable convolutional correction part consists of a third depthwise separable convolutional layer and a sixth correction linear unit (ReLU). The third convolutional part includes a sixth convolutional layer.

[0027] Based on the above scheme, the direct sampling network layer proposed in this application contains simple operations, and the accelerated processor can execute these simple operations, so that the neural network model containing the direct sampling network layer can be deployed in the accelerated processor.

[0028] In some embodiments, the neural network model further includes a seventh convolutional layer, a deconvolutional layer, and a first concatenation layer;

[0029] The seventh convolutional layer is used to adjust the number of feature channels in the image output by the last network layer; the adjusted number of feature channels is the same as the number of feature channels in the first multi-scale feature map.

[0030] The deconvolutional layer is used to adjust the resolution of the first multi-scale feature map; the resolution of the first multi-scale feature map after adjustment is the same as the resolution of the hand feature map output by the seventh convolutional layer.

[0031] The first stitching processing layer is used to stitch the first multi-scale feature map after resolution adjustment with the hand feature map output by the seventh convolutional layer to obtain a second multi-scale feature map, which includes spatial information of multiple scales of the hand image.

[0032] In some embodiments, the neural network model includes a pyramid pooling layer; the pyramid pooling layer includes a max pooling portion and a second splicing processing layer;

[0033] The max pooling part consists of multiple max pooling layers of different scales, which are used to perform multi-scale max pooling processing on the hand feature map input to the pyramid pooling layer.

[0034] The second splicing processing layer is used to splice the feature maps after multi-scale max pooling output by the max pooling part to obtain the first multi-scale feature map.

[0035] Based on the above scheme, the neural network model proposed in this application includes a pyramid pooling layer. It uses max pooling layers of different scales to perform multi-scale max pooling processing on the input hand feature map, and then splices the feature maps after multi-scale max pooling processing. The resulting first multi-scale feature map can effectively reduce the overfitting problem and improve the recognition accuracy.

[0036] In some embodiments, the display device further includes a display screen;

[0037] The display screen is used to display the image to be identified.

[0038] Secondly, embodiments of this application propose a gesture recognition method, including:

[0039] Obtain the image to be recognized;

[0040] A hand image is obtained from the image to be identified using a neural network model;

[0041] The neural network model obtains a hand feature map by repeatedly acquiring hand features in high-dimensional space and downsampling the hand features from the hand image. The hand feature map includes hand feature information in the hand image at different dimensions and resolutions.

[0042] The neural network model is used to perform multi-scale max pooling on the hand feature map, and the feature maps after multi-scale max pooling are fused to obtain a first multi-scale feature map, which includes spatial information of the hand image at multiple scales.

[0043] The neural network model obtains the gesture category probabilities corresponding to multiple prediction boxes based on the first multi-scale feature map; wherein, the multiple prediction boxes are obtained by segmenting the image to be recognized;

[0044] The hand gestures in the image to be identified are determined based on the probability of the gesture category corresponding to each of the multiple prediction boxes.

[0045] In some embodiments, the neural network model includes multiple network layers, each network layer including one or more cascaded identity mapping network layers and downsampling network layers;

[0046] The first network layer is an identity mapping network layer, used to obtain hand features in high-dimensional space based on the input image;

[0047] The downsampling network layer of the first network layer is used to downsample the hand features obtained in the high-dimensional space to obtain a hand feature map while reducing the resolution of the image input to the downsampling network layer, and input it into the identity mapping network layer of the second network layer.

[0048] The first network layer and the second network layer are any two adjacent network layers among the multiple network layers included in the neural network model.

[0049] In some embodiments, the identity mapping network layer includes a first convolutional linear correction part, a first depthwise separable convolutional correction part, and a first convolutional part; the input of the first convolutional linear correction part is skip-connected to the first convolutional part.

[0050] The first convolutional linear correction part consists of a first convolutional layer and a first correction linear unit (ReLU), and the first depth-separable convolutional correction part consists of a first depth-separable convolutional layer and a second correction linear unit (ReLU). The first convolutional part includes a second convolutional layer.

[0051] In some embodiments, the downsampling network layer includes:

[0052] The second convolution linear correction part, the second depthwise separable convolution correction part, and the second convolution part;

[0053] The second convolutional linear correction part consists of a third convolutional layer and a third correction linear unit (ReLU), and the second depthwise separable convolutional correction part consists of a second depthwise separable convolutional layer and a fourth correction linear unit (ReLU). The second convolutional part includes a fourth convolutional layer.

[0054] In some embodiments, the last network layer of the neural network model further includes a direct sampling network layer;

[0055] The direct sampling network layer is used to adjust the number of feature channels of the image input to the direct sampling network layer without changing the resolution of the image input to the direct sampling network layer.

[0056] In some embodiments, the direct sampling network layer includes:

[0057] The third convolutional linear correction part, the third depthwise separable convolutional correction part, and the third convolutional part;

[0058] The third convolutional linear correction part consists of a fifth convolutional layer and a fifth correction linear unit (ReLU). The third depthwise separable convolutional correction part consists of a third depthwise separable convolutional layer and a sixth correction linear unit (ReLU). The third convolutional part includes a sixth convolutional layer.

[0059] In some embodiments, the neural network model further includes a seventh convolutional layer, a deconvolutional layer, and a first concatenation layer;

[0060] The seventh convolutional layer is used to adjust the number of feature channels in the image output by the last network layer; the adjusted number of feature channels is the same as the number of feature channels in the first multi-scale feature map.

[0061] The deconvolutional layer is used to adjust the resolution of the first multi-scale feature map; the resolution of the first multi-scale feature map after adjustment is the same as the resolution of the hand feature map output by the seventh convolutional layer.

[0062] The first stitching processing layer is used to stitch the first multi-scale feature map after resolution adjustment with the hand feature map output by the seventh convolutional layer to obtain a second multi-scale feature map, which includes spatial information of multiple scales of the hand image.

[0063] In some embodiments, the neural network model includes a pyramid pooling layer; the pyramid pooling layer includes a max pooling portion and a second splicing processing layer;

[0064] The max pooling part consists of multiple max pooling layers of different scales, which are used to perform multi-scale max pooling processing on the hand feature map input to the pyramid pooling layer.

[0065] The second splicing processing layer is used to splice the feature maps after multi-scale max pooling output by the max pooling part to obtain the first multi-scale feature map.

[0066] In some embodiments, the method further includes:

[0067] The image to be identified is displayed.

[0068] Thirdly, embodiments of this application propose a gesture recognition device, including:

[0069] The first processing unit is configured to acquire an image to be identified and perform the following processing on the image to be identified using a neural network model:

[0070] A hand image is obtained from the image to be identified. The hand image is then subjected to multiple consecutive high-dimensional space acquisitions and downsampling processes to obtain a hand feature map. This hand feature map includes hand feature information at different dimensions and resolutions. Multi-scale max pooling is performed on the hand feature map, and the resulting feature maps are then fused to obtain a first multi-scale feature map. This first multi-scale feature map includes spatial information from multiple scales of the hand image. Based on the first multi-scale feature map, multiple prediction boxes are obtained, each corresponding to a specific gesture category probability. These multiple prediction boxes are obtained by segmenting the image to be identified.

[0071] The second processing unit is configured to identify the hand gestures in the image to be identified based on the probability of the gesture categories corresponding to the plurality of prediction boxes.

[0072] Fourthly, embodiments of this application also provide a computer storage medium storing computer program instructions that, when executed on a computer, cause the computer to perform the method described in the second aspect.

[0073] The technical effects of any of the implementation methods in the second to fourth aspects can be found in the technical effects of the corresponding implementation method in the first aspect, and will not be repeated here. Attached Figure Description

[0074] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1A A hardware configuration block diagram of a display device provided in an embodiment of this application;

[0076] Figure 1B A software structure block diagram of a display device provided in an embodiment of this application;

[0077] Figure 2A This is a schematic flowchart of a gesture recognition method provided in an embodiment of this application;

[0078] Figure 2B A schematic diagram of a gesture recognition process provided in an embodiment of this application;

[0079] Figure 3 A schematic diagram of a display interface for displaying an image to be recognized, provided as an embodiment of this application;

[0080] Figure 4 A schematic diagram of a display interface for displaying gesture recognition results provided in an embodiment of this application;

[0081] Figure 5 A schematic diagram of another display interface for displaying gesture recognition results provided in an embodiment of this application;

[0082] Figure 6A A schematic diagram of a gesture recognition process provided in an embodiment of this application;

[0083] Figure 6B A schematic diagram illustrating the processing flow of a hand feature map by an identity mapping network layer, provided in an embodiment of this application;

[0084] Figure 6C A schematic diagram illustrating the processing flow of a downsampling network layer for hand feature maps provided in an embodiment of this application;

[0085] Figure 7A A schematic diagram illustrating another gesture recognition process provided in an embodiment of this application;

[0086] Figure 7B A schematic diagram illustrating the processing flow of a direct sampling network layer for hand feature maps, provided in an embodiment of this application;

[0087] Figure 8 This is a schematic diagram of the structure of a display device provided in an embodiment of this application;

[0088] Figure 9 This is a schematic diagram of the structure of a gesture recognition device provided in an embodiment of this application. Detailed Implementation

[0089] To facilitate understanding of the technical solutions provided in the embodiments of this application, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0090] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be interchanged where appropriate so that embodiments of this application can be implemented in a sequence other than that shown in the illustrations or descriptions of this application. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0091] Currently, the accelerators configured in display devices have limited computing power and cannot run complex gesture recognition algorithms. Therefore, the gesture recognition process is still performed by the display device's central processing unit or image processor, which not only wastes the resources of the central processing unit or image processor but also leads to low efficiency in the gesture recognition process.

[0092] This application proposes a gesture recognition method and apparatus, and a neural network model that can be deployed on an accelerated processor. By repeatedly extracting features in high-dimensional space, performing downsampling, and max pooling, it can acquire hand feature information at different dimensions, resolutions, and scales. The algorithm, which involves feature extraction in high-dimensional space, downsampling, and max pooling, is simple and consumes fewer processing resources, thus making it suitable for accelerated processors with limited processing resources. Furthermore, compared to existing methods that use processors for gesture recognition, this method improves efficiency and reduces resource consumption.

[0093] Below, to facilitate understanding of the solutions proposed in the embodiments of this application, the solutions will be described in conjunction with different embodiments. It should be noted that the solutions proposed in this application can be applied to display devices. The display devices involved in this application can have a camera for capturing images including hands, and can also have a transceiver for receiving images including hands transmitted from other devices. They can also include a display screen for displaying images including hands captured by the camera or received by the transceiver, such as smartphones, tablets, laptops, PDAs, smart TVs, mobile internet devices (MIDs), wearable devices, etc. Wearable devices can include, for example, smartwatches or smart glasses, smart bracelets, etc.

[0094] As an example, the structure of the display device involved in this application is described in detail below. See [link to documentation]. Figure 1A The diagram shown is a schematic of a possible hardware configuration for a display device 200. In some embodiments, the display device includes at least one of a tuner / demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display component 260, an audio output interface 270, a memory, a power supply, and a user interface 280.

[0095] In some embodiments, the display component 260 includes a display screen assembly for presenting images, a driving component for driving image display, a component for receiving image signals output from a processor, and a user interface (UI) for displaying video content, image content, menu control interface, etc.

[0096] In some embodiments, the tuner 210 receives broadcast television signals via wired or wireless means and demodulates audio and video signals from multiple wireless or wired broadcast television signals. In some embodiments, the communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wireless Fidelity (Wi-Fi) component, a Bluetooth component, a wired Ethernet component, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The Wi-Fi component can be used, under the control of the controller 250, to broadcast detection frames and receive response frames or beacon frames from wireless hotspots. The display device 200 can transmit data with the target health detection device or the peer device 300 via the communicator 220.

[0097] In some embodiments, detector 230 is used to acquire signals of the external environment or interaction with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes or user interaction gestures, or to acquire images containing the user's hands.

[0098] In some embodiments, the external device interface 240 may include, but is not limited to, any one or more interfaces such as: High Definition Multimedia Interface (HDMI), Analog or Data High Definition Component Input Interface (Component), Composite Video Input Interface (CVBS), USB Input Interface (USB), RGB Port, Camera Interface, etc.

[0099] In some embodiments, the processor 250 and the tuner / demodulator 210 may be located in separate devices.

[0100] In some embodiments, the processor 250 includes at least one of a central processing unit (CPU), a video processor, an accelerator processor, an audio processor, a graphics processing unit (GPU), RAM (Random Access Memory), ROM (Read-Only Memory), a first to an nth interface for input / output, a communication bus, etc.

[0101] In some embodiments, the CPU is used to execute operating system and application instructions stored in memory, as well as various applications, data, and content based on various interactive instructions received from external input.

[0102] In some embodiments, a graphics processing unit (GPU) is used to generate various graphics objects. The GPU includes an arithmetic logic unit (ALU) that performs calculations based on various interactive instructions received from the user and displays various objects according to display attributes; it also includes a renderer that renders the various objects obtained from the ALU, and the rendered objects are used to display on a display component.

[0103] In some embodiments, a video processor is configured to receive external video signals and perform at least one of the following video processing operations according to a standard encoding / decoding protocol of the input signals: decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, image compositing, etc.

[0104] In some embodiments, the accelerator processor can be used to run deep learning recognition algorithms, perform AI operations, and implement AI applications.

[0105] In some embodiments, the video processor includes at least one of a demultiplexing module, a video decoding module, an image compositing module, a frame rate conversion module, and a display formatting module.

[0106] In some embodiments, an audio processor is configured to receive external audio signals, decompress and decode them according to a standard codec protocol of the input signals to obtain sound signals that can be played in a speaker.

[0107] In some embodiments, the user can input user commands through a graphical user interface (GUI) displayed on the display component 260, and the user input interface receives the user input commands through the graphical user interface (GUI). Alternatively, the user can input user commands by inputting specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0108] In some embodiments, the user interface 280 is an interface that can be used to receive control input (e.g., physical buttons on the peer device body, or others).

[0109] In some embodiments, the peer device's system may include a kernel, a command interpreter (shell), a file system, and applications. The kernel, shell, and file system together form the basic operating system structure, allowing users to manage files, run programs, and use the system. Upon power-up, the kernel starts, activates the kernel space, abstracts hardware, initializes hardware parameters, and runs and maintains virtual memory, the scheduler, signals, and inter-process communication (IPC). After the kernel starts, the shell and user applications are loaded. Applications are compiled into machine code after startup, forming a process.

[0110] In addition, this application also provides a software configuration block diagram of the Android system display device 200, as follows: Figure 1BAs shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (hereinafter referred to as the "Application Layer"), the Application Framework layer (hereinafter referred to as the "Framework Layer"), the Android Runtime and System Library layer (hereinafter referred to as the "System Runtime Layer"), and the kernel layer.

[0111] In some embodiments, at least one application runs in the application layer. These applications may be Windows programs, system settings programs, or clock programs that come with the operating system; they may also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0112] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions that applications within the application layer should take. Applications can access system resources and obtain system services during execution through the API interface.

[0113] In some embodiments, the Activity Manager is used to manage the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and navigating back. The Window Manager is used to manage all window programs, such as obtaining the screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling display window changes (e.g., shrinking the display window, shaking the display, distorting the display, etc.).

[0114] In some embodiments, the system runtime library layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system runs the C / C++ libraries contained in the system runtime library layer to implement the functions that the framework layer needs to perform.

[0115] In some embodiments, such as Figure 1B As shown, the kernel layer is the layer between hardware and software.

[0116] Different terminals have different hardware configurations and software structures, therefore the above... Figure 1A and Figure 1B These are all illustrative examples.

[0117] To reduce CPU and GPU usage and improve gesture recognition efficiency, this application proposes a gesture recognition method. The gesture recognition method proposed in this application will be described in detail below with reference to different embodiments.

[0118] First, see Figure 2AThe diagram shows a flowchart of the gesture recognition method proposed in this application. It should be noted that the gesture recognition method proposed in this application is applied to a display device, which includes an accelerometer processor. The method specifically includes:

[0119] 201, The display device acquires the image to be recognized.

[0120] In some embodiments, the image to be identified can be captured by a camera on the display device. The image to be identified can also be received by a transceiver on the display device. Receiving the image to be identified via a transceiver can include receiving the image via wired transmission or wireless transmission; this application does not specifically limit this. It should be noted that the image to be identified is an image containing the user's hand.

[0121] 202, The display device acquires a hand image from the image to be recognized.

[0122] In some embodiments, the display device may acquire a hand image from an image to be identified through one or more convolutional layers included in a neural network model.

[0123] 203, The display device performs multiple consecutive acquisitions of hand features in high-dimensional space and downsampling of hand features to obtain a hand feature map.

[0124] The hand feature map includes hand feature information from hand images at different dimensions and resolutions. For example, hand feature information can include color features, shape features, position features, and so on.

[0125] 204. The display device performs multi-scale max pooling on the hand feature map and fuses the multi-scale max pooled feature maps to obtain the first multi-scale feature map.

[0126] The first multi-scale feature map includes spatial information of the hand image at multiple scales. For example, the spatial information may include the position coordinates of the hand in the image, the size of the area occupied by the hand in the image, the hand outline, the hand color, the hand texture information, etc.

[0127] 205, The display device obtains the probability of the gesture category corresponding to each of the multiple prediction boxes based on the first multi-scale feature map.

[0128] The multiple predicted bounding boxes are obtained by segmenting the image to be recognized. The image is segmented to obtain multiple predicted bounding boxes, and then the probability of each predicted bounding box corresponding to a gesture category is obtained based on the first multi-scale feature map. For example, multiple gesture categories can be pre-configured in the display device; for instance, 12 gesture categories can be configured (e.g., including mute, thumbs up, heart, fist, OK, one, two, three, four, five, six, and eight). Then, the probability of the predicted bounding box for each of these 12 categories is determined based on the first multi-scale feature map.

[0129] 206. The display device identifies the hand gesture in the image to be identified based on the probability of the gesture category corresponding to the plurality of prediction boxes.

[0130] Multiple predicted bounding boxes correspond to different probability categories of gestures. The display device selects the predicted bounding box with the highest probability. Based on the gesture category with the highest probability corresponding to that predicted bounding box, the device determines the hand gesture in the image to be recognized. For example, the display device can use a non-maximum suppression (NMS) algorithm to select the predicted bounding box with the highest probability from multiple predicted bounding boxes.

[0131] Based on the above solution, compared to performing gesture recognition through a central processing unit or graphics processing unit, using an accelerator processor to perform the gesture recognition process takes less time, thus improving the efficiency of gesture recognition. For a clearer understanding of the solution proposed in the embodiments of this application, please refer to... Figure 2B This paper describes the gesture recognition process proposed in this application, taking into account the specific structure of the display device. The display device proposed in this application includes an input / output terminal, an accelerometer, and a central processing unit. The gesture recognition process includes:

[0132] 210, The input end obtains the image to be identified, which contains the hand.

[0133] For example, the image to be identified can be obtained through a camera or transceiver.

[0134] 220. The input terminal can transmit the image to be recognized to the accelerator processor through the internal interface of the display device.

[0135] For example, the input terminal can transfer the image to be recognized to the display device's memory via a Digital Camera Interface (DCMI), General-purpose Input / Output (GPIO), or Universal Serial Bus Video Class (UVC). This allows the accelerator processor to retrieve the image from memory.

[0136] 230, the accelerator processor performs gesture recognition on the image to be recognized through a neural network model.

[0137] For details, please refer to [link / reference]. Figure 2A Steps 202-205 in the previous section will not be repeated here.

[0138] 240. The accelerator processor sends the recognition results to the CPU through the internal interface of the display device.

[0139] The recognition results may include multiple predicted bounding boxes obtained by the accelerator processor through the neural network model for gesture recognition of the image to be recognized, as well as the probability of the gesture category corresponding to each of the multiple predicted bounding boxes.

[0140] 250. The CPU determines the highest probability prediction box and the gesture category in the image to be recognized based on the recognition results.

[0141] For details, please refer to [link / reference]. Figure 2A Step 206 in the process will not be repeated here.

[0142] 260. The CPU transmits the predicted bounding box with the highest probability and the gesture category to the output.

[0143] Optionally, the output can display the prediction box with the highest probability determined by the CPU on the screen.

[0144] To further understand the solution proposed in the embodiments of this application, the solution will be described below in conjunction with specific scenarios. As mentioned above, a display device can capture an image to be recognized via a camera or receive an image to be recognized via a transceiver. For ease of explanation, the following description will use the example of a display device capturing an image to be recognized via a camera. In some embodiments, the gesture recognition method proposed in this application may include:

[0145] The display device's camera captures an image to be recognized, which includes the user's hand. Optionally, after acquiring the image, the display device can display it on the screen; for example, see [reference needed]. Figure 3 The displayed interface is shown below. Figure 3 The image displayed in the shown interface includes the user's gesture as "five". Further, after acquiring the image to be recognized, the camera can transmit the image to the accelerator processor via the internal interface of the display device. The accelerator processor deploys a neural network model for gesture recognition of the image to be recognized. Optionally, the neural network model may include one or more convolutional layers. The accelerator processor acquires the hand image from the image to be recognized through one or more convolutional layers included in the neural network model.

[0146] Furthermore, the neural network model may also include multiple network layers, each of which includes one or more cascaded identity mapping network layers and downsampling network layers. Network layers are merely a logical division of the number of processing layers and do not constitute a specific limitation on this application. Network layers may also be named using other names, and this application does not impose any specific limitations on this.

[0147] Taking the identity mapping network layer and downsampling network layer of the first network level as an example: The identity mapping network layer of the first network level is used to obtain hand features in high-dimensional space based on the input hand image; the downsampling network layer of the first network level is used to downsample the hand features obtained in high-dimensional space while reducing the resolution of the hand image input to the downsampling network layer to obtain a hand feature map, which is then input into the identity mapping network layer of the second network level. It should be noted that the first network level and the second network level are any two adjacent network layers in the multiple network layers included in the neural network model. The accelerator processor can process the hand image through the multiple network layers included in the neural network model to obtain the hand feature map, which includes hand feature information of the hand image at different dimensions and resolutions.

[0148] Optionally, the last network layer in the neural network model may also include a direct sampling network layer. The accelerator processor can adjust the number of feature channels of the hand feature map input to the direct sampling network layer without changing the resolution of the hand feature map through the direct sampling network layer of the neural network model.

[0149] Furthermore, the neural network model can also include a pyramid pooling layer. The accelerator performs multi-scale max pooling on the hand feature map input to the pyramid pooling layer, and then fuses the feature maps after multi-scale max pooling to obtain a first multi-scale feature map. The first multi-scale feature map includes spatial information of the hand image at multiple scales. Further, after obtaining the first multi-scale feature map, the accelerator can obtain the probability of the gesture category corresponding to multiple prediction boxes based on the first multi-scale feature map. These multiple prediction boxes are obtained by segmenting the image to be recognized. As an example, the accelerator can pre-configure multiple gesture categories, for example, 10 gesture categories. Assuming that 100 prediction boxes are obtained after segmenting the image to be recognized, the accelerator predicts the probability of each of these 100 prediction boxes for the aforementioned 10 gesture categories. For example, the probability of the first prediction box for the first gesture category is a%, the probability of the first prediction box for the second gesture category is b%, and so on, obtaining the probability of each of the 100 prediction boxes for the 10 gesture categories. Optionally, the accelerator can also generate a confidence score for each prediction box before predicting its probability. The confidence score indicates whether the prediction box contains a hand. Optionally, a higher confidence score indicates a higher probability that the prediction box contains a hand. Of course, a lower confidence score also indicates a higher probability that the prediction box contains a hand.

[0150] In one possible example, after obtaining the gesture category probabilities corresponding to multiple predicted bounding boxes, the accelerator can output the predicted bounding boxes and their corresponding gesture category probabilities to the central processing unit (CPU). For example, the CPU can use a non-maximum suppression (NMS) algorithm to determine the predicted bounding box with the highest probability from the multiple predicted bounding boxes, and then determine the gesture category of the hand in the image to be recognized based on the predicted bounding box with the highest probability. In another possible example, the CPU can first exclude some predicted bounding boxes that do not contain hands based on confidence levels, and then determine the gesture category based on the predicted bounding box with the highest probability. For example, if the image to be recognized is divided into 100 predicted bounding boxes, and each predicted bounding box corresponds to a probability for 10 gesture categories, then the CPU receives a total of 1000 probabilities. The CPU determines the gesture category based on the highest probability among these 1000 probabilities. In some embodiments, after the CPU determines the gesture category, it can also display the corresponding predicted bounding box on the display screen as the recognition result. For example, see [link to relevant documentation]. Figure 4 The display interface shown is shown below. Alternatively, after the CPU determines the gesture category, it can also display the corresponding prediction box and the predicted gesture category on the display interface as the recognition result. For example, see [link to relevant documentation]. Figure 5The display interface shown is shown. In some other embodiments, the CPU may also be configured with operation instructions corresponding to different gesture categories. For example, if the operation instruction corresponding to gesture category 1 is a mute instruction, then after determining that the gesture category is gesture category 1, the CPU will execute the corresponding mute operation.

[0151] The above describes the gesture recognition method proposed in this application in conjunction with practical scenarios. To further understand the solution of this application, the gesture recognition solution proposed in this application will be described below with reference to some specific embodiments.

[0152] After the camera of the display device captures the image to be recognized, it transmits the image to the accelerator processor of the display device. The accelerator processor is equipped with a neural network model, which completes the gesture recognition process of the image to be recognized. In some embodiments, the gesture recognition process performed by the accelerator processor can be found in [reference needed]. Figure 6A The flowchart shown specifically includes:

[0153] A1, Input Image: Input the image to be recognized into the gesture recognition model. The image includes the user's hand. Optionally, the input image may contain color features. For example, the resolution of the input image may be 288*512, and the number of features may be 3, which can be the red, green, and blue (RGB) features of the input image.

[0154] A2, Convolutional Operation to Extract Hand Image: The neural network model may contain one or more convolutional layers for extracting hand images from the image to be identified. For example, the hand image is extracted from the image input in step A1 with a resolution of 255*512 and 3 features, based on the convolutional operation in step A2. The extracted hand image can have a resolution of 144*256 and 16 features.

[0155] A3, Identity Mapping: Neural network models can include identity mapping network layers to extract hand features in a high-dimensional space based on an input hand image.

[0156] In some embodiments, the identity mapping network layer may include a first convolutional linear correction part, a first depthwise separable convolutional correction part, and a first convolutional part, wherein the input of the first convolutional linear correction part is skip-connected to the output of the first convolutional part. The skip connection refers to adding hand features from the hand image input to the identity mapping network layer to the hand image output by the first convolutional part, such as adding information about the hand's color, contour, texture, and contrast. The neural network model determines whether the addition result affects gesture recognition; if it has no effect, the addition result is deleted; if it does, the addition result is retained. Optionally, the first convolutional linear correction part may consist of a first convolutional layer and a first corrected linear unit (ReLU), the first depthwise separable convolutional correction part may consist of a first depthwise separable convolutional layer and a second corrected linear unit (ReLU), and the first convolutional part may include a second convolutional layer.

[0157] As an example, see Figure 6B The illustrated identity mapping network layer processes the input hand image in four steps: First, the features are upscaled using a first convolutional layer (e.g., by a factor of 6), and then linearly corrected using a first Rectified Linear Unit (ReLU). Second, features are extracted in the high-dimensional space using a first depthwise separable convolutional layer, and then linearly corrected using a second Rectified Linear Unit (ReLU). Third, the features are downscaled using a second convolutional layer. Fourth, a skip connection operation is performed between the high-dimensional and low-dimensional features, that is, a skip connection operation is performed between the detailed information in the high-dimensional features and the low-dimensional hand features.

[0158] Optionally, in this step, the hand image can be processed by the identity mapping network layer once or multiple times. This application does not specifically limit the number of times the identity mapping network layer processing is performed.

[0159] For example, following the example of step A2, step A3 can obtain a hand feature map with a resolution of 144*256 and a number of features of the hand image of 16 through one or more identity mapping network layers, and then transmit the hand feature map to step A4.

[0160] A4, Downsampling: A neural network model can include a downsampling network layer to reduce the resolution of the feature map and to obtain a hand feature map by downsampling the hand features obtained in high-dimensional space.

[0161] In some embodiments, the downsampling network layer may include a second convolutional linear correction portion, a second depthwise separable convolutional correction portion, and a second convolutional portion. Optionally, the second convolutional linear correction portion may consist of a third convolutional layer and a third corrected linear unit (ReLU), the second depthwise separable convolutional correction portion may consist of a second depthwise separable convolutional layer and a fourth corrected linear unit (ReLU), and the second convolutional portion may include a fourth convolutional layer.

[0162] As an example, see Figure 6C The downsampling network layer shown can process the input hand image in three steps: First, the features are upscaled using a third convolutional layer, for example, by a factor of 6, and then linearly corrected in the high-dimensional space using a third Rectified Linear Unit (ReLU). Second, features are extracted from the hand feature map corrected by the ReLU in the high-dimensional space using a second depthwise separable convolutional layer. For example, the stride of the second depthwise separable convolutional layer can be 2. That is, hand features are extracted in the high-dimensional space using a second depthwise separable convolutional layer with a stride of 2. The extracted hand features are then linearly corrected using a fourth Rectified Linear Unit (ReLU). Third, the features are downscaled using a fourth convolutional layer. For example, the processed hand feature map obtained after processing the input hand feature map by the downsampling network layer can have a resolution of 72*128 and a feature count of 24. The processed hand feature map is then transferred to step A5.

[0163] A5, Identity Mapping: See step A3 for details, which will not be repeated here. Optionally, in this step, the hand image may be processed by the identity mapping network layer once or multiple times. This application does not specify the number of times the identity mapping network layer is processed.

[0164] For example, following the example of step A4, the resolution of the hand feature map processed by the identity mapping network layer once or multiple times in step A5 can be 72*128 and the number of features can be 24, and the processed hand feature map is then transmitted to step A6.

[0165] A6, Downsampling: Refer to step A4, which will not be repeated here. For example, continuing from the example in step A5 above, the hand feature map obtained after processing by the downsampling network layer in step A6 can have a resolution of 36*64 and a feature number of 32, and the processed hand feature map is then transmitted to step A7.

[0166] A7, Identity Mapping: Refer to step A3, which will not be repeated here. Optionally, in this step, the hand image can be processed by the identity mapping network layer once or multiple times. This application does not specify the number of times the identity mapping network layer processing is performed. For example, continuing from the example in step A6 above, the hand feature map obtained after one or more identity mapping network layer processing in step A7 can have a resolution of 36*64 and a feature number of 32, and the processed hand feature map is then transmitted to step A8.

[0167] A8, Downsampling: Refer to step A4, which will not be repeated here. For example, continuing from the example in step A7 above, the hand feature map obtained after processing by the downsampling network layer in step A8 can have a resolution of 18*32 and a feature number of 64, and the processed hand feature map is then transmitted to step A9.

[0168] A9, Identity Mapping: Refer to step A3, which will not be repeated here. Optionally, in this step, the hand image can be processed by the identity mapping network layer once or multiple times. This application does not specify the number of times the identity mapping network layer processing is performed. For example, continuing the example in step A8 above, the hand feature map obtained after one or more identity mapping network layer processing in step A9 can have a resolution of 18*32 and a feature number of 64, and the processed hand feature map is then transmitted to step A10.

[0169] A10, Downsampling: Refer to step A4, which will not be repeated here. For example, continuing the example in step A9 above, the hand feature map obtained after processing by the downsampling network layer in step A10 can have a resolution of 9*16 and a feature number of 160, and the processed hand feature map is then transmitted to step A11.

[0170] A11, Hand Feature Fusion in Pyramid Pooling Layer: The neural network model can also include a pyramid pooling layer to perform multi-scale max pooling on the input hand feature map, and then fuse the resulting feature maps to obtain a first multi-scale feature map. This first multi-scale feature map includes spatial information from multiple scales of the hand image. As an example, the pyramid pooling layer can employ max pooling operations at multiple scales, and then concatenate the feature maps output from each max pooling operation with the input feature map. For example, it can use max pooling operations at three scales: 6x6, 4x4, or 2x2, outputting three feature maps respectively. These three output feature maps are then concatenated with the input feature map, resulting in a feature map that fuses multi-scale features. This effectively reduces overfitting.

[0171] In some embodiments, the pyramid pooling layer includes a max pooling portion and a first concatenation layer. Optionally, the max pooling portion may consist of multiple max pooling layers of different scales, for example, it may include a 6*6 max pooling layer, a 4*4 max pooling layer, and a 2*2 max pooling layer, used to perform multi-scale max pooling processing on the hand feature map input to the pyramid pooling layer; the first concatenation layer is used to concatenate the multi-scale max pooled feature map output by the max pooling portion with the feature map input to the pyramid pooling layer to obtain a first multi-scale feature map.

[0172] For example, continuing with the example in step A11 above, the resolution of the first multi-scale feature map obtained after pyramid pooling layer processing can be 9*16 and the number of features can be 640.

[0173] A12, Prediction Part: The neural network model further includes a prediction part, used to obtain the gesture category probabilities corresponding to multiple prediction boxes based on the first multi-scale feature map.

[0174] Among them, multiple predicted bounding boxes are obtained by dividing the image to be recognized.

[0175] As an example, the accelerator can be pre-configured with multiple gesture categories, for instance, setting 12 gesture categories (which may include mute, thumbs up, heart shape, fist, OK, one, two, three, four, five, six, and eight). The probability of each prediction box for these 12 gesture categories is determined based on the first multi-scale feature map.

[0176] Finally, the accelerator can output the acquired multiple prediction boxes and the probability of each prediction box for multiple gesture categories to the CPU. The CPU can then use the NMS algorithm to determine the prediction box with the highest probability from the multiple prediction boxes, and determine the gesture category of the hand in the image to be recognized based on the prediction box with the highest probability. Based on the above scheme, since the NMS algorithm is complex, having the CPU determine the prediction box with the highest probability using the NMS algorithm can shorten the gesture recognition time and improve efficiency.

[0177] Existing accelerators cannot support operations such as division, fully connected operations, mean calculation, matrix completion, matrix dimension transformation, softmax activation function for logistic regression models, and channel splitting. However, the YOLO-V4 model currently used for gesture recognition includes splitting operations, making it unsuitable for deployment on accelerators. The neural network model proposed in this application only includes basic operations such as convolution, depthwise separable convolution, deconvolution, and max pooling. Accelerators can support these simple operations, allowing them to run the proposed neural network model effectively. Furthermore, since the YOLO-V4 model cannot be deployed on accelerators and requires CPU, while the proposed neural network model does not, CPU utilization is reduced from 32%-35% to 8%-14%, saving CPU resources. In addition, compared with the existing YOLO-V4 model run by the CPU, the neural network model run by the accelerated processor in this application can shorten the gesture recognition time to 26-29ms, thus improving the efficiency of gesture recognition.

[0178] It should be noted that the above Figure 6A The illustrated process is merely an example; in other embodiments, the process of accelerating the processor to perform gesture recognition can also be found in [reference needed]. Figure 7A Specifically, it includes:

[0179] Figure 7A For reference, see B1-B9. Figure 6A A1-A9 in the table will not be elaborated upon here.

[0180] B10, Direct Sampling: The neural network model may also include a direct sampling network layer, which is used to increase the number of feature channels of the image input to the direct sampling network layer without changing the resolution of the image input to the direct sampling network layer.

[0181] In some embodiments, the direct sampling network layer may include a third convolutional linear correction part, a third depthwise separable convolutional correction part, and a third convolutional part. Optionally, the third convolutional linear correction part may consist of a fifth convolutional layer and a fifth correction linear unit (ReLU), the third depthwise separable convolutional correction part may consist of a third depthwise separable convolutional layer and a sixth correction linear unit (ReLU), and the third convolutional part may include a sixth convolutional layer.

[0182] As an example, see Figure 7BThe illustrated direct sampling network layer processes the input hand feature map in three steps: First, the features are upscaled using a fifth convolutional layer (e.g., by a factor of 6), and then linearly corrected in the high-dimensional space using a fifth Rectified Linear Unit (ReLU). Second, features are extracted from the linearly corrected hand feature map in the high-dimensional space using a third depthwise separable convolutional layer. For example, the stride of the third depthwise separable convolutional layer can be 1, increasing the number of extracted features while changing the resolution of the feature map. Furthermore, the features in the high-dimensional space can be linearly corrected using the fifth Rectified Linear Unit (ReLU). Third, the features output from the second step are downscaled using a sixth convolutional layer.

[0183] B11, Identity Mapping: See step A3 for details, which will not be repeated here. Optionally, in this step, the hand image may be processed by the identity mapping network layer once or multiple times. This application does not specify the number of times the identity mapping network layer is processed.

[0184] B12, downsampling: See details for more information. Figure 6A Step A10 in the above will not be repeated here.

[0185] B13, Pyramid Pooling Hand Feature Fusion: The input hand feature map is processed through a pyramid pooling layer to obtain the first multi-scale feature map. In some embodiments, the processing procedure can be found in [reference needed]. Figure 6A Step A11 in the above will not be repeated here.

[0186] B14, Direct Sampling: By using a direct sampling network layer, the number of hand features is reduced while maintaining the resolution of the first multi-scale feature map output in step B13 above.

[0187] Since the pyramid pooling layer in step B13 increases the number of feature channels, excessive hand features not only reduce computation speed but also introduce redundant hand features that are not useful for gesture recognition. Therefore, adding a direct sampling network layer in step B14 reduces the number of hand features without changing the resolution of the first multi-scale feature map, thereby eliminating redundant hand features and improving computation speed.

[0188] B15, Deconvolution: The neural network model may also include a deconvolution layer. Through deconvolution operation, the resolution of the first multi-scale feature map output by step B14 above is adjusted.

[0189] B16, Convolution operation to extract hand features: The neural network model may also include a seventh convolutional layer, which is used to adjust the number of feature channels of the hand feature map output in step B11 above.

[0190] In this embodiment, the execution order of steps B15 and B16 is not limited. For example, B15 can be executed before B16, B16 can be executed before B15, or B15 and B16 can be executed simultaneously. It should be noted that the number of feature channels adjusted by the seventh convolutional layer in step B16 is the same as the number of feature channels in the first multi-scale feature map output by the deconvolutional layer in step B15. The resolution of the first multi-scale feature map adjusted in step B15 is the same as the resolution of the hand feature map output in step B16.

[0191] B17, Feature stitching: The neural network model also includes a second stitching processing layer, which is used to stitch together the feature maps output from steps B15 and B16 to obtain a second multi-scale feature map. The second multi-scale feature map includes spatial information of multiple scales from the hand image.

[0192] B18, Prediction Part: Used to obtain the gesture category probabilities corresponding to multiple prediction boxes based on the first multi-scale feature map and the second multi-scale feature map, respectively.

[0193] In some embodiments, the prediction process can be found in [reference]. Figure 6A The details of step A12 will not be repeated here.

[0194] In some embodiments, during the training process of the neural network model described above, the training samples can be images that conform to the display device's aspect ratio requirements, such as the required display ratio. For example, when the display device is a television, to meet the actual application requirements of a television, images with a 16:9 aspect ratio can be input to ensure that the images are not distorted during subsequent image processing and to improve the accuracy of gesture recognition.

[0195] Based on the same concept as the methods described above, this application provides a display device 800. The device 800 is capable of performing the various steps of the methods described above; to avoid repetition, they will not be repeated here. See also... Figure 8 The display device 800 includes an accelerator processor 801, a central processing unit 802, and a display screen 803.

[0196] Acceleration processor 801 is configured to acquire an image to be recognized and perform the following processing on the image to be recognized using a neural network model:

[0197] A hand image is obtained from the image to be identified. The hand image is then subjected to multiple consecutive high-dimensional space acquisitions and downsampling processes to obtain a hand feature map. This hand feature map includes hand feature information at different dimensions and resolutions. Multi-scale max pooling is performed on the hand feature map, and the resulting feature maps are then fused to obtain a first multi-scale feature map. This first multi-scale feature map includes spatial information from multiple scales of the hand image. Based on the first multi-scale feature map, multiple prediction boxes are obtained, each corresponding to a specific gesture category probability. These multiple prediction boxes are obtained by segmenting the image to be identified.

[0198] The central processing unit 802 is configured to identify hand gestures in the image to be identified based on the probability of the gesture category corresponding to the plurality of prediction boxes.

[0199] In some embodiments, the neural network model includes multiple network layers, each network layer including one or more cascaded identity mapping network layers and downsampling network layers;

[0200] The first network layer is an identity mapping network layer, used to obtain hand features in high-dimensional space based on the input image;

[0201] The downsampling network layer of the first network layer is used to downsample the hand features obtained in the high-dimensional space to obtain a hand feature map while reducing the resolution of the image input to the downsampling network layer, and input it into the identity mapping network layer of the second network layer.

[0202] The first network layer and the second network layer are any two adjacent network layers among the multiple network layers included in the neural network model.

[0203] In some embodiments, the identity mapping network layer includes a first convolutional linear correction part, a first depthwise separable convolutional correction part, and a first convolutional part; the input of the first convolutional linear correction part is skip-connected to the first convolutional part.

[0204] The first convolutional linear correction part consists of a first convolutional layer and a first correction linear unit (ReLU), and the first depth-separable convolutional correction part consists of a first depth-separable convolutional layer and a second correction linear unit (ReLU). The first convolutional part includes a second convolutional layer.

[0205] In some embodiments, the downsampling network layer includes:

[0206] The second convolution linear correction part, the second depthwise separable convolution correction part, and the second convolution part;

[0207] The second convolutional linear correction part consists of a third convolutional layer and a third correction linear unit (ReLU), and the second depthwise separable convolutional correction part consists of a second depthwise separable convolutional layer and a fourth correction linear unit (ReLU). The second convolutional part includes a fourth convolutional layer.

[0208] In some embodiments, the last network layer of the neural network model further includes a direct sampling network layer;

[0209] The direct sampling network layer is used to adjust the number of feature channels of the image input to the direct sampling network layer without changing the resolution of the image input to the direct sampling network layer.

[0210] In some embodiments, the direct sampling network layer includes:

[0211] The third convolutional linear correction part, the third depthwise separable convolutional correction part, and the third convolutional part;

[0212] The third convolutional linear correction part consists of a fifth convolutional layer and a fifth correction linear unit (ReLU). The third depthwise separable convolutional correction part consists of a third depthwise separable convolutional layer and a sixth correction linear unit (ReLU). The third convolutional part includes a sixth convolutional layer.

[0213] In some embodiments, the neural network model further includes a seventh convolutional layer, a deconvolutional layer, and a first concatenation layer;

[0214] The seventh convolutional layer is used to adjust the number of feature channels in the image output by the last network layer; the adjusted number of feature channels is the same as the number of feature channels in the first multi-scale feature map.

[0215] The deconvolutional layer is used to adjust the resolution of the first multi-scale feature map; the resolution of the first multi-scale feature map after adjustment is the same as the resolution of the hand feature map output by the seventh convolutional layer.

[0216] The first stitching processing layer is used to stitch the first multi-scale feature map after resolution adjustment with the hand feature map output by the seventh convolutional layer to obtain a second multi-scale feature map, which includes spatial information of multiple scales of the hand image.

[0217] In some embodiments, the neural network model includes a pyramid pooling layer; the pyramid pooling layer includes a max pooling portion and a second splicing processing layer;

[0218] The max pooling part consists of multiple max pooling layers of different scales, which are used to perform multi-scale max pooling processing on the hand feature map input to the pyramid pooling layer.

[0219] The second splicing processing layer is used to splice the feature maps after multi-scale max pooling output by the max pooling part to obtain the first multi-scale feature map.

[0220] In some embodiments, the display device further includes a display screen 803;

[0221] The display screen 803 is used to display the image to be identified.

[0222] Based on the same concept as the method described above, this application also provides a gesture recognition device 900. The device 900 is used to implement the various steps in the above method; to avoid repetition, they will not be described again here. See also Figure 9 The device 900 includes a first processing unit 901, a second processing unit 902, and a display unit 903.

[0223] The first processing unit 901 is configured to acquire an image to be identified and perform the following processing on the image to be identified using a neural network model:

[0224] A hand image is obtained from the image to be identified. The hand image is then subjected to multiple consecutive high-dimensional space acquisitions and downsampling processes to obtain a hand feature map. This hand feature map includes hand feature information at different dimensions and resolutions. Multi-scale max pooling is performed on the hand feature map, and the resulting feature maps are then fused to obtain a first multi-scale feature map. This first multi-scale feature map includes spatial information from multiple scales of the hand image. Based on the first multi-scale feature map, multiple prediction boxes are obtained, each corresponding to a specific gesture category probability. These multiple prediction boxes are obtained by segmenting the image to be identified.

[0225] The second processing unit 902 is configured to identify the hand gestures in the image to be identified based on the probability of the gesture categories corresponding to the plurality of prediction boxes.

[0226] Display unit 903 is used to display the image to be recognized.

[0227] This application also provides a computer-storable medium having a computer program stored thereon, which, when executed by a processor or controller, implements the steps of any of the methods described above.

[0228] While specific embodiments of this application have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this application is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this application, and all such changes and modifications fall within the scope of protection of this application. Although preferred embodiments of this application have been described, those skilled in the art, once they understand the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0229] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A gesture recognition method, characterized in that, include: Obtain the image to be recognized; A hand image is obtained from the image to be identified using a neural network model; The neural network model obtains a hand feature map from the hand image by repeatedly acquiring hand features in high-dimensional space and downsampling these features. The hand feature map includes hand feature information at different dimensions and resolutions within the hand image. The neural network model comprises multiple network layers, each layer including one or more cascaded identity mapping network layers and downsampling network layers. The identity mapping network layer of the first network layer acquires hand features in high-dimensional space based on the input image. The downsampling network layer of the first network layer downsamples the hand features obtained in high-dimensional space while reducing the resolution of the image input to the downsampling network layer, and inputs the resulting hand feature map to the identity mapping network layer of the second network layer. The first network layer and the second network layer are any two adjacent network layers in the neural network model. The last network layer in the neural network model... The network layer also includes a direct sampling network layer; the direct sampling network layer is used to adjust the number of feature channels of the input image of the direct sampling network layer without changing the resolution of the input image; the neural network model also includes a seventh convolutional layer, a deconvolutional layer, and a first stitching processing layer; the seventh convolutional layer is used to adjust the number of feature channels of the image output by the last network layer; the adjusted number of feature channels is the same as the number of feature channels of the first multi-scale feature map; the deconvolutional layer is used to adjust the resolution of the first multi-scale feature map; the resolution of the first multi-scale feature map after resolution adjustment is the same as the resolution of the hand feature map output by the seventh convolutional layer; the first stitching processing layer is used to stitch the first multi-scale feature map after resolution adjustment with the hand feature map output by the seventh convolutional layer to obtain a second multi-scale feature map, the second multi-scale feature map including spatial information of multiple scales of the hand image; The neural network model is used to perform multi-scale max pooling on the hand feature map, and the feature maps after multi-scale max pooling are fused to obtain a first multi-scale feature map, which includes spatial information of the hand image at multiple scales. The neural network model obtains the gesture category probabilities corresponding to multiple prediction boxes based on the first multi-scale feature map; wherein, the multiple prediction boxes are obtained by segmenting the image to be recognized; The hand gestures in the image to be identified are determined based on the probability of the gesture category corresponding to each of the multiple prediction boxes.

2. The method as described in claim 1, characterized in that, The identity mapping network layer includes a first convolutional linear correction part, a first depthwise separable convolutional correction part, and a first convolutional part; the input of the first convolutional linear correction part is skipped to the first convolutional part. The first convolutional linear correction part consists of a first convolutional layer and a first correction linear unit (ReLU), and the first depth-separable convolutional correction part consists of a first depth-separable convolutional layer and a second correction linear unit (ReLU). The first convolutional part includes a second convolutional layer.

3. The method as described in claim 1, characterized in that, The downsampling network layer includes: The second convolution linear correction part, the second depthwise separable convolution correction part, and the second convolution part; The second convolutional linear correction part consists of a third convolutional layer and a third correction linear unit (ReLU), and the second depthwise separable convolutional correction part consists of a second depthwise separable convolutional layer and a fourth correction linear unit (ReLU). The second convolutional part includes a fourth convolutional layer.

4. The method as described in claim 1, characterized in that, The direct sampling network layer includes: The third convolutional linear correction part, the third depthwise separable convolutional correction part, and the third convolutional part; The third convolutional linear correction part consists of a fifth convolutional layer and a fifth correction linear unit (ReLU). The third depthwise separable convolutional correction part consists of a third depthwise separable convolutional layer and a sixth correction linear unit (ReLU). The third convolutional part includes a sixth convolutional layer.

5. The method as described in claim 1, characterized in that, The neural network model includes a pyramid pooling layer; the pyramid pooling layer includes a max pooling component and a second splicing processing layer; The max pooling part consists of multiple max pooling layers of different scales, which are used to perform multi-scale max pooling processing on the hand feature map input to the pyramid pooling layer. The second splicing processing layer is used to splice the feature maps after multi-scale max pooling output by the max pooling part to obtain the first multi-scale feature map.

6. The method as described in claim 1, characterized in that, The method further includes: The image to be identified is displayed.

7. A display device, characterized in that, include: An accelerator processor is configured to acquire an image to be identified and perform the following processing on the image to be identified using a neural network model: A hand image is obtained from the image to be identified. The hand image undergoes multiple consecutive acquisitions of hand features in high-dimensional space and downsampling of these features to obtain a hand feature map. This hand feature map includes hand feature information at different dimensions and resolutions. Multi-scale max pooling is performed on the hand feature map, and the feature maps after multi-scale max pooling are fused to obtain a first multi-scale feature map. This first multi-scale feature map includes spatial information of the hand image at multiple scales. Multiple prediction boxes are obtained based on the first multi-scale feature map, corresponding to different gesture categories. These prediction boxes are obtained by dividing the image to be identified. The neural network model includes multiple network layers, each including one or more cascaded identity mapping network layers and downsampling network layers. The identity mapping network layer of the first network layer is used to acquire hand features in high-dimensional space based on the input image. The first network layer's downsampling network layer is used to downsample the hand features obtained in high-dimensional space to obtain a hand feature map while reducing the resolution of the image input to the downsampling network layer, and then input it to the second network layer's identity mapping network layer; the first network layer and the second network layer are any two adjacent network layers among the multiple network layers included in the neural network model; the last network layer included in the neural network model also includes a direct sampling network layer; the direct sampling network layer is used to adjust the number of feature channels of the image input to the direct sampling network layer without changing the resolution of the image input to the direct sampling network layer; the neural network model also includes a seventh convolutional layer, a deconvolutional layer, and a first concatenation processing layer; the seventh convolutional layer is used to adjust the number of feature channels of the image output by the last network layer; the adjusted number of feature channels is the same as the number of feature channels of the first multi-scale feature map; The deconvolutional layer is used to adjust the resolution of the first multi-scale feature map; the resolution of the first multi-scale feature map after adjustment is the same as the resolution of the hand feature map output by the seventh convolutional layer. The first stitching processing layer is used to stitch the first multi-scale feature map after the resolution adjustment with the hand feature map output by the seventh convolutional layer to obtain a second multi-scale feature map, wherein the second multi-scale feature map includes spatial information of multiple scales of the hand image. The central processing unit is configured to identify hand gestures in the image to be identified based on the probability of the gesture category corresponding to the plurality of prediction boxes.

Images