Hand detection method based on retina hand light infrared image

Abstract

The application discloses a hand detection method based on a RetinaHand light infrared image, and comprises the following steps: S100, generating a hand region image by using a hand detection network based on RetinaHand; and S200, performing enhancement processing on the generated hand region image. The method has the characteristics of short delay, accurate hand detection positioning and real-time generation support, and can be widely used in natural interaction in the fields of intelligent vehicles, intelligent homes and robots.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of computer vision, pattern recognition and artificial intelligence, and particularly relates to a RetinaHand-based lightweight infrared image hand detection method. BACKGROUND

[0002] Gesture, as a natural way of human interaction, is an important bridge for communication between people, people and machines, and even human-like intelligent machines and machines. It is in urgent need in many fields, such as deaf-mute communication, smart home, robots, medical and defense, etc. How to obtain high-precision and high-accuracy gesture recognition has become the key to gesture interaction research.

[0003] The general steps of gesture recognition are as follows: input the gesture image in the video stream into the gesture recognition system, first output the position coordinates of the hand in the whole picture through the gesture detection network, cut out the hand part using the coordinates, and then send it to the skeleton key point positioning network after pretreatment. The network needs to accurately give the position of 21 skeleton key points of the hand, and then connect the 21 key points into a gesture skeleton. Hand detection is the first step and a very important step in the whole dynamic gesture recognition process, because the accuracy and stability of the detection effect directly affect the performance of subsequent skeleton key point positioning and dynamic gesture classification and recognition using key point motion sequence. The role of hand detection is that for an input gesture image, the model can accurately give the position of the hand, and then cut the hand part according to the position, and send it to the subsequent skeleton key point positioning network for positioning. Many current hand detection methods are mainly for RGB image processing, and most of the current mainstream methods for detecting hands completely apply classic general object detection algorithms. On the one hand, no additional design is made for the characteristics of infrared images, so the precision of direct application to infrared data is difficult to guarantee. On the other hand, the model parameter quantity and calculation quantity are generally too large, and cannot be used in real time on mobile terminals. SUMMARY

[0004] In order to solve the above technical problems, the present disclosure discloses a RetinaHand-based lightweight infrared image hand detection method, which comprises the following steps:

[0005] S100: generating a hand region image by using a hand detection network based on RetinaHand;

[0006] S200: performing enhancement processing on the generated hand region image.

[0007] By the technical solution, the hand detection is realized based on the hand detection network of RetinaHand, and has the characteristics of short delay, accurate hand detection positioning, and support for real-time generation. The method is not only suitable for infrared image hand detection, but also suitable for dynamic gesture skeleton positioning in RGB video stream, RGB-Depth video stream or IR-Depth video stream, and can be widely used in natural interaction in the fields of intelligent vehicles, smart homes, robots and the like. The method can realize real-time, accurate and stable hand detection on a mobile terminal. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a flowchart of a RetinaHand-based lightweight infrared image hand detection method provided in an embodiment of the present disclosure;

[0009] Figure 2 is a hand detection network structure diagram based on RetinaHand provided in an embodiment of the present disclosure;

[0010] Figure 3 is an improved FPN implementation flowchart with a fusion attention mechanism provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0011] In order for those skilled in the art to understand the technical solutions disclosed in the present disclosure, the following will be combined with the embodiments and related drawings to describe the technical solutions of the embodiments. Figures 1 to 3 The technical solutions of the embodiments are described, and the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. The terms "first", "second", etc. used in the present disclosure are used to distinguish different objects, rather than to describe a specific order. In addition, "include" and "have" and any variations thereof are intended to cover and not exclusive. For example, a process, or method, or system, or product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, system, product or device.

[0012] In this document, reference to "embodiments" means that the specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of the present disclosure. The phrase appears at various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily mutually exclusive or alternative embodiments. Those skilled in the art can understand that the embodiments described herein can be combined with other embodiments.

[0013] Referring to Figure 1 In one embodiment, the present disclosure discloses a RetinaHand-based lightweight infrared image hand detection method, comprising the following steps:

[0014] S100: generating a hand region image by using a hand detection network based on RetinaHand;

[0015] S200: performing enhancement processing on the generated hand region image.

[0016] For this embodiment, the method can detect hands in infrared images in real time through the two steps of generating a hand region image by using a hand detection network based on RetinaHand and performing enhancement processing on the hand region image. For an input infrared gesture image video, hand detection is performed frame by frame, and once a hand is detected, a hand region image of a fixed resolution is obtained after image scaling of the hand region.

[0017] The hand detection algorithm belongs to a subtask in general target detection and can use a classic model in target detection. General target detection algorithms mainly have two types: single-stage represented by YOLO and SSD, simple model, faster speed, and more suitable for landing application; and double-stage represented by Faster R-CNN, more complex model, higher precision, but slower speed. Target detection algorithms can be divided into two types based on whether a prior anchor frame is needed: Anchor-base and Anchor-free. In the Anchor-free series algorithm, the most representative is CenterNet.

[0018] The method considers the lightweight deployment requirement and designs and implements a hand detection method based on RetinaHand, which belongs to a single-stage target detection network and can be applied to the lightweight hand detection task of the method.

[0019] The hand detection network structure based on RetinaHand is improved and upgraded from the RetinaFace framework, specifically in that more lightweight networks are introduced as backbone networks, the Feature Pyramid Networks (FPN) is improved, the positive and negative sample generation strategy is changed, the Neck part is simplified, and different loss functions are tried.

[0020] The entire process of RetinaHand from input to output is as follows: the input infrared image is 640x480, which is scaled by equal ratio, the long side of the input is scaled to 224, the short side is scaled to 168, and then the short side is completed to 224 using pure black pixels. Therefore, the image input to RetinaHand is 224x224, and after three stages of feature extraction by RetinaHand, the feature maps are down-sampled by 8 times, 16 times, and 32 times, respectively, and the sizes are 28x28, 14x14, and 7x7, respectively. Each pixel point on each layer of feature map corresponds to an 8x8, 16x16, and 32x32 block area on the original image, respectively. Next, K prediction boxes are generated on each pixel point of the three feature maps according to the pre-set prior box parameters. Each box needs 5 parameters (left upper corner horizontal and vertical coordinates, width and height, and confidence), so the three feature maps will obtain 28x28xKx(4+1), 14x14xKx(4+1), and 7x7xKx(4+1) outputs after the regression head module, that is, 28x28xK prediction boxes and the probability of belonging to the foreground or background. During training, the loss is calculated using the prediction and the real label, and during prediction, the non-maximum suppression algorithm is used to process the prediction boxes to obtain the final prediction result.

[0021] In another embodiment, the hand detection network structure based on RetinaHand includes a backbone network, a multi-scale feature fusion processing module, and a regression head module.

[0022] For this embodiment, the hand detection network based on RetinaHand conforms to the classic design process of Backbone, Neck, and Head in the target detection algorithm, and the network structure is as shown in Figure 2 The network structure mainly includes three main parts:

[0023] 1) The backbone network for feature extraction, usually referred to as backbone.

[0024] 2) The multi-scale feature fusion processing module FPN, also referred to as the neck module of the network.

[0025] 3) The regression head part, usually referred to as the head module, which is used to regress the specific category, coordinates, and other information of the target from the features processed by the neck module.

[0026] In another embodiment, step S100 further includes the following steps:

[0027] S101: Generation of prior anchor boxes and matching of anchor boxes and target boxes;

[0028] S102: The entire mapping process from input to output of the RetinaHand-based hand detection network;

[0029] S103: The obtained feature maps are processed by the bounding box regression branch and the confidence classification branch to regress the coordinates of the hand in the current coordinate system and the probabilities of the foreground and background.

[0030] In this embodiment, the regression head module comprises two parts: a target bounding box regression branch and a confidence classification branch. One branch is used to regress coordinates, and the other is used to regress the probabilities of the foreground and background. The object's coordinates in the current image coordinate system are specifically the horizontal and vertical coordinates of the top-left and bottom-right corners.

[0031] The RetinaHand-based hand detection network processes hand detection in three steps, as follows: Figure 2 As shown.

[0032] Step 1: Generation of prior anchor boxes and matching of anchor boxes with ground truth (GT). The basic principle of all single-stage object detection algorithms based on prior anchor boxes can be summarized as classification and regression after dense sampling of the original image. Therefore, generating anchor boxes is an essential step. Although the geometric meaning of the anchor box is relative to the original image, its specific generation needs to be combined with feature maps. For Retina-hand, three layers of feature maps in the network are retained, with downsampling ratios of 1 / 8, 1 / 16, and 1 / 32 relative to the original image, respectively.

[0033] Considering the characteristics of the infrared gesture image dataset used in this method and the need for speed, in one instance, the original size of the input infrared image is limited to 224x224. Therefore, the scales of the three feature maps are 28x28, 14x14, and 7x7, respectively. Each pixel in each feature map corresponds to an 8x8, 16x16, or 32x32 region in the original image, respectively. Traditional algorithms such as Faster R-CNN, SSD, and RetinaNet generate k anchor boxes with different scales and aspect ratios based on each pixel in the feature map; typically, k=9, representing three different scales and three different aspect ratios. Furthermore, given that the infrared gesture image data used in this method is nearly square, we can consider only the scale and ignore the aspect ratio, thus simplifying the anchor box design. Additionally, when processing the dataset, we can use the method of padding the shorter sides to force all annotations to be square.

[0034] After the anchor box is generated, only the dense sampling work for the original image is completed, and further, a target for supervised learning needs to be constructed for each sample, which specifically represents the position of the target box relative to the anchor box and the category of each anchor box. That is, to determine whether the anchor box belongs to the foreground or the background, if it belongs to the foreground, the specific position needs to be determined, and the position is represented by the bias of the anchor box center point relative to the target box center point. The bias is divided into two parts, the bias of the target box center point relative to the anchor box center point and the conversion of the width and height of the target box relative to the width and height of the anchor box, and the conversion specifically represents the scale ratio of the target box and the anchor box after logarithmic transformation.

[0035] Second step: the mapping process of the entire RetinaHand-based hand detection network from input to output.

[0036] Third step: these feature maps will respectively pass through the target box regression branch and the confidence classification branch to regress the final coordinates and foreground and background probabilities. For the present application, if the total number of anchor boxes is represented by N, the final output of the classification branch of the network model will be 2N, and the final output of the coordinate box regression branch will be 4N, which respectively represents the probability that each anchor box belongs to the foreground or the background, and if it belongs to the foreground, the center point of the target relative to the anchor box bias and the logarithmic conversion value of the target width and height relative to the anchor box width and height.

[0037] In another embodiment, the matching of the anchor box and the target box in step S101 further comprises: normalizing the width and height of the target box relative to the center point of the anchor box.

[0038] For this embodiment, it should be noted that, in order to eliminate the influence of the scale of the anchor box itself, all anchor boxes are treated equally, and the width and height of the target box relative to the center point of the anchor box need to be normalized. If not normalized, large anchor boxes can tolerate larger deviations, while small anchor boxes are very sensitive to deviations, which is not conducive to model training and learning, and converting the regression absolute scale to the regression relative scale can solve this problem.

[0039] In another embodiment, the matching of the anchor box and the target box in step S101 further comprises the following steps: converting the width and height of the target box relative to the width and height of the anchor box to the logarithmic space.

[0040] For this embodiment, another important step is to convert the width and height of the target box relative to the width and height of the anchor box to the logarithmic space. If not converted, the output width and height of the model can only be positive, which increases the requirements for the model and increases the optimization difficulty, and the conversion to the logarithmic space solves the problem.

[0041] In another embodiment, step S102 further comprises the following steps:

[0042] S1021: The input image is subjected to feature extraction by a backbone network composed of a stack of convolutional layers;

[0043] S1022: The features of some two layers in the middle of the backbone network and the features of the last layer are extracted and sent to a multi-scale feature fusion processing module for processing;

[0044] S1023: After FPN feature fusion, 3 layers of features are obtained, and at this time, the features are further subjected to feature extraction by a feature refining module composed of a large convolutional kernel to expand the receptive field of the feature map.

[0045] For this embodiment, the second step: the mapping process of the entire network from input to output, the input image 3x224x224 is first subjected to feature extraction by a backbone network composed of a stack of convolutional layers, the features of each layer in the middle of the network are extracted and sent to the subsequent FPN for processing, and here a total of three layers of features of the entire backbone network are extracted. For MobileNetV1x0.25 as the backbone network, the scales of the three layers of feature maps are 64x28x28, 128x14x14, and 256x7x7, respectively. After FPN feature fusion, 3 layers of features are obtained, and each layer will have a large number of prior anchor boxes. In order to improve the expression ability of the features, the feature map at this time will further be subjected to feature extraction by a feature refining module composed of a large convolutional kernel to expand the receptive field of the feature map.

[0046] In another embodiment, the feature processing and fusion module FPN is an improved FPN that fuses attention mechanisms.

[0047] For this embodiment, the FPN is an Attention-FPN. The feature pyramid, as a necessary component in the current mainstream target detection model, can effectively improve the positioning ability of the algorithm for targets of different scales. For the hand detection task, because the distance and orientation of the photographed object relative to the camera vary in actual scenes, the size of the hand varies greatly. The pixel of the target close to the camera can be up to 400x400, and the size of the farthest target is only 20x20, with a large scale variation. This requires the target detection network to have good detection ability for large and small targets. The traditional FPN realizes this by directly adding the up-sampled high-level features and the low-level features. This method designs and implements an improved FPN that fuses the Attention idea.

[0048] In another embodiment, the Query in the improved FPN that fuses attention mechanisms comes from the nonlinear transformation of the shallow feature map, and the Key and Value both come from the linear transformation of the deep feature map after up-sampling.

[0049] For this embodiment, inspired by MobileViT, the self-attention mechanism is extended and introduced into the FPN module. Here, Query, Key, and Value are no longer from the same input. Query comes from the nonlinear transformation of the shallow feature map, and Key and Value come from the linear transformation of the deep feature map after upsampling. The operation in the original FPN using element-wise addition is changed to fusion using the attention mechanism. From the principle of attention mechanism, this operation can be understood as expressing each pixel in the shallow feature map using the weighted sum of all pixels in the deep feature map. The benefit of this is that the use of deep attention mechanism to represent the shallow layer can effectively introduce global information for each pixel in the shallow feature map, while convolution focuses more on local information. Therefore, the feature map after fusion retains both global and local information, which is more conducive to model learning. Finally, after obtaining the new feature map of the shallow and deep features fused by the attention mechanism, the self-attention mechanism is used again to further transform the feature map to improve the expression ability of the feature.

[0050] The specific operation is as follows: upsample the relatively deep feature map, 7x7 sample to 14x14, then use 1x1 convolution to align the channel number with the channel of the previous layer, map 256 to 128, get 128x14x14, then in order to use the obtained feature map for Attention operation, here we refer to the method of MobileViT, first perform slicing operation on the feature map, perform self-attention operation on all pixels in each slice, then the final result is obtained after inverse transformation to get the same shape as the original input feature map, thus realizing an attention calculation process. Figure 3 The complete implementation process of Attention-FPN is shown.

[0051] In another embodiment, the loss function for regressing the target box coordinates in step S103 is the intersection over union loss.

[0052] For this embodiment, in order to improve the positioning accuracy, the loss function for regressing the target box coordinates is replaced from the mean absolute error loss to the intersection over union loss (IoU Loss). When using absolute error to measure the distance between output and target, each geometric quantity regressed is independent of each other, lacking the inherent geometric constraints between them. If the intersection over union between the predicted box and the real box is directly optimized, this geometric relationship can be modeled, which can also be regarded as direct optimization for evaluation indicators.

[0053] In another embodiment, the enhancement processing in step S200 includes infrared image denoising, Gamma correction, histogram equalization enhancement, and edge enhancement.

[0054] For this embodiment, the hand features in the enhanced hand region image will be more prominent, which is conducive to improving the accuracy of subsequent hand skeleton key point positioning and gesture action recognition.

[0055] Although the embodiments of the present application are described above with reference to the drawings, the present application is not limited to the above-described specific embodiments and application fields, and the above-described specific embodiments are merely illustrative and instructive, but not restrictive. Those skilled in the art can make many forms under the guidance of the present specification and without departing from the scope protected by the claims of the present application, which all belong to the protection of the present application.

Claims

1. A lightweight infrared image hand detection method based on RetinaHand, comprising the following steps: S100: A hand region image is generated using a RetinaHand-based hand detection network, wherein... For the input infrared gesture image video, hand detection is performed frame by frame. Once a hand is detected, the hand region is extracted and the image is scaled to obtain a hand region image with a fixed resolution. S200: Enhance the generated hand region image; Step S100 further includes the following steps: S101: Generation of prior anchor boxes and matching of anchor boxes and target boxes; S102: The entire mapping process from input to output of the RetinaHand-based hand detection network; S103: The obtained feature maps are processed by the bounding box regression branch and the confidence classification branch to regress the coordinates of the hand in the current coordinate system and the probabilities of the foreground and background. Step S102 further includes the following steps: S1021: The input image is processed through a backbone network consisting of stacked convolutional layers for feature extraction; S1022: Extract the features of two middle layers and the last layer of the backbone network and send them to the multi-scale feature fusion processing module for processing; S1023: After feature fusion, three layers of features are obtained. At this point, the features are further extracted by a feature refinement module composed of large convolutional kernels to expand the receptive field of the feature map.

2. The method as described in claim 1, wherein the RetinaHand-based hand detection network structure comprises a backbone network, a multi-scale feature fusion processing module, and a regression head module.

3. The method as described in claim 1, wherein the matching of the anchor box and the target box in step S101 further includes: The center point of the target frame relative to the anchor frame is normalized using width and height.

4. The method as described in claim 1, wherein the matching of the anchor frame and the target frame in step S101 further includes the following step: transforming the width and height of the target frame relative to the width and height of the anchor frame to logarithmic space.

5. The method as described in claim 1, wherein the multi-scale feature fusion processing module is an improved FPN with a fusion attention mechanism.

6. The method as described in claim 5, wherein the Query in the improved FPN with fused attention mechanism comes from the nonlinear transformation of the shallow feature map, and both Key and Value come from the linear transformation of the deep feature map after upsampling.

7. The method as described in claim 1, wherein the loss function for regressing the target box coordinates in step S103 is the cross-union loss.

8. The method of claim 1, wherein the enhancement process in step S200 includes: Infrared image denoising, Gamma correction, histogram equalization enhancement, and edge enhancement.

Images