Dynamic human hand detection and extraction method for large digital screen

Abstract

The invention belongs to the cross technical field of artificial intelligence and image processing, and provides a digital large screen-oriented dynamic hand detection and extraction method, which comprises the steps of to-be-detected picture acquisition, multi-channel data acquisition, hand region image acquisition, depth feature acquisition and target hand detection. According to the invention, through skin color segmentation and convolutional neural network identification, the hand region image is obtained, the data volume of subsequent identification is reduced, and the model identification efficiency is improved; by extracting the depth features, accurate detection of the human hand is realized, and the recognition accuracy of the model is improved.

Description

Technical Field

[0001] The present invention relates to the cross-technical field of artificial intelligence and image processing, and in particular to a dynamic human hand detection and extraction method for a digital large screen. Background Art

[0002] The basis of gesture interaction is to detect human hands, obtain information such as the position, size, and shape of human hands, and then recognize, understand, and track gestures to achieve interaction between humans and machines. The detection of human hands builds a bridge for communication and exchange between machines and humans, allowing machines to know human movements, understand human movement trajectories, understand human intentions, and interact with humans.

[0003] However, due to the complex changes in human hand movements and the complex background environment that will affect the accuracy of hand detection, it is still a big challenge to detect human hands in lighting changes, occlusion, self-occlusion and complex background environments. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a dynamic human hand detection and extraction method for digital large screens. Through skin color segmentation and convolutional neural network recognition, a hand area image is obtained to reduce the amount of data for subsequent recognition and improve model recognition efficiency; by extracting deep features, accurate detection of human hands is achieved to improve the recognition accuracy of the model.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A dynamic human hand detection and extraction method for a digital large screen, comprising:

[0007] Get the image to be detected;

[0008] Perform skin color segmentation on the image to be detected to obtain multi-channel data;

[0009] Inputting the multi-channel data into a preset convolutional neural network to obtain a hand area image;

[0010] Extracting depth features of the hand region image;

[0011] The deep features are input into a preset random forest network to obtain the target hand detection result.

[0012] Preferably, skin color segmentation is performed on the image to be detected to obtain multi-channel data, including:

[0013] Counting the three primary color channel values ​​of the image to be detected;

[0014] Map the three primary color channel values ​​to the HSV space to obtain an H value histogram;

[0015] The H value histogram is segmented using a preset H threshold to obtain the multi-channel data.

[0016] Preferably, before inputting the multi-channel data into a preset convolutional neural network to obtain a hand area image, the method further includes:

[0017] Calculate the skin color connected region area according to the multi-channel data;

[0018] Filtering out the skin color connected region whose area is smaller than a preset area threshold;

[0019] Calculate the minimum outer rectangle according to the area of ​​the skin color connected area after filtering;

[0020] Perform skin color pixel determination based on the outer minimum rectangle to obtain a skin color mask bitmap;

[0021] The outer minimum rectangle and the skin color mask bitmap are input into the convolutional neural network.

[0022] Preferably, the optimization process of the convolutional neural network includes:

[0023] Calculate a first area and a first center of gravity of the frame to be identified;

[0024] Calculate the second area and the second center of gravity of the next frame;

[0025] When the difference between the first area and the second area and between the first center of gravity and the second center of gravity meets a preset area threshold, saving the recognition results of the frame to be recognized and the next frame in the same historical classification result;

[0026] The recognition result with the highest number of occurrences in the historical classification results is output as the hand area image.

[0027] The present invention discloses the following technical effects:

[0028] The present invention provides a dynamic human hand detection and extraction method for digital large screens. Through skin color segmentation and convolutional neural network recognition, the problem of excessive recognition data volume is solved and the segmentation of hand area images is realized. Through deep features, the problem of low human hand recognition accuracy is solved and real-time and accurate tracking of human hands is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 A schematic diagram of a dynamic human hand detection and extraction method provided by an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of a dynamic hand detection and extraction architecture provided by an embodiment of the present invention;

[0032] Figure 3 A schematic diagram of a dynamic hand detection and extraction process provided by an embodiment of the present invention;

[0033] Figure 4 A flow chart of a human hand detection algorithm based on skin color detection and convolutional neural network provided in an embodiment of the present invention;

[0034] Figure 5 A random forest classification flow chart provided by an embodiment of the present invention;

[0035] Figure 6 A schematic diagram of the hand detection effect provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The purpose of the present invention is to provide a dynamic human hand detection and extraction method for digital large screens. Through skin color segmentation and convolutional neural network recognition, a hand area image is obtained to reduce the amount of data for subsequent recognition and improve model recognition efficiency; by extracting deep features, accurate detection of human hands is achieved to improve the recognition accuracy of the model.

[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] Figure 1 A schematic diagram of a dynamic human hand detection and extraction method provided by an embodiment of the present invention, Figure 2 A schematic diagram of a dynamic hand detection and extraction architecture provided by an embodiment of the present invention, Figure 3 A schematic diagram of a dynamic hand detection and extraction process provided by an embodiment of the present invention, such as Figure 1 , Figure 2 and Figure 3 As shown, the present invention provides a dynamic human hand detection and extraction method for a digital large screen, comprising:

[0040] Get the image to be detected;

[0041] Perform skin color segmentation on the image to be detected to obtain multi-channel data;

[0042] Inputting the multi-channel data into a preset convolutional neural network to obtain a hand area image;

[0043] Extracting depth features of the hand region image;

[0044] The deep features are input into a preset random forest network to obtain the target hand detection result.

[0045] Specifically, skin color segmentation is performed on the image to be detected to obtain multi-channel data, including:

[0046] Counting the three primary color channel values ​​of the image to be detected;

[0047] Map the three primary color channel values ​​to the HSV space to obtain an H value histogram;

[0048] The H value histogram is segmented using a preset H threshold to obtain the multi-channel data.

[0049] Furthermore, before the multi-channel data is input into a preset convolutional neural network to obtain a hand area image, the method further includes:

[0050] Calculate the skin color connected region area according to the multi-channel data;

[0051] Filtering out the skin color connected region whose area is smaller than a preset area threshold;

[0052] Calculate the minimum outer rectangle according to the area of ​​the skin color connected area after filtering;

[0053] Perform skin color pixel determination based on the outer minimum rectangle to obtain a skin color mask bitmap;

[0054] The outer minimum rectangle and the skin color mask bitmap are input into the convolutional neural network.

[0055] Specifically, the optimization process of the convolutional neural network includes:

[0056] Calculate a first area and a first center of gravity of the frame to be identified;

[0057] Calculate the second area and the second center of gravity of the next frame;

[0058] When the difference between the first area and the second area and between the first center of gravity and the second center of gravity meets a preset area threshold, saving the recognition results of the frame to be recognized and the next frame in the same historical classification result;

[0059] The recognition result with the highest number of occurrences in the historical classification results is output as the hand area image.

[0060] Specifically, skin color detection based on HSV color space: human skin color information is one of the significant features that distinguishes it from surrounding objects. Scientific research has found that human skin color information has very good clustering properties. Without considering the influence of brightness and saturation, each color has a corresponding hue value in the HSV color space, so the hue dimension in the HSV color space can be used to segment different colors. When performing skin color detection, the ambient light often changes, causing the brightness and saturation of human skin to change, but the hue of human skin generally does not change, so skin color can be segmented based on the H dimension in the HSV color space. In order to establish a good skin color model, it is necessary to collect hand images under various environmental conditions. In this embodiment, a total of 91 hand images with 121 skin color areas are collected, totaling 5910513 pixels. After calculation and statistics, it is found that the average grayscale value of these pixels is 129.12, and the average values ​​of the three RGB channels are 149.21, 124.25, and 101.02, respectively, where R, G, and B∈[0,255]. This result is converted to HSV space, and the distribution of H channel histogram is observed. It is found that the H values ​​in most histograms are evenly and concentratedly distributed in the range of 0 to 60, and a small part of H values ​​are distributed around 80, 180, and 360. From the HSV to RGB space transformation formula, it can be seen that the H value distribution in the range of 0 to 60 represents the normal skin color of the human body under normal circumstances. For a very small number of H values ​​distributed in other areas, considering their extremely small probability density and actual conditions, it can be determined that these H values ​​are caused by noise. In order to make the selected threshold have good noise resistance, the H threshold of skin color segmentation is selected as 6 to 48 after experimental measurement. Using this range can segment most of the gesture area, and has good noise resistance performance.

[0061] refer to Figure 4, Extraction of the main skin color area: The skin color area in the image is extracted using the skin color detection result. When extracting the skin color area, it is impossible to regard each skin color pixel judged by the skin color model as the skin color area to be judged in the next step. Therefore, this embodiment uses the method of judging the area of ​​the skin color area to limit the main skin color area. First, the area of ​​all skin color connected areas is calculated using the connected area, and then the non-main skin color areas are filtered out according to the set threshold. Finally, the outer minimum rectangle of the skin color area is calculated, and the hand area image is cut out on the original color image according to this rectangle. There are two main purposes of this step: 1) remove the interference of noise; 2) set the threshold according to the size of the hand in the image to effectively remove the non-hand area, reduce calculations, and improve detection efficiency. According to the possible size of the hand, the threshold is set to 5000 pixels. As training data, the cut-out skin color area is divided into two formats, namely the original color image data format and the skin color mask bitmap data format. For convolutional neural networks, the first format of data: the minimum outer rectangle determined by the skin color area is mapped to the original color image, and then the corresponding sub-area is intercepted on the color image, and finally input into the neural network training; the second format of data is judged based on whether the pixel is skin color, and the skin color pixel is set to 1, otherwise it is set to 0, which is equivalent to the skin color mask bitmap. The advantage of the first format of image is that it contains rich texture information, but because it contains a lot of texture information, it also leads to more interference factors, such as background information; the second format discards all texture information and only retains the shape and contour information of the image.

[0062] Furthermore, hand detection based on convolutional neural network: convolutional neural network is mainly used in the field of images. Using convolutional neural network to analyze images has the characteristics of local connection, weight sharing, maximum pooling, etc. compared with other artificial neural networks. These characteristics make it more convenient to process image information. Commonly used CNN network structures include LeNet, AlexNet and GoogLeNet, etc. The complexity of these three network structures increases successively. According to the needs of this embodiment, AlexNet network is selected. The AlexNet network structure has a total of 8 layers, and 1000 types of images are classified. The AlexNet network used in this embodiment belongs to a streamlined network structure, which consists of three parts in total, the leftmost is the input end, the middle is the convolutional neural network structure, and the rightmost is the output end. At the input end, two formats of data are input respectively, and classified by neural network respectively; the middle convolutional neural network has a total of 7 layers, including 4 convolutional layers, 3 fully connected layers, and outputs 2 categories of classification results; the output end receives the classification results of two data formats, including three processing modules, result fusion module, statistical optimization module and classification output module. In order to achieve sample diversity, a large amount of data was collected under various environmental conditions. For the positive sample hand data, various gestures should be made as much as possible, and the hands should be made to appear in various areas of the screen as much as possible; for the negative sample data, in addition to the frequently appearing negative samples of human faces, various negative sample data of similar skin color should also be collected, such as chairs, books, etc. In this embodiment, a total of 7,000 samples were collected, including 3,500 positive samples and 3,500 negative samples. Each type of sample has 3,000 training sets and 500 test sets. These sample sets are input into the neural network training. In the process of implementing human hand detection, sometimes due to the influence of factors such as noise or inaccurate models, classification errors may occur in rare cases. For example, the judgment of the human hand area in the first few frames is a hand, but the area may be judged as a non-hand area at this time due to changes in gestures. Under normal circumstances, a certain area cannot suddenly change from a hand to a non-hand. To this end, this embodiment adopts a statistical optimization method to optimize the classification results. The method takes into account the historical information of the classification results when making decisions. The specific steps are: first calculate the area and center of gravity of a certain area. When the area and center of gravity of the next frame are not much different from those of the frame, the two areas are regarded as the same area. Then, the historical classification results of a certain area can be saved in sequence, and a statistical range can be set based on this, and then the result with the highest number of occurrences in the current range can be taken as the result at this moment. The specific steps are: first, assume that the length of the statistical range is n, and then count the historical result sequence {t-n+1, ​​t-n+2...t} closest to the current time, and find the classification result with the highest number of occurrences. If the manual and non-manual results appear the same number of times, take the result closest to the current time.

[0063] Preferably, although the human hand detection based on optical images has certain robustness against illumination changes and occlusions and can achieve good results, the algorithm still has some problems to be solved. For example, when the hand overlaps with the face, neither the skin color detection nor the contour detection effect is ideal. In order to solve this problem, this embodiment introduces a method for calculating the depth pixel features of the human hand into the human hand detection method in the depth image. Since the depth image is also called the distance image, the value of its pixel reflects the distance information between the target object and the camera. The depth image is not affected by changes in illumination and environment, and can directly reflect the three-dimensional information of the object, which greatly simplifies the three-dimensional reconstruction, recognition, positioning and other problems of the object. Therefore, the gestures are annotated for different parts of the human body, and the depth offset comparison feature is extracted for different parts of the body. The feature uses the random distribution of the depth pixel offset to reflect the three-dimensional information of the local surface of the depth image, and at the same time links different local spaces, which has a better recognition effect.

[0064] refer to Figure 5 and Figure 6 , random decision forests are very suitable for discrete classification of body parts. The hand detection problem based on random decision forests designed in this embodiment can be regarded as a classification problem based on the hand and background of each pixel in the depth image, including four modules: RGBD database construction, hand marking, deep feature extraction, and random forest training. Among them, the RGBD database construction module is to construct image data based on the one-to-one mapping of optical images and depth images; the hand marking module is to map the hands marked with bright colors in the optical image to the depth image through the HSV color space to obtain the hand marks in the depth image; the depth feature extraction module is to extract the depth offset comparison features in the depth image; the random forest training module is to perform random forest training on the extracted depth features and hand marks to achieve hand detection.

[0065] The beneficial effects of the present invention are as follows:

[0066] The present invention obtains a hand area image through skin color segmentation and convolutional neural network recognition, reduces the amount of data for subsequent recognition, and improves the model recognition efficiency; by extracting deep features, accurate detection of human hands is achieved, and the recognition accuracy of the model is improved.

[0067] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0068] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A dynamic human hand detection and extraction method for digital large screens, characterized in that: include: Get the image to be detected; Perform skin color segmentation on the image to be detected to obtain multi-channel data; Inputting the multi-channel data into a preset convolutional neural network to obtain a hand area image; Extracting depth features of the hand region image; The deep features are input into a preset random forest network to obtain the target hand detection result.

2. A dynamic human hand detection and extraction method for a digital large screen according to claim 1, characterized in that: Perform skin color segmentation on the image to be detected to obtain multi-channel data, including: Counting the three primary color channel values ​​of the image to be detected; Map the three primary color channel values ​​to the HSV space to obtain an H value histogram; The H value histogram is segmented using a preset H threshold to obtain the multi-channel data.

3. The method for dynamic human hand detection and extraction for a digital large screen according to claim 1, characterized in that: Before inputting the multi-channel data into a preset convolutional neural network to obtain a hand area image, the method further includes: Calculate the skin color connected region area according to the multi-channel data; Filtering out the skin color connected region whose area is smaller than a preset area threshold; Calculate the minimum outer rectangle according to the area of ​​the skin color connected area after filtering; Perform skin color pixel determination based on the outer minimum rectangle to obtain a skin color mask bitmap; The outer minimum rectangle and the skin color mask bitmap are input into the convolutional neural network.

4. The method for dynamic human hand detection and extraction for a digital large screen according to claim 1, characterized in that: The optimization process of the convolutional neural network includes: Calculate a first area and a first center of gravity of the frame to be identified; Calculate the second area and the second center of gravity of the next frame; When the difference between the first area and the second area and between the first center of gravity and the second center of gravity meets a preset area threshold, saving the recognition results of the frame to be recognized and the next frame in the same historical classification result; The recognition result with the highest number of occurrences in the historical classification results is output as the hand area image.

Images