A lightweight multi-person key point detection method based on depth images

Abstract

The application discloses a kind of lightweight multi-person key point detection methods based on depth image, by extracting the feature of different levels and different sizes under depth image, multiple target human frame and key point coordinate can be accurately predicted simultaneously.The layered method is used to extract different channel features for depth image data in the application, the information of human body region is better obtained, the redundant background information is removed, the effective information content of feature is increased, and the calculation amount of extracting feature using neural network is reduced simultaneously.The detection method provided by the application simultaneously identifies the human frame and key point of multiple persons in depth image, without additional model to pre-extract human frame, the algorithm process is simple, the operation amount is low, and the lightweight deployment requirements of various low-power devices are met.

Description

TECHNICAL FIELD

[0001] The present application relates to a lightweight multi-person key point detection method for depth images, belonging to the technical field of computer vision. BACKGROUND

[0002] Human key point detection is an important research topic in the field of computer vision. In recent years, more and more life scenes use human key point detection algorithms to provide people with more convenient services. Human key point detection based on depth images is a recent research trend. Depth images can overcome the recognition inaccuracy caused by the problems of light and shadow in traditional optical images. Compared with traditional optical images, depth images avoid the collection of private information such as human face details, and can be applied in the fields of smart home, intelligent security, and elderly care.

[0003] Current multi-person key point detection methods based on depth images have problems such as low prediction accuracy and heavy algorithm process. For example, Chinese patent 202010175838.X only detects whether a human body appears in a depth image and the position of the human body, and does not identify and detect human key points. Chinese patent 202111082447.4 uses traditional image processing to extract depth image features, and then uses OpenCV open source library algorithm for human body detection, which has limited recognition accuracy. Chinese patent CN202010410104.5 trains two neural networks, uses one model to extract a human body frame as a human body recognition area, and then performs key point detection based on the human body frame area. The algorithm can only be applied to color image scenes, and the overall process is long, making it difficult to meet the lightweight deployment requirements of low-power devices. SUMMARY

[0004] The purpose of the present application is to provide a lightweight detection algorithm that can accurately and efficiently detect human key points.

[0005] To achieve the above purpose, the technical scheme of the present application provides a lightweight multi-person key point detection method based on depth images, characterized in that it comprises the following steps:

[0006] Step 1: Separate the multiple channels of all input depth images, layer different channels according to depth values, remove foreground and background redundant data under different channels, and obtain multi-channel depth maps under different depth ranges;

[0007] Step 2: Clean the multi-channel depth map;

[0008] Step 3: Use a basic convolutional neural network to extract the detail features of the depth image obtained in the previous step, sample and splice the features under different scales, and obtain multi-scale fusion features of the depth image, which comprises the following steps:

[0009] Step 3-1: The bottom layer feature extraction is performed using a convolutional neural network structure, and the preprocessed depth image in the previous step is input into 6 serial depth separable convolutional neural component groups to obtain 6 layers of feature representations {f1, f2, f3, f4, f5, f6} with different sizes;

[0010] Step 3-2: For the 6 different layers of features, the middle k layers of features are selected as the fusion candidate layers, and then the next layer of features to be fused is deconvolved and upsampled to obtain features with the same size as the previous layer of features. The two are added to obtain new features of adjacent upper and lower layers, and the overall fused features F are calculated by summing all the new features;

[0011] Step 3-3: The obtained fused features F are input into a convolutional neuron to further fuse the features of different layers and different sizes to obtain the final fused features F;

[0012] Step 4: The fused features F obtained in the previous step are input into the human body box coordinate prediction network and the human body key point coordinate prediction network to obtain the coordinate prediction values of the human body box and the key points, which includes the following steps:

[0013] Step 4-1: The fused features F obtained in the previous step are input into the human body box coordinate prediction network, and a set of 1*1 convolution is performed to obtain the human body box feature vector f bbox ;

[0014] Step 4-2: The human body box feature vector f bbox is input into a set of 1*1 convolutional neural network to extract the feature vector f center of the human body center point, and the human body center point prediction result is obtained through the Softmax function, indicating that the feature point is the center point of the human body;

[0015] Step 4-3: The human body box feature vector f bbox is input into a set of 1*1 convolutional neural network to extract the human body box width and height feature vector f wh , and the prediction values of the human body box width and height are obtained through the Softmax function, thereby obtaining the human body box region;

[0016] Step 4-4: The fused features F are input into the human body key point coordinate prediction network, and a set of convolutional neural network is used to extract the human body key point feature vector f kps ;

[0017] Step 4-5: According to the human body key point feature vector f kps , a set of convolutional neural network with 34 convolutional channels is used to extract the human body key point coordinate vector f coord, through the Softmax function to obtain the prediction result of the human body key point coordinates, which respectively represent the horizontal and vertical coordinates of several human body key points;

[0018] Step 4-6: input the human body key point feature vector f kps to a convolutional neural network with 2 convolutional channels, and then through the Softmax function to obtain the offset of the human body key point coordinate value, which is superimposed on the horizontal and vertical coordinates of several human body key points obtained in step 4-5;

[0019] Step 5: post-processing and correction of the coordinate prediction value to obtain the human body frame coordinate and key point coordinate value under the original image size, which specifically includes the following steps:

[0020] Step 4-3: the human body frame region obtained in step 4-3 and the coordinate value of several human body key points obtained in step 4-6 are scaled and transformed. During scaling and transformation, according to the size of the fusion feature map and the size of the actual input image, the scaling ratio is calculated, and the coordinates are mapped to the original image to obtain the coordinate value under the original image size. Then, the offset value obtained in step 4-6 is superimposed on the new coordinate as the corrected key point coordinate after scaling.

[0021] Preferably, the step 1 includes the following steps:

[0022] Step 1-1: separate the three channels of the input depth image to obtain separate channel graphs c1, c2 and c3;

[0023] Step 1-2: for channel graph c1, completely retain the original image pixel value;

[0024] For channel graph c2, use the background pixel range threshold {b min ,b max} to separate the front and back backgrounds. The pixels less than the lower threshold b min are all set to 0, and the pixels greater than the upper threshold b max are all set to 255, saved as channel graph c'2, and the edge redundant background information is maximized removed;

[0025] For channel graph c3, use the human pixel range threshold {h min ,h max} to separate the human scene. The pixels less than the lower threshold h min are all set to 0, and the pixels greater than the upper threshold h max are all set to 255, saved as channel graph c'3, and the human body region information is maximized saved;

[0026] Step 1-3: merge the three processed channel graphs into {c1, c'2, c'3} as a new three-channel depth image.

[0027] Preferably, the step 2 comprises the following steps:

[0028] Step 2-1: remove the salt and pepper noise in the three-channel depth image obtained in the last step by using a median filter function, and then remove the Gaussian noise in the three-channel depth image by using a Gaussian filter function;

[0029] Step 2-2: process the three-channel depth image using a bilateral filter function to retain the edge information in the three-channel depth image;

[0030] Step 2-3: fix the resolution of the three-channel depth image to a uniform width and height, and then normalize the three-channel depth image as shown in the following formula (1):

[0031]

[0032] In formula (1), img is a three-channel image matrix, img.mean is the mean value of all images in three channels, and img.std is the standard deviation of all images in three channels.

[0033] Preferably, the step 3-2, the feature f i-1 After deconvolution, the feature size is expanded to the same as the feature f i of the i-th layer, and then added to the feature of the i-th layer as a new feature as shown in the following formula (2):

[0034] f i =f i +TransConv(f i-1 ) (2)

[0035] In formula (2), TransConv() represents deconvolution, and the overall fused feature f

[0036] The method disclosed in the application can accurately predict multi-target human frame and key point coordinates by extracting features of different sizes at different levels of depth images.

[0037] Compared with the prior art, the application has the following advantages:

[0038] 1) The lightweight multi-person key point detection method based on depth images provided by the application extracts different channel features by using a hierarchical method for depth image data, better acquires human region information, removes redundant background information, increases the effective information content of the features, and reduces the calculation amount of extracting features by using a neural network;

[0039] 2) The detection method provided by the application simultaneously recognizes the human body frame and key points of multiple people in a depth image, does not need an additional model to pre-extract the human body frame, has a simple algorithm process and low calculation amount, and meets the lightweight deployment requirements of various low-computing-power devices. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A lightweight multi-person key point detection method flowchart;

[0041] Figure 2 A depth image preprocessing flowchart;

[0042] Figure 3 A fusion feature extraction flowchart under different scales;

[0043] Figure 4 A human body frame coordinate prediction and human body key point coordinate prediction flowchart. DETAILED DESCRIPTION

[0044] The application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not to limit the scope of the application. Furthermore, it should be understood that those skilled in the art can make various modifications or changes to the application after reading the content taught by the application, and these equivalent forms also fall within the scope defined by the appended claims.

[0045] As shown in Figure 1 , the lightweight multi-person key point detection method based on a depth image disclosed by the embodiment includes the following steps:

[0046] Step 1: Separate the multiple channels of the depth image, layer different channels according to the depth value, remove the foreground and background redundant data under different channels, and obtain the multi-channel depth map under different depth ranges, which specifically includes the following steps:

[0047] Step 1-1: Separate the three channels of the input depth image to obtain separate channel maps c1, c2 and c3;

[0048] Step 1-2: For the channel map c1, completely retain the original image pixel value;

[0049] For the channel map c2, use the background pixel range threshold {b min ,b max} to separate the foreground and background, set all pixels less than the lower limit b min of the threshold to 0 and all pixels greater than the upper limit b max of the threshold to 255, save as the channel map c′2, and maximize the removal of edge redundant background information;

[0050] For the channel map c3, use the human body pixel range threshold {hmin h max}perform human-scene separation, all pixels less than the lower threshold h min are set to 0, and all pixels greater than the upper threshold h max are set to 255, and saved as a channel map c'3, maximizing the human region information;

[0051] Step 1-3: The three processed channel maps are recombined as {c1, c'2, c'3} as a new three-channel depth image.

[0052] Step 2: The multi-channel depth image is cleaned, including denoising, standardization and other preprocessing operations, specifically including the following steps:

[0053] Step 2-1: A median filter function is used to remove salt and pepper noise in the three-channel depth image obtained in the previous step, and then a Gaussian filter function is used to remove Gaussian noise in the three-channel depth image;

[0054] Step 2-2: A bilateral filter function is used to process the three-channel depth image, preserving the edge information in the three-channel depth image;

[0055] Step 2-3: The resolution of the three-channel depth image is fixed to a uniform width and height, and in this embodiment, the fixed resolution size is 192 pixels x 192 pixels, and then the three-channel depth image is standardized, as shown in the following formula (1):

[0056]

[0057] In formula (1), img is a three-channel image matrix, img.mean is the mean of all images in three channels, and img.std is the standard deviation of all images in three channels.

[0058] Step 3: The detail features of the depth image obtained in the previous step are extracted using a basic convolutional neural network, and the features at different scales are sampled and spliced to obtain multi-scale fusion features of the depth image, specifically including the following steps:

[0059] Step 3-1: MobileNetV2 is used as the basic convolutional neural network structure for bottom feature extraction, and the depth image preprocessed in the previous step is input into 6 serial depth separable convolutional neuron components to obtain 6 layers of feature representations {f1, f2, f3, f4, f5, f6} at different sizes;

[0060] Step 3-2: For the features of 6 different layers, the middle k layer features are selected as the fusion candidate layer (in this embodiment, k is 4), and then the to-be-fused features of the next layer are respectively subjected to the deconvolution upsampling operation to obtain features with the same size as the features of the previous layer, and the two are added to obtain new features of the adjacent upper and lower layers, as shown in the following formula (2), the features of the i-1 layer fi i-1 are expanded in size through deconvolution to be the same as the features of the i layer fi i , and then added to the features of the i layer as new features:

[0061] f i i-1 = f i + TransConv(f i-1 ) (2)

[0062] The overall fused features F are calculated as shown in the following formula (3):

[0063]

[0064] Step 3-3: The obtained fused features F are input into the convolutional neurons to further fuse the features of different layers and different sizes to obtain the final fused features F.

[0065] Step 4: The fused features obtained in the previous step are respectively input into the human body box coordinate prediction network and the human body key point coordinate prediction network to obtain the coordinate prediction values of the human body box and the key points, which specifically includes the following steps:

[0066] Step 4-1: The fused features F obtained in the previous step are input into the human body box (bounding box, bbox) coordinate prediction network, and a set of 1*1 convolution is performed to obtain the human body box feature vector f bbox .

[0067] Step 4-2: The human body box feature vector f bbox is input into a set of 1*1 convolutional neural network to extract the feature vector f center of the human body center point, and the human body center point prediction result is obtained through the Softmax function, indicating that the feature point is the center point position of the human body;

[0068] Step 4-3: The human body box feature vector f bbox is input into a set of 1*1 convolutional neural network to extract the human body box width and height (wh) feature vector f wh , and the prediction values of the width and height of the human body box are obtained through the Softmax function, so as to obtain the human body box region;

[0069] Step 4-4: The fusion feature F is input into the human keypoint coordinate prediction network, and a set of convolutional neural networks is used to extract the human keypoint (kps) feature vector f kps ;

[0070] Step 4-5: According to the human keypoint feature vector f kps , a set of convolutional neural networks with 34 convolutional channels is used to extract the human keypoint coordinate (coord) vector f coord , and the Softmax function is used to obtain the prediction results of the human keypoint coordinates, which represent the horizontal and vertical coordinates of the 17 key points of the human body.

[0071] Step 4-6: In order to reduce errors, the human keypoint feature vector f kps is input into a set of convolutional neural networks with 2 convolutional channels, and the Softmax function is used to obtain the offset of the human keypoint coordinate value, which is added to the horizontal and vertical coordinates of the 17 key points of the human body obtained in step 4-5.

[0072] Step 5: Post-processing and correction of the coordinate prediction value to obtain the human frame coordinate and keypoint coordinate value in the original image size, which includes the following steps:

[0073] The human frame region obtained in step 4-3 and the coordinate value of the 17 key points of the human body obtained in step 4-6 are scaled and transformed. During scaling and transformation, the scaling ratio is calculated according to the size of the fusion feature map and the size of the actual input image, and the coordinates are mapped to the original image to obtain the coordinate value in the original image size. Then, the offset value obtained in step 4-6 is scaled and added to the new coordinates as the corrected keypoint coordinates.

Claims

1. A lightweight multi-person keypoint detection method based on depth images, characterized in that, Includes the following steps: Step 1: Separate the multi-channels of all input depth images, layer the different channels according to the depth value, remove redundant foreground and background data in different channels, and obtain multi-channel depth maps in different depth ranges; Step 2: Clean the multi-channel depth map; Step 3: Use a basic convolutional neural network to extract detailed features from the depth image obtained in the previous step, and sample and stitch features at different scales to obtain multi-scale fusion features of the depth image. This includes the following steps: Step 3-1: Use a convolutional neural network structure to extract low-level features. Input the preprocessed depth image from the previous step into 6 serial depth-separable convolutional neuron components to obtain feature representations {f1, f2, f3, f4, f5, f6} at 6 different sizes. Step 3-2: For the features of 6 different layers, select the middle k layers as fusion candidate layers, and then perform deconvolution upsampling operation on the features to be fused in the next layer to obtain features with the same size as the features of the previous layer. Add the two to obtain new features of adjacent upper and lower layers. Sum all the obtained new features to calculate the overall fusion feature F. Step 3-3: Input the obtained fusion feature F into the convolutional neuron to further fuse features of different sizes from different layers to obtain the final fusion feature F; Step 4: Input the fused feature F obtained in the previous step into the human bounding box coordinate prediction network and the human keypoint coordinate prediction network respectively to obtain the coordinate prediction values ​​of the human bounding box and keypoints. This includes the following steps: Step 4-1: Input the fused feature F obtained in the previous step into the human bounding box coordinate prediction network, and obtain the human bounding box feature vector f after a set of 1*1 convolutions. bbox ; Step 4-2: Convert the human body bounding box feature vector f bbox The feature vector f is extracted from the center point of the human body by inputting it into a set of 1*1 convolutional neural networks. center The prediction result of the human body center point is obtained by the Softmax function, which indicates that the feature point is the location of the human body center point. Step 4-3: Convert the human body bounding box feature vector f bbox The input is fed into a 1x1 convolutional neural network to extract the human body bounding box width and height feature vector f. wh The predicted values ​​of the width and height of the human body bounding box are obtained through the Softmax function, and the human body bounding box region can be obtained from this. Step 4-4: Input the fused feature F into the human keypoint coordinate prediction network, and extract the human keypoint feature vector f through a set of convolutional neural networks. kps ; Steps 4-5: Based on the human body key point feature vector f kps The human body key point coordinate vector f is extracted using a convolutional neural network with 34 convolutional channels. coord The Softmax function is used to obtain the predicted coordinates of key points on the human body, which represent the horizontal and vertical coordinates of several key points on the human body. Steps 4-6: Convert the human body key point feature vector f kps The input is fed into a convolutional neural network with 2 channels, and then the offset of the human body key point coordinate values ​​is obtained through the Softmax function. These offsets are then superimposed on the horizontal and vertical coordinates of several human body key points obtained in steps 4-5. Step 5: Post-process and correct the predicted coordinates to obtain the human bounding box coordinates and key point coordinates at the original image size. This includes the following steps: The human body bounding box region obtained in step 4-3 and the coordinate values ​​of several key points of the human body obtained in step 4-6 are scaled. During the scaling transformation, the scaling ratio is calculated based on the size of the fused feature map and the size of the actual input image. The coordinates are mapped to the original image to obtain the coordinate values ​​under the original image size. Then, the offset values ​​obtained in step 4-6 are scaled and superimposed on the new coordinates as the corrected key point coordinates.

2. The lightweight multi-person keypoint detection method based on depth images as described in claim 1, characterized in that, Step 1 includes the following steps: Step 1-1: Separate the three channels of the input depth image to obtain individual channel maps c1, c2, and c3; Steps 1-2: For channel image c1, completely preserve the original image pixel values; For channel image c2, a background pixel range threshold {b} is used. min b max Perform foreground and background separation; if the threshold value is less than the lower limit b... min All pixels are set to 0, which is greater than the threshold limit b. max All pixels are set to 255 and saved as channel image c′2 to maximize the removal of redundant background information at the edges; For channel image c3, a human pixel range threshold {h} is used. min h max Perform human-scene separation; if the threshold value is less than the lower limit h... min All pixels are set to 0, which is greater than the upper threshold h. max All pixels are set to 255 and saved as channel image c′3 to maximize the preservation of human body region information; Steps 1-3: Re-merge the processed three channel images into {c1, c′2, c′3} as a new three-channel depth image.

3. The lightweight multi-person keypoint detection method based on depth images as described in claim 1, characterized in that, Step 2 includes the following steps: Step 2-1: Use the median filter function to remove salt-and-pepper noise from the three-channel depth image obtained in the previous step, and then use the Gaussian filter function to remove Gaussian noise from the three-channel depth image; Step 2-2: Use a bilateral filtering function to process the three-channel depth image and preserve the edge information in the three-channel depth image; Steps 2-3: Fix the resolution of the three-channel depth image to a uniform width and height, and then normalize the three-channel depth image as shown in equation (1): In equation (1), img is a three-channel image matrix, img.mean is the mean of all images in the three channels, and img.std is the standard deviation of all images in the three channels.

4. The lightweight multi-person keypoint detection method based on depth images as described in claim 1, characterized in that, In step 3-2, the features f of the (i-1)th layer i-1 After deconvolution, the feature size is expanded to be similar to the feature size of the i-th layer. i The same features are then added to the features of the i-th layer to form a new feature, as shown in equation (2) below: f i =f i +TransConv(f i-1 ) (2) In equation (2), TransConv() represents deconvolution, and the overall fused features are...

Images