A human figure segmentation method based on deep learning

Abstract

The present invention provides a method for human figure segmentation based on deep learning. An image is input into the backbone of a deep neural network to generate high-order features corresponding to the image. Subsequently, multiple parallel branches are connected to the feature output end of the neural network to respectively predict the center position of the human figure in the image's circumscribed rectangular frame, the width and height, and the vertical coordinate offset of the dividing line that divides the human figure relative to the center position of the human figure's circumscribed rectangular frame. The horizontal dividing line is drawn by adding the above-mentioned predicted coordinate offset and the vertical coordinate of the center position, and the human figure frame is divided into three parts, each corresponding to the head, upper body, and lower body of the human figure. Compared with the traditional human figure segmentation method that performs detection and segmentation separately and divides the area by artificially constructed features, the method of the present invention is simple in steps, easy to implement, and has higher segmentation accuracy. At the same time, it has a huge advantage in execution efficiency and is more conducive to the subsequent analysis of various appearance attributes of the human figure.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a human figure segmentation method based on deep learning. Background Art

[0002] Pedestrian attribute analysis is a challenging area in computer vision. Its essence is to identify pedestrians in videos and automatically extract their structured attributes, including gender, age, orientation, wearing hats, glasses, masks, top and pants colors, backpacks, and luggage. Analysis of complete human images focuses on capturing global features, neglecting local details. This results in poor recognition of attributes represented by most local features. To address this issue, pedestrian attribute analysis introduces human figure segmentation. This method divides the complete human figure image into several image blocks, each corresponding to a different region of the figure. This region-specific feature analysis effectively improves the recognition rate of pedestrian attributes. Currently, there are two mainstream human figure segmentation methods: one that segments the figure based on manually designed features, and the other that uses deep learning to construct a detection network for each region of the human figure, directly detecting each region to achieve segmentation. The former requires extracting the circumscribed rectangular area of the human figure from the original image before execution, which has relatively low efficiency and segmentation accuracy. Although the latter is more efficient, the segmentation of the human figure causes the loss of common features in the areas where the same parts of different human figures are located, which increases the difficulty of detecting network training. Summary of the Invention

[0003] The purpose of the present invention is to provide a human figure segmentation method based on deep learning, which integrates human figure segmentation and human figure detection into the same neural network, so that the detection of complete human figures and human figure segmentation can be carried out simultaneously, thereby improving the technical problem of segmentation accuracy and execution efficiency without increasing the difficulty of network training.

[0004] To solve the above technical problems, the present invention provides the following technical solution: a human figure segmentation method based on deep learning, comprising the following steps:

[0005] Step 1: The original image is preprocessed and then input into the backbone of the neural network;

[0006] Step 2: Extract high-order feature maps of various sizes generated by the backbone of the neural network to form a feature pyramid, and then fuse them from top to bottom;

[0007] Step 3: The fused features are input into the prediction block of the neural network. The prediction results of the four parallel convolutional layers at the end of the prediction block are used to obtain the 2D coordinates of the center of the human figure box, the confidence score, and the vertical coordinates of the head, upper body, and the upper body boundary within the box.

[0008] Step 4: Use diou-nms to filter redundant rectangular frames. For the retained human-shaped frames, a dividing line is drawn based on the vertical coordinates of the two dividing lines corresponding to the human-shaped frames to split the detected human-shaped frames into three blocks. Each block contains the head, upper body, and lower body of the target human.

[0009] As an optional implementation method, the backbone of the neural network can directly use the current mainstream backbone networks such as: vggnet, resnet, shufflenet, mobilenet, darknet, senet, etc.

[0010] As an optional implementation method, the neural network can directly use the current mainstream object detection networks such as SSD, YOLO, RetinaNet, CenterNet, FCOS, etc. for human detection, and at the same time add a branch in the detection block of the network to predict the vertical coordinates of the human head and the dividing line between the upper and lower body.

[0011] As an optional implementation, the filtering of redundant rectangular boxes can be replaced by conventional nms or improved softer nms.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] Compared to the segmented approach of first locating the human figure and then using manually designed features to achieve segmentation, using a neural network to simultaneously locate the vertical coordinates of the head and the dividing line between the upper and lower body after detecting the human figure's position can achieve faster and more accurate segmentation;

[0014] Compared with the detection network that detects different regions of a human figure separately, the detection network that directly detects the human figure has lower computational complexity and is easier to build and train. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a macroscopic structural diagram of a neural network;

[0016] Figure 2 It is a structural diagram of the prediction block;

[0017] Figure 3 It is a flow chart of the method of the present invention.

[0018] Specific implementation form

[0019] The following is a further description of an implementation of the present invention with reference to the accompanying drawings.

[0020] The present invention provides a human figure segmentation method based on deep learning. This method realizes parallel processing of human figure detection and segmentation through neural network, with high segmentation accuracy and significant efficiency improvement. The macroscopic structure diagram of the neural network is shown in Figure 1 The complete flowchart is shown in Figure 3 , the specific implementation steps are as follows:

[0021] 1) The original image is scaled to 640x480 by bilinear interpolation, and the pixel values are shrunk from integers in the range of 0 to 255 to floating-point numbers in the range of 0 to 1 by linear scaling.

[0022] 2) Here we use the Mobilenet v2 network as the backbone network, removing the last convolutional layer and pooling layer to simplify the network.

[0023] 3) The preprocessed image is fed into the simplified Mobilenet V2 network, and the feature maps of the output of the 3rd, 5th, and 7th bottleneck modules with sizes of 80x60, 40x20, and 20x10 are used to construct the feature pyramid.

[0024] 4) Feature fusion from top to bottom: the upper layer features are upsampled and then concatenated with the next layer features. The final fused feature map of size 80x60 is input to the prediction block for subsequent prediction.

[0025] 5) The prediction block contains a main path and a residual bypass. The main path consists of three 1x1 convolutional layers connected in series, and the residual bypass is a convolutional layer with a kernel size of 1x1. The structure diagram is shown in Figure 2 The fused features are added together through the outputs of the main road and the residual bypass, and then passed through four parallel 1x1 convolutional layers to predict the confidence of each point on the feature map as the center of the human figure box, the offset from the actual center, the width and height, and the offset of the vertical coordinates of the upper and lower dividing lines relative to the current point.

[0026] 6) Based on the above prediction results and the coordinates of each point on the feature map, the coordinates of the upper left and lower right corners of all predicted rectangles and the positions of the two dividing lines are obtained. Then, diou-nms is used to filter out invalid rectangles, retaining only one rectangle for each detected human figure. The mathematical expression of diou-nms is as follows:

[0027]

[0028]

[0029]

[0030] In the above formula, s i is the confidence of the rectangle, B max ,B i They represent the rectangular box with the highest confidence and the other rectangular boxes respectively, and λ represents the confidence of nms. x ,hx ,w y ,h y ,w xy ,h xy Respectively represent the width and height of the rectangular box x, the width and height of the rectangular box y, and the width and height of the overlapping part of x and y. xy ,c xy denotes the Euclidean distance of the x and y center points and the length of the diagonal of the circumscribed rectangle, respectively, and μ is a weight factor. diou-nms adjusts the confidence of other rectangular boxes by calculating the diou between the rectangle with the highest confidence and the other rectangular boxes.

[0031] 7) For the last retained human frame i, the coordinates of the predicted center are and the corresponding dividing line offset Two dividing lines can be obtained

[0032]

[0033]

[0034] According to the scaling ratio of the current feature map and the original image, the coordinates of the detected rectangular frame and the corresponding dividing line are mapped to the original image to complete the detection and segmentation of the human figure.

[0035] Although the above describes the specific implementation methods of the present invention, it does not limit the scope of protection of the present invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A human figure segmentation method based on deep learning, characterized in that: The following steps are included: Step 1: The original image is preprocessed and then input into the backbone of the neural network; Step 2: Extract high-order feature maps of various sizes generated by the backbone of the neural network to form a feature pyramid, and then fuse them from top to bottom; Step 3: The fused features are input into the prediction block of the neural network. The prediction results of the four parallel convolutional layers at the end of the prediction block are used to obtain the 2D coordinates of the center of the human figure box, the confidence score, and the vertical coordinates of the head, upper body, and the upper body boundary within the box. Step 4: Use diou-nms to filter redundant rectangular frames. For the retained human-shaped frames, divide the detected human-shaped frames into three blocks based on the vertical coordinates of the two dividing lines corresponding to the human-shaped frames. Each block contains the head, upper body, and lower body of the target human. The preprocessing of the original image specifically includes: scaling the original image to 640x480 by bilinear interpolation, and shrinking the pixel values from integers in the range of 0 to 255 to floating-point numbers in the range of 0 to 1 by linear scaling; The prediction block includes a backbone path and a residual bypass path. The backbone path consists of three serially connected 1x1 convolutional layers, and the residual bypass is a convolutional layer with a kernel size of 1x1. The fused features are passed through the backbone path and the residual bypass, and the output features are added and then passed through four parallel 1x1 convolutional layers to predict the confidence of each point on the feature map as the center of the human figure box, the offset from the actual center, the width and height, and the offset of the vertical coordinates of the upper and lower dividing lines relative to the current point.

2. The method for human figure segmentation based on deep learning according to claim 1, characterized in that: The backbone of the neural network uses the current mainstream backbone networks: vggnet, resnet, shufflenet, mobilenet, darknet, and senet.

3. The method for human figure segmentation based on deep learning according to claim 1, characterized in that: The neural network uses the current mainstream target detection networks SSD, YOLO, RetinaNet, CenterNet, and FCOS for human detection, and adds a branch in the detection block of the network for predicting the vertical coordinates of the human head and the boundary between the upper and lower body.

4. The method for human figure segmentation based on deep learning according to claim 1, characterized in that: The redundant rectangular frames are filtered using conventional nms or improved softer nms.

Images