Human body contour point positioning method and system, device, storage medium

Abstract

The application discloses a human body contour point positioning method and system, equipment and a storage medium, comprising constructing a human body contour point positioning neural network model; obtaining a training data set, and training the neural network model using the training data set; inputting an image with a human body into the neural network model to obtain human body skeleton point coordinates and human body contour point coordinates on the image; and obtaining contour points after block connection of the human body based on the human body skeleton point coordinates and the human body contour point coordinates on the image. The neural network model of the application predicts contour points independent of blocks, and subsequent block connection is performed through rules. The contour points independent of blocks are clearly defined, and the contour point positions are also continuous when the human body posture continuously changes. These advantages of the contour points independent of blocks are beneficial to the training of the neural network to obtain good prediction results.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and specifically relates to a method, system, device, and storage medium for locating human body contour points. Background Technology

[0002] With the development of artificial intelligence technology, it has been widely used. For example, when an image of a user is captured by an image acquisition device, the key points of the user's human skeleton can be given based on the image. However, the detection results obtained based on this detection method are often not accurate enough. The detection results only include the coordinates of a few key points associated with the user's skeleton, and cannot give the coordinates of more and more detailed human contour points distributed in various parts of the user's body.

[0003] In a certain application scenario, for human image editing operations such as body shaping and slimming, it is necessary to accurately obtain the coordinates of multiple human skeletal points and multiple human contour points of the current user. However, the existing detection methods obtain too few key points in the detection results, and most of the key points obtained are the user's skeletal key points. Thus, the detection results can only display the user's two-dimensional skeleton, resulting in a low user experience. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a method, system, device, and storage medium for locating human body contour points, the method comprising:

[0005] Construct a neural network model for human contour point localization;

[0006] Obtain a training dataset and use the training dataset to train the neural network model;

[0007] The neural network model is input into an image containing a human body to obtain the coordinates of human skeleton points and human contour points on the image.

[0008] Based on the coordinates of human skeleton points and human contour points on the image, the contour points of the connected human body blocks are obtained.

[0009] Preferably, the step of training the neural network model based on the image containing the human body includes:

[0010] The neural network model includes a backbone network and two headnet networks, namely a human contour point headnet network and a human skeleton point headnet network.

[0011] Freeze the human body contour point headnet network, and train the backbone network and the human body skeleton point headnet network using the training dataset;

[0012] After training is completed, the backbone network and the human skeleton point headnet network are frozen, and the human contour point headnet network is trained using the training dataset.

[0013] After training is complete, the backbone network and the two headnet networks are trained using the training dataset.

[0014] Preferably, the training dataset includes a positive sample dataset and a negative sample dataset. The positive sample dataset is a collection of images labeled with human bounding boxes, human skeleton points, and human contour points, while the negative sample dataset is a collection of images without human figures.

[0015] Preferably, inputting an image containing a human body into the neural network model to obtain the coordinates of human skeleton points and human contour points on the image includes:

[0016] Obtain the human body bounding box from the image containing the human body, modify the human body bounding box to the same aspect ratio as the neural network model, and scale the image to a fixed size;

[0017] The fixed-size image is input into the neural network model to obtain heat maps of N human skeleton points and M human contour points;

[0018] Based on the heatmap of the human skeleton points and the heatmap of the human contour points, the coordinates of the human skeleton points, the confidence scores of the human skeleton points, the coordinates of the human contour points, and the confidence scores of the human contour points on the image of a fixed size are obtained respectively.

[0019] Based on the coordinates of human skeleton points and human contour points on the image of a fixed size, the coordinates of human skeleton points and human contour points on the image are obtained.

[0020] Preferably, obtaining the coordinates of human skeletal points, the confidence scores of human skeletal points, the coordinates of human contour points, and the confidence scores of human contour points on the image of a fixed size, based on the heatmap of the human skeletal points and the heatmap of the human contour points, includes:

[0021] Based on the heatmap of the human skeleton points and the heatmap of the human contour points, the position of the pixel with the largest response value on each single-channel heatmap is obtained.

[0022] A rectangular region is divided with the pixel with the largest response value as the center. The weighted average of the coordinates of the rectangular region is used as the coordinates of the human skeleton point or human contour point on the single-channel heatmap, and the largest response value is the corresponding confidence level.

[0023] Based on the human skeleton points or human contour points on the single-channel heatmap, the coordinates of the human skeleton points and human contour points on the image of a fixed size are calculated using the following formula:

[0024] x_netin=x_hm / W hm *W;

[0025] y_netin=y_hm / H hm *H;

[0026] In the formula: x_netin and y_netin are the coordinates of human skeleton points or human contour points, respectively; W hm H hm W and H are the width and height of the heatmap of the human skeleton points or the heatmap of the human contour points, respectively; W and H are the width and height of the fixed-size image, respectively.

[0027] Preferably, the contour points obtained after connecting the human body blocks are obtained based on the coordinates of the human skeleton points and the coordinates of the human contour points on the image, including:

[0028] The human body blocks include the shoulder and neck block, chest, waist and hip block, right upper arm block, right forearm block, right palm block, left upper arm block, left forearm block, left palm block, right thigh block, right calf block, right foot block, left thigh block, left calf block and left foot block;

[0029] Based on the coordinates of human skeleton points and human contour points on the image, the right upper arm block, right forearm block, and right palm block are connected to obtain an independent right arm contour.

[0030] Similarly, the left upper arm block, left forearm block, and left palm block are connected to obtain an independent left arm contour; the right thigh block, right calf block, and right foot block are connected to obtain an independent right leg contour; and the left thigh block, left calf block, and left foot block are connected to obtain an independent left leg contour.

[0031] Based on the coordinates of human skeletal points and human contour points of the shoulder and neck area and the chest, waist and hip area, the front and back sides of the shoulder and neck area and the chest, waist and hip area are determined respectively.

[0032] If the shoulder and neck area is in the front view, the outline of the independent right arm is connected to the left outline of the shoulder and neck area and the left outline of the chest, waist and hip area, and the outline of the independent left arm is connected to the other side outline of the shoulder and neck area and the other side outline of the chest, waist and hip area.

[0033] If the shoulder and neck area is the back, the outline of the independent left arm is connected to the left outline of the shoulder and neck area and the left outline of the chest, waist and hip area, and the outline of the independent right arm is connected to the other side outline of the shoulder and neck area and the other side outline of the chest, waist and hip area.

[0034] If the chest, waist, and hip area is in the front view, the left contour of the independent right leg contour is connected to the left contour of the chest, waist, and hip area, and the right contour of the independent right leg contour is connected to the crotch point of the chest, waist, and hip area; the right contour of the independent left leg contour is connected to the right contour of the chest, waist, and hip area, and the left contour of the independent left leg contour is connected to the crotch point of the chest, waist, and hip area.

[0035] If the chest, waist, and hip area is the back view, the left contour of the independent left leg is connected to the left contour of the chest, waist, and hip area, and the right contour of the independent left leg is connected to the crotch point of the chest, waist, and hip area; the right contour of the independent right leg is connected to the right contour of the chest, waist, and hip area, and the left contour of the independent right leg is connected to the crotch point of the chest, waist, and hip area.

[0036] By connecting the coordinates of the human skeleton points and the coordinates of the human contour points on the image, the contour points of the connected human body blocks are obtained.

[0037] Preferably, based on the coordinates of human skeletal points and human contour points of the shoulder and neck area and the chest, waist, and hip area, the front and back sides of the shoulder and neck area and the chest, waist, and hip area are determined respectively:

[0038] When the confidence scores of the contour points and skeletal points of the shoulder and neck area and the chest, waist and hip area are both greater than a preset threshold, the front and back orientation of the shoulder and neck area and the chest, waist and hip area are determined by the distance between the left and right contour points and the left and right skeletal points.

[0039] When the confidence level of the skeletal points of the shoulder and neck region and the chest, waist and hip region is greater than a preset threshold, the skeletal points of the shoulder and neck region and the chest, waist and hip region are used to determine the front and back orientation of the shoulder and neck region and the chest, waist and hip region.

[0040] This invention provides a system for locating human body contour points, comprising:

[0041] Build modules are used to construct neural network models for human contour point localization;

[0042] The training module is used to acquire a training dataset and train the neural network model using the training dataset.

[0043] The acquisition module is used to input an image containing a human body into the neural network model to obtain the coordinates of human skeleton points and human contour points on the image;

[0044] The calculation module is used to obtain the contour points of the human body after connecting the human body blocks based on the coordinates of the human skeleton points and the coordinates of the human body contour points on the image.

[0045] An embodiment of the present invention provides an apparatus comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a program, and when the program is executed by the processing unit, the processing unit performs the method described above.

[0046] An embodiment of the present invention provides a computer-readable storage medium storing a computer program executable by a device, which, when run on the device, causes the device to perform the method described above.

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0048] The neural network model of this invention predicts block-independent contour points, and then connects these blocks using rules. The block-independent contour points are clearly defined, and their positions change continuously as the human body posture changes. These advantages of block-independent contour points are beneficial for training the neural network to obtain good prediction results. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the human body contour point positioning method of the present invention;

[0050] Figure 2 This is a schematic diagram of the human skeleton points in the human contour point positioning method of the present invention.

[0051] Figure 3-5 This is a schematic diagram of the front, back and side views of the block-independent contour points of the human body contour point positioning method of the present invention.

[0052] Figure 6-8 This is a schematic diagram of the front, back and side views of the contour points of the block connection in the human body contour point positioning method of the present invention.

[0053] Figure 9 This is a schematic diagram of the central axis direction of the block-independent contour of the human body contour point positioning method of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Reference Figure 1 A method, system, device, and storage medium for locating human body contour points, the method comprising:

[0056] Construct a neural network model for human contour point localization;

[0057] Specifically, the neural network model includes a backbone network and two headnet networks, namely a human contour point headnet network and a human skeleton point headnet network. The backbone is constructed by using residual modules and downsampling stacks, while the headnets are constructed by depthwise separable convolutions and upsampling stacks.

[0058] Obtain the training dataset and use it to train the neural network model;

[0059] Specifically, the human contour point headnet network is frozen, and the backbone network and human skeleton point headnet network are trained using the training dataset;

[0060] After training, freeze the backbone network and the human skeleton point headnet network, and use the training dataset to train the human contour point headnet network.

[0061] After training is complete, the backbone network and two headnet networks are trained using the training dataset.

[0062] Furthermore, the training dataset includes a positive sample dataset and a negative sample dataset. The positive sample dataset is a collection of images with human bounding boxes, human skeleton points, and human contour points labeled, while the negative sample dataset is a collection of images without human figures.

[0063] The positive sample dataset collection process includes: obtaining a random human bounding box (bbox1) based on the labeled human region bounding boxes and human skeleton points (which may only contain part of the limbs, such as only the upper body or the whole body); expanding bbox1 by a certain proportion to obtain bbox2; randomly scaling, rotating, and translating bbox2 to obtain bbox3; and transforming the image in bbox3 into an image of network input size (H*W*C) image_netin by affine transformation (transforming the four vertices of bbox3 to the four vertices of the neural network model input image image_netin), simultaneously transforming the ground truth (GT) values ​​(point coordinates) of the relevant human contour points and human skeleton points. Based on the positions of the human contour points and human skeleton points on their respective heatmaps, the GT values ​​of their respective heatmaps are generated (the GT value of the human skeleton point's heatmap is denoted as heatmap_skl_gt, with a size of H). hm *W hm *N; The ground truth (GT) value of the human body contour points is denoted as heatmap_bdycntr_gt, with a size of H. hm *W hm *M). Specifically, for human contour points, the heatmap peak values ​​of occluded points and visible points are differentiated. Specifically, the heatmap peak value of occluded points is slightly smaller than that of visible points. For a keypoint (x_hm, y_hm), the value of a pixel (x, y) on its heatmap is:

[0064]

[0065] in:

[0066] σ is a parameter of the generated Gaussian heatmap;

[0067] w max The parameter used to adjust the peak value of the Gaussian heatmap.

[0068] The negative sample dataset collection includes: randomly selecting an image from the negative sample image set, extracting a region and scaling it to the size required by the input image of the neural network model, with all values ​​of the human contour points and human skeleton points on the heatmap being 0.

[0069] During training, positive and negative samples are mixed in a certain proportion.

[0070] Furthermore, the loss functions used for training are L2 loss for both the human skeleton point heatmap and the human contour point heatmap; the total loss function is:

[0071] loss = loss_skl + loss_bdycntr

[0072] Wherein, loss_skl is the loss function of the human skeleton point heatmap, which adopts L2 loss; loss_bdycntr is the loss function of the human contour point heatmap, which adopts L2 loss. Occluded points and visible points are treated differently in terms of their contribution to the loss to indicate that occluded points have greater uncertainty in annotation than visible points.

[0073] Specifically, loss_bdycntr is:

[0074]

[0075] Among them, w_kps k This is the weighted value of the k-th point of the human body contour in the loss. For visible and non-existent points, this value is 1, and for occluded points, this value is 0.8.

[0076] Input an image containing a human body into a neural network model to obtain the coordinates of human skeleton points and human contour points in the image;

[0077] Specifically, the human body bounding box in the image containing the human body is obtained, and the human body bounding box is modified to the same aspect ratio as the neural network model and the image is scaled to a fixed size;

[0078] By inputting images of a fixed size into a neural network model, heatmaps of N human skeleton points and M human contour points are obtained.

[0079] Based on the heatmaps of human skeleton points and human contour points, the coordinates of human skeleton points, the confidence scores of human skeleton points, the coordinates of human contour points, and the confidence scores of human contour points are obtained on images of fixed sizes.

[0080] Based on the coordinates of human skeleton points and human contour points on a fixed-size image, obtain the coordinates of human skeleton points and human contour points on the image.

[0081] Specifically, based on the heatmaps of human skeletal points and human contour points, the coordinates of human skeletal points, confidence scores of human skeletal points, coordinates of human contour points, and confidence scores of human contour points on images of fixed sizes are obtained, including:

[0082] Based on the heatmaps of human skeleton points and human contour points, the positions of the pixels with the largest response values ​​on each single-channel heatmap are obtained.

[0083] Divide a rectangular area centered on the pixel with the largest response value, and use the weighted average of the coordinates of the rectangular area as the coordinates of the human skeleton point or human contour point on the single-channel heatmap, with the largest response value being the corresponding confidence level.

[0084] Based on human skeleton points or human contour points on a single-channel heatmap, the coordinates of human skeleton points and human contour points on a fixed-size image are calculated using the following formula:

[0085] x_netin=x_hm / W hm *W;

[0086] y_netin=y_hm / H hm *H;

[0087] In the formula: x_netin and y_netin are the coordinates of human skeleton points or human contour points, respectively; W hm H hm W and H represent the width and height of the heatmap of human skeleton points or human contour points, respectively; W and H represent the width and height of a fixed-size image, respectively.

[0088] Based on the coordinates of human skeleton points and human contour points in the image, the contour points of the connected human body blocks are obtained.

[0089] Specifically, based on the coordinates of human skeleton points and human contour points in the image, the contour points obtained after connecting human body blocks include:

[0090] The human body blocks include the shoulder and neck block, chest, waist and hip block, right upper arm block, right forearm block, right palm block, left upper arm block, left forearm block, left palm block, right thigh block, right calf block, right foot block, left thigh block, left calf block and left foot block;

[0091] Based on the coordinates of human skeleton points and human contour points on the image, the right upper arm block, right forearm block, and right palm block are connected to obtain an independent right arm contour.

[0092] Similarly, connect the left upper arm block, left forearm block, and left palm block to obtain the independent left arm outline; connect the right thigh block, right calf block, and right foot block to obtain the independent right leg outline; and connect the left thigh block, left calf block, and left foot block to obtain the independent left leg outline.

[0093] Based on the coordinates of human skeletal points and human contour points in the shoulder and neck area and the chest, waist and hip area, the front and back views of the shoulder and neck area and the chest, waist and hip area are determined respectively.

[0094] If the shoulder and neck area is in the front view, the outline of the independent right arm is connected to the left outline of the shoulder and neck area and the left outline of the chest, waist and hip area. The outline of the independent left arm is connected to the other side outline of the shoulder and neck area and the other side outline of the chest, waist and hip area.

[0095] If the shoulder and neck area is the back, the outline of the independent left arm is connected to the left outline of the shoulder and neck area and the left outline of the chest, waist and hip area at the same time, and the outline of the independent right arm is connected to the other side outline of the shoulder and neck area and the other side outline of the chest, waist and hip area at the same time.

[0096] If the chest, waist, and hip area is in the front view, the left contour of the independent right leg is connected to the left contour of the chest, waist, and hip area, and the right contour of the independent right leg is connected to the crotch point of the chest, waist, and hip area; the right contour of the independent left leg is connected to the right contour of the chest, waist, and hip area, and the left contour of the independent left leg is connected to the crotch point of the chest, waist, and hip area.

[0097] If the chest, waist, and hip area is the back view, the left contour of the independent left leg is connected to the left contour of the chest, waist, and hip area, and the right contour of the independent left leg is connected to the crotch point of the chest, waist, and hip area; the right contour of the independent right leg is connected to the right contour of the chest, waist, and hip area, and the left contour of the independent right leg is connected to the crotch point of the chest, waist, and hip area.

[0098] By connecting the coordinates of the human skeleton points and the human contour points on the image, the contour points of the connected human body blocks are obtained.

[0099] Furthermore, based on the coordinates of human skeletal points and human contour points in the shoulder and neck area and the chest, waist, and hip area, the front and back views of the shoulder and neck area and the chest, waist, and hip area are determined respectively:

[0100] When the confidence scores of the contour points and skeletal points of the shoulder and neck area and the chest, waist and hip area are both greater than the preset threshold, the front and back orientation of the shoulder and neck area and the chest, waist and hip area is determined by the distance between the left and right contour points and the left and right skeletal points.

[0101] When the confidence level of the skeletal points in the shoulder and neck region and the thoracolumbar region is greater than a preset threshold, the skeletal points in the shoulder and neck region and the thoracolumbar region are used to determine the frontal and back orientation of the shoulder and neck region and the thoracolumbar region.

[0102] Example 1

[0103] The human skeleton points used in this embodiment are a set of 25 2D human keypoints. For details on the definition of these points, please refer to [link to documentation]. Figure 2 And Table 1.

[0104] Table 1

[0105] Point number Point meaning Point number Point meaning 0 Right hip joint 13 Right ear point 1 Right knee joint 14 tip of nose 2 Right ankle joint 15 Left ear point 3 Left hip joint 16 Right thumb base joint 4 Left knee joint 17 right little finger base joint 5 Left ankle joint 18 left thumb base joint 6 Top vertex 19 left little finger base joint 7 Right shoulder joint 20 bottom of right heel 8 Right elbow joint point 21 center point of the base of the right toe 9 right wrist point 22 left heel bottom 10 Left shoulder joint 23 Center point of the base of the left toe 11 Left elbow joint 24 Adam's apple 12 left wrist point

[0106] The individual human contour points for each block are fixed points, involving 14 blocks and a total of 67 key points. The 14 blocks are: 1. Shoulder and Neck Block, 2. Chest, Waist, and Hips Block, 3. Right Upper Arm Block, 4. Right Forearm Block, 5. Right Palm Block, 6. Left Upper Arm Block, 7. Left Forearm Block, 8. Left Palm Block, 9. Right Thigh Block, 10. Right Lower Leg Block, 11. Right Foot Block, 12. Left Thigh Block, 13. Left Lower Leg Block, and 14. Left Foot Block. For details of the corresponding points for each of the 67 blocks, please refer to [link to details]. Figure 3-5 And Table 2.

[0107] Table 2

[0108] Independent Block Corresponding contour point number 1. Neck and shoulder area 0～2，64～66 2. Chest, waist, and hip areas 16～22，44～50，33 3. Right upper arm block 4～6，12～15 4. Right forearm area 7～8，10～11 5. Right palm area 9 6. Left upper arm block 51～54，60～62 7. Left forearm area 55～56，58～59 8. Left palm area 57 9. Right thigh area 23～24，31～32 10. Right calf area 25～26，29～30 11 Right foot area 27～28 12 Left thigh area 34～35，42～43 13. Left calf area 36～37，40～41 14. Left foot area 38～39

[0109] Independent human body contour points in blocks do not have the following connections (some of the following connections are required in certain operations such as slimming arms): connection between shoulder and neck contour and upper arm contour, connection between upper arm contour and chest, waist and hip contour, connection between chest, waist and hip contour and thigh contour, etc.

[0110] Connecting human contour points to blocks refers to completing the connection of human contour points between individual blocks, such as... Figure 6-8 .

[0111] Some differences between block-independent human contour points and block-connected human contour points: Block-independent contour points are defined on a fixed side (left / right) of a fixed block. For example, point 6 is the elbow point on the left contour of the right arm. Block-connected contour points are not fixed on a fixed side of a fixed limb. For example, although point 6 is also an elbow contour point, it is not specified whether it is on the right arm or either arm. Point 6 is the elbow point on the outer contour of the arm connected to shoulder contour point 2. (This concept can be applied when the body is turned 90 degrees to the side. As the body moves slightly to the left or right, a point that was defined on the left arm one moment may be defined on the right arm the next moment.)

[0112] Subsequently, during neural network prediction, we instructed the neural network to predict block-independent points, while the connections between blocks were made using a rule-based strategy. The advantages are: block-independent contour points are clearly defined, and their positions change continuously as the human posture changes (unlike connected contour points which experience abrupt position changes, such as when the body is near a 90-degree sideways angle). These advantages of block-independent contour points facilitate neural network training to achieve better prediction results. The human contour point information here includes not only coordinate information but also occlusion information. There are three values ​​for occlusion markers: visible point, occluded point, and non-existent point (for non-existent points, the ground truth (GT) of the subsequent network heatmap response value will be set to 0).

[0113] A neural network model for locating human body contour points was constructed, and the original dataset was collected.

[0114] The original image set A is a collection of images with accurate human bounding boxes, human skeleton points, and human contour points (some images do not have human contour points labeled); the original image set B is a collection of images without human figures.

[0115] Positive Sample Creation: Based on the labeled human body bounding boxes and human skeleton points, obtain a random human bounding box (which may only contain part of the limbs, such as only the upper body or the whole body) bbox1; expand bbox1 by a certain proportion to obtain bbox2; randomly scale, rotate, and translate bbox2 to obtain bbox3; perform an affine transformation on the image in bbox3 (transforming the 4 vertices of bbox3 to the 4 vertices of the network input image image_netin) to create an image image_netin of network input size (H*W*C), simultaneously transforming the ground truth (GT) values ​​(point coordinates) of the relevant human contour points and human skeleton points. Based on the positions of the human contour points and human skeleton points on their respective heatmaps, generate the GT values ​​of their respective heatmaps (the GT value of the human skeleton point's heatmap is denoted as heatmap_skl_gt, with a size of H). hm *W hm *N; The ground truth (GT) value of the human body contour points is denoted as heatmap_bdycntr_gt, with a size of H. hm *W hm *M). Specifically, for human contour points, the heatmap peak values ​​of occluded points and visible points are differentiated. Specifically, the heatmap peak value of occluded points is slightly smaller than that of visible points. For a keypoint (x_hm, y_hm), the value of a pixel (x, y) on its heatmap is:

[0116]

[0117] in:

[0118] σ is a parameter of the generated Gaussian heatmap;

[0119] w max The parameter used to adjust the peak value of the Gaussian heatmap.

[0120] Negative sample creation: Randomly select an image from the negative sample image set, extract a region and scale it to the size required by the network input image, and set all the values ​​of the human contour points and human skeleton points on the heatmap to 0.

[0121] The size of heatmap_skl_gt is (H hm *W hm *N);

[0122] The size of heatmap_bdycntr_gt is (H hm *W hm *M);

[0123] During training, positive and negative samples are mixed in a certain proportion.

[0124] Specifically, the human contour point localization neural network model adopts an encoder-decoder combination, using one backbone network plus two headnet networks. One headnet (headnet_skl) outputs the heatmap_skeleton of human skeleton points, and the other headnet (headnet_bdycntr) outputs the heatmap_contour of human contour points.

[0125] Specifically, Backbone is mainly constructed by using residual modules and downsampling stacks, while HeadNet is constructed by depthwise separable convolutions and upsampling stacks.

[0126] During training, both the human skeleton point heatmap and the human contour point heatmap loss functions use L2 loss; the total loss function is:

[0127] loss = loss_skl + loss_bdycntr

[0128] Wherein, loss_skl is the loss function of the human skeleton point heatmap, which adopts L2 loss; loss_bdycntr is the loss function of the human contour point heatmap, which adopts L2 loss. Occluded points and visible points are treated differently in terms of their contribution to the loss to indicate that occluded points have greater uncertainty in annotation than visible points.

[0129] Specifically, loss_bdycntr is:

[0130]

[0131] Among them, w_kps k This is the weighted value of the k-th point of the human body contour in the loss. For visible and non-existent points, this value is 1, and for occluded points, this value is 0.8.

[0132] Furthermore, the training process includes:

[0133] Phase 1: Freeze the headnet with human contour points, using a dataset with human skeleton point annotations and a dataset without human figures, and train the headnet network parameters only for the backbone and human skeleton points;

[0134] Phase 2: Freeze the headnet with backbone and human skeleton points, unfreeze the headnet with human contour points, and train the headnet with human contour points using a training dataset containing human contour points and a dataset without human figures.

[0135] Phase 3: Unfreeze the backbone and two headnets, and fine-tune the entire network using data that includes both human contour points and human skeleton points, as well as data without human data.

[0136] After the model is trained, input an image (image_oriin) and the bounding box (bbox_oriin) of the human body in the image (consisting of four values: cx_oriin, cy_oriin, wid_oriin, and hei_oriin, representing the coordinates of the center point and the width and height of the bounding box, respectively) into the model. The output is:

[0137] The coordinates of the block-independent human contour points bdycntrsep_coord_oriin: an M*2 two-dimensional vector ([[x cs,0 y cs,0 ],[x cs,1 y cs,2 ],…,[x cs,M y cs,M ]]), where M is the number of contour points, which is 67 in this case;

[0138] Block-independent human contour point confidence bdycntrsep_conf: a one-dimensional vector of length M;

[0139] The coordinates of the human contour points connected by the block are bdycntrcon_coord_oriin: an M*2 two-dimensional vector ([[x cc,0 y cc,0 ],[x cc,1 y cc,2 ],…,[x cc,M y cc,M ]]);

[0140] Block connection human contour point confidence bdycntrcon_conf: a one-dimensional vector of length M;

[0141] The coordinates of human skeletal points skl_coord_oriin: an N*2 two-dimensional vector ([[xs,0 y s,0 ],[x s,1 y s,2 ],…,[x s,N y s,N ]]), where N is the number of bone points, which is 25 in this case;

[0142] The confidence score of human skeletal points skl_conf: a one-dimensional vector of length N.

[0143] The left and right outlines of the independent blocks mentioned below refer to left and right when viewed from the central axis of each block, such as... Figure 9 As shown, the specific process includes:

[0144] Step 1: Based on the input human bounding box bbox_oriin and the input image image_oriin, create the input data image_netin (an H*W*C image, where H is the height, W is the width, and C is the number of channels) for the human contour point localization network, and expand the human bounding box proportionally; further, use a method of lengthening the shorter side to make the aspect ratio of the human bounding box equal to the aspect ratio (W / H) of the network input image, thus obtaining the expanded human bounding box bbox_std(cx_std, cy_std, wid_std, hei_std) with a uniform aspect ratio; scale the image in bbox_std to a fixed-size image image_netin. Specifically, image_netin is obtained through affine transformation, and the corresponding affine transformation matrix is ​​aiffine_matrix (a 2*3 matrix): affine_matrix = [[wid_std / W, 0, cx_std-wid_std / 2], [0, hei_std / H, cy_std-hei_std / 2]]. Then, image_netin is obtained using the affine transformation of the image.

[0145] Note: A pixel coordinate (x_netin, y_netin) in image_netin corresponds to a pixel coordinate (x_oriin, y_oriin) in image_oriin.

[0146] [x_oriin,y_oriin] T =affine_matrix*[x_netin,y_netin,1] T .

[0147] Step 2: Input image_netin into the human contour point localization network to obtain a heatmap_skl of N human skeleton points (size H). hm *Whm *N) and M human contour points heatmap_cntr (size H) hm *W hm *M).

[0148] Step 3: Based on the human skeleton heatmap (heatmap_skl), obtain the coordinates (skl_coord_netin) and confidence scores (skl_conf) of the human skeleton points on the input image; based on the human contour heatmap (heatmap_cntr), obtain the coordinates (bdycntr_coord_netin) and confidence scores (bdycntr_conf) of the human contour points on the input image, which include:

[0149] Multichannel heatmap (H hm *W hm *N, or H hm *W hm A single channel heatmap (size H) on *M) hm *W hm This refers to the coordinates and confidence level of a 2D keypoint (skeleton point / contour point). The confidence level and coordinates on image_netin are obtained from a single-channel heatmap as follows:

[0150] For each single-channel heatmap, the location (x_max, y_max) of the pixel with the largest response value is obtained. Then, within a small region centered on this location (e.g., a 10*10 rectangular region), the weighted average of the coordinates of the points in the heatmap at each location is used as the final predicted location (x_hm, y_hm) of the 2D keypoint on the heatmap. At the same time, the maximum response value on the heatmap is used as the confidence level of the 2D point.

[0151] The 2D keypoint coordinates (x_netin, y_netin) on the input image from the network are obtained using the following conversion relationship:

[0152] x_netin=x_hm / W hm *W;

[0153] y_netin=y_hm / H hm *H.

[0154] Step 4: Calculate the coordinates of the human skeleton points (skl_coord_oriin) and the coordinates of the block-specific human contour points (bdycntrsep_coord_oriin) on the original input image (image_oriin). Obtain skl_coord_oriin and bdycntrsep_coord_oriin from skl_coord_netin and bdycntrsep_coord_netin using affine_matrix. The calculation process is as follows:

[0155] [x_oriin,y_oriin] T =affine_matrix*[x_netin,y_netin,1] T

[0156] Where (x_netin, y_netin) is a coordinate point in skl_coord_netin or bdycntrsep_coord_netin; (x_oriin, y_oriin) is the coordinate point on the original input image corresponding to (x_netin, y_netin) (the corresponding coordinate point in skl_coord_oriin or bdycntrsep_coord_oriin).

[0157] Step 5: Output the coordinates skl_coord_netin and confidence score skl_conf of the human skeleton points on the original input image, and the coordinates bdycntrsep_coord_netin and confidence score bdycntrsep_conf of the block-independent human contour points on the original input image.

[0158] Based on the model output, the steps to obtain the block-connected human contour points include:

[0159] Step 1: Connect the upper arm contour, forearm contour, and palm contour using simple connections to obtain the block-independent left arm contour and the block-independent right arm contour; connect the thigh contour, calf contour, and foot contour using simple connections to obtain the block-independent left leg contour and the block-independent right leg contour.

[0160] Specifically:

[0161] The left contour of the block-independent right upper arm is connected to the left contour of the block-independent right lower arm, and the right contour of the block-independent right upper arm is connected to the right contour of the block-independent right lower arm; similarly, the contours of the block-independent left upper arm and the block-independent left lower arm are connected.

[0162] Connect the left contour of the independent right thigh to the left contour of the independent right calf, and connect the right contour of the independent right thigh to the right contour of the independent right calf; similarly, connect the contours of the independent left thigh and the independent left calf.

[0163] The forearm block and palm block are connected unambiguously and naturally; the lower leg block and foot block are connected unambiguously and naturally.

[0164] Step Two: Based on the coordinates of the human skeletal points and human contour points in the shoulder and neck area and the chest, waist, and hip area, determine the front and back views of the shoulder and neck area and the chest, waist, and hip area respectively. Specifically:

[0165] When the confidence scores of the shoulder contour points (points 2 and 64) and the shoulder bone points (points 7 and 10) are greater than a pre-set threshold (e.g., 0.15), the front / back orientation of the shoulder region is determined by the distance between the left and right contour points and the left and right bone points (to obtain the connection relationship between the shoulder and hand). Specifically: if Distance(bdycntrsep_2, skl_7) + Distance(bdycntrsep_64, skl_10) <= Distance(bdycntrsep_2, skl_10) + Distance(bdycntrsep_64, skl_7), the shoulder region is biased towards the front; otherwise, the shoulder region is biased towards the back. The Distance(a, b) function refers to the distance between two points, a and b.

[0166] When the confidence scores of the contour points of the chest, waist, and hip area (points 21 (bdycntrsep_21) and 45 (bdycntrsep_45) and the hip bone points (points 0 (skl_0) and 3 (skl_3)) are greater than a pre-set threshold (e.g., 0.15), the front / back orientation of the hip area is determined by the distance between the left and right contour points and the left and right bone points (to obtain the connection relationship between the hip and the leg). Specifically: if Distance(bdycntrsep_21,skl_0) + Distance(bdycntrsep_45,skl_3) <= Distance(bdycntrsep_21,skl_3) + Distance(bdycntrsep_45,skl_0), the hip area is biased towards the front; otherwise, the hip area is biased towards the back.

[0167] When the above methods are insufficient to obtain the orientation of the front and back sides, further methods are employed, including:

[0168] When the confidence levels of the bone points of the shoulders and hips (point skl_7, point skl_10, point skl_0, point skl_3) are greater than a pre-set threshold (e.g., 0.15), use the bone points of the shoulders and hips to obtain the front-back bias of the shoulders and the front-back bias of the hips. Specifically, the following steps are performed in sequence:

[0169] Obtain the axis vector vec_shoulder_to_hip ([x_axis, y_axis]) from the midpoint of the shoulder bone points (0.5*skl_7 + 0.5*skl_10) to the midpoint of the hip bone points (0.5*skl_0 + 0.5*skl_3), with the value being (0.5*skl_0 + 0.5*skl_3) - (0.5*skl_7 + 0.5*skl_10);

[0170] Obtain the normal direction of the axis vector vec_shoulder_to_hip:

[0171] vertical_vec_shoulder_to_hip ([-y_axis, x_axis]);

[0172] Obtain the projected points of skl_7 and skl_10 on the normal direction vertical_vec_shoulder_to_hip of the axis vector vec_shoulder_to_hip: proj_skl_7 and proj_skl_10;

[0173] Obtain the projected points of skl_0 and skl_3 on the normal direction vertical_vec_shoulder_to_hip of the axis vector vec_shoulder_to_hip: proj_skl_0 and proj_skl_3;

[0174] Along the direction of vec_shoulder_to_hip: If proj_skl_7 is on the left side of proj_skl_10 (i.e., proj_skl_7 < proj_skl_10), then the shoulder area is biased towards the back; otherwise, the shoulder area is biased towards the front.

[0175] Along the direction of vec_shoulder_to_hip: If proj_skl_0 is on the left side of proj_skl_3 (i.e., proj_skl_0 < proj_skl_3), then the hip area is biased towards the back; otherwise, the hip area is biased towards the front.

[0176] Step 3: Connect the shoulder-neck contour and the arm contour, and connect the chest-waist-hip contour and the arm contour. Specifically:

[0177] If the shoulder area is oriented towards the front, the outline of the right arm (a separate block) connects simultaneously to the left outline of the shoulder and neck (a separate block) and the left outline of the chest, waist, and hips (a separate block). The other arm connects to the other side of the shoulder and neck outline and the other side of the chest, waist, and hips outline. That is:

[0178] Connect the left contour of the independent right arm block and the left contour of the independent shoulder and neck block (connect point 3 to point 2);

[0179] Connect the right contour of the independent right arm block and the left contour of the independent chest, waist and hip block (connect point 15 to point 16).

[0180] Connect the right contour of the block-independent left arm and the right contour of the block-independent shoulder and neck (connect point 63 to point 64);

[0181] Connect the left contour of the independent left arm block and the right contour of the independent chest, waist and hip block (connect point 51 to point 50).

[0182] If the shoulder area leans towards the back, then:

[0183] Connect the left contour of the block-independent left arm and the left contour of the block-independent shoulder and neck (connect point 51 to point 2);

[0184] Connect the right contour of the independent left arm block and the left contour of the independent chest, waist and hip block (connect point 63 to point 16).

[0185] Connect the right contour of the block-independent right arm and the right contour of the block-independent shoulder and neck (connect point 15 to point 64);

[0186] Connect the left contour of the independent right arm block and the right contour of the independent chest, waist and hip block (connect point 3 to point 50);

[0187] Step 4: Connect the chest, waist, hip contours with the leg contours. Specifically:

[0188] If the hip area is more forward, then:

[0189] Connect the left contour of the independent right leg block to the left contour of the independent chest, waist, and hip block; (connect point 23 to point 22).

[0190] Connect the right outline of the independent right leg of the block to the crotch point of the independent chest, waist and hip of the block; (connect point 32 to point 33);

[0191] Connect the right contour of the independent left leg block to the right contour of the independent chest, waist, and hip block; (connect point 43 to point 44);

[0192] Connect the left contour of the independent left leg of the block to the crotch point of the independent chest, waist and hip of the block; (connect point 34 to point 33);

[0193] If the hip area leans towards the back, then:

[0194] Connect the left contour of the independent left leg block to the left contour of the independent chest, waist, and hip block; (connect point 34 to point 22);

[0195] Connect the right outline of the independent left leg of the block to the crotch point of the independent chest, waist and hip of the block; (connect point 43 to point 33);

[0196] Connect the right contour of the independent right leg block to the right contour of the independent chest, waist, and hip block; (connect point 32 to point 44);

[0197] Connect the left contour of the independent right leg of the block to the crotch point of the independent chest, waist and hip of the block; (connect point 23 to point 33);

[0198] Step 5: Output the coordinates bdycntrcon_coord_netin and confidence level bdycntrcon_conf of the contour points after block connection (starting from the left contour of the shoulder and neck and ending at the right contour of the shoulder and neck).

[0199] This invention provides a system for locating human body contour points, comprising:

[0200] Build modules are used to construct neural network models for human contour point localization;

[0201] The training module is used to acquire the training dataset and use the training dataset to train the neural network model.

[0202] The acquisition module is used to input images containing human bodies into a neural network model to obtain the coordinates of human skeleton points and human contour points in the image.

[0203] The calculation module is used to obtain the contour points of the human body after connecting the human body blocks based on the coordinates of the human skeleton points and the human body contour points on the image.

[0204] An embodiment of the present invention provides an apparatus including at least one processing unit and at least one storage unit, wherein the storage unit stores a program, and when the program is executed by the processing unit, the processing unit performs the above-described method.

[0205] The present invention provides a computer-readable storage medium storing a computer program executable by a device, which, when run on the device, causes the device to perform the above-described method.

[0206] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for locating human body contour points, characterized in that, include: Construct a neural network model for human contour point localization; Obtain a training dataset and use the training dataset to train the neural network model; The neural network model is input with an image containing a human body to obtain the coordinates of human skeleton points and human contour points on the image; including: Obtain the human body bounding box from the image containing the human body, modify the human body bounding box to the same aspect ratio as the neural network model, and scale the image to a fixed size; The fixed-size image is input into the neural network model to obtain heat maps of N human skeleton points and M human contour points. Based on the heatmaps of the human skeleton points and the human contour points, the coordinates, confidence scores, coordinates, and confidence scores of the human skeleton points, contour points, and human contour points on the image of a fixed size are obtained, respectively. Specifically, this includes: obtaining the pixel position with the largest response value on each single-channel heatmap based on the heatmaps of the human skeleton points and the human contour points; dividing a rectangular region centered on the pixel position with the largest response value; using the weighted average of the coordinates of the points in the rectangular region as the coordinates of the human skeleton point or human contour point on the single-channel heatmap, with the largest response value being the corresponding confidence score; and calculating the coordinates of the human skeleton point and human contour point on the image of a fixed size based on the human skeleton point or human contour point on the single-channel heatmap, using the following formula: x_netin = x_hm / W hm * W: y_net = y_hm / H hm * H: In the formula: x_netin and y_netin are the coordinates of human skeleton points or human contour points, respectively; W hm H hm W and H are the width and height of the heatmap of the human skeleton points or the heatmap of the human contour points, respectively; W and H are the width and height of the fixed-size image, respectively. Based on the coordinates of human skeleton points and human contour points on the image of a fixed size, the coordinates of human skeleton points and human contour points on the image are obtained. Based on the coordinates of human skeleton points and human contour points on the image, the contour points of the connected human body blocks are obtained.

2. The human body contour point localization method as described in claim 1, characterized in that, The steps for training the neural network model based on the image containing the human body include: The neural network model includes a backbone network and two headnet networks, namely a human contour point headnet network and a human skeleton point headnet network. Freeze the human body contour point headnet network, and train the backbone network and the human body skeleton point headnet network using the training dataset; After training is completed, the backbone network and the human skeleton point headnet network are frozen, and the human contour point headnet network is trained using the training dataset. After training is complete, the backbone network and the two headnet networks are trained using the training dataset.

3. The human body contour point localization method as described in claim 2, characterized in that, The training dataset includes a positive sample dataset and a negative sample dataset. The positive sample dataset is a collection of images with human bounding boxes, human skeleton points, and human contour points labeled, while the negative sample dataset is a collection of images without human figures.

4. The human body contour point localization method as described in claim 1, characterized in that, Based on the coordinates of human skeleton points and human contour points in the image, the contour points obtained after connecting the human body blocks include: The human body blocks include the shoulder and neck block, chest, waist and hip block, right upper arm block, right forearm block, right palm block, left upper arm block, left forearm block, left palm block, right thigh block, right calf block, right foot block, left thigh block, left calf block and left foot block; Based on the coordinates of human skeleton points and human contour points on the image, the right upper arm block, right forearm block, and right palm block are connected to obtain an independent right arm contour. Similarly, the left upper arm block, left forearm block, and left palm block are connected to obtain an independent left arm contour; the right thigh block, right calf block, and right foot block are connected to obtain an independent right leg contour; and the left thigh block, left calf block, and left foot block are connected to obtain an independent left leg contour. Based on the coordinates of human skeletal points and human contour points of the shoulder and neck area and the chest, waist and hip area, the front and back sides of the shoulder and neck area and the chest, waist and hip area are determined respectively. If the shoulder and neck area is in the front view, the outline of the independent right arm is connected to the left outline of the shoulder and neck area and the left outline of the chest, waist and hip area, and the outline of the independent left arm is connected to the other side outline of the shoulder and neck area and the other side outline of the chest, waist and hip area. If the shoulder and neck area is the back, the independent left arm contour is connected to the left contour of the shoulder and neck area and the left contour of the chest, waist and hip area, and the independent right arm contour is connected to the other side contour of the shoulder and neck area and the other side contour of the chest, waist and hip area. If the chest, waist, and hip area is in the front view, the left contour of the independent right leg contour is connected to the left contour of the chest, waist, and hip area, and the right contour of the independent right leg contour is connected to the crotch point of the chest, waist, and hip area; the right contour of the independent left leg contour is connected to the right contour of the chest, waist, and hip area, and the left contour of the independent left leg contour is connected to the crotch point of the chest, waist, and hip area. If the chest, waist, and hip area is the back view, the left contour of the independent left leg is connected to the left contour of the chest, waist, and hip area, and the right contour of the independent left leg is connected to the crotch point of the chest, waist, and hip area; the right contour of the independent right leg is connected to the right contour of the chest, waist, and hip area, and the left contour of the independent right leg is connected to the crotch point of the chest, waist, and hip area. By connecting the coordinates of the human skeleton points and the coordinates of the human contour points on the image, the contour points of the connected human body blocks are obtained.

5. The human body contour point localization method as described in claim 4, characterized in that, Based on the coordinates of human skeletal points and human contour points in the shoulder and neck area and the chest, waist, and hip area, the front and back sides of the shoulder and neck area and the chest, waist, and hip area are determined respectively: When the confidence scores of the contour points and skeletal points of the shoulder and neck area and the chest, waist and hip area are both greater than a preset threshold, the front and back orientation of the shoulder and neck area and the chest, waist and hip area are determined by the distance between the left and right contour points and the left and right skeletal points. When the confidence level of the skeletal points of the shoulder and neck region and the chest, waist and hip region is greater than a preset threshold, the skeletal points of the shoulder and neck region and the chest, waist and hip region are used to determine the front and back orientation of the shoulder and neck region and the chest, waist and hip region.

6. A system for locating human body contour points, used to implement the method described in any one of claims 1 to 5, characterized in that, include: Build modules are used to construct neural network models for human contour point localization; The training module is used to acquire a training dataset and train the neural network model using the training dataset. The acquisition module is used to input an image containing a human body into the neural network model to obtain the coordinates of human skeleton points and human contour points on the image; The calculation module is used to obtain the contour points of the human body after connecting the human body blocks based on the coordinates of the human skeleton points and the coordinates of the human body contour points on the image.

7. A device, characterized in that, It includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program that, when executed by the processing unit, causes the processing unit to perform the method described in any one of claims 1 to 5.

8. A storage medium, characterized in that, It stores a computer program executable by the device, which, when run on the device, causes the device to perform the method described in any one of claims 1 to 5.

Images