A low-cost and high-accuracy human key point detection system
By designing multi-scale feature extraction and fusion modules in the top-down human key point detection technology, efficient human key point detection is achieved, solving the problems of low efficiency and complex model training in the existing technology, and ensuring the detection Accuracy and efficiency, suitable for low computing power equipment.
Patent Information
- Application Number
- CN202410479726.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-21
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-07-21
AI Technical Summary
Although the existing top-down human key point detection technology is highly accurate, it is relatively low in efficiency, and the model training process is complex, making it difficult to improve operational efficiency while ensuring accuracy.
A method for detecting key points in human body was designed, using the network structure of multi-scale feature extraction module, detection head module, human body position frame detection module, region of interest alignment module and human body key point detection module to realize human body position through one forward reasoning and the detection of key points, combined with multi-channel and multi-stage modes, multi-scale feature maps are analyzed, and the coordinates of human key points are obtained through multi-scale fusion.
It realizes improving operational efficiency on the basis of top-down, ensuring the accuracy of detection, and simplifies the algorithm process and reduces the amount of computing. It is suitable for lightweight deployment of low-computing equipment, and can quickly be used in dense crowd scenarios. Accurately identify the body frames and key points of multiple people.
Smart Images

Figure CN118506398B_ABST
Abstract
Description
Technical Field
[0001] This invention application is a divisional application of an invention patent application with the application date of July 21, 2023, application number: 202310899391.4, and named “A method and system for detecting key points of the human body”.
[0002] The present invention relates to the field of computer vision technology, and in particular to a method and system for detecting key points of a human body. Background Art
[0003] Human key point detection achieves human posture estimation by predicting the positions of the main skeleton points (key points) of the human body. Human posture estimation is one of the basic tasks in the field of computer vision and can be widely used in the fields of action recognition, pedestrian tracking, autonomous driving, human-computer interaction, etc. The technology adopted in the industry is divided into two approaches: bottom-up and top-down.
[0004] The bottom-up method directly detects the key points of the human body from the original image, and then groups them according to the relationship between different key points. The top-down method first detects the pedestrian area, and then performs key point detection on each pedestrian, which can be regarded as a combination of pedestrian detection and single-person posture estimation. Usually, the top-down technical solution is more accurate than the bottom-up method, but less efficient than the bottom-down method.
[0005] In order to achieve high-precision human key point detection, an invention patent for a multi-person human posture estimation method (CN111339903B) discloses a two-stage training method combining bottom-up and top-down methods to train a deep feature human key point detection model. First, the deep feature human key point detection model is trained by a bottom-up method, and then the deep feature human key point detection model is trained by a top-down method. Finally, the deep feature human key point detection model trained in the two stages is used to detect human key points. The deep feature human key point detection model is trained from the bottom up to output a human key point detection confidence heat map; the deep feature human key point detection model is trained from the top down to identify all human key points in the cropped image.
[0006] The above method can realize the recognition of multiple human key points in the image, but the model training process is complicated. The present invention is specially proposed. The present invention is improved on the basis of the top-down method. While ensuring the accuracy of the top-down technical solution, the operating efficiency is improved. Summary of the invention
[0007] The purpose of the present invention is to address the deficiencies in the prior art and to provide a method and system for detecting key points of a human body, which can accurately estimate key points of a human body and has a wide range of applications.
[0008] In a first aspect, the present application provides a method for detecting key points of a human body, comprising:
[0009] Step 1, constructing a human key point estimation network, wherein the human key point estimation network includes a multi-scale feature extraction module, a detection head module, a human position frame detection module, a region of interest alignment module and a human key point detection module;
[0010] The multi-scale feature extraction module is used to receive an input image and output feature maps of multiple scales;
[0011] The detection head module is used to receive feature maps and output detection head feature maps of multiple scales;
[0012] The human body position frame detection module is used to receive the detection head feature map and output predicted human body position frames of multiple scales;
[0013] The region of interest alignment module is used to receive the detection head feature map, the human body position frame and output alignment feature maps of multiple scales;
[0014] The human key point detection module is used to receive the alignment feature map and output the predicted coordinates of human key points of different scales;
[0015] Step 2, using the training data set to train the human body key point estimation network to obtain a human body key point estimation model;
[0016] Step 3, inputting the image to be detected into the human key point estimation model, and outputting the predicted coordinates of human key points of different scales;
[0017] Step 4: Fusing the predicted coordinates of the human body key points of different scales to obtain the coordinates of the human body key points.
[0018] Furthermore, the multi-scale feature map extraction module outputs feature maps of four scales, the detection head module outputs detection head feature maps of four scales, and the human body position frame detection module outputs predicted human body position frames of four scales; the region of interest alignment module outputs aligned feature maps of four scales, and the human body key point detection module outputs predicted coordinates of 17 human body key points of four different scales.
[0019] Furthermore, the multi-scale feature map extraction module includes a Backbone feature extraction module, a top-down multi-scale feature fusion module and a bottom-up multi-scale feature fusion module.
[0020] Furthermore, the Backbone feature extraction module uses multiple residual network blocks to obtain feature maps of at least 4 scales, and outputs feature maps of the last 4 scales with resolutions from small to large, which are: BONE end , BONEend-1 , BONE end-2 ,BONe end-3 ;
[0021] The top-down multi-scale feature fusion module outputs feature maps of four scales, with resolutions ranging from small to large. end , U.P. end-1 , U.P. end-2 , U.P. end-3 , among which, UP end Through BONE end Perform convolution operation to obtain; UP end-1 Through UP end Perform transposed convolution and add BONE end-1 After the convolution operation, UP end-2 Through UP end-1 Perform transposed convolution and add BONE end-2 After the convolution operation, UP end-3 Through UP end-2 Perform transposed convolution and add BONE end-3 After the convolution operation, we get:
[0022] The bottom-up multi-scale feature fusion module outputs feature maps of four scales, with resolutions from small to large, respectively. end , DOWN end-1 , DOWN end-2 , DOWN end-3 , where DOWN end-3 Through UP end-3 Convolution operation is performed; DOWN end-2 By DOWN end-3 Convolution operation and UP end-2 After the convolution operation, DOWN end-1 By DOWN end-2 Convolution operation and UP end-1 After the convolution operation, DOWN end By DOWN end-1 Convolution operation and UP end It is obtained after the convolution operation.
[0023] Furthermore, the detection head module performs convolution operations on the feature maps of the four scales respectively, and outputs the detection head feature maps of the four scales, which are: Head end , Head end-1 , Head end-2 , Head end-3 .
[0024] Furthermore, the human body position frame detection module performs convolution operations on the detection head feature maps of four scales to obtain predicted human body position frames of four scales, namely BOX end , BON end-1 , BOX end-2 , BOX end-3 , predict at least one anchor for each pixel in each scale, and each anchor predicts 6 values, namely the horizontal coordinate of the center position of the human body position box, the vertical coordinate of the center position of the human body position box, the offset of the human body position box relative to the center position, the offset of the human body position box relative to the center position, and the confidence box of the target existence. conf And the human category confidence human conf .
[0025] Furthermore, the number of channels of the feature map for predicting the human body position frame is 18, and 3 anchors are predicted for each pixel.
[0026] Furthermore, the region of interest alignment module uses the region of interest alignment technology to output four scale alignment feature maps, namely ALIGN end , ALIGN end-1 , ALIGN end-2 , ALIGN end-3 .
[0027] Furthermore, the human key point detection module uses convolution operation on the aligned feature maps of four scales to obtain human key point prediction feature maps of four scales, namely POINT end ,POINT end-1 ,POINT end-2 ,POINT end-3 , the number of channels of the human key point prediction feature map is 17.
[0028] Furthermore, the step 2 comprises:
[0029] Step 21, selecting the training data set, wherein the training data set is the coco data set;
[0030] Step 22, the human body position frame is a real human body position frame or the predicted human body position frame, the real human body position frame is a human body position frame marked in the coco data set, and the real human body position frame and the predicted human body position frame are obtained in real time during the training process;
[0031] When the IOU of the real human body position frame and the predicted human body position frame is ≤80%, the human body position frame input by the region of interest alignment module of the human key point estimation network is the real human body position frame;
[0032] When the IOU of the real human body position frame and the predicted human body position frame is greater than 80%, the human body position frame input by the region of interest alignment module of the human key point estimation network is the real human body position frame and the predicted human body position frame;
[0033] Step 23, using a gradient descent method to train the human body key point estimation network to obtain a human body key point estimation intermediate model;
[0034] Step 24, adjusting the region of interest alignment module of the human key point estimation intermediate model so that the human body position frame inputted therein is the predicted human body position frame, and obtaining a human body key point estimation model.
[0035] Furthermore, in step 2, the total error formula is:
[0036] Lossp total =αLoss box +(1-α)Loss point
[0037] Among them, Loss total is the total error, Loss box is the total error of the human body position frame, Loss point is the total error of the human body key points, and α∈(0,1) is used to weigh the total error of the human body position frame and the total error of the human body key points.
[0038] Furthermore, the total error formula of the human body position frame is:
[0039]
[0040] The SIoU is used to calculate the intersection between the real human body position frame and the predicted human body position frame;
[0041] Among them, D represents different scales, i and j represent the horizontal and vertical coordinates of the human position prediction feature map, k represents the sequence number of the anchor, and Box gt Indicates the real human position box, Box pred Represents the predicted human position box.
[0042] Furthermore, the total error formula of the human body key points is:
[0043]
[0044] Among them, D is a different scale, h is the serial number of the key point of the human body, m and n are the horizontal and vertical coordinates of the predicted feature map of the key point of the human body, respectively, m,n,s,h is the true Gaussian value of the key points of the human body, Predict coordinates for human key points.
[0045] Furthermore, the true Gaussian values of the human body key points are obtained by transforming the coordinates of the human body key points using Gaussian distribution according to the coordinates of the human body key points calibrated in the coco data set.
[0046] Furthermore, the key points of the human body are represented by Gaussian distribution, and the real position coordinates of the key points of the human body are mapped to the predicted feature maps of the key points of the human body at different scales, and the coordinates are rounded and set to ; the radius of the Gaussian circle is set to R; on the predicted feature map of the key points of the human body, the Gaussian function calculation value is filled with as the center and the radius R.
[0047] Furthermore, the specific steps of step 3 are:
[0048] Step 31, inputting the image to be detected into the multi-scale feature extraction module, and outputting feature maps of multiple scales;
[0049] Step 32, inputting the feature map into the detection head model, and outputting detection head feature maps of multiple scales;
[0050] Step 33, inputting the detection head feature map into the human body position frame detection module, and outputting predicted human body position frames of multiple scales;
[0051] Step 34, inputting the detection head feature map and the predicted human body position frame into the region of interest alignment module, and outputting alignment feature maps of multiple scales;
[0052] Step 35, inputting the alignment feature map into a human key point detection module, and outputting human key point prediction feature maps of multiple scales;
[0053] Step 36, in the human body key point prediction feature map at each scale, take the coordinates of the maximum value in each channel as the human body key point prediction coordinates, and obtain the human body key point prediction coordinates of different scales.
[0054] Furthermore, the step 4 comprises:
[0055] Step 41, mapping the predicted coordinates of the human body key points of different scales back to the image to be detected to obtain the mapped coordinates of the human body key points of different scales;
[0056] Step 42, averaging the mapping coordinates of the human body key points of different scales to obtain the coordinates of the human body key points.
[0057] In a second aspect, the present application also provides a human key point detection system, including a human key point estimation network construction module, a network training module, a human key point coordinate prediction module, and a human key point coordinate fusion module;
[0058] The human key point estimation network construction module is used to construct a human key point estimation network, and the human key point estimation network includes a multi-scale feature extraction module, a detection head module, a human position frame detection module, an area of interest alignment module and a human key point detection module;
[0059] The multi-scale feature extraction module is used to receive an input image and output feature maps of multiple scales;
[0060] The detection head module is used to receive feature maps and output detection head feature maps of multiple scales;
[0061] The human body position frame detection module is used to receive the detection head feature map and output predicted human body position frames of multiple scales;
[0062] The region of interest alignment module is used to receive the detection head feature map, the human body position frame and output alignment feature maps of multiple scales;
[0063] The human key point detection module is used to receive the alignment feature map and output the predicted coordinates of human key points of different scales;
[0064] The network training module trains the human key point estimation network using the training data set to obtain a human key point estimation model;
[0065] A human body key point coordinate prediction module, which outputs predicted coordinates of human body key points of different scales by inputting the image to be detected into the human body key point estimation model;
[0066] The human body key point coordinate fusion module obtains the human body key point coordinates by fusing the predicted coordinates of the human body key points of different scales.
[0067] The beneficial effects of the present invention are:
[0068] (1) The present invention is an improvement on the top-down approach, and adopts a forward reasoning structure, including human body position frame detection and human body key point detection. That is, only a single model is needed to realize human body position detection and human body key point detection, which ensures the accuracy of the top-down technical solution while improving the operating efficiency.
[0069] (2) A human key point detection network is designed, which integrates a multi-scale feature extraction module, a detection head module, a human position box detection module, a region of interest alignment module and a human key point detection module. The multi-scale feature map is analyzed using a multi-channel and multi-stage mode to obtain the predicted coordinates of human key points of different scales. Finally, the multi-scale predicted coordinates of human key points are fused in a multi-scale fusion manner to obtain more comprehensive and effective human key point coordinates, thereby achieving high-precision recognition of human key points.
[0070] (3) In application scenarios with dense crowds, the present invention can realize the simultaneous recognition of the human body frames and key points of multiple people, and can accurately and quickly estimate the key points of the human body without the need for additional models to pre-extract the human body frames. The algorithm flow is simple and the amount of calculation is low, which meets the lightweight deployment requirements of various low-computing power devices. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0072] Figure 1 Schematic diagram of a process of human key point detection method.
[0073] Figure 2 Network structure for human key point estimation.
[0074] Figure 3 Schematic diagram of the process of step S2 in a method for detecting key points of a human body.
[0075] Figure 4 It is a schematic diagram of the process of step S3.
[0076] Figure 5 This is a block diagram of a human key point detection system. DETAILED DESCRIPTION
[0077] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments, and all other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making creative work are within the scope of protection of the present invention.
[0078] like Figure 1 As shown, an embodiment of the present invention provides a method for detecting key points of a human body, comprising the following steps:
[0079] Step 1, constructing a human key point estimation network, which includes a multi-scale feature extraction module, a detection head module, a human position frame detection module, a region of interest alignment module and a human key point detection module;
[0080] The multi-scale feature extraction module is used to accept input images and output feature maps of multiple scales;
[0081] The detection head module is used to receive the feature map and output the detection head feature map of multiple scales;
[0082] The human body position frame detection module is used to receive the detection head feature map and output the predicted human body position frame of multiple scales;
[0083] The region of interest alignment module is used to accept the detection head feature map, the human body position frame and output the alignment feature map of multiple scales;
[0084] The human key point detection module is used to accept the aligned feature map and output the predicted coordinates of human key points of different scales.
[0085] Step 2: Use the training data set to train the human key point estimation network to obtain the human key point estimation model.
[0086] Step 3: Input the image to be detected into the human key point estimation model and output the predicted coordinates of human key points of different scales.
[0087] Step 4: Fuse the predicted coordinates of human key points of different scales to obtain the coordinates of human key points.
[0088] This embodiment improves on the top-down approach and adopts a forward reasoning structure, including body position frame detection and body key point detection. That is, only a single model is needed to realize body position detection and body key point detection, which ensures the accuracy of the top-down technical solution while improving the operating efficiency.
[0089] The semantic information of the same area covered by human key points of different scales is different. In order to realize multi-scale human key point feature analysis, a multi-scale feature map extraction module is designed. The multi-scale feature extraction module is used to output feature maps of at least two scales.
[0090] In this embodiment, the multi-scale feature map extraction module outputs feature maps of four scales, the detection head module outputs detection head feature maps of four scales, and the human body position frame detection module outputs predicted human body position frames of four scales; the region of interest alignment module outputs aligned feature maps of four scales, and the human body key point detection module outputs predicted coordinates of 17 human body key points of four different scales.
[0091] A human key point detection network is designed which integrates multi-scale feature extraction module, detection head module, human position box detection module, region of interest alignment module and human key point detection module. The multi-scale feature map is analyzed using a multi-channel and multi-stage mode to obtain the predicted coordinates of human key points of different scales. Finally, the multi-scale predicted coordinates of human key points are fused in a multi-scale fusion manner to obtain more comprehensive and effective human key point coordinates, thereby achieving high-precision recognition of human key points.
[0092] like Figure 2 As shown, the multi-scale feature map extraction module includes a Backbone feature extraction module, a top-down multi-scale feature fusion module, and a bottom-up multi-scale feature fusion module.
[0093] The Backbone feature extraction module uses multiple residual network blocks to obtain feature maps of at least 4 scales, and outputs feature maps of the last 4 scales with resolutions from small to large, which are: BONE end , BONE end-1 , BONE end-2 ,BONe end-3 ;
[0094] The top-down multi-scale feature fusion module outputs feature maps of four scales, with resolutions from small to large. end , U.P. end-1 , U.P. end-2 , U.P. end-3 , among which, UP end Through BONE end Perform convolution operation to obtain; UP end-1 Through UP end Perform transposed convolution and add BONE end-1 After the convolution operation, UP end-2 Through UP end-1 Perform transposed convolution and add BONE end-2 After the convolution operation, UP end-3 Through UP end-2 Perform transposed convolution and add BONE end-3 After the convolution operation, we get:
[0095] The bottom-up multi-scale feature fusion module outputs feature maps of four scales, with resolutions from small to large, respectively. end , DOWN end-1 , DOWN end-2 , DOWN end-3 , where DOWN end-3 Through UP end-3 Convolution operation is performed; DOWN end-2 By DOWN end-3 Convolution operation and UP end-2 After the convolution operation, DOWN end-1 By DOWN end-2 Convolution operation and UP end-1 After the convolution operation, DOWN end By DOWN end-1 Convolution operation and UP endIt is obtained after the convolution operation.
[0096] The image is input into the multi-scale feature map extraction module and the output is passed through the Backbone feature extraction module, the top-down multi-scale feature fusion module and the bottom-up multi-scale feature fusion module in sequence, and the feature maps of four scales are output, with resolutions from small to large, respectively. end , DOWN end-1 , DOWN end-2 , DOWN end-3 .
[0097] DOWN end , DOWN end-1 , DOWN end-2 , DOWN end-3 Input the detection head module, the detection head module performs convolution operations on the feature maps of four scales and outputs the detection head feature maps of four scales, namely: Head end , Head end-1 , Head end-2 , Head end-3 .
[0098] Head end , Head end-1 , Head end-2 , Head end-3 Input the human body position frame detection module, which performs convolution operations on the detection head feature maps of four scales to obtain the predicted human body position frames of four scales, namely BOX end , BOX end-1 , BOX end-2 , BOX end-3 , predict at least one anchor for each pixel in each scale, and each anchor predicts 6 values, namely the horizontal coordinate of the center position of the human body position box, the vertical coordinate of the center position of the human body position box, the offset of the human body position box relative to the center position, the offset of the human body position box relative to the center position, and the confidence box of the target existence. conf And the human category confidence human conf .
[0099] In this embodiment, the number of channels of the feature map for predicting the human body position frame is 18, and 3 anchors are predicted for each pixel.
[0100] The number of channels of each feature map of the predicted human position box is 18, which means that 3 anchors are predicted for each human position box and 6 values are predicted for each anchor.
[0101] Head end , Headend-1 , Head end-2 , Head end-3 , the human body position frame is input into the region of interest alignment module. The region of interest alignment module uses the region of interest alignment technology to output alignment feature maps of four scales, namely ALIGN end , ALIGN end-1 , ALIGN end-2 , ALIGN end-3 .
[0102] Specifically, in the prediction stage, the detection head feature maps of different scales and the predicted human position frames are aligned using the region of interest technology to obtain aligned feature maps of different scales. end and BOX end GET ALIGN end , through Head end-1 and BOX end-1 GET ALIGN end-1 , through Head end-2 and BOX end-2 GET ALIGN end-2 , through Head end-3 and BOX end-3 GET ALIGN end-3 .
[0103] The human key point detection module aligns feature maps of four scales ALIGN end , ALIGN end-1 , ALIGN end-2 , ALIGN end-3 By using convolution operation, we can get four scales of human key point prediction feature maps, namely POINT end ,POINT end-1 ,POINT end-2 ,POUNT end-3 , the number of channels of the human key point prediction feature map is 17, 17 represents 17 key points of the human body, that is, each channel represents a human key point.
[0104] In order to establish a high-precision human key point estimation model, through step 2, the human key point estimation network is trained using the coco dataset, such as Figure 3 As shown, step 2 includes:
[0105] Step 21, select a training data set, the training data set is the coco data set.
[0106] The available training datasets include public datasets such as COCO, MPII and CrowdPose. Human key point detection is a relatively complex problem. It is very difficult to establish a suitable dataset for a neural network model. Every joint of each person in the image must be located and marked, which is a trivial and time-consuming task. This embodiment uses the COCO dataset, which has about 80 categories of images and about 250,000 person instances.
[0107] Step 22, the human body position frame is a real human body position frame or a predicted human body position frame, the real human body position frame is the human body position frame marked in the coco data set, and the real human body position frame and the predicted human body position frame are obtained in real time during the training process;
[0108] When the IOU between the real human position frame and the predicted human position frame is ≤80%, the human position frame input by the region of interest alignment module of the human key point estimation network is the real human position frame;
[0109] When the IOU between the real human body position frame and the predicted human body position frame is greater than 80%, the human body position frame input into the region of interest alignment module of the human key point estimation network is the real human body position frame and the predicted human body position frame.
[0110] In the initial stage of human key point estimation network training, since the human position frame predicted by the model is inaccurate, the region of interest alignment module only receives the real human position frame, such as Figure 2 As shown, the input includes a real human body position frame, and the real human body position frame is input into the region of interest alignment module.
[0111] As the model is continuously trained, when the IoU value between the real human position frame and the predicted human position frame obtained by the model exceeds 80%, the region of interest alignment module will receive both the real human position frame and the predicted human position frame, thus increasing the diversity and accuracy of the samples.
[0112] Step 23, using the gradient descent method to train the human body key point estimation network to obtain the human body key point estimation intermediate model.
[0113] Step 24, adjusting the region of interest alignment module of the human body key point estimation intermediate model so that its input human body position frame is a predicted human body position frame, and obtaining a human body key point estimation model.
[0114] Through step 2.3, a trained human key point estimation intermediate model is obtained. The human body position frame input to the region of interest alignment module of the model is the real human body position frame and the predicted human body position frame. However, when using the trained model to detect human key points, there is no manually calibrated real human body position frame, only the predicted human body position frame obtained by the model. Based on this, through step 2.4, the input of the region of interest alignment module of the human key point estimation intermediate model is adjusted, and the real human body position frame is removed in the region of interest alignment module to obtain the final human key point estimation model.
[0115] In step 2, the total error formula during training is:
[0116] Loss total =αLoss box +(1-α)Loss point
[0117] Among them, Loss total is the total error, Loss box is the total error of the human body position frame, Loss point is the total error of the human body key points, and α∈(0,1) is used to weigh the total error of the human body position frame and the total error of the human body key points.
[0118] In this embodiment, constraints are imposed on both the position frame of the human body and the key point information of the human body predicted by the model, thus ensuring that the final human body position key point estimation model is more accurate.
[0119] The total error formula of the human body position frame is:
[0120]
[0121] SIoU is used to calculate the intersection between the real human body position frame and the predicted human body position frame;
[0122] Among them, D represents different scales, i and j represent the horizontal and vertical coordinates of the human position prediction feature map, k represents the sequence number of the anchor, and Box gt Indicates the real human position box, Box pred Represents the predicted human position box.
[0123] The total error formula of human key points is:
[0124]
[0125] Among them, D is a different scale, h is the serial number of the key point of the human body, m and n are the horizontal and vertical coordinates of the predicted feature map of the key point of the human body, respectively, m,n,s,h is the true Gaussian value of the key points of the human body, Predict coordinates for human key points.
[0126] In this embodiment, the mean square error is used to calculate the error between the real human body key points and the predicted human body key points.
[0127] The true Gaussian values of the key points of the human body are obtained by transforming the coordinates of the key points of the human body using Gaussian distribution according to the coordinates of the key points of the human body calibrated in the coco dataset.
[0128] Specifically, the key points of the human body are represented by Gaussian distribution, the real position coordinates of the key points of the human body are mapped to the predicted feature maps of the key points of the human body at different scales, and the coordinates are rounded and set to; the radius of the Gaussian circle is set to R; on the predicted feature map of the key points of the human body, the Gaussian function calculation value is filled with as the center and the radius R.
[0129] Through step 2, a trained human body position key point estimation model is obtained, and through step 3, the predicted coordinates of human body key points of different scales in the image to be detected are obtained, such as Figure 4 As shown, the specific steps of step 3 are:
[0130] Step 31, input the image to be detected into a multi-scale feature extraction module, and output feature maps of multiple scales;
[0131] Step 32, input the feature map into the detection head model, and output the detection head feature maps of multiple scales;
[0132] Step 33, inputting the detection head feature map into the human body position frame detection module, and outputting predicted human body position frames of multiple scales;
[0133] Step 34, input the detection head feature map and the predicted human body position frame into the region of interest alignment module, and output alignment feature maps of multiple scales;
[0134] Step 35, inputting the aligned feature map into a human key point detection module, and outputting human key point prediction feature maps of multiple scales;
[0135] Step 36, in the human body key point prediction feature map of each scale, the coordinates of the maximum value in each channel are taken as the human body key point prediction coordinates, and the human body key point prediction coordinates of different scales are obtained.
[0136] Step 4 is used to achieve the fusion of predicted coordinates of human key points of different scales. Step 4 includes:
[0137] Step 41, mapping the predicted coordinates of the human body key points of different scales back to the image to be detected to obtain the mapping coordinates of the human body key points of different scales;
[0138] Step 42, averaging the mapping coordinates of the key points of the human body at different scales to obtain the coordinates of the key points of the human body.
[0139] Specifically, step 41 is to predict the coordinates of the key points of the human body at different scales {X D,i ,Y D,i} Map back to the image to be detected to obtain the mapping coordinates of human key points of different scales
[0140] Step 42: Mapping coordinates of key points of the human body at different scales Find the mean as follows:
[0141]
[0142]
[0143] Get the final coordinates of the human body key points, Among them, i=0,…,16.
[0144] The 17 key points correspond to the following human body parts: 0: nose, 1: left eye, 2: right eye, 3: left ear, 4: right ear, 5: left shoulder, 6: right shoulder, 7: left elbow, 8: right elbow, 9: left wrist, 10: right wrist, 11: left hip, 12: right hip, 13: left knee, 14: right knee, 15: left ankle, 16: right ankle.
[0145] In application scenarios with dense crowds, this embodiment can realize the simultaneous recognition of the human body frames and key points of multiple people, and can accurately and quickly estimate the key points of the human body without the need for additional models to pre-extract the human body frames. The algorithm flow is simple and the amount of computation is low, meeting the lightweight deployment requirements of various low-computing power devices.
[0146] In a second aspect, the present embodiment further provides a human key point detection system, including a human key point estimation network construction module, a network training module, a human key point coordinate prediction module, and a human key point coordinate fusion module;
[0147] The human key point estimation network construction module is used to construct a human key point estimation network, which includes a multi-scale feature extraction module, a detection head module, a human position frame detection module, an area of interest alignment module and a human key point detection module;
[0148] The multi-scale feature extraction module is used to accept input images and output feature maps of multiple scales;
[0149] The detection head module is used to receive the feature map and output the detection head feature map of multiple scales;
[0150] The human body position frame detection module is used to receive the detection head feature map and output the predicted human body position frame of multiple scales;
[0151] The region of interest alignment module is used to accept the detection head feature map, the human body position frame and output the alignment feature map of multiple scales;
[0152] The human key point detection module is used to accept the aligned feature map and output the predicted coordinates of human key points of different scales;
[0153] The network training module uses the training data set to train the human key point estimation network to obtain the human key point estimation model;
[0154] The human body key point coordinate prediction module inputs the image to be detected into the human body key point estimation model and outputs the predicted coordinates of human body key points of different scales;
[0155] The human body key point coordinate fusion module obtains the human body key point coordinates by fusing the predicted coordinates of human body key points of different scales.
[0156] It should be noted that the modules (or units) in this embodiment are logical in nature. In specific implementation, multiple modules (or units) can be combined into one module (or unit), and one module (or unit) can also be split into multiple modules (or units).
[0157] A human body key point detection system in this implementation realizes automatic recognition and is low-cost, simple and convenient, and has obvious effects. Its application can accurately and quickly estimate the key points of the human body in application scenarios with dense crowds, which has the advantages of reducing the workload of staff and improving work efficiency.
[0158] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0159] The above description is only a preferred example of the invention and is not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.
Claims
1. A low-cost and high-accuracy human key point detection system, including a human key point estimation network construction module, a network training module, a human key point coordinate prediction module, and a human key point coordinate fusion module; The human key point estimation network construction module is used to construct a human key point estimation network, and the human key point estimation network includes a multi-scale feature extraction module, a detection head module, a human position frame detection module, an area of interest alignment module and a human key point detection module; The multi-scale feature extraction module is used to receive an input image and output feature maps of multiple scales; The detection head module is used to receive feature maps and output detection head feature maps of multiple scales; The human body position frame detection module is used to receive the detection head feature map and output predicted human body position frames of multiple scales; The region of interest alignment module is used to receive the detection head feature map, the human body position frame and output alignment feature maps of multiple scales; The human key point detection module is used to receive the alignment feature map and output the predicted coordinates of human key points of different scales; The network training module trains the human key point estimation network using the training data set to obtain a human key point estimation model; The step of obtaining a human key point estimation model by the network training module comprises: Step 2.1, selecting the training data set, wherein the training data set is the coco data set; Step 2.2, the human body position frame is a real human body position frame or the predicted human body position frame, the real human body position frame is a human body position frame marked in the coco data set, and the real human body position frame and the predicted human body position frame are obtained in real time during the training process; When the IOU of the real human body position frame and the predicted human body position frame is ≤80%, the human body position frame input by the region of interest alignment module of the human key point estimation network is the real human body position frame; When the IOU of the real human body position frame and the predicted human body position frame is greater than 80%, the human body position frame input by the region of interest alignment module of the human key point estimation network is the real human body position frame and the predicted human body position frame; Step 2.3, using a gradient descent method to train the human body key point estimation network to obtain a human body key point estimation intermediate model; Step 2.4, adjusting the region of interest alignment module of the human key point estimation intermediate model so that the human body position frame inputted by it is the predicted human body position frame, and obtaining a human body key point estimation model; A human body key point coordinate prediction module, which outputs predicted coordinates of human body key points of different scales by inputting the image to be detected into the human body key point estimation model; A human body key point coordinate fusion module obtains the human body key point coordinates by fusing the predicted human body key point coordinates of different scales; The multi-scale feature extraction module outputs feature maps of 4 scales, the detection head module outputs detection head feature maps of 4 scales, the human body position frame detection module outputs predicted human body position frames of 4 scales; the region of interest alignment module outputs alignment feature maps of 4 scales, and the human body key point detection module outputs predicted coordinates of 17 human body key points of 4 different scales; The multi-scale feature extraction module includes a Backbone feature extraction module, a top-down multi-scale feature fusion module and a bottom-up multi-scale feature fusion module; The Backbone feature extraction module uses multiple residual network blocks to obtain feature maps of at least 4 scales, and outputs the feature maps of the last 4 scales with resolutions from small to large, which are: BONE end , BONE end-1 , BONE end-2 , BONE end-3 ; The top-down multi-scale feature fusion module outputs feature maps of four scales, with resolutions ranging from small to large. end , U.P. end-1 , U.P. end-2 , U.P. end-3 , among which, UP end Through BONE end Perform convolution operation to obtain; UP end-1 Through UP end Perform transposed convolution and add BONE end-1 After the convolution operation, UP end-2 Through UP end-1 Perform transposed convolution and add BONE end-2 After the convolution operation, UP end-3 Through UP end-2 Perform transposed convolution and add BONE end-3 After the convolution operation, we get: The bottom-up multi-scale feature fusion module outputs feature maps of four scales, with resolutions from small to large, respectively. end , DOWN end-1 , DOWN end-2 , DOWN end-3 , where DOWN end-3 Through UP end-3 Convolution operation is performed; DOWN end-2 By DOWN end-3 Convolution operation and UP end-2 After the convolution operation, DOWN end-1 By DOWN end-2 Convolution operation and UP end-1 After the convolution operation, DOWN end By DOWN end-1 Convolution operation and UP end After the convolution operation, we get: The detection head module performs convolution operations on the feature maps of four scales and outputs the detection head feature maps of four scales, which are: Head end , Head end-1 , Head end-2 , Head end-3 ; The human body position frame detection module performs convolution operations on the detection head feature maps of four scales to obtain predicted human body position frames of four scales, namely BOX end , BOX end-1 , BOX end-2 , BOX end-3 , predict at least one anchor for each pixel in each scale, each anchor x hor predicts 6 values, namely the horizontal coordinate of the center position of the human body position frame, the vertical coordinate of the center position of the human body position frame, the offset of the human body position frame relative to the center position, the offset of the human body position frame relative to the center position, and the confidence box of the target existence. conf And the human category confidence human conf ; The human key point detection module uses convolution operation on the aligned feature maps of four scales to obtain human key point prediction feature maps of four scales, namely POINT end , POINT end-1 , POINT end-2 , POINT end-3 , the number of channels of the human key point prediction feature map is 17; The low-cost and high-accuracy human key point detection system is integrated into the application scenario of dense crowds, and can realize the simultaneous recognition of the human body frames and key points of multiple people. It can accurately and quickly estimate the human body key points without the need for additional models to pre-extract the human body frames. The algorithm flow is simple and the computational complexity is low, which meets the lightweight deployment requirements of various low-computing power devices.
2. A low-cost and high-accuracy human key point detection system according to claim 1, characterized in that: The specific steps of the human body key point coordinate prediction module are as follows: Step 31, inputting the image to be detected into the multi-scale feature extraction module, and outputting feature maps of multiple scales; Step 32, inputting the feature map into the detection head model, and outputting detection head feature maps of multiple scales; Step 33, inputting the detection head feature map into the human body position frame detection module, and outputting predicted human body position frames of multiple scales; Step 34, inputting the detection head feature map and the predicted human body position frame into the region of interest alignment module, and outputting alignment feature maps of multiple scales; Step 35, inputting the alignment feature map into a human key point detection module, and outputting human key point prediction feature maps of multiple scales; Step 36, in the human body key point prediction feature map at each scale, take the coordinates of the maximum value in each channel as the human body key point prediction coordinates, and obtain the human body key point prediction coordinates of different scales.
3. A low-cost and high-accuracy human key point detection system according to claim 1, characterized in that: In the step of obtaining the human key point estimation model by the network training module, the total error formula is: Loss total =αLoss box +(1-α)Loss point Among them, Loss total is the total error, Loss box is the total error of the human body position frame, Loss point is the total error of the key points of the human body, α∈(0,1) is the weight used to weigh the total error of the human position frame and the total error of the key points of the human body; The total error formula of the human body position frame is: The SIoU is used to calculate the intersection between the real human body position frame and the predicted human body position frame; Among them, D represents different scales, i and j represent the horizontal and vertical coordinates of the center position of the human body position frame, and k represents the sequence number of the anchor. represents the real human position frame, Represents the predicted human body position frame; The total error formula of the human body key points is: Among them, h is the serial number of the key point of the human body, m and n are the horizontal and vertical coordinates of the key point of the human body, and x m,n,D,h is the true Gaussian value of the key points of the human body, Predict coordinates for human key points.
4. A low-cost and high-accuracy human key point detection system according to claim 1, characterized in that: The human body key point coordinate fusion module steps include: Step 41, mapping the predicted coordinates of the human body key points of different scales back to the image to be detected to obtain the mapping coordinates of the human body key points of different scales; Step 42: averaging the mapping coordinates of the human body key points at different scales to obtain the coordinates of the human body key points.
