A human pose detection method and system based on target recognition

Abstract

The application discloses a human posture detection method and system based on target identification. The application uses key parts as object targets, and establishes a connection relationship based on a limb heat map to express human posture features, thereby reducing the missed detection rate under the shielding condition. Moreover, the posture features used by the application already contain a human boundary box, so when human posture features, human target features and part target features are needed at the same time subsequently, an accurate and non-regression fitting prediction boundary box can be obtained without additionally loading a target detection algorithm, and the detection speed is improved.

Description

Technical Field

[0001] This invention relates to the field of target recognition technology, and in particular to a human posture detection method and system based on target recognition. Background Technology

[0002] Existing research on abnormal behavior detection systems often focuses on open environments such as homes and shopping mall lobbies, or queuing scenarios and outdoor environments where sufficient feature information is available for anomaly detection. Research specifically targeting bus scenarios is relatively limited. For example... Figure 1 and Figure 2 As shown, the bus scene is characterized by severe occlusion of human images compared to other scenes, which leads to a greater need for subsequent behavioral judgment research and information requirements (meaning that after feature extraction, these features may be used for fall detection, boundary crossing detection, face recognition, fast movement detection, fighting detection, sitting and gripping detection, etc., which may depend on different features, such as gripping requires the bounding box of the hand, crossing the boundary requires the bounding box of the human body, falling requires posture information, and face recognition requires the bounding box of the head, etc.).

[0003] In the research of intelligent public transportation technology, detecting abnormal passenger behavior by relying on surveillance cameras inside the bus is a crucial task. Existing research on in-image behavior detection systems typically divides it into three stages: The first is feature extraction, which extracts complex image information into more refined information. This mainly includes moving object information and edge information detected using traditional image processing methods (this part mainly refers to methods before the widespread application of deep networks and is no longer covered in current research), human target information detected using object recognition algorithms (i.e., human bounding boxes), and human pose information detected using pose estimation algorithms (usually human keypoint skeletons). The second stage is feature processing. Different studies often require further processing of the detected features according to their respective methodologies. For example, if behavior needs to be determined based on video, frame-by-frame processing is required. For target matching, if human targets are used as features to represent behavior, further extraction of target features is required (such as calculating the aspect ratio, center change rate, effective area ratio, etc. of the bounding box); the third is behavior judgment, that is, using the features extracted above to determine what behavior the human body is in. Common methods include using experience to design logical judgment strategies (such as a rapid increase in the aspect ratio of the human bounding box may indicate that the human body has fallen, this method is often used for simple single behavior judgment), using trained SVM classifiers for judgment, and using deep networks for judgment (such as when judging behavior based on video, the continuous poses of the object will form a set of sequence information, often relying on residual networks or memory networks to complete the classification task).

[0004] Anomaly detection in bus scenarios is crucial for public safety and hazard warning. However, anomaly recognition relies on extracting human pose features from captured images or videos. Surveys of bus scenarios reveal that human targets are often severely occluded. Traditional human pose estimation algorithms often miss even partially occluded targets, resulting in a high false negative rate.

[0005] The existing detection methods include:

[0006] 1. Common methods for pose feature detection in specific scenarios: Deploy common bottom-up human pose estimation algorithms, which often represent pose information using a human keypoint skeleton composed of various human skeletal keypoints and their relationships. Deep convolutional networks are used to detect keypoints, and keypoints belonging to the same person are connected into a skeleton in a post-processing stage. The most common one is the OpenPose algorithm.

[0007] 2. Since subsequent behavior discrimination may sometimes require not only human pose features but also human target features (i.e., human bounding boxes), although pose features can be regressed from the skeleton to obtain a human bounding box, the error is relatively large. In this case, existing research will choose a top-down human pose estimation algorithm (i.e., first run a human target detector, then crop the image of each detected human bounding box, and perform single-person pose estimation on it. This can obtain accurate target features and pose features at the same time, and the false negative rate is low. The disadvantage is that the detection speed will slow down significantly as the number of people in the image increases, making it difficult to apply to real-time detection). Alternatively, a bottom-up human pose estimation algorithm and a human target recognition algorithm can be run at the same time to match the detected pose features with the human target (which can also obtain accurate target features and pose features at the same time, although it also runs two or even three networks (the matching process may also require deep networks to calculate). The detection speed is much slower than a single bottom-up human pose estimation algorithm, but better than a top-down pose estimation algorithm).

[0008] Current technologies for extracting human feature information in specific scenarios often employ a bottom-up human pose estimation algorithm, which can detect human pose features in images and performs well in real-time. However, the bottom-up human pose estimation algorithm works by first detecting all keypoints in the image and then using auxiliary information in the post-processing stage to regress and connect keypoints belonging to the same person to form a complete human skeleton. This leads to the following two problems:

[0009] Firstly, for reliable regression connections to be achieved in the post-processing stage, a sufficient number of keypoints on the human target need to be detected. That is, if the number of keypoints identifying a person is less than a specified threshold, these keypoints are considered false detections and discarded without further regression connection. This is primarily to reduce network sensitivity and overfitting, and improve accuracy. However, in scenarios with high occlusion in human images, many images may fail to detect enough keypoints due to occlusion, leading to their rejection in post-processing and a significant increase in the false negative rate. A diagram illustrating this is shown below. Figure 3 As shown in (a) and (b) in the figure.

[0010] Secondly, subsequent behavior discrimination may require multiple pieces of information. Sometimes, not only human posture information but also human target information is needed (for example, in passenger fall detection and prediction, human posture information can be used to represent the action, but human target information is also needed to determine whether the passenger is in the aisle area or the seating area, and hand target information is needed to determine whether the passenger is holding the handrail, etc.). Current research mainly uses three methods to meet this requirement: one is to use bounding boxes regressed from human posture features to represent human target features. However, since key points on the human body are located inside the human target, the resulting bounding boxes are often smaller than those directly detected (from...). Figure 3 This problem can also be seen in [the text], although simple and convenient, it is sometimes insufficient to represent human targets. Secondly, abandoning the bottom-up human pose estimation algorithm and choosing a top-down human pose estimation algorithm instead, because the latter is a two-step detection, detecting human target features in the first step, but due to its algorithm design issues, it is difficult to achieve real-time multi-person detection. Thirdly, simultaneously incorporating a bottom-up human pose estimation algorithm and a human target recognition algorithm, this can simultaneously detect human pose features and target features, and then match the two types of features after the detection stage, matching the pose features of the same human with the target features. The problem is that the detection capabilities of the two detection algorithms are different (in highly occluded scenes, target recognition is generally better than pose estimation), resulting in some features not matching the other feature, forcing the disposal of these features, or designing the subsequent judgment algorithm to be compatible with features from all three cases, which often reduces the subsequent judgment capability, and adding an extra network also reduces the detection speed. Summary of the Invention

[0011] The purpose of this invention is to provide a human posture detection method and system based on target recognition, so as to reduce the false negative rate under occlusion and improve the detection speed.

[0012] To achieve the above objectives, the present invention provides the following solution:

[0013] In one aspect, the present invention provides a human posture detection method based on target recognition. The method is based on a human posture detection network model, which includes: a feature extraction network, a key part prediction network, a limb heatmap prediction network, and a post-processing module. The output of the feature extraction network is connected to the input of the key part prediction network and the input of the limb heatmap prediction network, respectively. The outputs of the key part prediction network and the limb heatmap prediction network are both connected to the input of the post-processing module.

[0014] The method includes the following steps:

[0015] Acquire the image to be identified in the target scene;

[0016] The feature extraction network is used to extract features from the image to be identified to obtain feature data;

[0017] Based on the feature data, the key parts prediction network is used to predict key parts, thereby obtaining the key parts of the human target and the human target itself; the key parts include the head, torso, left hand, right hand, right foot, and left foot;

[0018] Based on the feature data, the limb heatmap prediction network is used to predict the limb heatmap and obtain the limb heatmap of the human target.

[0019] Using the post-processing module, regression connections are performed on key parts of the same human target based on the limb heatmap to determine the posture information of the human target.

[0020] Optionally, the feature extraction network includes: 20 convolutional layers;

[0021] The critical component prediction network includes: 5 convolutional layers, and two fully connected layers connected after the 5 convolutional layers;

[0022] The limb heatmap prediction network is an hourglass network, which includes 5 convolutional layers.

[0023] Optionally, the step of acquiring the image to be identified in the target scene further includes:

[0024] Construct datasets for key body parts and connected limbs;

[0025] The key part prediction network is trained based on the key part dataset;

[0026] The limb heatmap prediction network is trained based on the connected limb dataset.

[0027] Optionally, the construction of the key part dataset and the connected limb dataset specifically includes:

[0028] Acquire multiple sample images of the target scene;

[0029] In each sample image, the bounding boxes of the human body, head, torso, left hand, right hand, right foot, and left foot are labeled to obtain the key part labels for each sample image; wherein, the torso bounding box is a tiltable rectangle, and the bounding boxes of the human body, head, left hand, right hand, right foot, and left foot are all non-tiltable rectangles;

[0030] In each of the sample images, the bounding boxes of the human target—right upper arm, right lower arm, left upper arm, left lower arm, right thigh, right calf, left thigh, and left calf—are labeled to obtain the connected limb labels for each sample image; wherein, the bounding boxes of the right upper arm, right lower arm, left upper arm, left lower arm, right thigh, right calf, left thigh, and left calf are all tiltable rectangles;

[0031] The key part labels of each sample image are converted into an array to obtain the key part dataset;

[0032] The connected limb labels of each sample image are converted into heat maps that decay from the center to the edge, resulting in heat maps of the right upper arm, right lower arm, left upper arm, left lower arm, right thigh, right calf, left thigh, and left calf.

[0033] The heatmaps of the right upper arm, right lower arm, left upper arm, left lower arm, right thigh, right calf, left thigh, and left calf of the same human target in each sample image are integrated into a heatmap of the right arm, left arm, right leg, and left leg to obtain a connected limb dataset.

[0034] Optionally, the loss function used to train the key part prediction network is:

[0035]

[0036] in, This represents the loss function used to train the key part prediction network;

[0037] This indicates the loss of human target coordinates. x, y, w, and h are the x-coordinate, y-coordinate, width, and height of the center point of the predicted human bounding box in the i-th row and j-th column, respectively. , , and These represent the x-coordinate, y-coordinate, width, and height of the center point of the human bounding box in the i-th row and j-th column obtained from the annotation. This indicates whether a human target exists in the cell at row i and column j. If a human target exists, The value is 1 when there is no human target. The value is 0;

[0038] For positive samples targeting the human body, the confidence loss is... c is the confidence score of the predicted human bounding box in the i-th row and j-th column cell;

[0039] For negative sample confidence loss targeting human subjects. ; This indicates whether the cell in row i and column j does not contain a human target. The value is 1 when a human target is present. The value is 0;

[0040] For the loss of target coordinates in critical parts, , This indicates whether the cell in row i and column j contains the k-th key element. If the k-th key element exists, ... The value is 1 when there is no k-th key part. The value is 0, and the kth key part is the head, torso, left hand, right hand, right foot or left foot; , , and These represent the x-coordinate, y-coordinate, width, and height of the center point of the bounding box of the k-th critical component in the i-th row and j-th column of the predicted data. , , and These are the x-coordinate, y-coordinate, width, and height of the center point of the bounding box of the k-th key part in the i-th row and j-th column cell obtained from the annotation;

[0041] The confidence loss for positive samples of key target areas. ,in, This represents the confidence level of the bounding box of the k-th critical component in the i-th row and j-th column of the predicted data.

[0042] The negative sample confidence loss is used for targets in critical areas. ,in, This indicates whether the k-th key element does not exist in the cell at row i and column j. If the k-th key element does not exist, ... The value is 1 when there is a k-th critical part. The value is 0;

[0043] For the loss of trunk angle, 'a' represents the predicted torso angle of the cell in the i-th row and j-th column. The torso angle is the value of the cell in the i-th row and j-th column obtained from the annotation. This indicates whether the cell in row i and column j contains a torso. If a torso exists, The value is 1 when there is no torso. The value is 0;

[0044] , , , , , and These are the weight parameters for the loss of human target coordinates, the confidence loss of positive human target samples, the confidence loss of negative human target samples, the loss of key part target coordinates, the confidence loss of positive key part target samples, the confidence loss of negative key part target samples, and the torso angle loss.

[0045] Optionally, the loss function used to train the limb heatmap prediction network is:

[0046]

[0047] in, This is the loss function used to train the limb heatmap prediction network; Let p be the value of the predicted v-th heatmap. The value at point p is the value of the v-th heatmap obtained by annotation. The v-th heatmap is either the right arm heatmap, the left arm heatmap, the right leg heatmap, or the left leg heatmap.

[0048] Optionally, the step of using the post-processing module to perform regression connections on key parts of the same human target based on the limb heatmap to determine the posture information of the human target specifically includes:

[0049] Calculate the Euclidean distance between the midpoint of each side of the head bounding box and the midpoint of each side of the torso bounding box in the predicted image to be identified.

[0050] Establish the connection between the head target and the torso target by the midpoint of the edge of the head bounding box with the shortest Euclidean distance and the midpoint of the edge of the torso bounding box.

[0051] Using the four vertices of the torso bounding box as centers, and the length of the longest side of the torso bounding box as the boundary, respectively... Construct four first circular regions with radius 1;

[0052] The radius with the largest sum of heatmap values ​​in each of the four first circular regions is selected as the target radius, resulting in four first target radii; the sum of heatmap values ​​is the sum of the heatmap values ​​of each point on the radius.

[0053] Using the center point of the preset key part bounding box as the center, and the length of the longer side of the torso bounding box as the center... Construct a second circular area with a radius equal to the specified radius; key components are pre-defined as the right hand, left hand, right foot, and left foot.

[0054] The radius of the region with the largest sum of heatmap values ​​in the second circular region is selected as the target radius, thus obtaining the second target radius;

[0055] Calculate the corresponding preset key parts With the The intersection of the extensions of the lines is the inflection point corresponding to the preset key part; the right hand, left hand, right foot, and left foot correspond to The first target radius is obtained in the first circular region centered at the top right, top left, bottom right, and bottom left vertices of the torso bounding box.

[0056] Establish the connection relationship between the vertices of the torso bounding box corresponding to the preset key parts, the inflection points of the preset key parts, and the center point of the preset key parts in sequence; the vertices of the torso bounding boxes corresponding to the right hand, left hand, right foot, and left foot are the upper right vertex, upper left vertex, lower right vertex, and lower left vertex of the torso bounding box, respectively.

[0057] Optionally, the step of calculating the Euclidean distance between the midpoint of each side of the predicted head bounding box and the midpoint of each side of the torso bounding box in the image to be identified further includes:

[0058] The image obtained after confidence competition is obtained by predicting the human bounding boxes in the image to be identified, including human bounding boxes with confidence scores less than a confidence threshold, and key part bounding boxes contained within human bounding boxes with confidence scores less than a confidence threshold.

[0059] Calculate the Intersection over Union (IOU) value between any two human bounding boxes in the image after confidence competition;

[0060] Remove the bounding boxes of human figures with smaller C values ​​from the two bounding boxes with IOU values ​​greater than the IOU threshold, as well as the bounding boxes of key parts contained within the bounding box with the smaller C value, to obtain the image after IOU value competition:

[0061] The formula for calculating the C value is:

[0062]

[0063] in, express value, This represents the confidence level of the human body bounding box. , , , , and The confidence scores are for the head bounding box, torso bounding box, right-hand bounding box, left-hand bounding box, right-foot bounding box, and left-foot bounding box, respectively. The confidence weights are for key location positions.

[0064] On the other hand, the present invention also provides a human posture detection system based on target recognition. The system is based on a human posture detection network model, which includes: a feature extraction network, a key part prediction network, a limb heatmap prediction network, and a post-processing module. The output of the feature extraction network is connected to the input of the key part prediction network and the input of the limb heatmap prediction network, respectively. The outputs of the key part prediction network and the limb heatmap prediction network are both connected to the input of the post-processing module.

[0065] The system includes:

[0066] The image acquisition module is used to acquire the image to be recognized in the target scene;

[0067] The feature extraction module is used to extract features from the image to be identified using the feature extraction network to obtain feature data;

[0068] The key part prediction module is used to predict key parts based on the feature data using the key part prediction network, thereby obtaining the key parts of the human target and the human target.

[0069] The limb heatmap prediction module is used to predict limb heatmaps based on the feature data and using the limb heatmap prediction network to obtain limb heatmaps of human targets.

[0070] The posture information acquisition module is used to determine the posture information of the human target by performing regression connections on key parts of the same human target based on the limb heat map using the post-processing module.

[0071] In another aspect, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned method.

[0072] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0073] This invention discloses a human pose detection method and system based on target recognition. The method includes the following steps: acquiring an image to be identified in a target scene; using a feature extraction network to extract features from the image to obtain feature data; based on the feature data, using a key part prediction network to predict key parts, obtaining key parts of the human target and the human target itself; the key parts include the head, torso, left hand, right hand, right foot, and left foot; based on the feature data, using a limb heatmap prediction network to predict limb heatmaps, obtaining limb heatmaps of the human target; and using a post-processing module to perform regression connections on the key parts of the same human target based on the limb heatmaps to determine the pose information of the human target. This invention uses key parts as the target and establishes connections based on limb heatmaps to express human pose features, thereby reducing the false negative rate under occlusion conditions. Moreover, the pose features used in this invention already include human bounding boxes, so when human pose features, human target features, and part target features are needed simultaneously, accurate, non-regression-fitted predicted bounding boxes can be obtained without additional target detection algorithms, improving detection speed. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 This is a schematic diagram of a bus carriage scene in the existing technology;

[0076] Figure 2 This is a schematic diagram of the exterior of a bus in the prior art;

[0077] Figure 3 This is a schematic diagram of the detection results of a bottom-up human pose estimation algorithm in the prior art; Figure 3 (a) in the figure is a schematic diagram of the key point detection results of the bottom-up human pose estimation algorithm in the prior art. Figure 3 (b) in the figure is a schematic diagram of the post-processing result of the bottom-up human pose estimation algorithm in the prior art;

[0078] Figure 4 This is a schematic diagram of the structure of the human pose detection network model provided in an embodiment of the present invention;

[0079] Figure 5 This is a schematic diagram of a sample image of a bus carriage scene provided in an embodiment of the present invention;

[0080] Figure 6 A schematic diagram illustrating key parts provided for embodiments of the present invention;

[0081] Figure 7 This is a schematic diagram of the annotation of connecting limbs provided in an embodiment of the present invention;

[0082] Figure 8 This is a schematic diagram of cell segmentation provided in an embodiment of the present invention;

[0083] Figure 9 A visual representation of the key component labels provided in this embodiment of the invention;

[0084] Figure 10 This is a visual representation of the limb-connecting tag provided in an embodiment of the present invention;

[0085] Figure 11 A flowchart illustrating a human pose detection method based on target recognition provided in an embodiment of the present invention;

[0086] Figure 12 This is a schematic diagram of the regression connection between the head target and the torso target provided in an embodiment of the present invention;

[0087] Figure 13 This is a schematic diagram illustrating the regression connection between the torso target and the right hand target provided in an embodiment of the present invention;

[0088] Figure 14 This is a schematic diagram of finding limb connection paths based on heatmaps, provided in an embodiment of the present invention. Detailed Implementation

[0089] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0090] The purpose of this invention is to provide a human posture detection method and system based on target recognition, so as to reduce the false negative rate under occlusion and improve the detection speed.

[0091] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0092] Example 1

[0093] Embodiment 1 of the present invention provides a human posture detection method based on target recognition, such as Figure 4 As shown, the method of this embodiment of the invention is based on a human pose detection network model, which includes: a feature extraction network (BackBone), a key part prediction network, a limb heatmap prediction network, and a post-processing module; wherein, the output of the feature extraction network is connected to the input of the key part prediction network and the input of the limb heatmap prediction network, respectively, and the outputs of the key part prediction network and the limb heatmap prediction network are both connected to the input of the post-processing module.

[0094] For example, before using the human pose detection network model of the present invention, it is necessary to establish a dataset of key parts and a dataset of connected limbs, and train the human pose detection network model, as follows:

[0095] The collected images are labeled and the labels are converted into a label format suitable for network training and testing. This process involves three steps: dataset collection, dataset labeling, and dataset conversion. The dataset required by the network consists of two parts: a dataset of key body parts and a dataset of connected limbs.

[0096] The first step is dataset collection. Surveillance video was captured in a bus scene, and the video was segmented into images. Images with relatively reasonable passenger distribution were selected to form the original passenger image dataset for the bus scene, such as... Figure 5 As shown.

[0097] Next is the dataset annotation, which is divided into two parts: key body part annotation and connected limb annotation. The key body part annotation uses annotation software to mark the bounding boxes of the human body, head, torso, left hand, right hand, right foot, and left foot. The torso bounding box uses tiltable rectangles to obtain torso tilt angle features, while the other key body parts are labeled with ordinary rectangles, such as... Figure 6 As shown. Sub-labels are used to indicate the relationship between the human body bounding box and the bounding boxes of key parts; that is, bounding boxes belonging to the same person use the same sub-label. The limb annotations are connected using annotation software to mark the bounding boxes of the human body in the image: right upper arm, right lower arm, left upper arm, left lower arm, right thigh, right calf, left thigh, and left calf. All are marked with tiltable rectangles, such as... Figure 7 As shown, sub-labels are used to indicate the relationship between the eight bounding boxes; that is, bounding boxes belonging to the same person use the same sub-label.

[0098] Finally, the dataset is transformed, converting the labeled dataset into labels suitable for network training. The key part label transformation method involves segmenting the image into... A cell, such as Figure 8 As shown, the annotations of key parts are converted to... The array labels. This indicates that the center point falls on the th Line 1 Human pose object in column cell, This indicates how many human pose objects can be detected in a single cell. A human pose object is represented as:

[0099]

[0100]

[0101]

[0102] in These represent the confidence level, the x-coordinate of the center point, the y-coordinate of the center point, the width of the bounding box, the height of the bounding box, and the angle of the bounding box, respectively. These represent the bounding boxes for the human body, head, torso, right hand, left hand, right foot, and left foot, respectively, with values ​​normalized relative to the image size. The confidence level is 1 or 0, where 1 indicates a true label and 0 indicates a false label. Figure 9 As shown; the method for converting limb labels involves transforming the rectangular labeled area into a heatmap that decays from the center to the edge, with the heatmap values ​​being... Where x and y represent the coordinates of the corresponding pixel position relative to the rectangular label, and boxw represents the length of the longer side of the rectangular label. Finally, the heatmap is reorganized into 4 images. The single-value heatmaps are: right arm heatmap (obtained by reshaping the right upper arm and right lower arm heatmaps), left arm heatmap (obtained by reshaping the left upper arm and left lower arm heatmaps), right leg heatmap (obtained by reshaping the right thigh and right calf heatmaps), and left leg heatmap (obtained by reshaping the left thigh and left calf heatmaps). Figure 10 As shown.

[0103] like Figure 11 As shown, the method includes the following steps:

[0104] Step 1: Obtain the image to be identified in the target scene. For example, in this embodiment of the invention, the target scene is a bus carriage scene.

[0105] Step 2: The feature extraction network is used to extract features from the image to be identified, obtaining feature data. The feature extraction network consists of 20 convolutional layers, which performs preliminary processing on complex image information to obtain more refined features. Specifically, the input image's pixel information is used... An array representation.

[0106] Step 3: Based on the feature data, use the key part prediction network to predict the key parts and obtain the key parts of the human target and the human target; the key parts include the head, torso, left hand, right hand, right foot and left foot.

[0107] The key component prediction network consists of five convolutional layers followed by two fully connected layers, used to predict key components of the human body and the human body itself. The final network output is... The tensor. The training data is the key part dataset mentioned above. The loss function is... in, The loss is the human target coordinates, where x, y, w, and h are the x and y coordinates and width and height values ​​of the predicted human bounding box center point in the i-th row and j-th column cell. This indicates whether a human target exists in the cell at row i and column j; 1 indicates yes, 0 indicates no. Let c be the confidence loss for positive human target samples, and c be the confidence score for predicting the human bounding box in the i-th row and j-th column cell. For negative sample confidence loss targeting human subjects. This indicates whether the human target does not exist in the cell at row i and column j; 1 indicates yes, 0 indicates no. For the loss of target coordinates in critical parts, This indicates whether the key element (k-th critical part) exists in the cell at row i and column j; 1 indicates yes, 0 indicates no. , , and These represent the x-coordinate, y-coordinate, width, and height of the center point of the bounding box of the k-th critical component in the i-th row and j-th column of the predicted data. , , and These are the x-coordinate, y-coordinate, width, and height of the center point of the bounding box of the k-th key part in the i-th row and j-th column cell obtained from the annotation; The confidence loss for positive samples of key target areas. This represents the confidence level of the bounding box of the k-th critical component in the cell of the i-th row and j-th column obtained from the prediction. This indicates whether the key element (k-th critical part) exists in the cell at row i and column j; 1 indicates yes, 0 indicates no. The negative sample confidence loss is used for targets in critical areas. This indicates whether the key element in the i-th row and j-th column does not exist; 1 indicates yes, 0 indicates no. 'a' represents the predicted torso angle for the cell in the i-th row and j-th column. , , , , , and These are the weight parameters for the loss of human target coordinates, the confidence loss of positive human target samples, the confidence loss of negative human target samples, the loss of key part target coordinates, the confidence loss of positive key part target samples, the confidence loss of negative key part target samples, and the torso angle loss.

[0108] Step 4: Based on the feature data, use the limb heatmap prediction network to predict the limb heatmap and obtain the limb heatmap of the human target.

[0109] The limb heatmap prediction network is responsible for predicting limb heatmaps used to assist in regression connections of key areas. Its structure should be an hourglass network consisting of 5 convolutional layers, with the final output layer showing the following data shape. The training data is the limb heatmap dataset used in the training process described above. The loss function is... 'v' indicates one of the following: right arm heatmap, left arm heatmap, right leg heatmap, and left leg heatmap. This corresponds to the value at point p in the heatmap.

[0110] Step 5: Using the post-processing module, perform regression connections on key parts of the same human target based on the limb heatmap to determine the posture information of the human target.

[0111] The post-processing involves performing confidence and IOU competitions between the predicted human targets and their corresponding key body parts, and using limb heatmaps to perform regression connections on key body parts belonging to the same person.

[0112] The confidence competition specifically involves comparing the confidence scores of the predicted bounding boxes for all human targets. All human targets and their associated key body parts that are below the confidence threshold will be discarded.

[0113] The IOU competition specifically involves calculating the IOU value between human bounding boxes after confidence-based competition. If the IOU value exceeds a set IOU threshold, competition occurs, and the C value is calculated. Only keep the one with the largest C value. This is the confidence weight for the key location, used to adjust the weight of the key location and the human body bounding box in the competition for IOU.

[0114] Connection Strategy: The four single-value heatmaps predicted by the limb heatmap prediction network are used to perform regression connections between the human body target that has undergone IOU competition and its corresponding key body part target. Specifically, the head target is first regressed and connected to the torso target, such as... Figure 12 As shown, calculate the midpoints of the four sides of the head bounding box (i.e., Figure 12 (Midpoint of the header border) The midpoints of the four sides of the torso bounding box (i.e. Figure 12 (Midpoint of the torso frame) The Euclidean distance between the head and torso targets is used to establish a connection between them by finding the midpoints of the edges of the shortest head bounding box and the midpoints of the edges of the torso bounding box. Then, regression connections are performed between the torso target and the hand and foot targets. First, the right hand target is connected to the torso target, such as... Figure 13 As shown, the four vertices of the torso bounding box (i.e. Figure 13 (the vertices of the torso border) And the center point of the right bounding box (i.e. Figure 13 (center point of the right border in the middle) Centered on the torso bounding box, the length of the longer side is... Create a circular region with radius . and ,like Figure 14 As shown, in the circular area Create a line segment with the center of the circle as the starting point and the radius as the length. Rotate the line segment and calculate the sum of the heatmap values ​​on it. The line segment with the largest sum of heatmap values ​​is denoted as... In the circular area Create a line segment with the center of the circle as the starting point and the radius as the length. Rotate the line segment and calculate the sum of the heatmap values ​​on it. The line segment with the largest sum of heatmap values ​​is denoted as... Finally, calculate and Intersection of extensions The right elbow point (the inflection point corresponding to the right hand) is the point on the right arm. The line connecting the limb vertex (the upper right vertex of the torso bounding box), the right elbow point, and the center point of the right hand represents the regression connection between the right-hand target and the limb target. The regression connections between the left hand, left foot, and right foot targets and the limb targets are the same as those between the right hand target and the limb target, and will not be described further here.

[0115] Finally, the post-processed human target and the connection relationships between it and its key parts are recorded as the predicted pose object.

[0116] Example 2

[0117] Embodiment 2 of the present invention provides a human posture detection system based on target recognition. The system is based on a human posture detection network model, which includes a feature extraction network, a key part prediction network, a limb heatmap prediction network, and a post-processing module. The output of the feature extraction network is connected to the input of the key part prediction network and the input of the limb heatmap prediction network, respectively. The outputs of the key part prediction network and the limb heatmap prediction network are both connected to the input of the post-processing module.

[0118] The system includes:

[0119] The image acquisition module is used to acquire the image to be recognized in the target scene;

[0120] The feature extraction module is used to extract features from the image to be identified using the feature extraction network to obtain feature data.

[0121] The key part prediction module is used to predict key parts based on the feature data using the key part prediction network, thereby obtaining the key parts of the human target and the human target.

[0122] The limb heatmap prediction module is used to predict limb heatmaps based on the feature data using the limb heatmap prediction network, thereby obtaining limb heatmaps of human targets.

[0123] The posture information acquisition module is used to determine the posture information of the human target by performing regression connections on key parts of the same human target based on the limb heat map using the post-processing module.

[0124] The specific implementation methods of each module in Embodiment 2 of the present invention are the same as those in Embodiment 1, and will not be repeated here.

[0125] Example 3

[0126] Embodiment 3 of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0127] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0128] Firstly, this embodiment of the invention uses target-based representation to characterize pose features, and its detection method is similar to target detection. Compared to estimation networks that use keypoints to represent information, key body parts are stronger and easier to detect than local keypoint information. Since the human body is used as the indivisible basic element, compared to using keypoints as the indivisible basic element, the representation of pose objects using the human body and its associated key body parts and their connection relationships does not need to consider the discard problem during regression connections. Therefore, it has a stronger ability to obtain complete pose information, higher recall, and lower false negative rate.

[0129] Secondly, since the pose features in the embodiments of the present invention already include human body bounding boxes, when human body pose features, human body target features, and part target features are needed simultaneously in the future, it is not necessary to carry out additional target detection algorithms to obtain accurate, non-regression fitting predicted bounding boxes, which is beneficial to the real-time operation of the detection system.

[0130] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0131] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A human pose detection method based on target recognition, characterized in that, The method is based on a human posture detection network model, and the human posture detection network model comprises a feature extraction network, a key part prediction network, a limb heat map prediction network and a post-processing module; the output end of the feature extraction network is connected with the input end of the key part prediction network and the input end of the limb heat map prediction network respectively, and the output end of the key part prediction network and the output end of the limb heat map prediction network are connected with the input end of the post-processing module; The method comprises the following steps: acquiring a to-be-recognized picture in a target scene; extracting features of the to-be-recognized picture by using the feature extraction network to obtain feature data; predicting key parts by using the key part prediction network according to the feature data to obtain key parts of a human target and the human target; the key parts comprise a head, a trunk, a left hand, a right hand, a right foot and a left foot; predicting a limb heat map by using the limb heat map prediction network according to the feature data to obtain the limb heat map of the human target; based on the limb heat map, the post-processing module is used to regress and connect the key parts of the same human target to determine posture information of the human target, specifically comprising: calculating the Euclidean distance between the midpoint of each edge of the head bounding box in the to-be-recognized picture and the midpoint of each edge of the trunk bounding box respectively; establishing the connection relationship between the head target and the trunk target by taking the midpoint of the edge of the head bounding box with the shortest Euclidean distance and the midpoint of the edge of the trunk bounding box; respectively, with the four vertices of the trunk bounding box as the centers, and the length of the long side of the trunk bounding box as the radius, four first circular regions are constructed; respectively, with the four vertices of the trunk bounding box as the centers, and the length of the long side of the trunk bounding box as the radius, four first circular regions are constructed; selecting the radius with the maximum heat map value sum in each of the four first circular regions as a target radius to obtain four first target radii; the heat map value sum is the sum of the heat map values of each point on the radius; A second circular region is constructed with the center point of the preset key part bounding box as the center and the length of the long side of the trunk bounding box as the radius. The preset key parts include the right hand, the left hand, the right foot and the left foot. selecting the radius with the maximum heat map value sum in the second circular region as a target radius to obtain a second target radius; The intersection point of the first target radius corresponding to the preset key part and the extension line of the first target radius is a turning point corresponding to the preset key part. The first target radius corresponding to the right hand, the left hand, the right foot and the left foot is respectively obtained in a first circular region with the right upper vertex, the left upper vertex, the right lower vertex and the left lower vertex of the trunk bounding box as the center. The first target radius corresponding to the right hand, the left hand, the right foot and the left foot is respectively obtained in a first circular region with the right upper vertex, the left upper vertex, the right lower vertex and the left lower vertex of the trunk bounding box as the center. establishing the connection relationship of the vertex of the trunk bounding box corresponding to the preset key part-the inflection point corresponding to the preset key part-the center point of the preset key part in turn; the vertices of the trunk bounding box corresponding to the right hand, the left hand, the right foot and the left foot are the right upper vertex, the left upper vertex, the right lower vertex and the left lower vertex of the trunk bounding box respectively. 2.The target recognition based human pose detection method of claim 1, wherein, The feature extraction network comprises 20 convolutional layers; The key part prediction network comprises 5 convolutional layers and two fully connected layers connected after the 5 convolutional layers; The limb heat map prediction network is an hourglass network, and the hourglass network comprises 5 convolutional layers. 3.The target recognition based human pose detection method of claim 1, wherein, Before acquiring the to-be-recognized picture in the target scene, the following steps are further included: constructing a key part data set and a connected limb data set; training the key part prediction network based on the key part data set; training the limb heat map prediction network based on the connected limb data set. 4.The target recognition based human pose detection method of claim 3, wherein, The construction of the key part data set and the connected limb data set specifically comprises: acquiring a plurality of sample pictures in a target scene; annotating a human body bounding box, a head bounding box, a trunk bounding box, a left hand bounding box, a right hand bounding box, a right foot bounding box and a left foot bounding box of a human body target in each of the sample pictures to obtain key part labels of each of the sample pictures; wherein the trunk bounding box is a tiltable rectangle, and the human body bounding box, the head bounding box, the left hand bounding box, the right hand bounding box, the right foot bounding box and the left foot bounding box are all non-tiltable rectangles; annotating a right upper arm bounding box, a right lower arm bounding box, a left upper arm bounding box, a left lower arm bounding box, a right thigh bounding box, a right shank bounding box, a left thigh bounding box and a left shank bounding box of a human body target in each of the sample pictures to obtain connection limb labels of each of the sample pictures; wherein the right upper arm bounding box, the right lower arm bounding box, the left upper arm bounding box, the left lower arm bounding box, the right thigh bounding box, the right shank bounding box, the left thigh bounding box and the left shank bounding box are all tiltable rectangles; converting the key part labels of each of the sample pictures into an array form to obtain a key part data set; converting the connection limb labels of each of the sample pictures into heat maps that decay from the center to the edge to obtain a right upper arm heat map, a right lower arm heat map, a left upper arm heat map, a left lower arm heat map, a right thigh heat map, a right shank heat map, a left thigh heat map and a left shank heat map; integrating the right upper arm heat map, the right lower arm heat map, the left upper arm heat map, the left lower arm heat map, the right thigh heat map, the right shank heat map, the left thigh heat map and the left shank heat map of the same human body target in each of the sample pictures into a right arm heat map, a left arm heat map, a right leg heat map and a left leg heat map to obtain a connection limb data set. 5.The target recognition based human pose detection method of claim 3, wherein, a loss function for training the key part prediction network is: wherein, represents a loss function for training the keypoint prediction network; represents a human target coordinate loss, x, y, w and h are respectively a horizontal coordinate of a center point, a vertical coordinate of a center point, a width value and a height value of a human bounding box of the i-th row and the j-th column cell predicted, , , and are respectively a horizontal coordinate of a center point, a vertical coordinate of a center point, a width value and a height value of a human bounding box of the i-th row and the j-th column cell labeled, represents whether there is a human target in the i-th row and the j-th column cell, when there is a human target, the value of is 1, when there is no human target, the value of is 0; a human target positive sample confidence loss, c is the confidence of the predicted human bounding box of the i-th row and j-th column cell. a human target negative sample confidence loss, denotes whether a human target is absent in the cell at the ith row and jth column, and has a value of 1 when a human target is absent has a value of 0 when a human target is present;​​ For key part target coordinate loss, , represents whether the kth key part exists in the cell of the ith row and the jth column, when the kth key part exists, the value of is 1, when the kth key part does not exist, the value of is 0, and the kth key part is head, trunk, left hand, right hand, right foot or left foot; , , and are respectively the center point horizontal coordinate, the center point vertical coordinate, the width value and the height value of the kth key part bounding box of the cell of the ith row and the jth column obtained by prediction, , , and are respectively the center point horizontal coordinate, the center point vertical coordinate, the width value and the height value of the kth key part bounding box of the cell of the ith row and the jth column obtained by labeling; a confidence loss for keypoint target positive samples, wherein, represents the confidence of the kth keypoint bounding box of the ith row and jth column cell predicted. a keypoint target negative sample confidence loss, wherein, represents whether the kth keypoint is absent from the cell in the ith row and jth column, and when the kth keypoint is absent, has a value of 1 when the kth keypoint is present, has a value of 0; is a trunk angle loss, a is a predicted trunk angle of the i-th row and j-th column cell, is an annotated trunk angle of the i-th row and j-th column cell; indicates whether there is a trunk in the i-th row and j-th column cell, when there is a trunk, the value of is 1, when there is no trunk, the value of is 0; 、 、 、 、 、 and are weight parameters of the human body target coordinate loss, the human body target positive sample confidence loss, the human body target negative sample confidence loss, the key part target coordinate loss, the key part target positive sample confidence loss, the key part target negative sample confidence loss, and the torso angle loss, respectively. 6.The target recognition based human pose detection method of claim 3, wherein, a loss function for training the limb heat map prediction network is: wherein, is a loss function for training the limb heat map prediction network; is a value of the vth heat map at p predicted by the limb heat map prediction network, is a value of the vth heat map at p annotated, the vth heat map being a right arm heat map, a left arm heat map, a right leg heat map, or a left leg heat map. 7.The target recognition based human pose detection method of claim 1, wherein, the method further comprises: obtaining a picture after confidence competition by removing a human body bounding box whose confidence is less than a confidence threshold and a key part bounding box contained in the human body bounding box whose confidence is less than the confidence threshold in the predicted to-be-recognized picture; calculating an IOU value between any two human body bounding boxes in the picture after confidence competition; obtaining a picture after IOU value competition by removing a human body bounding box with a smaller C value and a key part bounding box contained in the human body bounding box with the smaller C value between the two human body bounding boxes whose IOU value is greater than an IOU value threshold; a calculation formula of the C value is: wherein, represents value, represents a confidence of a human body bounding box, , , , , and are confidence of a head bounding box, a torso bounding box, a right hand bounding box, a left hand bounding box, a right foot bounding box and a left foot bounding box, respectively; is a key part position confidence weight.

8. A human pose detection system based on target recognition, characterized by, the system is based on a human posture detection network model, and the human posture detection network model comprises a feature extraction network, a key part prediction network, a limb heat map prediction network and a post-processing module; output ends of the feature extraction network are connected with input ends of the key part prediction network and the limb heat map prediction network respectively, and output ends of the key part prediction network and the limb heat map prediction network are connected with an input end of the post-processing module; the system comprises: a to-be-recognized picture acquisition module configured to acquire a to-be-recognized picture in a target scene; The feature extraction module is configured to perform feature extraction on the picture to be recognized by using the feature extraction network to obtain feature data. The key part prediction module is configured to perform key part prediction by using the key part prediction network based on the feature data to obtain a key part of the human body target and the human body target. The limb heat map prediction module is configured to perform limb heat map prediction by using the limb heat map prediction network based on the feature data to obtain a limb heat map of the human body target. The posture information acquisition module is configured to perform regression connection on the key part of the same human body target based on the limb heat map by using the post-processing module to determine posture information of the human body target, and specifically includes: Calculate the Euclidean distance between the midpoint of each edge of the head bounding box in the predicted picture to be recognized and the midpoint of each edge of the torso bounding box; Establish the midpoint of the edge of the head bounding box with the shortest Euclidean distance and the midpoint of the edge of the torso bounding box as the connection relationship between the head target and the torso target; respectively, with the four vertices of the trunk bounding box as the centers, and the length of the long side of the trunk bounding box as the radius, four first circular regions are constructed; respectively, with the four vertices of the trunk bounding box as the centers, and the length of the long side of the trunk bounding box as the radius, four first circular regions are constructed; Select the radius with the maximum heat map value sum in each of the four first circular regions as a target radius to obtain four first target radii; the heat map value sum is the sum of the heat map values of each point on the radius; A second circular region is constructed with the center point of the preset key part bounding box as the center and the length of the long side of the trunk bounding box as the radius. The preset key parts include the right hand, the left hand, the right foot, and the left foot. Select the radius with the maximum heat map value sum in the second circular region as a target radius to obtain a second target radius; Calculate the first target radius corresponding to the preset key part and the... The intersection of the extensions of the lines is the inflection point corresponding to the preset key part; the right hand, left hand, right foot, and left foot correspond to The first target radius is obtained in the first circular region centered at the top right, top left, bottom right, and bottom left vertices of the torso bounding box. Establish the connection relationship of the top point of the torso bounding box corresponding to the preset key part-the inflection point corresponding to the preset key part-the center point of the preset key part in turn; the top points of the torso bounding boxes corresponding to the right hand, the left hand, the right foot and the left foot are the right top point, the left top point, the right bottom point and the left bottom point of the torso bounding box, respectively.

9. An electronic device, comprising: The computer program stored in the memory and executable on the processor, when the processor executes the computer program, realizes the method in any one of claims 1 to 7.

Images