Multi-person gesture recognition method based on quality score

Abstract

The present invention discloses a multi-person gesture recognition method based on quality scores, belonging to the field of target detection and gesture estimation. The method determines the detected gesture positioning accuracy based on the calculated quality score, and uses this to evaluate the post-processing heat map. The quality score is introduced into the heat map post-processing score to solve the difference in accuracy requirements between different key points, improve the quality of the detection frame generated by the target detector, solve the problem of gesture recognition of small targets, and obtain more accurate gesture estimation results. By jointly predicting and dividing key points, even if key points are occluded or key points are not within the candidate frame in a multi-person scene, key points with interfering information can still be correctly divided. The multi-person gesture recognition method based on quality scores of the present invention optimizes the target detector post-processing, key point division and connection methods, and effectively improves detection accuracy.

Description

Technical Field

[0001] The invention relates to a multi-person posture recognition method based on quality score, and belongs to the field of human posture recognition. Background Art

[0002] In recent years, computer vision research has focused on enabling machines to acquire visual understanding capabilities. With the development of artificial intelligence, the importance of computer vision has become increasingly prominent. The goal of artificial intelligence is to enable machines to possess human-like intelligence, encompassing capabilities such as vision, hearing, and language. Humans perceive the world through multiple senses, with vision being the most important, as it provides the majority of information. Not only do humans understand the world through vision, but vision is also essential for interacting with it. Therefore, vision is a core research area in computer vision.

[0003] As research in human pose recognition deepens, the increasingly complex application environments of pose estimation place higher demands on pose recognition technology. In the field of computer vision, pose estimation for single-person scenes is rarely used in real life, while multi-person pose estimation has gradually become a research hotspot. However, multi-person scenes present issues such as occlusion of different objects and high variability within a given action class. Given the highly flexible nature of human poses, pose recognition algorithms based on graphical structures and motion models struggle to maintain robustness.

[0004] Among the many algorithmic frameworks for human pose estimation, methods that directly learn and recognize human actions from image measurements have significantly improved the accuracy of human pose recognition. However, they face many challenges, such as foreground occlusion, background chaos, lighting, pose complexity, multiple overlapping people, and the high computational complexity of similarity calculations. They cannot adapt to changes in the scene, making them difficult to apply in real-world scenarios. The complexity of top-down methods increases linearly with the number of people in the image, making them unsuitable for real-time detection scenarios that require continuous operation. In contrast, bottom-up methods rely on heat maps to detect key points of all objects in a single scene and then group them into single person detections through post-processing. However, in this framework, the performance of the object detector affects the accuracy of pose detection. Even after post-processing steps such as pixel-level non-maximum suppression (NMS), linear integration, and grouping, the pose recognition results still rely on the quality of the candidate boxes. During the key point segmentation process, heat maps cannot distinguish between two similar key points of the same type. If these problems are not properly addressed, the accuracy of multi-person pose recognition will be significantly affected. Summary of the Invention

[0005] In order to solve the problems of occlusion and inaccurate key point division in the current multi-person posture estimation scenario and improve recognition accuracy, the present invention provides a multi-person posture recognition method based on quality score. The technical solution is as follows:

[0006] Step 1: Obtain a multi-person pose estimation scene graph to be identified;

[0007] Step 2: For the multi-person pose estimation scene graph, generate a preliminary human detection area using the object detection network;

[0008] Step 3: Input the preliminary human detection area into the symmetric space transformation network to obtain candidate boxes of different targets;

[0009] Step 4: Calculate the confidence of the candidate boxes of the different targets, use the quality scores predicted by OKS-Net to filter the candidate boxes, determine the detection accuracy based on the quality scores and filter the candidate boxes;

[0010] Step 5: Input the filtered image area into the pose generator, which estimates the connection lines of the corresponding target's overall key points;

[0011] Step 6: Through joint prediction and PAF optimization connection, more accurate posture recognition can be achieved by regression and classification of key points of the human body.

[0012] Optionally, the quality score calculation process includes:

[0013] The OKS-Net adopts the channel attention mechanism to adaptively perceive the occlusion of different postures, and learns the key point similarity between the estimated posture and the true value by sharing the posture features with the heat map regression;

[0014] The expression for OKS calculation is:

[0015]

[0016] Among them, d i represents the Euclidean distance between the labeled and predicted joint points, δ is the activation function, s represents the scale factor of the target, and k i Represents the key point constant for controlling attenuation, v i Visibility flag, which has three values: 0 means the point is not marked, 1 means it is marked but obscured, and 2 means it is marked and visible;

[0017] The quality score is expressed as taking the pose estimation feature map F as input, using the maximum pooling layer and the fully connected layer of the pose feature, obtaining the reweighting coefficient of each channel, and reweighting the pose feature by multiplying the coefficient:

[0018] Score=FC(Max(Fpool ))

[0019] Among them, Score is the quality score, FC is the output of the fully connected layer, Max(F pool ) is the calculation of the maximum pooling layer of the feature map;

[0020] Finally, the reweighted features are input into 3 convolutional layers and 3 fully connected layers, and the predicted quality score is calculated and output by the fully connected layer. The score interval is normalized between 0 and 1.

[0021] Optionally, the joint prediction in step 6 includes: after obtaining the detected target, performing feature extraction, taking the feature image U as input, and outputting the transformation matrix θ after convolution; secondly, finding the mapping position relationship according to the transformation matrix θ, and using the interpolation method to calculate the pixel value of each interpolation point to obtain the output V. For a w×h image input, positioning is generated according to each human 2D key point, while maintaining the position information and direction information between the limb areas, and the key points outside the box are connected to the target by the vector information between the key points to group the detected key points into a skeleton, because each bounding box has a related posture, thereby ensuring the correct division of the key points.

[0022] Optionally, in step 6, after detecting all key points, PAF converts the pose estimation task into solving the problem of optimal matching between different objects and key points and maximizing the weights of all edges through the relationship between key points and targets. The objective function is described as:

[0023]

[0024] Among them, F represents solving the global object-keypoint matching graph, Represents the edge weight value connecting key points i and j, Indicates whether to retain edges in the final matching graph

[0025] During the solution process, the following formula is used to constrain each posture detection object to match at most one k th Key Points:

[0026]

[0027]

[0028]

[0029] in, represents the j key point of target K, M represents the total number of detected targets, N K Indicates the number of candidate points for key point k;

[0030] The result graph F in the formula is decomposed into K subgraphs, such as:

[0031] F k =((H,J (k) ), ε (k) )

[0032] in, Target K key point set, represents the set of key point retained edges, H represents the result graph feature map, F k represents the kth subgraph in the result graph F, Indicates the edge connecting key points i and j. In the result of a given matching graph, for China Weighted Center The key points represented are divided into i th objects, as its k th nodes.

[0033] Optionally, in step 1, the multi-person posture estimation scene graph contains at least two target human target objects, the scene graph to be identified is a batch of pictures, or a continuously played video, and video frames are intercepted at time intervals as the posture estimation scene graph.

[0034] Optionally, in step 2, a deep learning network is used as the target detection network for feature extraction.

[0035] Optionally, the target detection network uses YOLOv3 as the backbone network of the detection network, performs feature extraction on the input multi-person posture estimation scene graph, learns human body features through the network, regresses the candidate box of the target detection through a sliding window, and uses the area within the candidate box as the preliminary human detection area generated in this step as output.

[0036] Optionally, the step three includes: performing spatial transformation on each human body detection area so that the human body is located in the center of the detection frame.

[0037] Optionally, the spatial transformation includes: moving and cropping.

[0038] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the multi-person gesture recognition method according to any one of the above items is implemented.

[0039] The beneficial effects of the present invention are:

[0040] (1) The present invention addresses the problem of inaccurate key point positioning caused by the different visibility and size of different key points, the difference in contribution of different key points, etc., which will produce biased quality scores, and the lack of strong correlation between the classification confidence and positioning confidence obtained only from the heat map post-processing score. The quality score is introduced into the heat map post-processing score to solve the size differences in accuracy requirements between different key points, improve the quality of detection frames generated by the target detector, solve the problem of posture recognition of small targets, and obtain more accurate posture estimation results;

[0041] (2) This invention addresses the issue of regional multi-person pose frameworks relying on the quality of the detection frame by dividing key points through joint prediction. Even in multi-person scenes where key points are occluded or not within the candidate frame, key points with interfering information can still be correctly divided. This approach optimizes the post-processing of the target detector, key point division, and connection, effectively improving detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 It is a regional multi-person pose estimation framework diagram.

[0044] Figure 2 This is an overall flow chart of the multi-person gesture recognition method based on quality score of the present invention.

[0045] Figure 3 This is a framework diagram of the quality score-based multi-person posture recognition method of the present invention.

[0046] Figure 4 It is the OKS-net network structure diagram of the method of the present invention.

[0047] Figure 5 This is a schematic diagram of the quality score screening candidate frame principle of the present invention.

[0048] Figure 6 This is a diagram showing the principle of optimizing key point division in the posture detection method of the present invention.

[0049] Figure 7 This is a schematic diagram of the key point division principle of the posture estimation PAF of the present invention.

[0050] Figure 8 This is a graph of the joint prediction posture estimation result of the present invention.

[0051] Figure 9 This is the result of the ablation experiment on the Campus dataset.

[0052] Figure 10 It is the key point score map of the visualization experiment.

[0053] Figure 11 It is a graph showing the key points connected in the joint estimation of the visualization experiment. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0055] First, the relevant contents involved in this application are introduced as follows:

[0056] The present invention is based on the optimization of the Regional Multi-person Pose Estimation framework. The Regional Multi-person Pose Estimation (RMPE) is a posture recognition framework for multi-person scenes. Regional Multi-person Pose Estimation is different from Single Person Pose Estimation (SPPE). RMPE relies on human body region detection. When there are many people to be estimated, it can more accurately estimate the posture of multiple people, such as Figure 1 As shown in Figure 2, the framework consists of three parts: Symmetric Transformative Network (SSTN), Non-maximum Suppression Network (NMS) and Pose-Guided Person Generation (PGPG). The framework structure is as follows: Figure 1 shown.

[0057] (1) Symmetric Space Transformation Network

[0058] To achieve multi-person pose recognition, the human pose detector first divides the input image into distinct single-object regions based on the person detection frames. Each region is then fed into a symmetric spatial transformation module, which automatically generates pose detection regions. Spatial transformations such as translation and cropping are used to center the person in the detection frame. During training, parallel SPPE is used to avoid local minima and improve the quality of candidate frames for the single-person regions. The estimated human pose and image are then inversely transformed and restored to the original image.

[0059] The symmetric space transformer network is a special structure in neural networks. It can perform geometric transformations on input data so that the network can remain robust after transformations such as rotation, translation, and scaling. The symmetric space transformer network consists of three components: a localization network, a grid generator, and a sampler.

[0060] The localization network accepts input data and generates parameters for controlling the transformation. The grid generator uses these parameters to generate a grid for transforming the input data. Finally, the sampler uses the generated grid to extract the transformed input data. By introducing the STN, the robustness of the neural network to the input data can be improved, allowing the neural network to accurately predict input data of different directions and sizes.

[0061] STN performs affine transformation on the two-dimensional data in the network. For example, θ1, θ2 and θ3 are used to represent three two-dimensional vectors. and The coordinates before and after the corresponding vector transformation, the posture generated after the transformation corresponds to the new mapping can be expressed as:

[0062]

[0063] After the pose estimation network, the generated pose is mapped to the original input image. Therefore, to map the input pose to the original image and represent it with its coordinates, it is necessary to add the SDTN network to calculate the γ used in the conversion process and generate the grid candidate box based on γ:

[0064]

[0065] The SDTN network is the reverse operation of the STN network, so we can get:

[0066] [γ1γ2]=[θ1θ2] -1

[0067] γ3=-1×[γ1γ2]θ3

[0068] To back propagate through SDTN, can be derived as:

[0069]

[0070] in, and It can be directly deduced from the formula that after the extraction operation of the human body area is completed, the SPPE network is used to estimate the human body posture more finely. During the network training process, SSTN is maintained in parallel with SPPE to obtain high-quality candidate frames.

[0071] Replacing the output regression loss with a parallel SPPE at the center improves the accuracy of the candidate regions, but at a cost to system performance. While the STN can transform some inputs, it cannot accurately place the person at the true label's location. Furthermore, the coordinate space discrepancy between the input and the label affects its pose estimation capabilities, resulting in a decrease in the performance of the main branch SPPE. To ensure the accuracy of the STN and SPPE networks, the weights in the parallel SPPE framework need to be frozen. This ensures that non-central poses produce a larger error, encourages the STN to generate central poses, and maintains the performance of the main branch SPPE.

[0072] To further improve the STN's ability to extract dominant regions of humans, a parallel SPPE branch is added to the training network. This branch does not include the SDTN and shares the STN with the original network. In the parallel SPPE, the pose label is fixed at the center and compared with the position of the label of the true pose. During the training phase, the training network layers are frozen, keeping the weights fixed. The pose error at the center is backpropagated to the STN module. If the extracted STN pose deviates from the center, the parallel branch amplifies the loss and propagates in reverse, helping the STN to correct the accuracy of region learning and extract the correct human body region. During the network testing phase, the parallel SPPE branch is discarded. During network training, the parallel SPPE can be regarded as a regularizer. The STN maintains the center pose of the extracted human body region, ensuring that the local minimum is reached as much as possible during training. The compensation from the SDTN reduces the network error. The STN is trained to move the person to the center of the extraction region, allowing the SPPE to accurately estimate the pose and avoiding local minima that affect the detection results.

[0073] (2) Posture parameter non-maximum suppression network

[0074] Through the symmetric spatial transformation network, multiple candidate boxes are generated for each single target region. Indiscriminately inputting these multiple candidate boxes into the pose generator will result in a large amount of redundant computation. Therefore, by parameterizing pose non-maximum suppression, a metric is established to measure the similarity between poses, and the generated detection candidate boxes are refined to eliminate redundant poses and reduce the number of parameters. In multi-person pose estimation, whether high-quality target detection regions can be provided as input will directly affect the final results of the key point detection network. A better understanding of the quality of detection boxes is an issue that requires special attention in the field of target detection. By correcting the detection box confidence scores and using the overlap between detected boxes to refine the scores instead of eliminating low-scoring boxes, a higher AP can be achieved in the evaluation.

[0075] Non-maximum suppression (NMS) is used after all bounding boxes have been regressed and adjusted. NMS is then applied to each class in the image to prevent multiple bounding boxes from appearing for the same object. Furthermore, NMS requires setting a threshold based on experience, depending on the scenario. After adjusting the predicted box positions, repeated NMS is performed based on the number of classes, using NMS to find all bounding boxes for each class. NMS is not performed during training in YOLOv3. Because training is not required, gradient retention is also unnecessary. The detection network generates many duplicate object boxes. NMS removes redundant object boxes and retains those with high confidence. This involves three steps: First, select the object box A with the highest confidence from the candidate box list. Second, remove the object box whose IoU with A exceeds the threshold from the candidate box list. A is marked and moved to the retained box list. Finally, the above steps are repeated for each class until the candidate box list is empty.

[0076] (3) Posture guidance generator

[0077] Finally, the highest-scoring candidate box is fed into the pose generator, which then augments the training samples with the guided region generator. Simulating bounding boxes generates more training samples, achieving the task of pose keypoint detection. Pose generation involves two steps. The first step is pose integration, where the image and target pose are fed into a convolutional network to generate an initial, but rough, image of a person with the target pose. First, an approximate human pose is obtained, and the coordinates of the corresponding keypoints are generated. The pose is encoded as a heatmap at different locations, with each heatmap corresponding to multiple pixels surrounding a keypoint. Pixels in the heatmap that do not contain keypoint information are filled with zeros and concatenated together as input to the model. The second step is image refinement, which refines the initial blurry result by removing the fully connected layers.

[0078] However, using an adaptive non-maximum suppression algorithm to segment the bounding box's IoU and learn the correction score from the ground truth is not very effective. This is because different types of keypoints have different visibility and sizes, and the differences in their contributions will result in biased confidence scores. The classification confidence derived solely from the heatmap post-processing scores and the keypoint positioning confidence are not strongly correlated, leading to inaccurate keypoint positioning.

[0079] Example 1:

[0080] This embodiment provides a multi-person gesture recognition method based on quality score, including:

[0081] Step 1: Obtain a multi-person pose estimation scene graph to be identified;

[0082] Step 2: For the multi-person pose estimation scene graph, use the object detection network to generate preliminary human detection areas;

[0083] Step 3: Input the preliminary human detection area into the symmetric space transformation network to obtain candidate boxes of different targets;

[0084] Step 4: Calculate the confidence of the candidate boxes of different targets, use the quality scores predicted by OKS-Net to filter the candidate boxes, judge the detection accuracy based on the quality scores and filter the candidate boxes;

[0085] Step 5: Input the filtered image area into the pose generator, which estimates the connection lines of the corresponding target's overall key points;

[0086] Step 6: Through joint prediction and PAF optimization connection, more accurate posture recognition can be achieved by regression and classification of key points of the human body.

[0087] Example 2:

[0088] This embodiment provides a multi-person gesture recognition method based on quality score, see Figure 2 , the method comprising:

[0089] Step 1: Get the scene graph of multiple people’s poses to be identified:

[0090] In step 1, a multi-person pose estimation scene graph contains at least two human target objects. The scene graph to be identified can be a batch of pictures or a continuously played video. Video frames are intercepted at certain intervals as the pose estimation scene graph.

[0091] Step 2: Generate the target human detection area based on the multi-person pose estimation scene graph obtained in step 1, which specifically includes the following processes:

[0092] The object detection network generates a detection region for each target in the multi-person pose estimation scene image input in step 1. Targets in the input image are represented by rectangular boxes. The object detection network employs a deep learning approach, using YOLOv3 as the detection network backbone. Feature extraction is performed on the input multi-person pose estimation scene image. The network learns human features and regresses candidate bounding boxes for target detection using a sliding window. The area within the candidate boxes serves as the output for the preliminary human detection region generated in this step.

[0093] Step 3: Input the target human detection area obtained in step 2 into the symmetric space transformation network to obtain the target detection area, which specifically includes the following process:

[0094] Each human detection region is spatially transformed, such as moving and cropping, to center the person within the detection frame. In the symmetric spatial transformation network, the parallel SPPE branch and the previous branch share the same STN to process candidate frames. The training label is set to the true pose of a single person, and the layers and weights of this branch are fixed during training. When the error between the output and the true result is large, a large loss is applied to help the STN perform the correct transformation, resulting in high-quality human frames and refined candidate frames as output.

[0095] Step 4: Obtain the detection boxes of different targets according to step 3, calculate their confidence scores, and filter candidate boxes. Specifically, the following steps are performed:

[0096] After the input image passes through the object detection network, multiple detection frames with different confidence levels are generated. OKS-Net evaluates the quality of each detection frame and uses the quality score predicted by OKS-Net instead of NMS to screen candidate frames. The detection accuracy is judged based on the quality score and the candidate frames are filtered.

[0097] In addition to calculating the heatmap score of each key point, the introduced OKS-Net network calculates the quality score in parallel, such as Figure 5 As shown in the figure, through self-supervised learning, the network model generates its own intermediate representations and uses quality scores to assist training, providing more reliable candidate boxes. The screening strategy mainly regresses scores from visible key points to reduce the impact of occluded key points.

[0098] OKS-Net, such as Figure 4 As shown in the figure, an end-to-end quality prediction module is constructed. A channel-attention mechanism is used to adaptively perceive occlusions of different poses, ensuring that the occluded pose predictions receive the correct score. By sharing pose features with heatmap regression, keypoint similarities between the estimated pose and the true value are learned. A maximum pooling layer and a fully connected layer are used to calculate the reweighting coefficients for each channel, and the pose features are reweighted by multiplying them. Finally, the channel-attention reweighted features are input into three convolutional layers and three fully connected layers. The predicted quality score is calculated and output by the fully connected layer. The score range is normalized between 0 and 1, allowing the entire network to perform end-to-end training and inference.

[0099] After the corresponding pose annotations are given, the predicted keypoint positions are obtained from the heat map using standard post-processing techniques, and the loss function is calculated as:

[0100] Loss = λ heatmap L heatmap +λ OKS L OKS

[0101] Among them, L heatmapis the heatmap loss, L OKS is the predicted OKS (Object Keypoint Similarity) loss, where λ heatmap The weight is set to 2000, λ OKS Set to 0 or 1 to ensure that data imbalance is alleviated during training. This inference process also conforms to the traditional pose estimation framework. For each input image, the network will generate K pose heat maps and corresponding OKS predictions. The heat map is used to select the locations of K key points to depict the pose quality score, which has a good effect.

[0102] For a single target, the network generates multiple candidate boxes for target detection, and the detection area is evaluated by the confidence of the candidate boxes. In key point evaluation, the definition of OKS is the same as the similarity metric that plays the same role as OKS in target detection, namely:

[0103]

[0104] Among them, δ is the activation function, s represents the scale factor of the target, k i Represents the key point constant for controlling attenuation, v i Visibility flag, which has three values: 0 means the point is not marked, 1 means it is marked but obscured, and 2 means it is marked and visible.

[0105] Since the key point scores in the heat map belong to the classification confidence, and in the non-fixed position confidence, the correlation of its positioning accuracy cannot be effectively associated with the classification accuracy; secondly, different types of key points have different impacts on the weight of pose estimation, such as hips and eyes, and their accuracy requirements are significantly different. The heat map post-processing score will be heavily biased towards the higher OKS and cannot produce the best score. The heat map post-processing score is not suitable as a criterion for key point selection. Therefore, for each pose key point confidence, the score value generated by OKS is between 0-1, and the candidate box IOU loss is extended from the bounding box to the key point. The output quality score is used as the pose evaluation standard and the confidence with the highest score is used to reflect the detection quality. The final human detection area candidate box is output as the input of the pose estimation network.

[0106] Step 5: Input the filtered image area into the pose generator, which estimates the connection lines of the corresponding key points of the target. Specifically, it includes:

[0107] The pose generator regresses the position information of the key points and uses the key point coordinates as the target for the final network regression. This method directly obtains the direct position information of each coordinate point. In the heat map corresponding to each key point, the key point at each position is regarded as a class, and its coordinates are represented by a probability map. Each pixel position in the image is given a probability, indicating the probability that the point belongs to the corresponding category key point. The closer the pixel is to the key point, the closer the probability is to 1, and the farther the pixel is from the key point, the closer the probability is to 0. This can be simulated by a Gaussian function. If the same pixel position is at different distances from different key points, that is, the probability of the position relative to different key points is different, then its final coordinates can be obtained by taking the maximum value or the average value.

[0108] Step 6: Through joint prediction and PAF optimization connection, regression and classification of key points of the human body are achieved for more accurate posture recognition, including:

[0109] According to the key point information saved in step 5, the joint prediction and PAF optimization connection method are used to achieve more accurate posture recognition by regression and classification of key points of the human body, such as Figure 6 shown.

[0110] The joint prediction method takes the feature image U as input after the detection target is obtained, and outputs the transformation matrix θ after convolution. Secondly, the mapping position relationship is found based on θ, and the pixel value of each difference point is calculated using interpolation to obtain the output V. For a w×h image input, positioning is generated based on each 2D key point of the human body, while maintaining the position and direction information between the limb areas. Figure 8 As shown, there is no restriction on the key points to be within the predicted bounding box. Since the key point information contains its own vectorial properties, the segmented parts are connected as a whole based on the confidence level. Key points outside the bounding box due to occlusion and other reasons can still be correctly identified and divided. The joint prediction connects the key points outside the box with the target by the vector information between the key points to group the detected key points into a skeleton. Because each bounding box has an associated posture, the correct division of the key points is guaranteed.

[0111] PAF (Part Affinity Fields) associates key points and limbs based on the most bipartite matching, and ultimately obtains all the postures of all people in the image. The Hungarian algorithm is used to obtain the best match. Instead of using the complete graph, the minimum number of edges is selected to obtain the spanning tree skeleton of the human posture, decomposing the matching problem into a set of bipartite matching sub-problems and independently determining the matches in adjacent tree nodes. The position and direction information of the limb area is obtained. Each PAF is a two-dimensional vector field for each limb. For each pixel in the area belonging to a specific limb, a two-dimensional vector encodes the direction from one part of the limb to another. Each type of limb has a corresponding PAF that connects its two related body parts.

[0112] The 2D confidence map of the key point part consists of j elements, represented as S = {S1, S2, ... S j}, where S j ∈R wxh , j∈{1......J}; the coding graph of the connection between parts is represented as L={L1,L2,...L C}, the element L in L c ∈R wxhx2 , c∈{1......C}.

[0113] PAF divides the key points into Figure 11 As shown in the figure, for the key points that are not divided, k represents the target to be detected (k = 0, 1, 2...), c represents the limb number, j1 and j2 represent the first and second key points. If the point falls on the limb c key point of a certain k individual, for the limb coordinates of j1 and j2 use Represents the unit vector pointing between j1 and j2, which is expressed as follows:

[0114]

[0115] According to the range formed by the width and height of the limb direction, any point falling within the area will be assigned a value Otherwise, the value is 0.

[0116] PAF uses the relationship between key points and targets. After detecting all key points, the pose estimation task is transformed into: solving the problem of optimal matching between different objects and key points and maximizing the weights of all edges. The objective function can be described as:

[0117]

[0118] Among them, F represents solving the global object-keypoint matching graph, Represents the edge weight value connecting key points i and j, Indicates whether to retain edges in the final matching graph During the solution process, the following formula is used to constrain each posture detection object to match at most one k th Key Points:

[0119]

[0120]

[0121]

[0122] in, represents the j key point of target K. M represents the total number of detected targets, N K The result graph F in the formula can be decomposed into K subgraphs, such as:

[0123] F k =((H,J (k) ), ε (k) )

[0124] in, In the results of a given matching graph, for China Weighted Center The key points represented will be divided into i th objects, as its k th Therefore, in this way, the candidate region of each pose detection object can be obtained, and objects that do not match any node will be removed.

[0125] for Figure 11 The midpoint P2 is divided into the detection frame area of ​​other targets due to the interference caused by the occlusion of the adjacent target action and the area range limitation of the detection frame. However, in the actual division process of the key points, it can be determined that each key point belongs to only one posture detection object. Therefore, in the process of dividing each key point according to the confidence score in different objects, it is only necessary to ensure that the key point with the highest score in an object is divided into the object. If the key point is also the primary candidate point of other objects, then copy the point and assign it to other objects. After the multi-target key point division is completed by joint prediction and the multi-target key point connection is completed by PAF, a more accurate target posture estimation skeleton map is output.

[0126] The present invention achieves more accurate multi-person posture recognition by optimizing the regional multi-person posture estimation framework. In order to improve the quality of the detection frame, the detected posture positioning accuracy is judged according to the calculated quality score, and the post-processing heat map is evaluated on this basis. At the same time, through joint prediction, the key points that are blocked or incorrectly divided outside the boundary box are divided to improve the detection accuracy. The key point positioning is inaccurate due to the problems such as different types of key points having different visibility and sizes, and the difference in contributions of different key points, which will produce biased quality scores, and the classification confidence and positioning confidence obtained only by the heat map post-processing score have no strong correlation. The quality score is introduced into the heat map post-processing score to solve the size differences in accuracy requirements between different key points, improve the quality of the detection frame generated by the target detector, and solve the posture recognition of small targets; in order to solve the problem that the regional multi-person posture framework depends on the quality of the detection frame, the key points are divided through joint prediction. Even if there are problems such as key point occlusion and key points not being in the candidate frame in multi-person scenes, the key points with interference information can still be correctly divided, such as Figure 3 Through the above methods, the post-processing, key point division and connection of the target detector are optimized, which effectively improves the detection accuracy.

[0127] In order to verify the effectiveness of the multi-person gesture recognition method based on quality score proposed in this invention, the following experiment is conducted:

[0128] 1. Experimental environment and parameters

[0129] The method of the present invention is implemented using PyCharm 2020.2.3, and is run on a laptop with an Intel Core i7-8750 processor, 2.2 GHz, 12 cores, 16 GB of memory, and an NVIDIA Geforce GTX 1060 with Max-Q Design graphics card, and a performance comparison analysis is performed with multiple methods.

[0130] The data processing method is to first scale the input image to a fixed size of 256×256, use standard data preprocessing to process the input data, and use random scale data augmentation and random inversion with a probability of 0.5 during training. The training settings are based on a learning rate of 1e-2 and a model iteration number of 200. The first 100 iterations L OKS Set it to 0 to obtain a good network backbone for pose estimation, then set it to 1 every 50 iterations to train OKS-Net, with the learning rate reduced by a factor of 0.1. The Adam optimizer was used during training to ensure stability and fast convergence. During testing, the input image was first resized to the required size, preserving the aspect ratio. The pose score was then calculated by averaging the predicted OKS scores for the original and flipped inputs.

[0131] 2. Experimental Dataset

[0132] The experiments used the COCO 2017 dataset and the Campus dataset. The COCO dataset contains over 200,000 images, 250,000 instances of people, and 17 keypoints. The training set consists of 57,000 images and over 1 million labeled keypoints, while the validation set and test set consist of 5,000 and 20,000 images, respectively. The Campus dataset uses cameras to capture human activities in outdoor environments and consists of multiple people performing different actions.

[0133] The dataset was split into training and test sets. To avoid overfitting the training data, the 2D portion was trained on the COCO keypoint dataset, and the 3D portion was trained using the camera parameters of the campus dataset. Synthetic 3D pose and 2D heatmap pairs were generated. The model was trained on the split training set, and the experimental results were compared on the test and validation sets.

[0134] 3. Experiment and results analysis

[0135] Experiment 1: Ablation experiment

[0136] To verify the impact of the added quality score and joint prediction network on the RMPE framework's feature extraction capabilities and human pose prediction accuracy, we constructed separate network structures and conducted ablation experiments. These experiments were conducted on the COCO 2017 test set and the campus dataset, analyzing the different components of the method. After introducing OKS-Net, a quality score loss was added to the loss function. The model was trained using a scale normalization loss, leveraging scale invariance to balance the impact of the loss formula.

[0137] By selecting representative scenes from the campus dataset, experiments are conducted on the detection results in single-person scenes, multi-person uncrowded scenes, multi-person crowded scenes, and multi-person occlusion scenes.

[0138] In different scenes, the pose detection results in the single-person scene are all good. Since the scene is not affected by complex environments such as occlusion, the pose estimation task is relatively simple. There are problems in the picture such as occlusion and the shooting angle causing the target to become smaller. Figure 9 The algorithm missed detection of some pose detection objects, and the key point detection and segmentation of occluded objects were inaccurate. By incorporating quality scores and a joint prediction method, the algorithm accurately identified the missed objects in the image and optimized the segmentation of occluded objects, effectively improving detection accuracy. In scenes with multiple people at medium and long distances, the optimized algorithm clearly demonstrated excellent results in crowded areas with multiple people, achieving more accurate key point detection and segmentation.

[0139] Different LOSSs were used in the experiment, among which AP50 This refers to an OKS threshold of 0.5. The experimental results in Table 1 show that after introducing the quality score into the basic network architecture, the model trained using the L1 loss performs better. The visibility and size of different key points of the same target, as well as those of similar key points of different targets, vary, and their contribution to the loss also varies. Therefore, quality score prediction is the most appropriate criterion for measuring human pose, enabling the model to better learn pose detection. Secondly, in multi-person pose detection, for scenarios where the same target key point is not within the candidate box, or where multiple target key points are similar or adjacent, a joint prediction method is introduced. By optimizing the key point segmentation strategy, the problem of loss of detection accuracy in occlusion and crowded scenes is addressed, thereby improving overall detection accuracy.

[0140] Table 1 Ablation experiment results

[0141]

[0142] Experiment 2: Comparative experiment

[0143] To verify the advantages of the posture recognition method of the present invention over other methods, the present invention is compared with some mainstream methods. The experiment compares the performance of different backbone networks and compares the competitiveness of the algorithms by parameter quantity, accuracy, and recall rate. The comparison results on the COCO dataset are shown in the table.

[0144] Table 2 contains a comparison between the proposed method and the current bottom-up method. Experimental results show that the proposed method is comparable to the existing method in terms of AP. After adding the OKS quality score to select the pose detection candidate box, the proposed method optimizes the key point connection strategy for the occluded and adjacent targets through joint prediction. The model has a better performance in AP and AP. 50 The improvement is the most significant. While slightly increasing the number of parameters, the detection accuracy is improved, and the posture in multi-person scenes can be predicted more accurately.

[0145] In experiments on a campus dataset, we assumed that everyone's identity was known and labeled every object in the input image. We compared this dataset with several state-of-the-art methods. Due to varying scenes, the number of objects in an image varies, as does the level of environmental occlusion and crowding. Therefore, we conducted pose detection experiments in single-person and multi-person scenarios, yielding the following results:

[0146] Table 2 Comparative experimental results of COCO dataset

[0147]

[0148]

[0149] Table 3 Comparative experimental results of campus dataset

[0150]

[0151] The detection accuracy scores in Table 3 show that the present invention performs well in single-person scenarios. Furthermore, in multi-person scenarios, the identities of different targets are not labeled and are set as unknown. All detected target body parts within the scene that are contained in the same space are considered to be in the same state. Experimental results show that the present invention can effectively handle both self-occlusion and natural occlusion in multi-person scenarios, performing relatively well in pose detection and demonstrating good generalization capabilities.

[0152] Experiment 3: Visualization Experiment

[0153] The visualization of the ablation experiment can more intuitively show the effect. The experimental results are scored according to the accuracy. Figure 10 The figure shows the comprehensive prediction scores of the 17 key points corresponding to all numbered targets. The images in this dataset contain images of different scales, different scenes, and different numbers of people, with varying degrees of occlusion and crowding.

[0154] The visualization results show that the degree of occlusion varies in different scenarios. In some cases where multiple people are occluded or even almost completely occluded, some individual numbered targets are not detected, resulting in a small phase difference. However, the detection of most numbered targets is relatively accurate, and the overall score of the detection of the corresponding numbered object pose joints has improved.

[0155] In the multi-person pose estimation framework, key point detection is performed by locating the human body area within the detection frame. Different key point division and connectivity processes require different targets to be divided separately. Since the detection frame division process in the figure is determined based on IOU, the detection frame generated for each target can contain most of the human body area. However, there are still some scenes where a few key points are divided outside the detection frame. Therefore, for Figure 11 In the figure, points p1 and p2 belonging to the right detection box area are divided into the left detection box. Of course, in scenes with more severe occlusion or overlap, there will be errors in the division of key points even if the high accuracy of the detection box is maintained.

[0156] Without the joint prediction method to connect the key points in the figure, we can see that in areas with multiple key points scattered, the result of the detection frame division leads to incorrect connection or even failure to connect the key points. However, after adding the joint prediction algorithm, based on the position and direction information corresponding to each point, even if the detection frame divides the area incorrectly, it is still possible to locate the key points with occlusion (such as key point p2) and connect them to the corresponding target.

[0157] Through the above experiments, it can be seen that the mass score-based fever multi-person posture recognition method provided by the present invention improves and optimizes the regional multi-person posture estimation framework algorithm, optimizes the candidate box selection strategy in the multi-person posture estimation problem, and can effectively connect the key points of the occluded area. It has achieved good results in the multi-person posture estimation scenario and can effectively deal with many problems caused by overlap or occlusion in multi-person scenes, such as target overlap, key point connection errors, invisible target key points, etc., and the experimental performance has achieved obvious advantages compared with the regional multi-person posture estimation framework experiment.

[0158] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-person gesture recognition method based on quality score, characterized in that: The method comprises: Step 1: Obtain a multi-person pose estimation scene graph to be identified; Step 2: Generate preliminary human detection regions using a target detection network for the multi-person pose estimation scene graph; Step 3: Input the preliminary human detection area into the symmetric space transformation network to obtain candidate boxes of different targets; Step 4: Calculate the confidence of the candidate boxes of the different targets, use the quality scores predicted by OKS-Net to filter the candidate boxes, determine the detection accuracy based on the quality scores and filter the candidate boxes; Step 5: Input the filtered image area into the pose generator, which estimates the connection lines of the corresponding target's overall key points; Step 6: Through joint prediction and PAF optimization connection mode, regression and classification of human key points are achieved for more accurate posture recognition; The calculation process of the quality score includes: The OKS-Net adopts the channel attention mechanism to adaptively perceive the occlusion of different postures, and learns the key point similarity between the estimated posture and the true value by sharing the posture features with the heat map regression; The quality score is expressed as taking the pose estimation feature map F as input, using the maximum pooling layer and the fully connected layer of the pose feature, obtaining the reweighting coefficient of each channel, and reweighting the pose feature by multiplying the coefficient: Among them, Score is the quality score, FC is the output of the fully connected layer, Calculate the maximum pooling layer of the feature map; Finally, the reweighted features are input into three convolutional layers and three fully connected layers. The predicted quality score is calculated and output by the fully connected layers. The score interval is normalized between 0 and 1. The joint prediction in step 6 includes: after obtaining the detection target, performing feature extraction, and converting the feature image As input, after convolution, the output transformation matrix ; Secondly, according to the transformation matrix Find the mapping position relationship and use interpolation to calculate the pixel value of each interpolation point to get the output , for a The image input generates positioning according to each 2D key point of the human body, while maintaining the position information and direction information between the limb areas. The key points outside the box are connected to the target by the vector information between the key points to group the detected key points into a skeleton, because each bounding box has an associated posture, thereby ensuring the correct division of the key points.

2. The multi-person gesture recognition method according to claim 1, characterized in that: The expression for OKS calculation is: in, represents the Euclidean distance between the labeled and predicted joint points, is the activation function, represents the scale factor of the target, represents the key point constant for controlling attenuation, Visibility flag, which has three values: 0 means the point is not marked, 1 means it is marked but obscured, and 2 means it is marked and visible.

3. The multi-person gesture recognition method according to claim 2, characterized in that: In step 6, PAF transforms the pose estimation task into solving the problem of optimal matching between different objects and key points and maximizing the weights of all edges after detecting all key points through the relationship between key points and targets. The objective function is described as: in, F Represents the object-keypoint matching graph for solving the global problem, b Represents key points i and j The weight of the connected edges, Indicates whether to retain edges in the final matching graph ; During the solution process, the following formula is used to constrain each posture detection object to match at most one k th Key Points: in, Indicates the target K of j Key points, M Indicates the total number of detected targets, N K Represents key points k The number of candidate points; Result graph in formula F Decompose into K subgraphs, such as: in, Target K key point set represents the set of key point preserving edges, Represents the result graph feature map, represents the kth subgraph in the result graph F, Represents key points i and j The connected edges, in the result of a given matching graph, for China Weighted Center The key points represented are divided into objects, as its nodes.

4. The multi-person gesture recognition method according to claim 1, wherein: In step 1, the multi-person pose estimation scene graph contains at least two human target objects, the scene graph to be identified is a batch of pictures, or a continuously played video, and video frames are intercepted at time intervals as the pose estimation scene graph.

5. The multi-person gesture recognition method according to claim 1, characterized in that: In step 2, a deep learning network is used as the target detection network to perform feature extraction.

6. The multi-person gesture recognition method according to claim 5, characterized in that: The target detection network uses YOLOv3 as the backbone network of the detection network, extracts features from the input multi-person pose estimation scene graph, learns human features through the network, regresses the candidate box of the target detection through a sliding window, and uses the area within the candidate box as the preliminary human detection area generated in this step as the output.

7. The multi-person gesture recognition method according to claim 5, characterized in that: The step 3 includes: performing spatial transformation on each human body detection area so that the human body is located in the center of the detection frame.

8. The multi-person gesture recognition method according to claim 7, characterized in that: The spatial transformation includes: moving and cutting. 9 . A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the multi-person gesture recognition method according to claim 1 .

Images