A lightweight method and device for detecting key points on the whole body of multiple people.

Abstract

This invention provides a lightweight method and apparatus for detecting keypoints across multiple individuals' entire bodies, comprising the following steps: obtaining predicted coordinates of facial, hand, and foot keypoints from the MPII human pose dataset, and combining these predicted coordinates with body keypoint coordinates to form ground truth labels; using the labeled individual data to form a full-body keypoint detection dataset; designing the first 22 layers of a residual 50-convolutional block attention network as the backbone of the full-body keypoint detection network, and combining it with N lightweight and powerful keypoint detection sub-networks to form a full-body keypoint detection network; then training the detection network on the full-body keypoint detection dataset, using two loss functions, label loss and distillation loss, for auxiliary training; and deploying the trained model for real-time detection and recognition. The detection method of this invention exhibits high accuracy and low parameter count performance.

Description

Technical Field

[0001] This invention relates to the field of key point detection of the whole body, and more specifically, to a lightweight method and device for detecting key points of multiple people's whole body. Background Technology

[0002] Human landmark detection has been an open problem studied for decades. Initially, most research focused on facial landmark detection, but it later evolved to include single and multi-person human landmark detection, including body, foot landmarks, and the more challenging hand landmark detection. A recent and more challenging research effort is to integrate all these landmark detection tasks into a single algorithm to achieve full-body (body, face, hands, and feet) landmark detection.

[0003] Full-body keypoint detection can be directly applied to scenarios including abnormal behavior detection, augmented reality, virtual reality, medical applications, and sports analytics. It can also provide more detailed information for 3D human keypoint detection and mesh reconstruction, person re-identification and tracking, and motion recognition. Despite these high demands, the first method to provide full-body keypoint detection was an early version of OpenPose. It first obtains the coordinates of all body keypoints from the input image in a bottom-up manner, then runs additional face and hand keypoint detectors for each detected person. As a multi-network approach, it directly uses existing body, face, and hand keypoint detection algorithms. However, it suffers from the following problems: if the body detector fails, recovery is impossible, especially when only the face or hands are visible in the image. Furthermore, its runtime is proportional to the number of people in the image, making full-body keypoint detection prohibitively expensive for multi-person and real-time applications. Osokin explored the design of networks for full-body keypoint detection, significantly optimizing the network design and post-processing code based on the OpenPose method. He employed an extended MobileNet v1 feature extractor with depthwise separable convolutions and a lightweight refinement stage with residual connections, ultimately improving the accuracy-to-network complexity ratio of full-body keypoint detection by more than 6.5 times. Hidalgo proposed the first single-network method for 2D full-body keypoint detection, simultaneously locating keypoints on the body, face, hands, and feet. This method follows a bottom-up detection approach, maintaining stable real-time detection performance regardless of the number of people in the image, significantly outperforming OpenPose in both speed and global accuracy. Furthermore, it eliminates the need for additional networks for each hand and face, greatly accelerating its speed in multi-person scenes.

[0004] However, although existing full-body keypoint detection methods have achieved certain detection results, they still have some limitations. (1) They cannot fully utilize the contextual information and spatial information related to keypoints, resulting in less than ideal detection accuracy. (2) The accuracy of facial and hand keypoint detection is still limited, especially in cases of severe background blur, small figures, and extreme gestures, where detection may fail. (3) The training efficiency of the models is low, the deployment cost of the models is high, and the number of parameters is large, which is not conducive to promoting the application of full-body keypoint detection on platforms with limited hardware and software resources.

[0005] In view of this, the present invention is hereby proposed. Summary of the Invention

[0006] In view of this, the present invention discloses a lightweight method for detecting key points on the whole body of multiple people. This method fully extracts contextual information related to key points at multiple scales by global key point information distillation and lightweight multi-scale dense connection residual design, and teaches the high-value information learned by the expert network to the student network, thereby achieving detection performance with high accuracy and low parameter quantity.

[0007] Specifically, the present invention is achieved through the following technical solutions:

[0008] This invention provides a lightweight method for detecting key points on the entire body of multiple individuals, comprising the following steps:

[0009] Predicted coordinates of facial key points, hand key points, and foot key points are obtained from the MPII human pose dataset, and the predicted coordinates are combined with the coordinates of body key points to form real labels. The labeled personal data is then used to form a full-body key point detection dataset.

[0010] The first 22 layers of the residual 50 convolutional block attention network are designed as the backbone network of the whole body keypoint detection network, which is then combined with N lightweight and powerful keypoint detection sub-networks to form the whole body keypoint detection network.

[0011] The detection network was then trained on the whole-body keypoint detection dataset, and two loss functions, label loss and distillation loss, were used for auxiliary training. The trained model was then deployed for real-time detection and recognition.

[0012] The detection method of this invention proposes a lightweight method for detecting and recognizing key points in multiple people's whole body based on global key point information distillation and multi-scale dense feature enhancement. Through global key point information distillation and lightweight multi-scale dense connection residual design, the relevant contextual information of multi-scale key points is fully extracted. Furthermore, multiple detection tasks (such as face key point detection, body key point detection, hand key point detection, and foot key point detection) are jointly trained, so that multiple key point detection problems can be solved simultaneously by a single network. In addition, the high-value information learned by the expert network is taught to the student network, thereby achieving convenient detection method, complete detection functions, low deployment cost, fast detection speed and high accuracy.

[0013] By jointly training multiple detection tasks (such as facial keypoint detection, body keypoint detection, hand keypoint detection, and foot keypoint detection), a single network can solve the problem of multiple keypoint detection simultaneously. This has the advantages of convenient detection method, low deployment cost, fast detection speed, and high accuracy.

[0014] Preferably, the state-of-the-art full-body keypoint detection algorithm uses predicted coordinates of facial, hand, and foot keypoints from the MPII (Max Planck Institute Informatics Human Pose Dataset). These predicted coordinates are then combined with the actual coordinates of 16 body keypoints for each individual from the MPII dataset to create full-body keypoint labels. Based on these labels, actual labels for each individual's Part Affinity Field (PAF) are created. The labeled individual data is then used to form the full-body keypoint detection dataset. Existing technologies do not yet address full-body keypoint datasets, and the full-body keypoint detection dataset established using this invention can further improve the accuracy of the detection network during subsequent training.

[0015] Preferably, during training, the backbone network is used to extract features from human images in the dataset, obtaining a low-level human image feature map (LF) of size 46×46×128. In terms of accuracy, because the Openpose backbone network uses the first 10 layers of VGG19, this invention uses a more advanced: an improved 50-layer Residual Nets + Convolutional Block Attention Module (ResNet-50+CBAM) with its first 22 layers. This deeper network, combined with knowledge distillation techniques to assist model training, improves accuracy.

[0016] Preferably, the key point detection subnetwork n=1 takes a 46×46×128 LF as input to generate the initial joint-related region prediction result R1.

[0017] Preferably, the keypoint detection subnetwork n∈{2,...,N-1} uses LF and the prediction result R of the previous adjacent stage. (n-1) As input, it produces increasingly accurate PAF prediction results R. n Finally, the subnetwork n=N uses the prediction results R from LF and the previous adjacent stage. (N-1) As input, it generates the final accurate predicted coordinates R of the whole body key points. N .

[0018] In terms of deployment cost, this invention uses a more lightweight modular design, typically employing four sub-networks in practice. These sub-networks consist of lightweight, densely connected residual blocks composed of 3x3, 1x1, and 3x3 convolutional layers. In contrast, Openpose uses six sub-networks, which are either 7x7 convolutional layers or complex ordinary residual blocks composed of multiple 3x3 convolutional layers (at least three). Furthermore, because this invention incorporates knowledge distillation, its detection speed and accuracy are not affected.

[0019] In summary, detection speed is related to the number of model parameters. Since our final deployment uses a more advanced residual 50-convolutional block attention network and a lightweight module design, it offers a significant speed advantage over the original, complex OpenPose, and is also more lightweight, enabling detection of more than 15 key points across the entire body. Therefore, the unique detection model design of this invention is not disclosed in existing technologies.

[0020] Preferably, during assisted training, the label loss L 标签 The label loss L applies to all subnetworks, where for subnetwork n∈{1,...,N-1}, the label loss L 标签 This is used to calculate the difference between the actual values ​​of whole-body joint-related regions and the predicted values ​​of whole-body joint-related regions by each subnetwork. For subnetwork n=N, the label loss L... 标签 This is used to calculate the difference between the actual values ​​of keypoints throughout the body and the predicted values ​​of keypoints throughout the body predicted by the keypoint detection subnetwork.

[0021] Preferably, L 标签 The calculation process is as follows:

[0022]

[0023] n represents the number of subnetworks, f represents the total number of joint-related regions, and k represents the total number of key points throughout the body. The representation of the subnetwork n's prediction results for the joint-related region f is shown below. The true value of f represents the joint-related region. This represents the prediction results for N pairs of key points k in the subnetwork. This represents the true value of the key point k.

[0024] Preferably, the distillation loss L 蒸馏 It operates on the keypoint detection subnetwork n∈{1,...,N-1} to calculate the difference between the intermediate prediction 46×46×384 feature map of the keypoint detection subnetwork N and the intermediate prediction 46×46×384 feature maps of each keypoint detection subnetwork n∈{1,...,N-1}, thereby performing global keypoint information distillation.

[0025] Distillation loss L 蒸馏 The calculation process is as follows:

[0026]

[0027] Where n represents the number of subnetworks, r n The feature map r represents the intermediate prediction of subnetwork n, which is 46×46×384. N The feature map representing the intermediate predictions of the expert network is 46×46×384.

[0028] The solution of this invention uses knowledge distillation for auxiliary training, thus achieving detection performance comparable to complex subnetworks even in such a lightweight network configuration.

[0029] Furthermore, to better integrate with the detection network of this invention and make it more suitable for its application scenarios, the knowledge distillation method employed in this invention uses global attitude distillation. This involves distilling the high-dimensional feature map (46×46×384) output from the intermediate layer of the keypoint detection network, which contains more information and preserves the connections and coordinates between keypoints. In contrast, existing keypoint knowledge distillation techniques only distill the coordinate information of keypoints, distilling only the 468×468×15 (number of keypoints) feature map output from the keypoint detection network. This results in less information being distilled. The distillation method of this invention preserves both the connections between keypoints and the keypoint information itself, improving the efficiency of attitude distillation. Existing distillation methods are unsuitable for the solution of this invention and fail to achieve its objectives.

[0030] In summary, the distillation method of this invention is perfectly suited to network model design. Only by designing the distillation target and the distillation loss function to contain more information and establish more connections can it be more suitable for the large network architecture with multiple similar sub-networks of this invention. Of course, the design of the network model itself also requires a lot of creative work. In order to improve the model's performance, a lot of practical operations have been carried out on the selection and improvement design of the backbone network and the efficient and lightweight design of each sub-network module.

[0031] Preferably, the overall loss function L 总 The calculation formula is expressed as follows:

[0032] L 总 =a×L 标签 +b×L 蒸馏 ;

[0033] Where a and b are hyperparameters, and by default a = 1 and b = 2.

[0034] The present invention also provides a detection device capable of detecting key points throughout the body, comprising:

[0035] The dataset construction module is used to obtain the predicted coordinates of facial key points, hand key points, and foot key points from the MPII human pose dataset, and combine the predicted coordinates with the coordinates of body key points to form real labels. The labeled personal data is then used to form a full-body key point detection dataset.

[0036] Network module construction: The first 22 layers of the residual 50 convolutional block attention network are designed as the backbone network of the whole body keypoint detection network, which is then combined with N lightweight and powerful keypoint detection sub-networks to form the whole body keypoint detection network.

[0037] Training and recognition module: This module trains the detection network on the whole-body keypoint detection dataset and uses two loss functions, label loss and distillation loss, for auxiliary training. The trained model is then deployed for real-time detection and recognition.

[0038] In summary, the key improvements of this invention are mainly as follows:

[0039] 1. A full-body keypoint detection dataset, MPII-FullPose, was constructed.

[0040] 2. The first 22 layers of the Residual 50-layer Residual Nets + Convolutional Block Attention Module (ResNet-50+CBAM) are improved and designed as the backbone network of the whole-body keypoint detection network. This is then combined with N lightweight and powerful keypoint detection sub-networks to design a complete whole-body keypoint detection network.

[0041] 3. Two loss functions, label loss and distillation loss, are used to guide the training of the entire model, which improves the training efficiency of the model.

[0042] 4. Deploy the trained model and add monitoring camera equipment to detect and identify all key points of the human body in the captured video in real time, or test it on the MPII-FullPose test dataset to evaluate the model's accuracy, deployment cost, detection speed and other indicators. Attached Figure Description

[0043] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0044] Figure 1 This is a flowchart illustrating the overall implementation of the detection method based on an embodiment of the present invention.

[0045] Figure 2 This is a diagram showing the overall framework of the whole-body key point detection network based on an embodiment of the present invention.

[0046] Figure 3 This is a framework structure diagram of the detection subnetwork for each key point in an embodiment of the present invention;

[0047] Figure 4 This is a flowchart illustrating a computer device provided in an embodiment of the present invention. Detailed Implementation

[0048] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure; rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0049] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms. Unless the context clearly indicates otherwise, it should be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0050] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, depending on the context. For example, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0051] Example

[0052] This embodiment provides a lightweight method for detecting and recognizing key points in multiple people's whole body based on global key point information distillation and multi-scale dense feature enhancement, as detailed below:

[0053] First, the most advanced full-body keypoint detection algorithm (e.g., Single-Network algorithm

[10] ) was used to obtain the predicted coordinates of 68 facial keypoints, 42 hand keypoints (21 per hand), and 6 foot keypoints (3 per foot) of 25,204 people from the MPII Human Pose Dataset (Max Planck Institute Informatics Human Pose Dataset);

[0054] Next, the predicted coordinates are combined with the coordinates of 16 real body key points of each person in the MPII human pose dataset to create full-body key point labels, and on this basis, the real labels of each person's joint-related region (PartAffinity Field, PAF) are created.

[0055] The data from 22,246 individuals was finally labeled as the training set, and the data from 2,958 individuals was used as the test set, resulting in the full-body keypoint detection dataset MPII-FullPose. Then, the first 22 layers of the residual 50-layer convolutional block attention network (50-layer ResidualNets+Convolutional Block Attention Module, ResNet-50+CBAM) were improved and designed as the backbone network of the full-body keypoint detection network. This backbone network was then combined with N lightweight and powerful keypoint detection sub-networks to design a complete full-body keypoint detection network.

[0056] Then, two loss functions are used: label loss L... 标签 and distillation loss L 蒸馏 Guide the entire model training process. 标签 It acts on all subnetworks, where for subnetwork n∈{1,...,N-1}, L 标签 This is used to calculate the difference between the actual values ​​of whole-body joint-related regions and the predicted values ​​of whole-body joint-related regions by each subnetwork. For subnetworks n=N, L... 标签 Used to calculate the difference between the actual values ​​of whole-body keypoints and the predicted values ​​of whole-body keypoints predicted by this sub-network. L 蒸馏 It operates on subnetworks n∈{1,...,N-1} to calculate the difference between the intermediate prediction 46×46×384 feature map of subnetwork N and the intermediate prediction 46×46×384 feature maps of each subnetwork n∈{1,...,N-1}, thereby performing global key point information distillation.

[0057] Finally, the trained model is deployed and monitored by cameras to detect and identify all key points of the human body in the captured video in real time, or it is tested on the MPII-FullPose test dataset to evaluate the model's accuracy, deployment cost, detection speed and other metrics.

[0058] The implementation flowchart of the lightweight multi-person full-body keypoint detection and recognition method based on global keypoint information distillation and multi-scale dense feature enhancement is as follows: Figure 1 As shown.

[0059] The overall framework of a lightweight multi-person full-body keypoint detection and recognition method based on global keypoint information distillation and multi-scale dense feature enhancement is as follows: Figure 2 As shown.

[0060] The backbone network is an improved, powerful feature extraction network with the first 22 layers of a 50-layer Residual Nets + Convolutional Block Attention Module (ResNet-50+CBAM). This network is used to extract features from human images in the dataset, obtaining a low-level human image feature (LF) map of size 46×46×128. Then, subnetwork n=1 takes the 46×46×128 LF map as input to generate the initial joint-related region prediction result R1. Subnetwork n∈{2,...,N-1} takes the LF map and the prediction result R1 from the previous adjacent stage as input. (n-1) As input, it produces increasingly accurate PAF prediction results R.n Finally, the subnetwork n=N uses the prediction results R from LF and the previous adjacent stage. (N-1) As input, it generates the final accurate predicted coordinates R of the whole body key points. N During model training, we use label loss Li. 标签 and distillation loss L 蒸馏 Two loss functions guide the training of the entire model.

[0061] L 标签 The calculation formula is expressed as follows:

[0062]

[0063] Where n represents the number of subnetworks, f represents the total number of joint-related regions, and k represents the number of key points throughout the body. The representation of the subnetwork n's prediction results for the joint-related region f is shown below. The true value of f represents the joint-related region. This represents the prediction results for N pairs of key points k in the subnetwork. This represents the true value of the key point k.

[0064] L 标签 It acts on all subnetworks, where for subnetwork n∈{1,...,N-1}, L 标签 This is used to calculate the difference between the actual values ​​of whole-body joint-related regions and the predicted values ​​of whole-body joint-related regions by each subnetwork. For subnetworks n=N, L... 标签 This is used to calculate the difference between the actual values ​​of key points throughout the body and the predicted values ​​of key points throughout the body predicted by this subnetwork.

[0065] L 蒸馏 The calculation formula is expressed as follows:

[0066]

[0067] Where n represents the number of subnetworks, r n The feature map r represents the intermediate prediction of subnetwork n, which is 46×46×384. N The feature map representing the intermediate predictions of the expert network is 46×46×384.

[0068] L 蒸馏 It operates on subnetworks n∈{1,...,N-1} to calculate the difference between the intermediate prediction 46×46×384 feature map of subnetwork N and the intermediate prediction 46×46×384 feature maps of each subnetwork n∈{1,...,N-1}, thereby performing global key point information distillation.

[0069] The overall loss function L 总 The calculation formula is expressed as follows:

[0070] L 总 =a×L 标签 +b×L 蒸馏 (3)

[0071] Where a and b are hyperparameters, and by default a = 1 and b = 2.

[0072] The framework structure of each proposed sub-network is as follows: Figure 3 As shown, each subnetwork consists of five lightweight densely connected residual blocks and two 1×1 convolutional layers. Specifically, the lightweight densely connected residual blocks m∈{1,...,5} sequentially use a 3×3 convolutional layer, a 1×1 convolutional layer, and a 3×3 convolutional layer, employing both dense and residual connections. This enhances the model's ability to represent image features and image-related spatial features while reducing computational cost. The two 1×1 convolutional layers, convolution 6 and convolution 7, are sequentially connected. Convolution 6 is used to extract depth features from the 46×46×384 feature map generated by the lightweight densely connected residual blocks of m=5, producing a 46×46×512 depth keypoint feature map. Convolution 7 directly generates predicted values ​​for the coordinates of whole-body keypoints or joint-related regions based on the depth keypoint feature map.

[0073] The trained model was deployed in an experimental environment using Ubuntu 16.04 LTS 64-bit operating system, a minimum CPU performance of 1.80GHz x 8, a minimum memory size of 8GB, a minimum GPU memory size of 6GB, the deep learning framework of Caffe, and the operating language of Python 3.5. Common surveillance camera devices such as the TP-LINK wireless camera (TL-IPC45AW Full-Color Plus) were added, and the shooting angle was adjusted to ensure real-time capture of most human figures within a certain range. The data source information in the configuration file and deployment code was then adjusted to camera capture. This allows for real-time detection and recognition of all key points of the entire human body appearing in the captured video or images, thereby evaluating the model's accuracy, deployment cost, detection speed, and other metrics. Specific detection results are shown in Table 1 below.

[0074] Table 1 Test Results

[0075]

[0076] Table 1 above shows the experimental results compared with mainstream full-body keypoint detection methods. This invention achieves an average detection accuracy of 81.4% on the MPII-FullPose test set. Specifically, it demonstrates superior detection performance and faster detection speed (15.2 fps) for keypoints in the head, shoulders, elbows, wrists, hands, hips, knees, ankles, and feet compared to other models. Furthermore, compared to the OpenPose method with the same network architecture, this invention requires only 4 stages for deployment, resulting in lower deployment costs. Overall, the invention exhibits superior performance.

[0077] In addition to providing the aforementioned lightweight method for detecting key points across multiple individuals' entire bodies, this invention also provides the aforementioned lightweight device for detecting key points across multiple individuals' entire bodies, specifically comprising:

[0078] The dataset construction module is used to obtain the predicted coordinates of facial key points, hand key points, and foot key points from the MPII human pose dataset, and combine the predicted coordinates with the coordinates of body key points to form real labels. The labeled personal data is then used to form a full-body key point detection dataset.

[0079] Network module construction: The first 22 layers of the residual 50 convolutional block attention network are designed as the backbone network of the whole body keypoint detection network, which is then combined with N lightweight and powerful keypoint detection sub-networks to form the whole body keypoint detection network.

[0080] Training and recognition module: This module trains the detection network on the whole-body keypoint detection dataset and uses two loss functions, label loss and distillation loss, for auxiliary training. The trained model is then deployed for real-time detection and recognition.

[0081] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of each unit, please refer to the previous method implementation examples, which will not be repeated here.

[0082] Figure 4 This is a schematic diagram of the structure of a computer device disclosed in this invention. (Reference) Figure 4 As shown, the computer device 400 includes at least a memory 402 and a processor 401; the memory 402 is connected to the processor via a communication bus 403 and is used to store computer instructions executable by the processor 401. The processor 401 is used to read computer instructions from the memory 402 to implement the steps of the detection method described in any of the above embodiments.

[0083] For the above-described apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The apparatus embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0084] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal disks or removable disks), magneto-optical disks, and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0085] Finally, it should be noted that although this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily used to describe the features of specific embodiments of a particular invention. Certain features described in the various embodiments of this specification may also be implemented in combination in a single embodiment. On the other hand, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0086] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0087] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0088] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A lightweight method for detecting key points on the entire body of multiple individuals, characterized in that, Includes the following steps: Predicted coordinates of facial key points, hand key points, and foot key points are obtained from the MPII human pose dataset. These predicted coordinates are then combined with the coordinates of body key points to form real labels. A full-body key point detection dataset is then formed using labeled personal data. The first 22 layers of the residual 50 convolutional block attention network are designed as the backbone network of the whole body keypoint detection network, which is then combined with N lightweight and powerful keypoint detection sub-networks to form the whole body keypoint detection network. The detection network is then trained on the whole-body keypoint detection dataset, and two loss functions, label loss and distillation loss, are used for auxiliary training. The trained model is then deployed for real-time detection and recognition. During training, the backbone network is used to extract features from human images in the dataset, obtaining a low-level human image feature map (LF) of size 46×46×128. The key point detection subnetwork n=1 takes a 46×46×128 LF as input to generate the initial joint-related region prediction result R1; The keypoint detection subnetwork n∈{2,...,N-1} uses LF and the prediction results R of the previous adjacent stage. (n-1) As input, it produces increasingly accurate PAF prediction results R. n Finally, the subnetwork n=N uses the prediction results R from LF and the previous adjacent stage. (N-1) As input, it generates the final accurate predicted coordinates R of the whole body key points. N ; During assisted training, for a subnetwork n∈{1, ..., N-1}, the label loss... Used to calculate the difference between the actual values ​​of the whole-body joint-related regions and the predicted values ​​of the whole-body joint-related regions predicted by each sub-network; For subnetwork n=N, the label loss... Used to calculate the difference between the actual values ​​of keypoints throughout the body and the predicted values ​​of keypoints throughout the body predicted by the keypoint detection subnetwork; The calculation process is as follows: ； n represents the number of subnetworks, f represents the total number of joint-related regions, and k represents the total number of key points throughout the body. The representation of the subnetwork n's prediction results for the joint-related region f is shown below. The true value of f represents the joint-related region. This represents the prediction results for N pairs of key points k in the subnetwork. This represents the true value of key point k; Distillation loss It operates on the keypoint detection subnetwork n∈{1,...,N-1} to calculate the difference between the intermediate prediction 46×46×384 feature map of the keypoint detection subnetwork N and the intermediate prediction 46×46×384 feature maps of each keypoint detection subnetwork n∈{1,...,N-1}, thereby performing global keypoint information distillation. Distillation loss The calculation process is as follows: ； Where n represents the number of subnetworks. The feature map representing the intermediate prediction of subnetwork n, which is 46×46×384. The feature map representing the intermediate predictions of the expert network is 46×46×384.

2. The detection method according to claim 1, characterized in that, Overall loss function The calculation formula is expressed as follows: ； in , These are hyperparameters; by default, a=1 and b=2.

3. The detection apparatus of the detection method according to any one of claims 1-2, characterized in that, include: The dataset construction module is used to obtain the predicted coordinates of facial key points, hand key points, and foot key points from the MPII human pose dataset, and combine the predicted coordinates with the coordinates of body key points to form real labels. The labeled personal data is used to form a full-body key point detection dataset. Network module construction: The first 22 layers of the residual 50 convolutional block attention network are designed as the backbone network of the whole body keypoint detection network, which is then combined with N lightweight and powerful keypoint detection sub-networks to form the whole body keypoint detection network. Training and recognition module: This module trains the detection network on the whole-body keypoint detection dataset and uses two loss functions, label loss and distillation loss, for auxiliary training. The trained model is then deployed for real-time detection and recognition.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the detection method as described in any one of claims 1-2.

Images