Human key point prediction model training and detection method, device, equipment and medium
By adding a one-dimensional heatmap prediction branch to the human keypoint prediction model and training it, the model loss is optimized, solving the problems of high learning difficulty and high computational cost in existing technologies, and achieving lightweight, efficient and accurate human keypoint detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-31
- Publication Date
- 2026-03-24
AI Technical Summary
Existing human key point prediction models are difficult to learn and computationally intensive, making it difficult to balance detection accuracy and efficiency.
A one-dimensional heatmap prediction branch is added to the already trained human keypoint prediction model to form a model containing one-dimensional and two-dimensional heatmap prediction branches. The model loss is obtained through image sample training to optimize the model, thereby reducing the learning difficulty and computational load.
It achieves efficient and accurate prediction of human key points in a lightweight model, balancing detection accuracy and efficiency, and is suitable for real-time human key point detection.
Smart Images
Figure CN115690843B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of artificial intelligence and live streaming technology, and in particular to a training method for a human keypoint prediction model, a human keypoint detection method, an apparatus, an electronic device, and a computer-readable storage medium. Background Technology
[0002] With the development of artificial intelligence technology, many applications such as short videos and live streaming provide functions such as deformation and beautification of the human body in the collected images, which involves the detection of key points of the human body.
[0003] Current human body key point prediction models are difficult to learn and computationally intensive, making it hard to balance the accuracy and efficiency of human body key point detection. Summary of the Invention
[0004] Therefore, it is necessary to provide a training method for a human keypoint prediction model, a human keypoint detection method, a device, an electronic device, and a computer-readable storage medium to address the above-mentioned technical problems.
[0005] Firstly, this application provides a method for training a human keypoint prediction model. The method includes:
[0006] Obtain a human key point prediction model trained in the first training stage; the human key point prediction model trained in the first training stage includes a two-dimensional heatmap prediction branch.
[0007] Based on the human key point prediction model trained in the first training, a one-dimensional heat map prediction branch is added to obtain a human key point prediction model to be trained in the second training, which includes a one-dimensional heat map prediction branch and a two-dimensional heat map prediction branch.
[0008] Image samples containing human bodies are input into the human body key point prediction model to be trained in the second stage, and the one-dimensional heat map prediction information output by the one-dimensional heat map prediction branch and the two-dimensional heat map prediction information output by the two-dimensional heat map prediction branch are obtained.
[0009] Based on the one-dimensional heatmap prediction information, the first human body key point location prediction information corresponding to the one-dimensional heatmap prediction branch is obtained, and based on the two-dimensional heatmap prediction information, the second human body key point location prediction information corresponding to the two-dimensional heatmap prediction branch is obtained.
[0010] The model loss is obtained based on the one-dimensional heatmap prediction information, the two-dimensional heatmap prediction information, the first human body key point location prediction information, the second human body key point location prediction information, and the one-dimensional heatmap annotation information and two-dimensional heatmap annotation information corresponding to the image sample.
[0011] The human keypoint prediction model to be trained a second time is performed based on the model loss. When the training completion condition is met, a human keypoint prediction model containing the one-dimensional heatmap prediction branch applied to human keypoint prediction is obtained.
[0012] Secondly, this application provides a method for detecting key points on the human body. The method includes:
[0013] Obtain a trained human keypoint prediction model that includes a one-dimensional heatmap prediction branch; wherein the human keypoint prediction model is trained according to the training method for human keypoint prediction models described above.
[0014] The image to be detected, containing a human body, is input into the trained human key point prediction model containing a one-dimensional heatmap prediction branch. The corresponding human key point position prediction information is obtained based on the one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch.
[0015] Based on the predicted location information of human body key points, the detection result of human body key point location in the image to be detected is obtained.
[0016] Thirdly, this application provides a training device for a human keypoint prediction model. The device includes:
[0017] The model acquisition module is used to acquire a human key point prediction model that has been trained in the first stage; the human key point prediction model that has been trained in the first stage includes a two-dimensional heat map prediction branch.
[0018] The model acquisition module is used to add a one-dimensional heatmap prediction branch based on the first-trained human key point prediction model to obtain a second-trained human key point prediction model containing a one-dimensional heatmap prediction branch and a two-dimensional heatmap prediction branch.
[0019] The image input module is used to input image samples containing human bodies into the human body key point prediction model to be trained in the second stage, and to obtain the one-dimensional heat map prediction information output by the one-dimensional heat map prediction branch and the two-dimensional heat map prediction information output by the two-dimensional heat map prediction branch.
[0020] The information acquisition module is used to obtain the first human key point location prediction information corresponding to the one-dimensional heat map prediction branch based on the one-dimensional heat map prediction information, and to obtain the second human key point location prediction information corresponding to the two-dimensional heat map prediction branch based on the two-dimensional heat map prediction information.
[0021] The loss acquisition module is used to acquire the model loss based on the one-dimensional heatmap prediction information, the two-dimensional heatmap prediction information, the first human body key point position prediction information, the second human body key point position prediction information, and the one-dimensional heatmap annotation information and two-dimensional heatmap annotation information corresponding to the image sample.
[0022] The model training module is used to perform a second training on the human keypoint prediction model to be trained a second time based on the model loss, and when the training completion condition is met, a human keypoint prediction model containing the one-dimensional heatmap prediction branch applied to human keypoint prediction is obtained.
[0023] Fourthly, this application provides a human body key point detection device. The device includes:
[0024] The model acquisition module is used to acquire a trained human keypoint prediction model that includes a one-dimensional heatmap prediction branch; wherein the human keypoint prediction model is trained according to the training device for the human keypoint prediction model as described above.
[0025] The model processing module is used to input the image to be detected containing a human body into the trained human key point prediction model containing a one-dimensional heatmap prediction branch, and to obtain the corresponding human key point position prediction information based on the one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch.
[0026] The result acquisition module is used to obtain the human key point location detection result of the image to be detected based on the human key point location prediction information.
[0027] Fifthly, this application provides an electronic device. The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:
[0028] A human keypoint prediction model trained in the first training stage is obtained; the first-trained human keypoint prediction model includes a two-dimensional heatmap prediction branch; a one-dimensional heatmap prediction branch is added based on the first-trained human keypoint prediction model to obtain a human keypoint prediction model to be trained in the second training stage, which includes a one-dimensional heatmap prediction branch and a two-dimensional heatmap prediction branch; an image sample containing a human body is input into the human keypoint prediction model to be trained in the second training stage, and the one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch and the two-dimensional heatmap prediction information output by the two-dimensional heatmap prediction branch are obtained; the corresponding information of the one-dimensional heatmap prediction branch is obtained based on the one-dimensional heatmap prediction information. The system generates first human keypoint location prediction information and second human keypoint location prediction information corresponding to the two-dimensional heatmap prediction branch based on the two-dimensional heatmap prediction information. It then obtains model loss based on the one-dimensional heatmap prediction information, the two-dimensional heatmap prediction information, the first human keypoint location prediction information, the second human keypoint location prediction information, and the one-dimensional and two-dimensional heatmap annotation information corresponding to the image samples. Finally, it performs a second training on the human keypoint prediction model to be trained based on the model loss, and obtains a human keypoint prediction model containing the one-dimensional heatmap prediction branch applied to human keypoint prediction when the training completion condition is met.
[0029] Sixthly, this application provides an electronic device. The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:
[0030] A trained human keypoint prediction model containing a one-dimensional heatmap prediction branch is obtained; wherein the human keypoint prediction model is trained according to the training method of the human keypoint prediction model described above; an image to be detected containing a human body is input into the trained human keypoint prediction model containing a one-dimensional heatmap prediction branch, and the corresponding human keypoint position prediction information is obtained according to the one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch; the human keypoint position detection result of the image to be detected is obtained according to the human keypoint position prediction information.
[0031] Seventhly, this application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:
[0032] A human keypoint prediction model trained in the first training stage is obtained; the first-trained human keypoint prediction model includes a two-dimensional heatmap prediction branch; a one-dimensional heatmap prediction branch is added based on the first-trained human keypoint prediction model to obtain a human keypoint prediction model to be trained in the second training stage, which includes a one-dimensional heatmap prediction branch and a two-dimensional heatmap prediction branch; an image sample containing a human body is input into the human keypoint prediction model to be trained in the second training stage, and the one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch and the two-dimensional heatmap prediction information output by the two-dimensional heatmap prediction branch are obtained; the corresponding information of the one-dimensional heatmap prediction branch is obtained based on the one-dimensional heatmap prediction information. The system generates first human keypoint location prediction information and second human keypoint location prediction information corresponding to the two-dimensional heatmap prediction branch based on the two-dimensional heatmap prediction information. It then obtains model loss based on the one-dimensional heatmap prediction information, the two-dimensional heatmap prediction information, the first human keypoint location prediction information, the second human keypoint location prediction information, and the one-dimensional and two-dimensional heatmap annotation information corresponding to the image samples. Finally, it performs a second training on the human keypoint prediction model to be trained based on the model loss, and obtains a human keypoint prediction model containing the one-dimensional heatmap prediction branch applied to human keypoint prediction when the training completion condition is met.
[0033] Eighthly, this application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:
[0034] A trained human keypoint prediction model containing a one-dimensional heatmap prediction branch is obtained; wherein the human keypoint prediction model is trained according to the training method of the human keypoint prediction model described above; an image to be detected containing a human body is input into the trained human keypoint prediction model containing a one-dimensional heatmap prediction branch, and the corresponding human keypoint position prediction information is obtained according to the one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch; the human keypoint position detection result of the image to be detected is obtained according to the human keypoint position prediction information.
[0035] The training method, detection method, device, electronic device, and storage medium of the aforementioned human keypoint prediction model involve obtaining a first-trained human keypoint prediction model containing a two-dimensional heatmap prediction branch and adding a one-dimensional heatmap prediction branch to obtain a second-trained human keypoint prediction model containing one-dimensional and two-dimensional heatmap prediction branches. Image samples containing human bodies are input into the model to obtain one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch and two-dimensional heatmap prediction information output by the two-dimensional heatmap prediction branch. First human keypoint position prediction information and second human keypoint position prediction information are obtained based on the one-dimensional heatmap prediction information and the two-dimensional heatmap prediction information, respectively. Model loss is obtained based on the one-dimensional and two-dimensional heatmap prediction information, the first and second human keypoint position prediction information, and the one-dimensional and two-dimensional heatmap annotation information corresponding to the image samples. The model is then trained a second time based on this model loss. When the training completion conditions are met, a human keypoint prediction model containing a one-dimensional heatmap prediction branch applied to human keypoint prediction is obtained. This scheme adds a one-dimensional heatmap prediction branch to a trained human keypoint prediction model that includes a two-dimensional heatmap prediction branch. Then, it trains the human keypoint prediction model that includes both one-dimensional and two-dimensional heatmap prediction branches. This reduces the learning difficulty and computational load of the model, allowing both one-dimensional and two-dimensional heatmap prediction branches to accurately predict human keypoints. In application, the lighter one-dimensional heatmap prediction branch in the model is used for prediction, achieving efficient and accurate prediction of human keypoints while balancing the accuracy and efficiency of human keypoint detection. Attached Figure Description
[0036] Figure 1 This is a diagram illustrating the application environment of the relevant methods in the embodiments of this application;
[0037] Figure 2 This is a flowchart illustrating the training method of the human key point prediction model in the embodiments of this application.
[0038] Figure 3 This is a flowchart illustrating the steps for obtaining model loss in an embodiment of this application;
[0039] Figure 4 This is a schematic diagram of the human body key point prediction model in the embodiments of this application;
[0040] Figure 5 This is a flowchart illustrating the human body key point detection method in the embodiments of this application;
[0041] Figure 6 This is a flowchart illustrating the process of detecting key points on the human body in an application example of this application;
[0042] Figure 7 This is a structural block diagram of the training device for the human key point prediction model in the embodiments of this application;
[0043] Figure 8 This is a structural block diagram of the human body key point detection device in the embodiments of this application;
[0044] Figure 9 This is a diagram showing the internal structure of the electronic device in an embodiment of this application;
[0045] Figure 10 This is an internal structural diagram of an electronic device according to another embodiment of this application. Detailed Implementation
[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0047] The training method for the human keypoint prediction model and the human keypoint detection method provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown may include a terminal and a server, with the terminal communicating with the server via a network. Specifically, the training method for the human keypoint prediction model provided in this application embodiment can be executed by the server, and the human keypoint detection method provided in this application embodiment can be applied to the terminal. That is, the server can train the human keypoint prediction model to obtain a trained human keypoint prediction model containing a one-dimensional heatmap prediction branch. Then, the server can send the human keypoint prediction model to the terminal, which uses the human keypoint prediction model containing the one-dimensional heatmap prediction branch to detect human keypoints in the image to be detected. Compared with traditional technologies, this application can detect human keypoints accurately in real time. The terminal can also perform deformation, beautification, and other processing on the human body in the image based on the human keypoint detection results, thereby improving its processing efficiency and accuracy. In this application environment, the terminal can be, but is not limited to, various personal computers, laptops, smartphones, and tablets; the server can be implemented using a standalone server or a server cluster composed of multiple servers.
[0048] The following is based on, Figure 1 The application environment shown will be illustrated in the following descriptions, along with the various embodiments and corresponding figures, to illustrate the training method of the human key point prediction model and the human key point detection method provided in this application.
[0049] In one embodiment, such as Figure 2 As shown, a training method for a human keypoint prediction model is provided, which may include:
[0050] Step S201: Obtain the human key point prediction model trained in the first step.
[0051] Step S202: Based on the human key point prediction model trained in the first step, add a one-dimensional heat map prediction branch to obtain a human key point prediction model to be trained in the second step, which includes a one-dimensional heat map prediction branch and a two-dimensional heat map prediction branch.
[0052] Steps S201 and S202 described above are steps related to forming the human keypoint prediction model to be trained a second time. Specifically, in step S201, a human keypoint prediction model trained in the first stage is first obtained. This first-trained human keypoint prediction model is a human keypoint prediction model that includes a two-dimensional heatmap prediction branch. That is, the first-trained human keypoint prediction model has the ability to predict human keypoints. It has a two-dimensional heatmap prediction branch, which can predict human keypoints based on a two-dimensional heatmap. It can output the corresponding two-dimensional heatmap prediction information from the two-dimensional heatmap prediction branch based on the input image. The two-dimensional heatmap prediction information refers to the two-dimensional heatmap corresponding to the predicted human keypoints output by the two-dimensional heatmap prediction branch. Based on the two-dimensional heatmap prediction information, the human keypoint detection result can be obtained. The human keypoint detection result is specifically the position coordinates of the human keypoints on the image. Step S202 involves adding a one-dimensional heatmap prediction branch to the first-trained human keypoint prediction model, thus forming a second-trained human keypoint prediction model containing both a one-dimensional and a two-dimensional heatmap prediction branch. The one-dimensional heatmap prediction branch outputs one-dimensional heatmap prediction information, which refers to the one-dimensional heatmap corresponding to the predicted human keypoints. Therefore, adding the one-dimensional heatmap prediction branch yields a human keypoint prediction model containing both branches. This model requires further training in subsequent steps. To distinguish it from the first-trained model containing the two-dimensional heatmap prediction branch, it is referred to as the second-trained human keypoint prediction model containing both branches. In summary, steps S201 and S202 above involve adding a one-dimensional heatmap prediction branch to the already converged human keypoint prediction model that includes a two-dimensional heatmap prediction branch for training.
[0053] Step S203: Input the image sample containing the human body into the human key point prediction model to be trained for the second time, and obtain the one-dimensional heat map prediction information output by the one-dimensional heat map prediction branch and the two-dimensional heat map prediction information output by the two-dimensional heat map prediction branch.
[0054] Step S204: Obtain the prediction information of the first human body key point location corresponding to the one-dimensional heat map prediction branch based on the one-dimensional heat map prediction information, and obtain the prediction information of the second human body key point location corresponding to the two-dimensional heat map prediction branch based on the two-dimensional heat map prediction information.
[0055] Steps S203 and S204 involve inputting image samples containing human bodies into the second human keypoint prediction model to be trained, thereby generating relevant data information. Specifically, in step S203, image samples containing human bodies are input into the second human keypoint prediction model to be trained. This human keypoint prediction model includes a one-dimensional heatmap prediction branch and a two-dimensional heatmap prediction branch. The one-dimensional heatmap prediction branch outputs one-dimensional heatmap prediction information corresponding to the image sample, and the two-dimensional heatmap prediction branch outputs two-dimensional heatmap prediction information corresponding to the image sample. Next, in step S204, further, based on the one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch, corresponding human keypoint position prediction information (denoted as first human keypoint position prediction information) is obtained, and based on the two-dimensional heatmap prediction information output by the two-dimensional heatmap prediction branch, corresponding human keypoint position prediction information (denoted as second human keypoint position prediction information) is obtained. Both the first and second human keypoint position prediction information can be the position coordinates of the human keypoints on the image.
[0056] As an example, step S204 specifically includes: for each human key point corresponding to a one-dimensional heat map prediction information, obtaining first human key point location prediction information based on the location information that maximizes the thermal data value in the one-dimensional heat map prediction information; and for each human key point corresponding to a two-dimensional heat map prediction information, obtaining second human key point location prediction information based on the location information that maximizes the thermal data value in the two-dimensional heat map prediction information.
[0057] In this embodiment, each human key point to be predicted can correspond to a one-dimensional heatmap prediction information and a two-dimensional heatmap prediction information. Therefore, for each human key point, its first human key point location prediction information can be calculated based on its corresponding one-dimensional heatmap prediction information, and its second human key point location prediction information can be calculated based on its corresponding two-dimensional heatmap prediction information. Specifically, for each human key point, the first human key point location prediction information is obtained based on the location information that maximizes the heat data value in its one-dimensional heatmap prediction information, and the second human key point location prediction information is obtained based on the location information that maximizes the heat data value in its two-dimensional heatmap prediction information. Specifically, the one-dimensional and two-dimensional heatmap prediction information can be processed by the argmax function to obtain the corresponding first and second human key point location prediction information. This allows for efficient and gradual calculation of the first and second human key point location prediction information based on the one-dimensional and two-dimensional heatmap prediction information.
[0058] Step S205: Obtain the model loss based on the one-dimensional heatmap prediction information, the two-dimensional heatmap prediction information, the first human body key point location prediction information, the second human body key point location prediction information, and the one-dimensional heatmap annotation information and two-dimensional heatmap annotation information corresponding to the image samples.
[0059] This step involves obtaining the model loss. Specifically, through the aforementioned steps S203 and S204, we can obtain the one-dimensional heatmap prediction information and the first human keypoint location prediction information from the one-dimensional heatmap prediction branch, as well as the two-dimensional heatmap prediction information and the second human keypoint location prediction information from the two-dimensional heatmap prediction branch. In addition, we also need to obtain the one-dimensional and two-dimensional heatmap annotation information corresponding to the image samples. The one-dimensional heatmap annotation information is a one-dimensional heatmap generated based on the location annotations of human keypoints in the image samples, used to supervise the output of the one-dimensional heatmap prediction branch. The two-dimensional heatmap annotation information is a two-dimensional heatmap generated based on the location annotations of human keypoints in the image samples, used to supervise the output of the two-dimensional heatmap prediction branch. This yields one-dimensional heatmap prediction information from the one-dimensional heatmap prediction branch, prediction information of the first human key point location, two-dimensional heatmap prediction information from the two-dimensional heatmap prediction branch, prediction information of the second human key point location, and one-dimensional and two-dimensional heatmap annotation information generated from the location annotation of human key points in the image samples. Then, based on this information, a model loss is obtained to characterize the supervision of the output of the one-dimensional heatmap prediction branch and the output of the two-dimensional heatmap prediction branch in the model.
[0060] Specifically, in one embodiment, such as Figure 3 As shown, step S205 further includes:
[0061] Step S301: Based on the consistency between the one-dimensional heat map prediction information and the one-dimensional heat map annotation information, obtain the first model loss.
[0062] In this step, for the supervision of the one-dimensional heatmap prediction branch, the first model loss is obtained based on the consistency between the output one-dimensional heatmap prediction information and the one-dimensional heatmap annotation information. Specifically, for the calculation of the first model loss in step S301, the first model loss can be obtained based on the mean squared error between the one-dimensional heatmap prediction information and the one-dimensional heatmap annotation information. That is, the mean squared error (MSELoss) is used to calculate the first model loss, and the specific calculation formula can be expressed as follows:
[0063]
[0064] Among them, mse 1D Let y1 represent the loss of the first model, n represent the number of human body keypoints, i represent the index of the human body keypoint, and y1 represent the loss of the first model. i This represents the annotation information of a one-dimensional heat map, y i ′ represents one-dimensional heatmap prediction information.
[0065] Step S302: Based on the consistency between the two-dimensional heat map prediction information and the two-dimensional heat map annotation information, obtain the second model loss.
[0066] In this step, for the supervision of the two-dimensional heatmap prediction branch, the second model loss is obtained based on the consistency between the output two-dimensional heatmap prediction information and the two-dimensional heatmap annotation information. Specifically, for the calculation of the second model loss in step S302, the second model loss can also be obtained based on the mean squared error between the two-dimensional heatmap prediction information and the two-dimensional heatmap annotation information. That is, the mean squared error (MSELoss) can also be used to calculate the second model loss. The specific calculation formula can be expressed as follows:
[0067]
[0068] Among them, mse 2D y2 represents the loss of the second model. i This represents information labeled on a two-dimensional heat map. This represents the predicted information from a two-dimensional heat map.
[0069] Step S303: Based on the consistency between the first human body key point location prediction information and the second human body key point location prediction information, obtain the third model loss.
[0070] In this step, for the one-dimensional heatmap prediction branch, the output of the two-dimensional heatmap prediction branch is also used to supervise it. Specifically, the third model loss is obtained based on the consistency between the first human key point position prediction information from the one-dimensional heatmap prediction branch and the second human key point position prediction information from the two-dimensional heatmap prediction branch. Specifically, for the calculation of the third model loss in step S303, the third model loss can be obtained based on the absolute error between the first and second human key point position prediction information. That is, the absolute error L1 loss can be used to calculate the third model loss. The specific calculation formula can be expressed as: Where L1 represents the loss of the third model, z' represents the predicted location information of the second human body key point, and z' represents the predicted location information of the first human body key point.
[0071] Step S304: Obtain the model loss based on the first model loss, the second model loss, and the third model loss.
[0072] This step combines the first model loss, the second model loss, and the third model loss to obtain the total model loss. Specifically, the first model loss, the second model loss, and the third model loss can be weighted to obtain the total model loss, such as loss = mse 2D +mse 1D +L1.
[0073] The scheme in this embodiment can supervise the one-dimensional and two-dimensional heatmap prediction branches using one-dimensional and two-dimensional heatmap annotation information respectively. It also applies consistency constraints to the output results of the one-dimensional and two-dimensional heatmap prediction branches so that the output results of the two-dimensional heatmap prediction branch can supervise the output results of the one-dimensional heatmap prediction branch. This reduces the learning difficulty and computational load of the model and enables both the one-dimensional and two-dimensional heatmap prediction branches to accurately detect key points of the human body.
[0074] Step S206: Perform a second training on the human keypoint prediction model to be trained according to the model loss. When the training completion condition is met, a human keypoint prediction model containing a one-dimensional heatmap prediction branch applied to human keypoint prediction is obtained.
[0075] In this step, the human keypoint prediction model to be trained a second time is performed based on the model loss obtained in step S205. The model parameters are continuously updated, and when the training completion condition is met, a trained human keypoint prediction model containing a one-dimensional heatmap prediction branch and a two-dimensional heatmap prediction branch can be obtained. The training completion condition can be, for example, that the model loss is less than or equal to a preset model loss value. During model training, both the one-dimensional and two-dimensional heatmap prediction branches are trained simultaneously. Upon completion of training, a human keypoint prediction model containing both one-dimensional and two-dimensional heatmap prediction branches is obtained. The one-dimensional heatmap prediction branch is then applied to human keypoint prediction. During training, both branches are effective, but after training, only the one-dimensional branch is used for human keypoint prediction. This results in a human keypoint prediction model that includes the one-dimensional heatmap prediction branch applied to human keypoint prediction. Compared to traditional techniques, the human keypoint prediction model obtained in this application has a more lightweight one-dimensional structure. Furthermore, the one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch can be expanded to increase the output dimension according to computational accuracy requirements, achieving sub-pixel level accuracy and enabling real-time accurate prediction of human keypoints.
[0076] In the training method of the human keypoint prediction model in this embodiment, a human keypoint prediction model containing a two-dimensional heatmap prediction branch, which has undergone first training, is obtained, and a one-dimensional heatmap prediction branch is added to obtain a human keypoint prediction model to be trained in the second stage, which contains one-dimensional and two-dimensional heatmap prediction branches. An image sample containing a human body is input into the model, and one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch and two-dimensional heatmap prediction information output by the two-dimensional heatmap prediction branch are obtained. The first human keypoint position prediction information and the second human keypoint position prediction information are obtained based on the one-dimensional heatmap prediction information. The model loss is obtained based on the one-dimensional and two-dimensional heatmap prediction information, the first and second human keypoint position prediction information, and the one-dimensional and two-dimensional heatmap annotation information corresponding to the image sample. The model is then trained in the second stage based on the model loss. When the training completion condition is met, a human keypoint prediction model containing a one-dimensional heatmap prediction branch applied to human keypoint prediction is obtained. This scheme adds a one-dimensional heatmap prediction branch to a trained human keypoint prediction model that includes a two-dimensional heatmap prediction branch. Then, it trains the human keypoint prediction model that includes both one-dimensional and two-dimensional heatmap prediction branches. This reduces the learning difficulty and computational load of the model, allowing both one-dimensional and two-dimensional heatmap prediction branches to accurately predict human keypoints. In application, the lighter one-dimensional heatmap prediction branch in the model is used for prediction, achieving efficient and accurate prediction of human keypoints while balancing the accuracy and efficiency of human keypoint detection.
[0077] In one embodiment, the human keypoint prediction model trained in the first step further includes a feature extraction network. Step S203, which involves inputting image samples containing human bodies into the human keypoint prediction model to be trained in the second step, and obtaining one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch and two-dimensional heatmap prediction information output by the two-dimensional heatmap prediction branch, specifically includes:
[0078] Image samples containing human bodies are input into the human body keypoint prediction model to be trained in the second stage. The feature extraction network in the human body keypoint prediction model to be trained in the second stage obtains a shared feature map based on the image samples. The shared feature map is then passed to the one-dimensional heatmap prediction branch and the two-dimensional heatmap prediction branch, respectively. The one-dimensional heatmap prediction branch outputs one-dimensional heatmap prediction information based on the shared feature map, and the two-dimensional heatmap prediction branch outputs two-dimensional heatmap prediction information based on the shared feature map.
[0079] Specifically, such as Figure 4 As shown, the human keypoint prediction model trained in the first stage can specifically include a feature extraction network and a two-dimensional heatmap prediction branch. Adding a one-dimensional heatmap prediction branch to this first-stage trained human keypoint prediction model yields a human keypoint prediction model to be trained in the second stage. That is, the human keypoint prediction model to be trained in the second stage can specifically include a feature extraction network, a one-dimensional heatmap prediction branch, and a two-dimensional heatmap prediction branch. Therefore, during the second training of the human keypoint prediction model to be trained in the second stage, image samples containing human bodies are input into the model. The feature extraction network extracts features from the image samples to obtain a shared feature map. This shared feature map is shared by both the one-dimensional heatmap prediction branch and the two-dimensional heatmap prediction branch. The feature extraction network can pass this shared feature map to both branches. The one-dimensional heatmap prediction branch can predict n human keypoints based on this shared feature map. Specifically, it can perform reshape and conv1×1 operations on the shared feature map to obtain and output the corresponding one-dimensional heatmap prediction information. The graph prediction information can be processed by the argmax function to obtain the corresponding coordinates (e.g., x1, y1, etc.), which is the predicted location information of the first human body key point. Correspondingly, in the two-dimensional heatmap prediction branch, the two-dimensional heatmap prediction branch can predict the key points of n human bodies respectively based on the shared feature map, obtain and output the corresponding two-dimensional heatmap prediction information. These two-dimensional heatmap prediction information can also be processed by the argmax function to obtain the coordinates (e.g., x1, y1, etc.) of each human body key point, which is the predicted location information of the second human body key point. Thus, the efficient acquisition of one-dimensional heatmap prediction information, two-dimensional heatmap prediction information, and the predicted location information of the first and second human body key points can be achieved by the one-dimensional and two-dimensional heatmap prediction branches.
[0080] For the feature extraction network, such as Figure 4 As shown, in some embodiments, a Feature Pyramid Network (FPN) structure can be specifically adopted. FPN effectively fuses feature maps of different depths through top-down and bottom-up processes, which can improve the accuracy of keypoint detection. In a specific implementation, for the one-dimensional heatmap prediction branch, if the input size of the image sample is (H, W), then the output size of the one-dimensional heatmap prediction branch can be (kH, kW), where k = [1, 2, ...]. For the two-dimensional heatmap prediction branch, its output size is (H / m, W / m), where m is the downsampling factor of the feature pyramid network, such as 4, 8, etc. The output method used in this application has an error of [0, 1 / 2k), which can reach the sub-pixel level.
[0081] Furthermore, in one embodiment, the feature extraction network may include a cascaded first feature extraction network and a second feature extraction network. In the above embodiment, the feature extraction network in the human keypoint prediction model to be trained obtains a shared feature map based on image samples, and passes the shared feature map to the one-dimensional heatmap prediction branch and the two-dimensional heatmap prediction branch respectively. Specifically, this may include:
[0082] The first feature extraction network in the human keypoint prediction model to be trained obtains an initial shared feature map based on the image samples, and passes the initial shared feature map to the second feature extraction network. The second feature extraction network obtains a shared feature map based on the initial shared feature map, and passes the shared feature map to the one-dimensional heatmap prediction branch and the two-dimensional heatmap prediction branch respectively.
[0083] In this embodiment, the feature extraction network may include a cascaded first feature extraction network and a second feature extraction network. Both the first and second feature extraction networks can employ feature pyramid networks, meaning the feature extraction network can include two cascaded feature pyramid networks. Thus, the first feature extraction network (i.e., the first feature pyramid network) in the model first obtains an initial shared feature map based on image samples. Then, the first feature pyramid network passes this initial shared feature map to the second feature extraction network (i.e., the second feature pyramid network). The second feature pyramid network obtains a shared feature map based on the initial shared feature map and passes this shared feature map to the one-dimensional heatmap prediction branch and the two-dimensional heatmap prediction branch respectively to complete the output of the corresponding prediction information. In other words, in this embodiment, the input of the second feature extraction network is the output of the first feature extraction network. It can achieve more refined prediction based on the feature map output by the first feature extraction network, further improving the accuracy and stability of keypoint prediction. Furthermore, the cascaded feature pyramid network structure can improve the accuracy and stability of human keypoint prediction in large-pose scenes such as side-viewing.
[0084] In one embodiment, such as Figure 5 As shown, a method for detecting key points on the human body is provided, which can be applied to... Figure 1 In the terminal, the method may include the following steps:
[0085] Step S501: Obtain a trained human keypoint prediction model that includes a one-dimensional heatmap prediction branch.
[0086] The human body key point prediction model is trained according to the training method of the human body key point prediction model provided in the above embodiments of this application.
[0087] Step S502: Input the image to be detected containing a human body into the trained human key point prediction model containing a one-dimensional heatmap prediction branch, and obtain the corresponding human key point position prediction information based on the one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch.
[0088] Step S503: Based on the predicted information of human body key point positions, obtain the detection results of human body key point positions in the image to be detected.
[0089] In this embodiment, after the server trains a human keypoint prediction model containing a one-dimensional heatmap prediction branch for human keypoint prediction according to the training method of the human keypoint prediction model provided in the above embodiments of this application, it can send the human keypoint prediction model to the terminal. The terminal obtains the trained human keypoint prediction model containing the one-dimensional heatmap prediction branch. Then, the terminal can obtain a detection image containing a human body and input it into the human keypoint prediction model to obtain the one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch in the model. Then, the one-dimensional heatmap prediction information can be processed by the argmax function to obtain the corresponding human keypoint position prediction information, such as the position coordinates of each human keypoint. Then, the human keypoint position detection result of the detection image can be obtained according to the human keypoint position prediction information. Specifically, the human keypoint position detection result can be the marking of the predicted human keypoint position coordinates on the detection image.
[0090] More specifically, such as Figure 6 As shown, after acquiring the original image, the terminal can first use a human detector to detect whether the original image contains a human body. If no human body is found, the process ends. If a human body is found, the human detector extracts the human body region and crops the detected human body region from the original image to obtain a detection image containing the human body. Then, based on this detection image containing the human body, human keypoint detection is performed. That is, the detection image containing the human body is input into the aforementioned trained human keypoint prediction model that includes a one-dimensional heatmap prediction branch. According to the one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch, the corresponding human keypoint position prediction information is obtained. Based on the human keypoint position prediction information, the human keypoint position detection result of the detection image is obtained, such as the marking of the position coordinates of the human keypoints on the detection image. Thus, the human keypoint position detection result of the detection image can also be subjected to the inverse transformation of the human body region cropping in the previous step, and the cropped human keypoint position detection result is mapped back to the original image to obtain the human keypoint position detection result of the original image.
[0091] The solution in this application example can predict human key points in real time. Specifically, it can be applied to the detection of human key points in images captured by cameras in live streaming scenarios. Based on the detection results, further processing of the human body in the image can be performed to enhance features such as leg length and body shaping, which helps to improve the efficiency, accuracy and stability of image processing for features such as leg length and body shaping.
[0092] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0093] Based on the same inventive concept, this application also provides a related apparatus for implementing the aforementioned related methods. The solution provided by this apparatus is similar to the implementation scheme described in the above methods; therefore, the specific limitations in one or more related apparatus embodiments provided below can be found in the limitations of the related methods described above, and will not be repeated here.
[0094] In one embodiment, such as Figure 7 As shown, a training device for a human keypoint prediction model is provided. The device 700 may include:
[0095] The model acquisition module 701 is used to acquire a human key point prediction model that has been trained in the first stage; the human key point prediction model that has been trained in the first stage includes a two-dimensional heat map prediction branch.
[0096] The model obtains module 702, which is used to add a one-dimensional heat map prediction branch based on the first trained human key point prediction model to obtain a human key point prediction model to be trained in the second stage, which includes a one-dimensional heat map prediction branch and a two-dimensional heat map prediction branch.
[0097] Image input module 703 is used to input image samples containing human bodies into the human body key point prediction model to be trained in the second stage, and to obtain the one-dimensional heat map prediction information output by the one-dimensional heat map prediction branch and the two-dimensional heat map prediction information output by the two-dimensional heat map prediction branch.
[0098] The information acquisition module 704 is used to obtain the first human key point position prediction information corresponding to the one-dimensional heat map prediction branch based on the one-dimensional heat map prediction information, and to obtain the second human key point position prediction information corresponding to the two-dimensional heat map prediction branch based on the two-dimensional heat map prediction information.
[0099] The loss acquisition module 705 is used to acquire the model loss based on the one-dimensional heat map prediction information, the two-dimensional heat map prediction information, the first human body key point position prediction information, the second human body key point position prediction information, and the one-dimensional heat map annotation information and the two-dimensional heat map annotation information corresponding to the image sample.
[0100] The model training module 706 is used to perform a second training on the human key point prediction model to be trained according to the model loss, and when the training completion condition is met, a human key point prediction model containing the one-dimensional heatmap prediction branch applied to human key point prediction is obtained.
[0101] In one embodiment, the loss acquisition module 705 is configured to acquire a first model loss based on the consistency between the one-dimensional heatmap prediction information and the one-dimensional heatmap annotation information; acquire a second model loss based on the consistency between the two-dimensional heatmap prediction information and the two-dimensional heatmap annotation information; acquire a third model loss based on the consistency between the first human key point location prediction information and the second human key point location prediction information; and obtain the model loss based on the first model loss, the second model loss, and the third model loss.
[0102] In one embodiment, the loss acquisition module 705 is used to acquire the first model loss based on the mean square error between the one-dimensional heat map prediction information and the one-dimensional heat map annotation information; acquire the second model loss based on the mean square error between the two-dimensional heat map prediction information and the two-dimensional heat map annotation information; and acquire the third model loss based on the absolute error between the first human key point position prediction information and the second human key point position prediction information.
[0103] In one embodiment, the first-trained human keypoint prediction model further includes a feature extraction network; an image input module 703 is used to input the image sample containing the human body into the second-trained human keypoint prediction model, whereby the feature extraction network in the second-trained human keypoint prediction model obtains a shared feature map based on the image sample, and transmits the shared feature map to the one-dimensional heatmap prediction branch and the two-dimensional heatmap prediction branch respectively, whereby the one-dimensional heatmap prediction branch outputs the one-dimensional heatmap prediction information based on the shared feature map, and the two-dimensional heatmap prediction branch outputs the two-dimensional heatmap prediction information based on the shared feature map.
[0104] In one embodiment, the feature extraction network includes a cascaded first feature extraction network and a second feature extraction network; the image input module 703 is used to obtain an initial shared feature map from the image samples by the first feature extraction network in the human keypoint prediction model to be trained, and to pass the initial shared feature map to the second feature extraction network, which then obtains the shared feature map from the initial shared feature map, and passes the shared feature map to the one-dimensional heatmap prediction branch and the two-dimensional heatmap prediction branch respectively.
[0105] In one embodiment, the information obtaining module 704 is used to obtain the first human key point location prediction information based on the location information that maximizes the thermal data value in the one-dimensional heat map prediction information corresponding to each human key point; and to obtain the second human key point location prediction information based on the location information that maximizes the thermal data value in the two-dimensional heat map prediction information corresponding to each human key point.
[0106] In one embodiment, such as Figure 8 As shown, a human body key point detection device is provided, the device 800 may include:
[0107] The model acquisition module 801 is used to acquire a trained human key point prediction model that includes a one-dimensional heatmap prediction branch; wherein the human key point prediction model is trained according to the training device for the human key point prediction model as described above.
[0108] The model processing module 802 is used to input the image to be detected containing a human body into the trained human key point prediction model containing a one-dimensional heatmap prediction branch, and obtain the corresponding human key point position prediction information based on the one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch.
[0109] The result acquisition module 803 is used to obtain the human key point location detection result of the image to be detected based on the human key point location prediction information.
[0110] Each module in the aforementioned related devices can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each module.
[0111] In one embodiment, an electronic device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9As shown, this electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database can be used to store data such as image samples. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a training method for a human keypoint prediction model.
[0112] In one embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10 As shown, the electronic device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a method for detecting key points on a human body. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.
[0113] Those skilled in the art will understand that Figure 9 and Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.
[0114] In one embodiment, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.
[0115] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.
[0116] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0117] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0118] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0119] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A training method for a human body key point prediction model, characterized in that, The method includes: Obtain a human key point prediction model trained in the first training stage; the human key point prediction model trained in the first training stage includes a two-dimensional heatmap prediction branch; perform human key point prediction based on the two-dimensional heatmap, and obtain the corresponding two-dimensional heatmap prediction information by outputting the two-dimensional heatmap prediction branch according to the input image, the two-dimensional heatmap prediction information is used to obtain the position coordinates of human key points on the image; Based on the human key point prediction model trained in the first training, a one-dimensional heat map prediction branch is added to obtain a human key point prediction model to be trained in the second training, which includes a one-dimensional heat map prediction branch and a two-dimensional heat map prediction branch. Image samples containing human bodies are input into the human body key point prediction model to be trained in the second stage, and the one-dimensional heat map prediction information output by the one-dimensional heat map prediction branch and the two-dimensional heat map prediction information output by the two-dimensional heat map prediction branch are obtained. Based on the one-dimensional heatmap prediction information, the first human body key point location prediction information corresponding to the one-dimensional heatmap prediction branch is obtained, and based on the two-dimensional heatmap prediction information, the second human body key point location prediction information corresponding to the two-dimensional heatmap prediction branch is obtained. The model loss is obtained based on the one-dimensional heatmap prediction information, the two-dimensional heatmap prediction information, the first human body key point location prediction information, the second human body key point location prediction information, and the one-dimensional heatmap annotation information and two-dimensional heatmap annotation information corresponding to the image sample. The human keypoint prediction model to be trained a second time is performed based on the model loss. When the training completion condition is met, a human keypoint prediction model containing the one-dimensional heatmap prediction branch applied to human keypoint prediction is obtained.
2. The method according to claim 1, characterized in that, The step of obtaining the model loss based on the one-dimensional heatmap prediction information, two-dimensional heatmap prediction information, first human body key point location prediction information, second human body key point location prediction information, and the one-dimensional and two-dimensional heatmap annotation information corresponding to the image samples includes: Based on the consistency between the one-dimensional heatmap prediction information and the one-dimensional heatmap annotation information, the first model loss is obtained; Based on the consistency between the two-dimensional heatmap prediction information and the two-dimensional heatmap annotation information, the second model loss is obtained; Based on the consistency between the first human body key point location prediction information and the second human body key point location prediction information, the third model loss is obtained; The model loss is obtained based on the first model loss, the second model loss, and the third model loss.
3. The method according to claim 2, characterized in that, The step of obtaining the first model loss based on the consistency between the one-dimensional heatmap prediction information and the one-dimensional heatmap annotation information includes: The first model loss is obtained based on the mean square error between the one-dimensional heat map prediction information and the one-dimensional heat map annotation information. The step of obtaining the second model loss based on the consistency between the two-dimensional heatmap prediction information and the two-dimensional heatmap annotation information includes: The second model loss is obtained based on the mean square error between the two-dimensional heat map prediction information and the two-dimensional heat map annotation information. The step of obtaining the third model loss based on the consistency between the first human body key point location prediction information and the second human body key point location prediction information includes: The third model loss is obtained based on the absolute error between the first human body key point location prediction information and the second human body key point location prediction information.
4. The method according to any one of claims 1 to 3, characterized in that, The human keypoint prediction model trained in the first stage also includes a feature extraction network; The step of inputting image samples containing human bodies into the human keypoint prediction model to be trained in the second stage, and obtaining the one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch and the two-dimensional heatmap prediction information output by the two-dimensional heatmap prediction branch, includes: The image sample containing the human body is input into the human keypoint prediction model to be trained in the second stage. The feature extraction network in the human keypoint prediction model to be trained in the second stage obtains a shared feature map based on the image sample, and passes the shared feature map to the one-dimensional heatmap prediction branch and the two-dimensional heatmap prediction branch respectively. The one-dimensional heatmap prediction branch outputs the one-dimensional heatmap prediction information based on the shared feature map, and the two-dimensional heatmap prediction branch outputs the two-dimensional heatmap prediction information based on the shared feature map.
5. The method according to claim 4, characterized in that, The feature extraction network includes a cascaded first feature extraction network and a second feature extraction network; the feature extraction network in the human keypoint prediction model to be trained obtains a shared feature map based on the image samples, and passes the shared feature map to the one-dimensional heatmap prediction branch and the two-dimensional heatmap prediction branch respectively, including: The first feature extraction network in the human keypoint prediction model to be trained obtains an initial shared feature map based on the image samples, and passes the initial shared feature map to the second feature extraction network. The second feature extraction network obtains the shared feature map based on the initial shared feature map, and passes the shared feature map to the one-dimensional heatmap prediction branch and the two-dimensional heatmap prediction branch respectively.
6. The method according to any one of claims 1 to 3, characterized in that, The step of obtaining the first human key point location prediction information corresponding to the one-dimensional heat map prediction branch based on the one-dimensional heat map prediction information, and obtaining the second human key point location prediction information corresponding to the two-dimensional heat map prediction branch based on the two-dimensional heat map prediction information, includes: For each key point of the human body, the location prediction information of the first key point of the human body is obtained based on the location information that maximizes the thermal data value in the one-dimensional heat map prediction information. For each key point on the human body, the predicted location information of the second key point is obtained based on the location information that maximizes the thermal data value in the predicted location information of the two-dimensional heat map.
7. A method for detecting key points on the human body, characterized in that, The method includes: Obtain a trained human keypoint prediction model that includes a one-dimensional heatmap prediction branch; wherein the human keypoint prediction model is trained by the training method of the human keypoint prediction model according to any one of claims 1 to 6. The image to be detected, containing a human body, is input into the trained human key point prediction model containing a one-dimensional heatmap prediction branch. The corresponding human key point position prediction information is obtained based on the one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch. Based on the predicted location information of human body key points, the detection result of human body key point location in the image to be detected is obtained.
8. A training device for a human body key point prediction model, characterized in that, The device includes: The model acquisition module is used to acquire a human key point prediction model trained in the first training stage; the human key point prediction model trained in the first training stage includes a two-dimensional heatmap prediction branch; human key points are predicted based on the two-dimensional heatmap, and the corresponding two-dimensional heatmap prediction information is output by the two-dimensional heatmap prediction branch according to the input image, and the two-dimensional heatmap prediction information is used to obtain the position coordinates of human key points on the image. The model acquisition module is used to add a one-dimensional heatmap prediction branch based on the first-trained human key point prediction model to obtain a second-trained human key point prediction model containing a one-dimensional heatmap prediction branch and a two-dimensional heatmap prediction branch. The image input module is used to input image samples containing human bodies into the human body key point prediction model to be trained in the second stage, and to obtain the one-dimensional heat map prediction information output by the one-dimensional heat map prediction branch and the two-dimensional heat map prediction information output by the two-dimensional heat map prediction branch. The information acquisition module is used to obtain the first human key point location prediction information corresponding to the one-dimensional heat map prediction branch based on the one-dimensional heat map prediction information, and to obtain the second human key point location prediction information corresponding to the two-dimensional heat map prediction branch based on the two-dimensional heat map prediction information. The loss acquisition module is used to acquire the model loss based on the one-dimensional heatmap prediction information, the two-dimensional heatmap prediction information, the first human body key point position prediction information, the second human body key point position prediction information, and the one-dimensional heatmap annotation information and two-dimensional heatmap annotation information corresponding to the image sample. The model training module is used to perform a second training on the human keypoint prediction model to be trained a second time based on the model loss, and when the training completion condition is met, a human keypoint prediction model containing the one-dimensional heatmap prediction branch applied to human keypoint prediction is obtained.
9. A human body key point detection device, characterized in that, The device includes: The model acquisition module is used to acquire a trained human keypoint prediction model that includes a one-dimensional heatmap prediction branch; wherein the human keypoint prediction model is trained by the training device of the human keypoint prediction model according to claim 8. The model processing module is used to input the image to be detected containing a human body into the trained human key point prediction model containing a one-dimensional heatmap prediction branch, and to obtain the corresponding human key point position prediction information based on the one-dimensional heatmap prediction information output by the one-dimensional heatmap prediction branch. The result acquisition module is used to obtain the human key point location detection result of the image to be detected based on the human key point location prediction information.
10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 6 or claim 7.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 6 or claim 7.