A key point detection model training method and a key point detection method

Abstract

The present disclosure relates to a key point detection model training method and a key point detection method. The key point detection model training method comprises: obtaining a plurality of sample images, each of which comprises corresponding annotation information, the annotation information comprising annotation box information for representing a target object in the sample image and key point information for representing the target object; inputting the sample image into a preset model for target detection and key point detection based on the target detection result to obtain a sample detection result, the sample detection result comprising target object prediction information and key point prediction information; determining a training loss according to the difference between the sample detection result and the annotation information and adjusting the parameters of the preset model to obtain a key point detection model. The present disclosure can overcome the problem that the heat map cannot distinguish similar key points with close spatial distance in key point detection, and improve the efficiency of key point detection.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a key point detection model training method and a key point detection method. Background Technology

[0002] Keypoint detection (specifically 2D keypoint detection) is a technique for identifying and labeling the locations of key points in an image, which is crucial for machine vision to understand people and objects in images. Traditional keypoint detection methods can be broadly categorized into top-down and bottom-up approaches. Top-down methods typically involve two steps: first, detecting each target in the image, and then performing keypoint detection on each target separately. The computational complexity of this type of method increases linearly with the number of targets in the image. Therefore, although it can achieve high accuracy, its two-step process and variable detection time make it unsuitable for real-time detection tasks. Bottom-up methods typically use heatmaps to detect all keypoints at once, and then use a complex post-processing procedure to classify the keypoints into their respective targets. However, bottom-up methods cannot distinguish between keypoints of the same type that are very close in location, and the post-processing is usually non-differentiable, thus preventing the use of neural networks for parallel computation, which also limits the execution efficiency of this type of method to some extent. Summary of the Invention

[0003] To address at least one of the aforementioned technical problems, this disclosure provides a keypoint detection model training method, a keypoint detection method, an apparatus, a storage medium, and an electronic device.

[0004] According to one aspect of this disclosure, a keypoint detection model training method is provided, comprising: acquiring multiple sample images, each sample image including corresponding annotation information, the annotation information including bounding box information for representing a target object in the sample image, and keypoint information for representing the target object; inputting the sample images into a preset model for target detection and keypoint detection based on the target detection results, obtaining sample detection results, the sample detection results including target object prediction information and keypoint prediction information; determining a target detection loss based on the difference between the target object prediction information and the bounding box information; determining a keypoint detection loss based on the difference between the keypoint information and the keypoint prediction information; and adjusting the parameters of the preset model based on the target detection loss and the keypoint detection loss to obtain a keypoint detection model.

[0005] In some possible implementations, the target object prediction information includes the predicted category information of the target object, the position information of the prediction box corresponding to the target object, and the confidence level of the prediction box, wherein the confidence level of the prediction box represents the probability that the target object exists within the prediction box; the key point prediction information includes the predicted category information of the key point, the predicted position information of the key point, and the confidence level of the key point, wherein the confidence level of the key point represents the accuracy of the key point prediction.

[0006] In some possible implementations, the bounding box information of the target object includes the location information and category information of the corresponding target object; determining the target detection loss based on the difference between the target object prediction information and the bounding box information includes: determining a target object classification loss component based on the difference between the predicted category information of the target object and the category information of the target object; determining a target object prediction box loss component based on the difference between the location information of the prediction box corresponding to the target object and the location information of the target object; determining a target object confidence loss component based on the confidence of the detection box, the location information of the prediction box corresponding to the target object, and the location information of the target object; and determining the target detection loss based on the target object classification loss component, the target object prediction box loss component, and the target object confidence loss component.

[0007] In some possible implementations, the key point information includes the location information, category information, weight, and confidence ground truth of the corresponding key point; the confidence ground truth of the key point indicates whether the key point is inside its corresponding sample image; when the key point is inside its corresponding sample image, the confidence ground truth of the key point is a first preset value; when the key point is outside its corresponding sample image, the confidence ground truth of the key point is a second preset value.

[0008] In some possible implementations, determining the keypoint detection loss based on the difference between the keypoint information and the keypoint prediction information includes: determining a keypoint confidence loss component based on the confidence of the keypoint, the ground truth value of the keypoint confidence, and the total number of keypoints in the sample image; determining a keypoint location loss component; and determining the keypoint detection loss based on the keypoint confidence loss component and the keypoint location loss component.

[0009] In some possible implementations, the key point information further includes the visibility value of the key point, which indicates whether the key point is occluded; if the key point is not occluded, the visibility value of the key point is a third preset value; if the key point is occluded, the visibility value of the key point is a fourth preset value; determining the key point position loss component includes: determining the size of the target object corresponding to the key point; and determining the key point position loss component based on the predicted position information, position information, weight, size of the corresponding target object, visibility value, and the total number of key points in the sample image.

[0010] According to a second aspect of this disclosure, a keypoint detection method is provided, comprising: acquiring an image to be detected; inputting the image to be detected into a keypoint detection model to obtain keypoints; wherein the keypoint detection model is obtained according to the keypoint detection model training method described in any one of the first aspects of this disclosure.

[0011] According to a third aspect of this disclosure, a keypoint detection model training apparatus is provided. The apparatus includes: a sample image acquisition module for acquiring multiple sample images, each sample image including corresponding annotation information, the annotation information including bounding box information representing a target object in the sample image and keypoint information representing the target object; a detection module for inputting the sample images into a preset model for target detection and keypoint detection based on the target detection results, obtaining sample detection results, the sample detection results including target object prediction information and keypoint prediction information; a target detection loss determination module for determining a target detection loss based on the difference between the target object prediction information and the bounding box information; a keypoint detection loss determination module for determining a keypoint detection loss based on the difference between the keypoint information and the keypoint prediction information; and a parameter update module for adjusting the parameters of the preset model based on the target detection loss and the keypoint detection loss to obtain a keypoint detection model.

[0012] In some possible implementations, the target object prediction information includes the predicted category information of the target object, the position information of the prediction box corresponding to the target object, and the confidence level of the prediction box, wherein the confidence level of the prediction box represents the probability that the target object exists within the prediction box; the key point prediction information includes the predicted category information of the key point, the predicted position information of the key point, and the confidence level of the key point, wherein the confidence level of the key point represents the accuracy of the key point prediction.

[0013] In some possible implementations, the bounding box information of the target object includes the location information and category information of the corresponding target object. The target detection loss determination module is used to determine the target detection loss based on the difference between the target object prediction information and the bounding box information, including: determining a target object classification loss component based on the difference between the predicted category information of the target object and the category information of the target object; determining a target object prediction box loss component based on the difference between the location information of the prediction box corresponding to the target object and the location information of the target object; determining a target object confidence loss component based on the confidence of the detection box, the location information of the prediction box corresponding to the target object, and the location information of the target object; and determining the target detection loss based on the target object classification loss component, the target object prediction box loss component, and the target object confidence loss component.

[0014] In some possible implementations, the key point information includes the location information, category information, weight, and confidence ground truth of the corresponding key point; the confidence ground truth of the key point indicates whether the key point is inside its corresponding sample image; when the key point is inside its corresponding sample image, the confidence ground truth of the key point is a first preset value; when the key point is outside its corresponding sample image, the confidence ground truth of the key point is a second preset value.

[0015] In some possible implementations, the keypoint detection loss module is used to determine the keypoint detection loss based on the difference between the keypoint information and the keypoint prediction information, including: determining a keypoint confidence loss component based on the confidence of the keypoint, the ground truth value of the keypoint confidence, and the total number of keypoints in the sample image; determining a keypoint location loss component; and determining the keypoint detection loss based on the keypoint confidence loss component and the keypoint location loss component.

[0016] In some possible implementations, the key point information further includes the visibility value of the key point, which indicates whether the key point is occluded; if the key point is not occluded, the visibility value of the key point is a third preset value; if the key point is occluded, the visibility value of the key point is a fourth preset value; the key point detection loss module determines the key point position loss component by: the size of the target object corresponding to the key point; and by determining the key point position loss component based on the predicted position information, position information, weight, size of the corresponding target object, visibility value, and the total number of key points in the sample image.

[0017] According to a fourth aspect of this disclosure, an electronic device is provided, including at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements a keypoint detection model training method as described in any one of the first aspects or a keypoint detection method as described in any one of the second aspects by executing the instructions stored in the memory.

[0018] According to a fifth aspect of this disclosure, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or at least one program being loaded and executed by a processor to implement the keypoint detection model training method as described in any of the first aspects or the keypoint detection method as described in any of the second aspects.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0020] Implementing this disclosure has the following beneficial effects: This disclosure obtains sample detection results by inputting sample images into a preset model for object detection and keypoint detection based on the object detection results. These sample detection results include target object prediction information and keypoint prediction information. The training loss is determined based on the difference between the sample detection results and the labeled information, and the parameters of the preset model are adjusted to obtain a keypoint detection model. This disclosure overcomes the problem in traditional keypoint detection where heatmaps cannot distinguish similar keypoints that are spatially close, and significantly reduces the computational load compared to post-processing calculations in traditional keypoint detection methods.

[0021] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0022] To more clearly illustrate the technical solutions and advantages in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating a training method for a keypoint detection model according to an embodiment of the present disclosure is shown.

[0024] Figure 2 This diagram illustrates a method for determining target detection loss according to an embodiment of the present disclosure.

[0025] Figure 3 This diagram illustrates a method for determining keypoint detection loss according to an embodiment of the present disclosure.

[0026] Figure 4 The diagram shows a flowchart of a method for determining the loss components of key point locations according to an embodiment of this disclosure.

[0027] Figure 5 A flowchart illustrating a key point detection method according to an embodiment of this disclosure is shown.

[0028] Figure 6 This diagram shows a block diagram of a key point detection model training device according to an embodiment of the present disclosure;

[0029] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0030] Figure 8 A block diagram of another electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0031] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0033] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0034] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0035] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0036] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0037] Figure 1 This diagram illustrates a flowchart of a keypoint detection model training method according to an embodiment of the present disclosure. Figure 1 As shown, the above method includes:

[0038] S10: Acquire multiple sample images, each of which includes corresponding annotation information. The annotation information includes bounding box information for characterizing the target object in the sample image, and key point information for characterizing the target object.

[0039] In the embodiments of this disclosure, the above-mentioned sample images may be obtained from monitoring equipment, camera equipment, or mature datasets. This disclosure does not limit the method of obtaining the above-mentioned sample images.

[0040] In a specific embodiment, taking the key point detection scenario of a vehicle as an example, the above sample image is an image frame containing a vehicle recorded by the monitoring device, the above target object is a vehicle, and the above key points can be certain positions on the vehicle, such as the position of the wheels, the position of the rearview mirror, etc.

[0041] In this embodiment of the disclosure, the bounding box information of the target object includes the location information and category information of the corresponding target object, and the key point information includes the location information, category information, weight, etc. of the key point. The weight represents the importance of the key point. In the key point recognition process, by assigning different weights to the key points, the trained key point detection model can detect key points with larger weights more accurately and can be better adapted to practical applications.

[0042] S20: Input the above sample images into a preset model to perform target detection and key point detection based on the target detection results, and obtain sample detection results. The above sample detection results include target object prediction information and key point prediction information.

[0043] In this embodiment of the disclosure, the preset model can be a model that can be used for object detection in the field of machine learning, including but not limited to convolutional neural network series and YOLO series models.

[0044] In this embodiment of the disclosure, the target object prediction information includes the target object prediction category information, the position information of the prediction box corresponding to the target object, and the confidence level of the prediction box. The confidence level of the prediction box represents the probability that the target object exists within the prediction box.

[0045] In this embodiment of the disclosure, the key point prediction information includes the prediction category information of the key point, the prediction location information of the key point, and the confidence level of the key point. The confidence level of the key point represents the accuracy of the key point prediction.

[0046] In a specific embodiment, the above sample detection results can be represented as: Res = [Obj_1; Obj_2; ...; Obj_N] (N is a positive integer), Obj_i = [cl s_i, bbox_i], where Obj_i represents the i-th target object (i is a positive integer), and cl s_i has M dimensions (M is a positive integer) and represents the category of target Obj_i (which of the M categories it belongs to). bbox_i = [cx, cy, w, h, conf, KP_1, ..., KP_O] represents the predicted bounding box of target Obj_i, where cx and cy represent the x and y coordinates of the geometric center of the predicted bounding box, w and h represent the width and height of the bounding box, conf represents the confidence score for the bounding box (ranging from 0 to 1), and KP_i = [x_i, y_i, c_i] represents the i-th keypoint, where x_i and y_i are the x and y coordinates of the i-th keypoint, and c_i represents the confidence score of the i-th keypoint.

[0047] Based on the above configuration, the key point detection model provided in this disclosure can detect the target object and key points on the target object in the above sample image, and at the same time give the confidence score to evaluate the detection effect.

[0048] S30: Determine the target detection loss based on the difference between the target object prediction information and the bounding box information.

[0049] As can be seen from the above, the annotation box information of the target object includes the location information and category information of the corresponding target object.

[0050] Figure 2 This diagram illustrates a method for determining target detection loss according to an embodiment of the present disclosure. Figure 2 As shown, the above method includes:

[0051] S301: Based on the difference between the predicted category information of the target object and the category information of the target object, determine the classification loss component of the target object.

[0052] In this embodiment of the disclosure, the above-mentioned target detection supports the detection of multiple targets. For classification loss classification, only the classification loss of positive sample images can be calculated. The positive samples are sample images containing target objects.

[0053] S302: Based on the difference between the position information of the prediction box corresponding to the target object and the position information of the target object, determine the loss component of the prediction box of the target object.

[0054] In this embodiment of the disclosure, the location loss component can be calculated using IOU loss (IOU = intersection / union). By calculating the intersection-union ratio of the target's bounding box and the predicted bounding box, the quality of the predicted bounding box can be more accurately reflected.

[0055] This disclosure does not limit the specific calculation formula of the loss component of the target object prediction box based on the prediction box and the annotation box.

[0056] S303: Based on the confidence of the detection box, the position information of the prediction box corresponding to the target object, and the position information of the target object, determine the target object confidence loss component.

[0057] In the embodiments of this disclosure, the aforementioned target object confidence loss component may employ algorithms including but not limited to the binary cross-entropy algorithm, and this disclosure does not impose any restrictions on it.

[0058] S304: Determine the target detection loss based on the target object classification loss component, the target object prediction box loss component, and the target object confidence loss component.

[0059] In this embodiment, the object detection loss consists of a classification loss component, a bounding box prediction loss component, and a confidence loss component. Furthermore, weights can be assigned to each of these loss components based on training performance and actual needs, and then the results are summed to obtain the object detection loss. This disclosure does not limit the specific weights of each loss component in the object detection loss.

[0060] Based on the above configuration, this disclosure can obtain the training loss of the preset model for object detection, and adjust and update the parameters of the preset model based on the loss.

[0061] S40: Determine the key point detection loss based on the difference between the key point information and the key point prediction information.

[0062] As can be seen from the above, the key point information in this embodiment includes the location information, category information, and weight of the key point. The key point prediction information includes the predicted category information of the key point, the predicted location information of the key point, and the confidence level of the key point. The confidence level of the key point represents the accuracy of the key point prediction.

[0063] Figure 3 This diagram illustrates a method for determining keypoint detection loss according to an embodiment of the present disclosure. Figure 3 As shown, the above method includes:

[0064] S401: Based on the confidence level of the above key points, the true value of the confidence level of the above key points, and the total number of key points in the above sample images, determine the key point confidence loss component.

[0065] In this embodiment of the disclosure, the key point information further includes the key point confidence truth value, which indicates whether the key point is inside its corresponding sample image; when the key point is inside its corresponding sample image, the key point confidence truth value is a first preset value; when the key point is outside its corresponding sample image, the key point confidence truth value is a second preset value.

[0066] In a specific embodiment, the first preset value is 1, the second preset value is 0, and the confidence loss is calculated using the binary cross-entropy algorithm, as shown in the following formula:

[0067]

[0068] Where p represents the confidence score of the nth keypoint in the model output (n is a positive integer between 1 and N). N represents the number of keypoints, y_i represents the true confidence score of the ith keypoint, and p(y_i) represents the predicted information of the ith keypoint.

[0069] Based on the above configuration, the sample images are divided into positive and negative samples by setting the confidence ground truth of key points, and the confidence ground truth is calculated according to their sample characteristics, which can balance the influence of sample images on the above preset model.

[0070] S402: Determine the loss components at key point locations.

[0071] Figure 4 The diagram shown is a flowchart illustrating a method for determining the loss components of key point locations according to an embodiment of this disclosure. Figure 4 As shown, the above method includes:

[0072] S4021: Determine the size of the target object corresponding to the above key points.

[0073] The size of the target object can be determined by the object's annotation information and can be represented as the product of its length and width.

[0074] S4022: Based on the predicted location information, location information, weight, size of the corresponding target object, visibility value, and total number of key points in the sample images, determine the location loss components of the key points.

[0075] In this embodiment of the disclosure, the key point information further includes the visibility value of the key point, which indicates whether the key point is occluded; when the key point is not occluded, the visibility value of the key point is a third preset value; when the key point is occluded, the visibility value of the key point is a fourth preset value.

[0076] In a specific embodiment, the third preset value is 1, the fourth preset value is 0, and the calculation formula for the key point position loss component is as follows:

[0077]

[0078] Among them, d n k represents the Euclidean distance between the predicted value and the true value of the nth keypoint. n Let represent the weight of the nth keypoint, s represent the size of the target, and δ represent the visibility value of the nth keypoint. In a specific embodiment, keypoint detection is performed on the front and rear wheels of the vehicle, with the weight of the front wheel keypoint set to 0.7 and the weight of the rear wheel keypoint set to 0.3.

[0079] S403: Based on the above-mentioned key point confidence loss component and the above-mentioned key point location loss component, determine the above-mentioned key point detection loss.

[0080] In this embodiment, the keypoint detection loss consists of a location loss component and a confidence loss component. Furthermore, weights can be assigned to each loss component based on training performance and actual needs, and then the results are summed to obtain the keypoint detection loss. This disclosure does not limit the specific weights of each loss component in the keypoint detection loss.

[0081] Based on the above configuration, different weights are assigned to different key points, symbolizing the importance of the key points. By assigning different weights to similar key points, the problem that traditional heatmaps cannot distinguish similar key points is solved. At the same time, the above key point position loss takes into account the size of the target object where the key point is located. Therefore, the size of the target does not interfere with the recognition effect of the key point and has scale invariance.

[0082] S50: Adjust the parameters of the preset model based on the target detection loss and the key point detection loss to obtain the key point detection model.

[0083] In this embodiment of the disclosure, the parameters of the preset model are updated according to the target detection loss and the key point detection loss. The number of times the preset model is updated can be set. When the preset number of updates is reached, the update of the preset model is completed, the training ends, and the required key point detection model is obtained.

[0084] Based on the above configuration, the technical solution of this disclosure integrates keypoint detection into object detection. On the one hand, it incorporates keypoint representation into the object detection bounding box, predicts keypoints for each object, and predicts a confidence level for each keypoint, overcoming the problem that traditional heatmap representation cannot distinguish similar keypoints that are spatially close. On the other hand, it reuses the non-maximum suppression post-processing process in object detection, which greatly reduces the amount of computation compared with the post-processing calculation in traditional keypoint detection methods. In addition, the keypoint similarity loss function of this disclosure is used for model training, so that the trained model can be applied to a variety of keypoint detection scenarios. For example, the detection scenario can be the detection of keypoints of vehicles of different sizes and orientations in an intelligent driving scenario.

[0085] Figure 5 This diagram illustrates a flowchart of a key point detection method according to an embodiment of the present disclosure, as shown below. Figure 5 As shown, the above method includes:

[0086] S01: Acquire the image to be detected.

[0087] Taking the key point detection scenario of the front and rear wheels of a vehicle as an example, the target object in the above-mentioned image to be detected is a vehicle, and the front and rear wheels of the vehicle are the key points to be detected. This disclosure does not limit the method for acquiring the above-mentioned image to be detected, nor does it limit the application scenarios of the above-mentioned key point detection.

[0088] S02: Input the above image to be detected into the key point detection model to obtain the key points.

[0089] The above key point detection model is obtained based on the key point detection model training method in any of the above embodiments. It can perform key point detection on the above image to be detected and obtain the positions of the front and rear wheels.

[0090] Based on the above configuration, the embodiments of this disclosure can quickly identify the required key points in an image, especially key points with high similarity. It can be applied to, but is not limited to, the detection of key points of vehicles of different sizes and orientations in intelligent driving scenarios, reducing labor costs and improving detection accuracy.

[0091] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0092] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further.

[0093] Figure 6 This diagram illustrates a block diagram of a keypoint detection model training device according to an embodiment of the present disclosure, such as... Figure 6 As shown, the above-mentioned device includes:

[0094] The sample image acquisition module 101 is used to acquire multiple sample images, each of which includes corresponding annotation information. The annotation information includes bounding box information for characterizing the target object in the sample image, and key point information for characterizing the target object.

[0095] The detection module 102 is used to input the sample image into a preset model for target detection and key point detection based on the target detection results, and obtain sample detection results, which include target object prediction information and key point prediction information.

[0096] The target detection loss determination module 103 is used to determine the target detection loss based on the difference between the target object prediction information and the bounding box information.

[0097] The key point detection loss determination module 104 is used to determine the key point detection loss based on the difference between the key point information and the key point prediction information.

[0098] The parameter update module 105 is used to adjust the parameters of the preset model according to the target detection loss and the key point detection loss to obtain the key point detection model.

[0099] In some possible implementations, the target object prediction information includes the predicted category information of the target object, the position information of the prediction box corresponding to the target object, and the confidence level of the prediction box, wherein the confidence level of the prediction box represents the probability that the target object exists within the prediction box; the key point prediction information includes the predicted category information of the key point, the predicted position information of the key point, and the confidence level of the key point, wherein the confidence level of the key point represents the accuracy of the key point prediction.

[0100] In some possible implementations, the bounding box information of the target object includes the location information and category information of the corresponding target object. The target detection loss determination module 103 is used to determine the target detection loss based on the difference between the target object prediction information and the bounding box information, including: determining a target object classification loss component based on the difference between the predicted category information of the target object and the category information of the target object; determining a target object prediction box loss component based on the difference between the location information of the prediction box corresponding to the target object and the location information of the target object; determining a target object confidence loss component based on the confidence of the detection box, the location information of the prediction box corresponding to the target object, and the location information of the target object; and determining the target detection loss based on the target object classification loss component, the target object prediction box loss component, and the target object confidence loss component.

[0101] In some possible implementations, the key point information includes the location information, category information, weight, and confidence ground truth of the corresponding key point; the confidence ground truth of the key point indicates whether the key point is inside its corresponding sample image; when the key point is inside its corresponding sample image, the confidence ground truth of the key point is a first preset value; when the key point is outside its corresponding sample image, the confidence ground truth of the key point is a second preset value.

[0102] In some possible implementations, the keypoint detection loss module 104 is used to determine the keypoint detection loss based on the difference between the keypoint information and the keypoint prediction information, including: determining a keypoint confidence loss component based on the confidence of the keypoint, the ground truth value of the confidence of the keypoint, and the total number of keypoints in the sample image; determining a keypoint location loss component; and determining the keypoint detection loss based on the keypoint confidence loss component and the keypoint location loss component.

[0103] In some possible implementations, the key point information further includes the visibility value of the key point, which indicates whether the key point is occluded; if the key point is not occluded, the visibility value of the key point is a third preset value; if the key point is occluded, the visibility value of the key point is a fourth preset value; the key point detection loss module 104 determines the key point position loss component by including: the size of the target object corresponding to the key point; and determining the key point position loss component based on the predicted position information, position information, weight, size of the corresponding target object, visibility value, and the total number of key points in the sample image.

[0104] This disclosure also proposes a computer-readable storage medium storing at least one instruction or at least one program segment. When the at least one instruction or at least one program segment is loaded and executed by a processor, it implements the aforementioned keypoint detection model training method or keypoint detection method. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0105] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured for the aforementioned keypoint detection model training method or keypoint detection method.

[0106] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0107] Figure 7 This diagram illustrates a block diagram of an electronic device according to an embodiment of the present disclosure. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, or other terminal.

[0108] Reference Figure 7 The electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0109] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0110] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0111] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0112] Multimedia component 808 includes a screen that provides an output interface between the aforementioned electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0113] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0114] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0115] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0116] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as Wi-Fi, 2G, 3G, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the aforementioned communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IRDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0117] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform a keypoint detection model training method or a keypoint detection method.

[0118] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by the processor 820 of the electronic device 800 to complete the above-described key point detection model training method or key point detection method.

[0119] Figure 8 A block diagram of another electronic device according to an embodiment of the present disclosure is shown. For example, electronic device 1900 may be provided as a server. (Refer to...) Figure 8 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the aforementioned keypoint detection model training method or keypoint detection method.

[0120] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0121] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to complete the above-described key point detection model training method or key point detection method.

[0122] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0123] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0124] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0125] The computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information of computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this disclosure.

[0126] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0127] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0128] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions specified in the blocks may occur in a different order than those specified in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0130] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A keypoint detection model training method, characterized in that, The method includes: Multiple sample images are acquired, each of which includes corresponding annotation information. The annotation information includes bounding box information for characterizing the target object in the sample image, and key point information for characterizing the target object. The sample image is input into a preset model for target detection and key point detection based on the target detection results to obtain sample detection results, which include target object prediction information and key point prediction information. Based on the difference between the target object prediction information and the bounding box information, determine the target object classification loss component, the target object prediction box loss component, and the target object confidence loss component. The target detection loss is determined based on the target object classification loss component, the target object prediction box loss component, and the target object confidence loss component. The key point detection loss is determined based on the difference between the key point information and the key point prediction information. The parameters of the preset model are adjusted according to the target detection loss and the key point detection loss to obtain the key point detection model; The step of determining the keypoint detection loss based on the difference between the keypoint information and the keypoint prediction information includes: Based on the confidence level of the key points, the true confidence level of the key points, and the total number of key points in the sample image, the key point confidence loss component is determined. Based on the predicted location information, location information, weight, size of the corresponding target object, visibility value, and total number of key points in the sample image, the key point location loss component is determined. The keypoint detection loss is determined based on the keypoint confidence loss component and the keypoint location loss component.

2. The method according to claim 1, characterized in that, The target object prediction information includes the predicted category information of the target object, the position information of the prediction box corresponding to the target object, and the confidence level of the prediction box. The confidence level of the prediction box represents the probability that the target object exists in the prediction box. The key point prediction information includes the predicted category information of the key point, the predicted location information of the key point, and the confidence level of the key point. The confidence level of the key point represents the accuracy of the key point prediction.

3. The method according to claim 2, characterized in that, The bounding box information of the target object includes its corresponding location information and category information; determining the target object classification loss component, the target object prediction box loss component, and the target object confidence loss component based on the difference between the target object prediction information and the bounding box information includes: Based on the difference between the predicted category information of the target object and the category information of the target object, the classification loss component of the target object is determined; Based on the difference between the position information of the prediction box corresponding to the target object and the position information of the target object, the loss component of the prediction box of the target object is determined; Based on the confidence of the detection box, the position information of the prediction box corresponding to the target object, and the position information of the target object, the confidence loss component of the target object is determined.

4. The method according to claim 2 or 3, characterized in that, The key point information includes the location information, category information, weight, and confidence ground truth of the corresponding key point; The confidence ground value of the key point indicates whether the key point is inside its corresponding sample image; When the key point is located inside its corresponding sample image, the confidence truth value of the key point is a first preset value; When the key point is outside its corresponding sample image, the confidence truth value of the key point is a second preset value.

5. The method according to claim 4, characterized in that, The visibility value indicates whether the key point is occluded; When the key point is not obscured, the visibility value of the key point is a third preset value; When the key point is obscured, the visibility value of the key point is a fourth preset value.

6. A key point detection method, characterized in that, The method includes: Acquire the image to be detected; The image to be detected is input into the key point detection model to obtain the key points; The keypoint detection model is obtained by the keypoint detection model training method according to any one of claims 1-5.

7. A key point detection model training device, characterized in that, The device includes: The sample image acquisition module is used to acquire multiple sample images, each of which includes corresponding annotation information. The annotation information includes bounding box information for characterizing the target object in the sample image, and key point information for characterizing the target object. The detection module is used to input the sample image into a preset model for target detection and key point detection based on the target detection results, and obtain sample detection results, which include target object prediction information and key point prediction information. The target detection loss determination module is used to determine the target object classification loss component, the target object prediction box loss component, and the target object confidence loss component based on the difference between the target object prediction information and the bounding box information; and to determine the target detection loss based on the target object classification loss component, the target object prediction box loss component, and the target object confidence loss component. The key point detection loss determination module is used to determine the key point detection loss based on the difference between the key point information and the key point prediction information. The parameter update module is used to adjust the parameters of the preset model according to the target detection loss and the key point detection loss to obtain the key point detection model. The keypoint detection loss determination module is specifically used for: determining a keypoint confidence loss component based on the confidence level of the keypoint, the ground truth value of the keypoint confidence level, and the total number of keypoints in the sample image; determining a keypoint location loss component based on the predicted location information, location information, weight, size of the corresponding target object, visibility value, and the total number of keypoints in the sample image; and determining the keypoint detection loss based on the keypoint confidence loss component and the keypoint location loss component.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the keypoint detection model training method as described in any one of claims 1-5, or the keypoint detection method as described in claim 6.

9. An electronic device, characterized in that, It includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the keypoint detection model training method as described in any one of claims 1-5, or the keypoint detection method as described in claim 6, by executing the instructions stored in the memory.

Images