Key point detection method, device, equipment and medium

Abstract

The present application provides a key point detection method, device, equipment and medium, wherein the method includes: inputting the image to be detected into the key point prediction model, obtaining the coordinate value of the prediction box in the image to be detected, determining the coordinate value of each predicted key point in the object to be detected according to the coordinate value of the prediction box, obtaining the predicted mask value and predicted fine-tuning value of each predicted key point according to the key point prediction model, and obtaining the target coordinate value of the predicted key point according to the coordinate value, mask value and predicted fine-tuning value of the predicted key point. In this technical solution, the positioning result of the key point is obtained by combining the coordinate value, predicted mask value and predicted fine-tuning value of the key point predicted by the key point prediction model, which can output a higher quality key point positioning result and improve the key point prediction effect.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a key point detection method, device, equipment and medium. Background Art

[0002] In the field of image recognition, human skeleton key points are a very popular direction, which can be applied to fields such as somatosensory games, motion recognition and motion anomaly detection. The skeleton key point detection algorithm mainly identifies the positions of various skeleton key points of the human body, such as eyes, nose, knees, waist and shoulders.

[0003] In the prior art, when detecting key points of human skeleton, a single person region is usually obtained from an image using a target detection algorithm, and then skeleton key points are detected in the region.

[0004] However, this method in the prior art usually has low detection accuracy or a long detection process, resulting in poor detection effect of key points of human skeleton. Summary of the invention

[0005] The present application provides a key point detection method, device, equipment and medium, which are used to solve the problem of poor key point detection effect in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a key point detection method, comprising:

[0007] Inputting the image to be detected into the key point prediction model, obtaining the coordinate value of the prediction box in the image to be detected, the prediction box is circumscribed to the object to be detected in the image to be detected, and the key point prediction model is obtained by training with preset label data;

[0008] Determining the coordinate value of each predicted key point in the object to be detected according to the coordinate value of the predicted frame, wherein the object to be detected includes at least two predicted key points;

[0009] According to the key point prediction model, a prediction mask value and a prediction fine-tuning value of each predicted key point are obtained, wherein the prediction mask value is used to indicate the position sequence of the predicted key point in the object to be detected, and the prediction fine-tuning value is used to indicate the predicted position offset of the predicted key point;

[0010] The target coordinate value of the predicted key point is obtained according to the coordinate value of the predicted key point, the predicted mask value and the predicted fine-tuning value.

[0011] In a possible design of the first aspect, determining the coordinate value of each predicted key point in the object to be detected according to the coordinate value of the predicted frame includes:

[0012] Acquire the size, coordinate value and proportion value of the prediction box, where the proportion value is used to indicate the ratio of the coordinate value of the prediction key point to the size of the prediction box;

[0013] The coordinate value of each predicted key point is determined according to the coordinate value, size and proportion value of the predicted box.

[0014] In another possible design of the first aspect, obtaining a predicted mask value and a predicted fine-tuning value of each predicted key point according to the key point prediction model includes:

[0015] According to the key point prediction model, a prediction mask value of each predicted key point is obtained;

[0016] Determining a central key point among the at least two predicted key points according to the predicted mask value of each predicted key point;

[0017] The coordinate value of the central key point is obtained, and the predicted fine-tuning value is obtained according to the coordinate value of the central key point and the coordinate value of each predicted key point.

[0018] In another possible design of the first aspect, acquiring the target coordinate value of the predicted key point according to the coordinate value, the predicted mask value, and the predicted fine-tuning value of the predicted key point includes:

[0019] Obtaining a preset size of a priori frame in the image to be detected;

[0020] According to the predicted mask value of each predicted key point, the coordinate value of the central key point is determined;

[0021] The target coordinate value of each predicted key point is calculated based on the coordinate value of the predicted key point, the coordinate value of the central key point, the predicted fine-tuning value of the predicted key point, and the preset size of the priori box.

[0022] In another possible design of the first aspect, before inputting the image to be detected into the key point prediction model and obtaining the coordinate value of the prediction box in the image to be detected, the method further includes:

[0023] Acquire an image to be annotated, annotate each real key point of the target object in the image to be annotated, and determine the coordinate value and mask value of each real key point;

[0024] According to the coordinate values ​​and mask values ​​of each real key point, a real box is marked in the image to be marked to obtain a marked image, wherein the real box is circumscribed to the target object in the image to be marked;

[0025] According to the annotated image, preset label data is obtained.

[0026] In yet another possible design of the first aspect, determining the mask value of each real key point includes:

[0027] According to the coordinate values ​​of each real key point, the position of each real key point in the target object is obtained;

[0028] According to the position of each real key point in the target object, the mask value of each real key point is determined.

[0029] In another possible design of the first aspect, acquiring preset label data according to the annotated image includes:

[0030] Inputting the annotated image into the model to be trained, and obtaining the predicted coordinate values ​​and predicted mask values ​​of each key point in the annotated image;

[0031] Calculate and obtain fine-tuning data according to the predicted coordinate values, the predicted mask values, the coordinate values ​​of each real key point, and the preset size of the priori box;

[0032] According to the real frame, normalizing the coordinate values ​​of each real key point in the annotated image to obtain the normalized coordinate values;

[0033] The preset label data is obtained according to the fine-tuning data, the coordinate values ​​after normalization and the mask values ​​of the real key points.

[0034] In another possible design of the first aspect, before inputting the image to be detected into the key point prediction model and obtaining the coordinate value of the prediction box in the image to be detected, the method further includes:

[0035] According to the coordinate value after the normalization processing and the preset coordinate value prediction loss function, the real box and the preset real box prediction loss function, the mask value and the preset mask value prediction loss function, the fine-tuning data and the preset fine-tuning data prediction loss function, the model to be trained is trained to obtain a key point prediction model.

[0036] In a second aspect, an embodiment of the present application provides a key point detection device, comprising:

[0037] A coordinate value acquisition module is used to input the image to be detected into a key point prediction model to obtain the coordinate value of a prediction box in the image to be detected, where the prediction box is circumscribed to the object to be detected in the image to be detected, and the key point prediction model is obtained by training with preset label data and sample images;

[0038] A key point acquisition module, used to determine the coordinate value of each predicted key point in the object to be detected according to the coordinate value of the predicted frame, and the object to be detected includes at least two predicted key points;

[0039] A prediction value acquisition module, used to acquire a prediction mask value and a prediction fine-tuning value of each predicted key point according to the key point prediction model, wherein the mask value is used to indicate the position sequence of the predicted key point in the object to be detected, and the prediction fine-tuning value is used to indicate the predicted position offset of the predicted key point;

[0040] The target value acquisition module is used to acquire the target coordinate value of the predicted key point according to the coordinate value, mask value and predicted fine-tuning value of the predicted key point.

[0041] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, and a memory communicatively connected to the processor;

[0042] The memory stores computer-executable instructions;

[0043] The processor executes the computer-executable instructions stored in the memory to implement the above method.

[0044] In a fourth aspect, an embodiment of the present application provides a readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the above method.

[0045] The key point detection method, device, equipment and medium provided in the embodiments of the present application obtain the key point positioning results by combining the coordinate values, predicted mask values ​​and predicted fine-tuning values ​​of the key points predicted by the key point prediction model, and can output higher quality key point positioning results and improve the key point prediction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the description, are used to explain the principles of the present application;

[0047] Figure 1 A schematic diagram of a key point detection method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the network structure of the key point prediction model provided in the embodiment of the present application;

[0049] Figure 3 A schematic diagram of an image to be detected provided in an embodiment of the present application;

[0050] Figure 4 A mask schematic diagram provided for an embodiment of the present application;

[0051] Figure 5 A schematic diagram of mask sorting provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the structure of a key point detection device provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0054] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] The key point detection of human skeleton is mainly to identify the position of the key points of the human skeleton, such as the position of the eyes, nose, shoulders and waist in the image. The key point detection methods of human skeleton used in the prior art are mainly divided into two categories. One is a top-down detection method, which mainly uses the target detection algorithm to obtain a single person area from the image, and then performs skeleton key point detection on the single person area. However, when the number of human bodies contained in the image is large, the detection time will be longer, and the detection accuracy of the skeleton key points is also easily affected by the previous target detection algorithm. The other is a bottom-up detection method, which mainly effectively utilizes the information of the entire picture, detects all the key points in the picture, and then uses the algorithm to classify the key points. Although the accuracy of this method will not be affected, the subsequent classification operation is relatively complicated and the detection efficiency is not high.

[0057] In response to the above problems, the key point detection method, device, equipment and medium provided in the embodiments of the present application predict the coordinate values, mask values ​​and fine-tuning values ​​of the predicted key points in the image to be detected through a key point prediction model. It can more accurately predict the target coordinate values ​​of the key points based on the use of the entire image information, thereby improving the prediction effect of the key points.

[0058] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0059] Figure 1This is a flow chart of a key point detection method provided in an embodiment of the present application. The method can be applied to a computer device, taking a computer device as an execution subject as an example, Figure 1 As shown, the method may specifically include the following steps:

[0060] S101, input the image to be detected into the key point prediction model to obtain the coordinate value of the prediction box in the image to be detected.

[0061] The prediction box is circumscribed to the object to be detected in the image to be detected, and the key point prediction model is obtained by training with preset label data. For example, the preset label data can be an image annotated in advance by the user, for example, the user annotates in advance the positions of the key points of the object in the image, and annotates the circumscribed box corresponding to the object, i.e., the real box.

[0062] In this embodiment, the image to be detected may include a human body, that is, the object to be detected. The key points of the object to be detected refer to the skeleton key points of the human body, and the coordinate values ​​of the prediction box may be the coordinate values ​​of the center point of the prediction box.

[0063] Among them, the key point prediction model includes multiple output branches. Exemplarily, the key point prediction model includes a Detect branch, a Refine branch and an F_out branch. Among them, the feature map output by the Detect branch can be used to predict the prediction box in the image to be detected. The feature map output by the Refine branch can be used to obtain the predicted fine-tuning value of each predicted key point, and the feature map output by the F_out branch can be used to predict an ordered mask map to obtain a predicted mask value.

[0064] For example, Figure 2 This is a schematic diagram of the network structure of the key point prediction model provided in the embodiment of the present application. Figure 2 As shown, a color image of size W*H*3 can be input into the model, where W represents width, H represents height, and 3 represents the number of channels. The format of the subsequent sizes is similar to this.

[0065] The network structure includes the first branch Detect1, the second branch Refine1, the third branch Detect2, the fourth branch Refine2 and the fifth branch F_OUT, the backbone network, the first encoding layer Encode1, the second encoding layer Encode2, the first feature extraction layer F1, the second feature extraction layer F2, the first decoding layer Decode1, the second decoding layer Decode2, and the upsample UPsample. Detect1 outputs a feature map of W1*H1*41, where the feature map of W1*H1*41 is used to predict the prediction box in the image to be detected. The feature map of W1*H1*36 is used to predict the coordinate values ​​of 18 skeletal key points of the human body, and Refine1 outputs a feature map of W1*H1*36 to determine the predicted fine-tuning value of each predicted key point. The F_OUT branch outputs a feature map of W1*H1*18 to predict the predicted mask value of each predicted key point. Detect2 has similar functions to Detect1, and they are combined to obtain the Detect branch. Refine2 has similar functions to Refine1, and they are combined to obtain the Refine branch.

[0066] For example, Figure 3 A schematic diagram of an image to be detected provided in an embodiment of the present application, such as Figure 3 As shown, the image to be detected 31 includes a human body image, wherein the prediction box 311 is circumscribed to the human body image.

[0067] In the embodiment of the present application, the coordinate value of the predicted box can be obtained by obtaining the value of the Detect branch and then decoding it, as follows:

[0068]

[0069]

[0070]

[0071]

[0072] In the above formula, Represents the coordinate value of the j-th prediction box on the x-axis, Represents the model's predicted value for the j-th prediction box on the x-axis, represents the width of the i-th prior box, Represents the coordinate value of the i-th prior box on the x-axis. Represents the coordinate value of the j-th prediction box on the y-axis, Represents the model's predicted value for the j-th prediction box on the y-axis, represents the height of the i-th prior box, Represents the coordinate value of the i-th prior box on the y-axis. represents the width of the j-th prediction box, l w is the predicted width value of the model, represents the width of the j-th prior box, represents the height of the j-th prediction box, l h is the height value predicted by the model, Represents the height of the j-th prior box.

[0073] S102: Determine the coordinate value of each predicted key point in the object to be detected according to the coordinate value of the predicted frame.

[0074] The object to be detected includes at least two predicted key points. For example, the object to be detected is a human body, which includes 18 skeletal key points, such as eyes, nose, knees, arms, shoulders and waist.

[0075] In this embodiment, the coordinate value of the i-th prediction key point can be used It indicates that, Indicates the coordinate of the i-th predicted key point on the x-axis, PT_y i Represents the coordinate of the i-th key point on the y-axis. The coordinate value of the i-th predicted key point can be calculated by the following formula:

[0076]

[0077]

[0078] In the above formula, pre x Indicates the coordinate value of the prediction box on the x-axis, y Indicates the coordinate value of the prediction box on the y-axis, w Indicates the width of the prediction box, h Indicates the height of the prediction box. and Indicates the proportion value. For example, the coordinate value of the center point of the prediction box can be taken as (pre x , pre y ).

[0079] The ratio refers to the ratio of the coordinate value of the predicted key point to the size of the prediction box, which can be obtained by predicting the detect branch in the key point prediction model.

[0080] S103. Obtain a predicted mask value and a predicted fine-tuning value for each predicted key point according to the key point prediction model.

[0081] The predicted mask value is used to indicate the position order of the predicted key points in the object to be detected, and the predicted fine-tuning value is used to indicate the predicted position offset of the predicted key points.

[0082] In this embodiment, the predicted mask value of each predicted key point can be obtained through the F_OUT branch prediction in the key point prediction model, and the predicted fine-tuning value of each predicted key point can be obtained through the Refine branch in the key point prediction model.

[0083] For example, Figure 4 The mask schematic diagram provided in the embodiment of the present application takes the human body including 18 skeletal key points as an example, and the mask schematic diagram can be obtained according to the coordinates of each skeletal key point of the human body. Figure 4 Each mask in (each mask corresponds to a skeleton key point) is sorted. Figure 5 The mask sorting diagram provided in the embodiment of the present application is as follows: Figure 5 As shown, after sorting each mask by label, the mask value (ie, label) of each skeleton key point is obtained.

[0084] S104. Obtain target coordinate values ​​of the predicted key points according to the coordinate values ​​of the predicted key points, the predicted mask values, and the predicted fine-tuning values.

[0085] In this embodiment, the position of the predicted key point on the object to be detected can be determined according to the predicted mask value, and then the coordinate value of the predicted key point is fine-tuned according to the predicted fine-tuning value to obtain the target coordinate value of the predicted key point.

[0086] For example, the target coordinate value can be represented by (last_pt_x, last_pt_y). Among them, last_pt_x represents the value of the predicted key point on the x-axis, and last_pt_y represents the value of the predicted key point on the y-axis, which can be calculated by the following formula:

[0087]

[0088]

[0089] In the above formula, represents the predicted fine-tuning value of the i-th predicted key point on the x-axis, represents the predicted fine-tuning value of the i-th predicted key point on the y-axis, w bbox Indicates the width of the prior box, h bbox Indicates the height of the prior box, x detect Indicates the coordinate value of the predicted key point on the x-axis, y detect Indicates the coordinate value of the predicted key point on the y-axis, x maskIndicates the coordinate value of the center key point on the x-axis, y mask Indicates the coordinate value of the center key point on the y-axis.

[0090] Among them, the prior frame refers to the one obtained based on historical experience analysis. The width and height of the prior frame are predetermined. When predicting the prediction frame, this predetermined width and height can be used for processing to assist the prediction of the prediction frame. The central key point refers to the one selected from multiple prediction key points based on the prediction mask value.

[0091] The embodiment of the present application predicts the coordinate values, mask values ​​and fine-tuning values ​​of the predicted key points in the image to be detected through a key point prediction model. It can more accurately predict the target coordinate values ​​of the key points based on the use of the entire image information. The algorithm principle is simple and can improve the prediction effect of the key points.

[0092] Furthermore, in some embodiments, the above step S102 can be implemented by the following steps:

[0093] Get the size, coordinates and proportion of the prediction box;

[0094] According to the coordinate value, size and proportion of the prediction box, the coordinate value of each predicted key point is determined.

[0095] The ratio value is used to indicate the ratio of the coordinate value of the predicted key point to the size of the predicted box.

[0096] For example, the dimensions of the real frame include width W and height H, the coordinate values ​​of the real frame are (X, Y) (i.e., the coordinates of the center point of the real frame), and the coordinates of the key point of the human knee are (X9, Y9) as an example. The corresponding ratio values ​​of the key point of the human knee are ratio_X9 and ratio_Y9, which can be calculated by the following formula:

[0097] ratio_X9=((X9-(XW / 2)) / W

[0098] ratio_Y9=((Y9-(YH / 2)) / H

[0099] In this embodiment, the ratio value ratio_x corresponding to the i-th predicted key point can be used. i , the width of the prediction box and the coordinate value of the prediction box on the x-axis (that is, the coordinate value of the center point of the prediction box on the x-axis) are calculated to obtain the coordinate value of the i-th prediction key point on the x-axis. According to the ratio value ratio_y corresponding to the i-th prediction key point i , the height of the prediction box and the coordinate value of the prediction box on the y-axis (that is, the coordinate value of the center point of the prediction box on the y-axis) are calculated to obtain the coordinate value of the i-th prediction key point on the y-axis.

[0100] The embodiment of the present application realizes positioning by detecting the target and obtaining the ratio of key points to the target detection frame, which can obtain high-quality key point positioning results and improve the prediction effect of key points.

[0101] In some embodiments, the above step S103 can be implemented by the following steps:

[0102] According to the key point prediction model, the predicted mask value of each predicted key point is obtained;

[0103] Determine a central key point among at least two predicted key points according to the predicted mask value of each predicted key point;

[0104] Get the coordinate value of the central key point, and get the predicted fine-tuning value based on the coordinate value of the central key point and the coordinate value of each predicted key point.

[0105] In this embodiment, the predicted fine-tuning value of each predicted key point can be obtained through the Refine branch in the key point prediction model. Exemplarily, the predicted fine-tuning value may include the position offset refine_x of the i-th predicted key point on the x-axis. i and the position offset on the y-axis refine_y i Among them, refine_x i and refine_y i It can be calculated by the following formula:

[0106] refine_x i =(X gt -x mask -x detect ) / w bbx

[0107] refine_y i =(Y gt -y mask -y detect ) / h bbx

[0108] In the above formula, X gt Indicates the value of the corresponding real key point on the x-axis, Y gt Indicates the value of the corresponding real key point on the y-axis, x mask Indicates the coordinate value of the center key point on the x-axis, y mask Indicates the coordinate value of the center key point on the y-axis, x detect Indicates the value of the predicted key point on the x-axis, y detect Indicates the value of the predicted key point on the y-axis, w bbox Indicates the width of the prior box, h bbox Indicates the height of the prior box.

[0109] The embodiment of the present application obtains a predicted fine-tuning value by combining the coordinate value of each predicted key point and the predicted mask value, and can output a high-quality key point positioning result, thereby improving the prediction effect of the key point.

[0110] In some embodiments, the above step S104 can be implemented by the following steps:

[0111] Get the preset size of the prior box in the image to be detected;

[0112] According to the predicted mask value of each predicted key point, the coordinate value of the central key point is determined;

[0113] According to the coordinate values ​​of the predicted key points, the coordinate values ​​of the central key points, the predicted fine-tuning values ​​of the predicted key points, and the preset size of the prior box, the target coordinate values ​​of each predicted key point are calculated.

[0114] In this embodiment, the preset size of the prior frame includes the width and height of the prior frame. Each predicted key point has a corresponding serial number as a predicted mask value. For example, taking the human body having 18 bone key points as an example, the mask value of the first bone key point is 0, and the mask value of the 18th bone key point is 17. Among them, the mask value of the knee is 8, and the knee can be used as the central key point.

[0115] Among them, the target coordinate value can be expressed by (last_pt_x, last_pt_y). Through the above formula for calculating the target coordinate value, the coordinate value of the predicted key point, the coordinate value of the center key point, the predicted fine-tuning value of the predicted key point, and the size of the prior box are put into the formula to calculate the target coordinate value.

[0116] The embodiment of the present application obtains the target coordinate value information of the key point based on the coordinate value, predicted fine-tuning value and predicted mask value of each predicted key point, so as to obtain a more accurate key point positioning result and improve the prediction effect of the key point.

[0117] In some embodiments, the above key point detection method may further include the following steps:

[0118] Obtain the image to be annotated, annotate each real key point of the target object in the image to be annotated, and determine the coordinate value and mask value of each real key point;

[0119] According to the coordinate values ​​and mask values ​​of each real key point, a real box is marked in the image to be marked to obtain a marked image;

[0120] Obtain preset label data based on the annotated image.

[0121] Among them, the real box is circumscribed to the target object in the image to be labeled.

[0122] In this embodiment, taking the human body as the target object in the image to be annotated, the human body has 18 skeletal key points (i.e., real key points). These real key points are marked in advance, and their coordinate values ​​and mask values ​​are accurate and can be used as a reference in the model training process. Correspondingly, the real frame can also be used as a reference in the model training process.

[0123] In some real-time methods, different real key points have different corresponding mask values. For example, the mask value of the knee point is 8, while the mask value of the left foot is 10. Therefore, the position of each real key point in the target object can be determined according to the coordinate value of each real key point, and then the mask value of each real key point can be determined according to the position of each real key point in the target object.

[0124] The embodiment of the present application obtains annotated images by annotating annotated images, so that the key point prediction model obtained by training with preset label data can effectively utilize the entire image information, improve the positioning accuracy of key points, and improve the prediction effect of key points.

[0125] In some embodiments, the above step of "obtaining preset label data according to the annotated image" can be specifically implemented by the following steps:

[0126] Input the annotated image into the model to be trained to obtain the predicted coordinate values ​​and predicted mask values ​​of each key point in the annotated image;

[0127] The fine-tuning data is calculated based on the predicted coordinate values, the predicted mask values, the coordinate values ​​of each real key point and the preset size of the prior box;

[0128] According to the real frame, the coordinate values ​​of each real key point in the annotated image are normalized to obtain the normalized coordinate values;

[0129] The preset label data is obtained based on the fine-tuning data, the normalized coordinate values ​​and the mask values ​​of the real key points.

[0130] In this embodiment, the preset label data includes two parts, one part includes a labeled image and a mask image, the labeled image includes the coordinate value and mask value of each real key point, and the other part includes fine-tuning data.

[0131] Among them, the above Figure 3 For example, the coordinates of each bone key point in the human body can be marked in the image, and the coordinate values ​​of 18 bone key points can be obtained (X0, Y0), (X1, Y1), (X2, Y2)... (X17, Y17), and then the coordinate values ​​can be obtained. Figure 4 The mask diagram shown in Figure 4The masks in are labeled to obtain Figure 5 The mask sorting diagram is shown in Figure 1. Figure 5 The 18 masks in are converted into 18 new mask maps, each of which includes a mask. Figure 3 , Figure 4 , Figure 5 And these new mask maps can be used as part of the preset label data.

[0132] Furthermore, the coordinate values ​​of these skeletal key points need to be normalized. Specifically, the external matrix of the human body (i.e., the real frame) can be marked first, and then the 18 skeletal key points can be normalized based on the external matrix.

[0133] Among them, the normalization processing mainly refers to comparing the coordinate value of the i-th key point on the x-axis with the width of the external matrix to obtain the new coordinate value of the i-th key point on the x-axis, and comparing the coordinate value on the y-axis with the height of the external matrix to obtain the new coordinate value of the i-th key point on the y-axis.

[0134] For example, taking the human knee as an example, its coordinate value is (X8, Y8), and its new coordinate value after normalization is (ratio_X8, ratio_Y8).

[0135] ratio_X8=(X8-(Xw / 2)) / w

[0136] ratio_Y8=(Y8-(Yh / 2)) / h

[0137] In the above formula, (X, Y) is the coordinate of the center point of the circumscribed matrix, w is the width of the circumscribed matrix, and h is the height of the circumscribed matrix.

[0138] In this embodiment, another part of the fine-tuning data is obtained dynamically, which is obtained based on the predicted coordinate values ​​of the model Detect branch and the F_OUT branch and the coordinate values ​​of the real key points.

[0139] In some embodiments, the model to be trained can be trained based on the normalized coordinate value and the preset coordinate value prediction loss function, the true frame and the preset true frame prediction loss function, the mask value and the preset mask value prediction loss function, the fine-tuning data and the preset fine-tuning data prediction loss function to obtain the key point prediction model.

[0140] In this embodiment, see Figure 2, a residual network (Residual Network, ResNet) such as the common resnet18 can be used as the backbone network, which includes four layers, namely layer1, layer2, layer3 and layer4. Layer1 and layer2 perform basic feature extraction, while layer3 and layer4 output semantic information. For example, layer3 outputs semantic information to Detect1 for regression of bounding rectangles and key points; outputs semantic information to refine1 for fine-tuning; outputs semantic information to F1 and then inputs it into the Decode1 module, combined with the high-level semantic information of the Decode2 module, for the final prediction of the mask map. Among them, detect1 and refine1 are responsible for predicting the skeleton key points of small human targets, detect2 and refine2 are responsible for predicting the skeleton key points of large human targets, and realize multi-scale prediction.

[0141] Among them, in the process of model training, four loss functions need to be used, namely the preset coordinate value prediction loss function loss_pt, the preset true box prediction loss function loss_rect, the preset mask value prediction loss function loss_mask and the preset fine-tuning data prediction loss function loss_refine. Among them:

[0142]

[0143] In the above formula, m∈{cx, cy, w, h} means to substitute cx, cy, w, h into the formula for calculation respectively. Indicates whether the i-th preset box and the j-th real box match. If they match, the value is 1, and if they do not match, the value is 0. is the model's predicted value, Represents the encoding value obtained based on the predicted box and the true box. The details are as follows:

[0144]

[0145]

[0146]

[0147]

[0148] In the above formula, g represents the true box value, and d represents the prior box value.

[0149]

[0150] In the above formula, m∈{x, y} means that the proportion of the x-axis coordinate value and the y-axis coordinate value of each key point is substituted into the formula for calculation, and the maximum value of i is the total number of key points.

[0151]

[0152] In the above formula, Represents the predicted mask map of the i-th key point, mask i Represents the true mask map of the i-th key point.

[0153]

[0154] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0155] Figure 6 This is a schematic diagram of the structure of the key point detection device provided in the embodiment of the present application. The device can be integrated in a computer device, or it can be independent of the computer device and cooperate with the computer device to implement the present solution. Figure 6 As shown, the key point detection device 60 includes a coordinate value acquisition module 61, a key point acquisition module 62, a predicted value acquisition module 63 and a target value acquisition module 64.

[0156] Among them, the coordinate value acquisition module 61 is used to input the image to be detected into the key point prediction model to obtain the coordinate value of the prediction box in the image to be detected. The key point acquisition module 62 is used to determine the coordinate value of each predicted key point in the object to be detected according to the coordinate value of the prediction box. The predicted value acquisition module 63 is used to obtain the predicted mask value and predicted fine-tuning value of each predicted key point according to the key point prediction model. The target value acquisition module 64 is used to obtain the target coordinate value of the predicted key point according to the coordinate value, mask value and predicted fine-tuning value of the predicted key point.

[0157] Among them, the prediction box is connected to the object to be detected in the image to be detected, the key point prediction model is obtained by training with preset label data and sample images, the object to be detected includes at least two predicted key points, the mask value is used to indicate the position order of the predicted key points in the object to be detected, and the predicted fine-tuning value is used to indicate the predicted position offset of the predicted key points.

[0158] In some embodiments, the key point acquisition module may be specifically used for:

[0159] Get the size, coordinates and proportion of the prediction box;

[0160] According to the coordinate value, size and proportion of the prediction box, the coordinate value of each predicted key point is determined.

[0161] The ratio value is used to indicate the ratio of the coordinate value of the predicted key point to the size of the predicted box.

[0162] In some embodiments, the above-mentioned prediction data acquisition module can be specifically used for:

[0163] According to the key point prediction model, the predicted mask value of each predicted key point is obtained;

[0164] Determine a central key point among at least two predicted key points according to the predicted mask value of each predicted key point;

[0165] Get the coordinate value of the central key point, and get the predicted fine-tuning value based on the coordinate value of the central key point and the coordinate value of each predicted key point.

[0166] In some embodiments, the target value acquisition module may be specifically used for:

[0167] Get the size of the prior frame in the image to be detected;

[0168] According to the predicted mask value of each predicted key point, the coordinate value of the central key point is determined;

[0169] According to the coordinate values ​​of the predicted key points, the coordinate values ​​of the central key points, the predicted fine-tuning values ​​of the predicted key points, and the size of the prior box, the target coordinate values ​​of each predicted key point are calculated.

[0170] In some embodiments, the key point detection device further includes a label data acquisition module, which is used to:

[0171] Obtain the image to be annotated, annotate each real key point of the target object in the image to be annotated, and determine the coordinate value and mask value of each real key point;

[0172] Annotate the real box in the image to be annotated to obtain the annotated image, where the real box is circumscribed to the target object in the image to be annotated;

[0173] According to the annotated image, obtain the preset label data.

[0174] In some embodiments, the above-mentioned tag data acquisition module can be specifically used for:

[0175] Get the position of each real key point in the target object;

[0176] According to the position of each real key point in the target object, the mask value of each real key point is determined.

[0177] In some embodiments, the above-mentioned tag data acquisition module can be specifically used for:

[0178] Input the annotated image into the model to be trained, obtain the size of the prior box, the predicted coordinate values ​​and predicted mask values ​​of each key point in the annotated image;

[0179] The fine-tuning data is calculated based on the predicted coordinate values, predicted mask values, coordinate values ​​of each real key point and the size of the prior box;

[0180] According to the real frame, the coordinate values ​​of each real key point in the annotated image are normalized to obtain the normalized coordinate values;

[0181] The preset label data is obtained based on the fine-tuning data, the normalized coordinate values ​​and the mask values ​​of the real key points.

[0182] In some embodiments, the key point detection device further includes a training module for:

[0183] According to the normalized coordinate value and the preset coordinate value prediction loss function, the true box and the preset true box prediction loss function, the mask value and the preset mask value prediction loss function, the fine-tuning data and the preset fine-tuning data prediction loss function, the model to be trained is trained to obtain a key point prediction model.

[0184] The device provided in the embodiment of the present application can be used to execute the method in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0185] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the coordinate value acquisition module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0186] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0187] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 7 As shown, the computer device 70 includes: at least one processor 71 , a memory 72 , a bus 73 and a communication interface 74 .

[0188] The processor 71 , the communication interface 74 and the memory 72 communicate with each other via the bus 73 .

[0189] The communication interface 74 is used to communicate with other devices, including a communication interface for data transmission and a display interface or operation interface for human-computer interaction.

[0190] The processor 71 is used to execute the computer-executable instructions stored in the memory 72, and specifically can execute the relevant steps in the method described in the above embodiment.

[0191] The processor may be a central processing unit. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0192] The memory is used to store computer executable instructions and may include high-speed RAM memory, or may also include non-volatile memory, such as at least one disk memory.

[0193] This embodiment also provides a readable storage medium, in which computer execution instructions are stored. When at least one processor of a computer device executes the computer instructions, the computer device executes the key point detection methods provided by the various implementations described above.

[0194] This embodiment also provides a program product, which includes computer instructions, which are stored in a readable storage medium. At least one processor of a computer device can read the computer instructions from the readable storage medium, and at least one processor executes the computer instructions so that the computer device implements the key point detection method provided by the various embodiments described above.

[0195] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship; in the formula, the character " / " indicates that the previous and next associated objects are in a "division" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0196] It is to be understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not intended to limit the scope of the embodiments of the present application. In the embodiments of the present application, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A key point detection method, characterized in that: include: Inputting the image to be detected into the key point prediction model, obtaining the coordinate value of the prediction box in the image to be detected, the prediction box is circumscribed to the object to be detected in the image to be detected, and the key point prediction model is obtained by training with preset label data; Determining the coordinate value of each predicted key point in the object to be detected according to the coordinate value of the predicted frame, wherein the object to be detected includes at least two predicted key points; According to the key point prediction model, a prediction mask value and a prediction fine-tuning value of each predicted key point are obtained, wherein the prediction mask value is used to indicate the position sequence of the predicted key point in the object to be detected, and the prediction fine-tuning value is used to indicate the predicted position offset of the predicted key point; Obtaining a preset size of a priori frame in the image to be detected; According to the predicted mask value of each predicted key point, the coordinate value of the central key point is determined; The target coordinate value of each predicted key point is calculated based on the coordinate value of the predicted key point, the coordinate value of the central key point, the predicted fine-tuning value of the predicted key point, and the preset size of the priori box.

2. The method according to claim 1, characterized in that Determining the coordinate value of each predicted key point in the object to be detected according to the coordinate value of the predicted frame includes: Acquire the size, coordinate value and proportion value of the prediction box, where the proportion value is used to indicate the ratio of the coordinate value of the prediction key point to the size of the prediction box; The coordinate value of each predicted key point is determined according to the coordinate value, size and proportion value of the predicted box.

3. The method according to claim 1, characterized in that The step of obtaining a predicted mask value and a predicted fine-tuning value of each predicted key point according to the key point prediction model includes: According to the key point prediction model, a prediction mask value of each predicted key point is obtained; Determining a central key point among the at least two predicted key points according to the predicted mask value of each predicted key point; The coordinate value of the central key point is obtained, and the predicted fine-tuning value is obtained according to the coordinate value of the central key point and the coordinate value of each predicted key point.

4. The method according to claim 1, characterized in that: Before inputting the image to be detected into the key point prediction model and obtaining the coordinate value of the prediction box in the image to be detected, the method further includes: Acquire an image to be annotated, annotate each real key point of the target object in the image to be annotated, and determine the coordinate value and mask value of each real key point; According to the coordinate values ​​and mask values ​​of each real key point, a real box is marked in the image to be marked to obtain a marked image, wherein the real box is circumscribed to the target object in the image to be marked; According to the annotated image, preset label data is obtained.

5. The method according to claim 4, characterized in that Determining the mask value of each real key point includes: According to the coordinate values ​​of each real key point, the position of each real key point in the target object is obtained; According to the position of each real key point in the target object, the mask value of each real key point is determined.

6. The method according to claim 4, characterized in that The step of obtaining preset label data according to the annotated image includes: Inputting the annotated image into the model to be trained, and obtaining the predicted coordinate values ​​and predicted mask values ​​of each key point in the annotated image; Calculate and obtain fine-tuning data according to the predicted coordinate values, the predicted mask values, the coordinate values ​​of each real key point, and the preset size of the priori box; According to the real frame, normalizing the coordinate values ​​of each real key point in the annotated image to obtain the normalized coordinate values; The preset label data is obtained according to the fine-tuning data, the coordinate values ​​after normalization and the mask values ​​of the real key points.

7. The method according to claim 6, characterized in that Before inputting the image to be detected into the key point prediction model and obtaining the coordinate value of the prediction box in the image to be detected, the method further includes: According to the coordinate value after the normalization processing and the preset coordinate value prediction loss function, the real box and the preset real box prediction loss function, the mask value and the preset mask value prediction loss function, the fine-tuning data and the preset fine-tuning data prediction loss function, the model to be trained is trained to obtain a key point prediction model.

8. A key point detection device, characterized in that: include: A coordinate value acquisition module is used to input the image to be detected into a key point prediction model to obtain the coordinate value of a prediction box in the image to be detected, where the prediction box is circumscribed to the object to be detected in the image to be detected, and the key point prediction model is obtained by training with preset label data and sample images; A key point acquisition module, used to determine the coordinate value of each predicted key point in the object to be detected according to the coordinate value of the predicted frame, and the object to be detected includes at least two predicted key points; A prediction value acquisition module, used to acquire a prediction mask value and a prediction fine-tuning value of each predicted key point according to the key point prediction model, wherein the mask value is used to indicate the position sequence of the predicted key point in the object to be detected, and the prediction fine-tuning value is used to indicate the predicted position offset of the predicted key point; A target value acquisition module is used to obtain the preset size of the prior frame in the image to be detected; determine the coordinate value of the center key point according to the predicted mask value of each predicted key point; and calculate the target coordinate value of each predicted key point according to the coordinate value of the predicted key point, the coordinate value of the center key point, the predicted fine-tuning value of the predicted key point, and the preset size of the prior frame.

9. A computer device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores computer instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images