Training method and detection method of target key point detection model, device and medium

Abstract

The application discloses a target key point detection model training method, a detection method, equipment and a medium. The method comprises the following steps: acquiring a detection target, detection information of the detection target, a plurality of image features and a plurality of multi-scale image coding features; determining target image features based on the plurality of image features and the plurality of multi-scale image coding features, and constructing a learnable query vector group for the detection target based on detection target position information; outputting predicted key point information through an initial target key point detection model based on the target image features and the learnable query vector group; and training the initial target key point detection model based on the predicted key point information and labeled key point information. The training process of the open set detection and the posture detection is decoupled, the ability of the target key point detection is added on the basis of the open set detection network, the training process is simplified on the basis of retaining the universal ability of the open set detection model, and the training cost is saved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a target key point detection model training method, a target key point detection model detection method, a device and a medium. BACKGROUND

[0002] The current open set detection network needs a large amount of data for training, and contains a large number of mechanisms designed for detection. The open set detection itself is already very complex. Therefore, when the open set detection network is needed to estimate the pose of the target, the open set detection network needs to have the ability of target detection and target key point detection at the same time, which will make the training process of the open set detection network very complex and need a lot of time cost. At the same time, the training process will also be limited by the labeled data, so that the ability of open set detection is limited.

[0003] Therefore, the prior art still needs to be improved and improved. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a target key point detection model training method, a target key point detection model detection method, a device and a medium to solve the problems of the prior art.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a target key point detection model training method, which specifically comprises:

[0006] detecting a training image in a preset training image set by a trained target detection model to obtain a detection target in the training image, detection information of the detection target, a plurality of image features and a plurality of scale image coding features, wherein the detection information includes detection target position information and a target category, and the training image carries labeled key point information of the detection target;

[0007] determining a target image feature based on the plurality of image features and the plurality of scale image coding features, and generating a learnable query vector group for the detection target based on the target category and the detection target position information of the detection target;

[0008] outputting predicted key point information of the detection target by an initial target key point detection model based on the target image feature and the learnable query vector group;

[0009] training the initial target key point detection model based on the predicted key point information and the labeled key point information of the detection target in the training image to obtain a target key point detection model.

[0010] The method for training the target key point detection model, wherein the target detection model comprises an image feature extraction module and a feature enhancement encoder, the multiple image features are image features obtained by each network layer in the image feature extraction module in a feature extraction process on a training image, and the multiple scale image coding features are image coding features obtained by a network layer in the feature enhancement encoder in an interactive enhancement process on image features output by a last network layer in the image feature extraction module and a preset prompt feature.

[0011] The method for training the target key point detection model, wherein a first number of the multiple image features is greater than a second number of the multiple scale image coding features, and the determining of the target image feature based on the multiple image features and the multiple scale image coding features specifically comprises:

[0012] selecting a second number of image features from the multiple image features;

[0013] selecting, from the selected image features, an image feature corresponding to each scale image coding feature, and determining the target image feature based on each scale image coding feature and the image feature corresponding thereto.

[0014] The method for training the target key point detection model, wherein the first number is 24, the second number is 4, and the four image features are image features obtained by any four network layers in the image feature extraction module.

[0015] The method for training the target key point detection model, wherein the selecting, from the selected image features, an image feature corresponding to each scale image coding feature specifically comprises:

[0016] determining a network layer of the image feature extraction module corresponding to each image feature in the selected image features in a layer sequence number of the image feature extraction module;

[0017] determining, based on the layer sequence number and a resolution of each scale image coding feature, an image feature corresponding to each scale image coding feature, wherein for any two scale image coding features, the layer sequence number of the image feature corresponding to the image coding feature with a larger resolution is smaller than the layer sequence number of the image feature corresponding to the image coding feature with a smaller resolution.

[0018] The method for training the target key point detection model, wherein the determining of the target image feature based on each scale image coding feature and the image feature corresponding thereto specifically comprises:

[0019] aligning the size of each scale image coding feature and the image feature corresponding thereto to obtain each scale aligned image coding feature;

[0020] The image coding features of each scale after alignment are spliced with the image features corresponding to the image coding features of each scale to obtain target image features, wherein the number of the target image features is the same as the number of the multi-scale image coding features.

[0021] The training method of the target key point detection model, wherein the generating of the learnable query vector group for the detection target based on the target category and the detection target position information of the detection target comprises:

[0022] determining the key point vector group based on the target category;

[0023] determining a position encoding vector based on the detection target position information of the detection target, and taking the position encoding vector as a reference position point of the key point vector group to obtain the learnable query vector group.

[0024] The training method of the target key point detection model, wherein the generating of the learnable query vector group for the detection target based on the target category and the detection target position information of the detection target comprises:

[0025] determining the number of target key points according to the target category, and randomly initializing the key point vectors of the number of target key points to obtain the key point vector group.

[0026] The training method of the target key point detection model, wherein the prediction key point information comprises prediction key point position information and prediction key point visibility, the labeled key point information comprises labeled key point position information and labeled key point visibility; and the training of the initial target key point detection model based on the prediction key point information and the labeled key point information of the detection target in the training image to obtain the target key point detection model comprises:

[0027] constructing a position loss term based on the prediction key point position information of each prediction key point in the prediction key point information and the labeled key point position information of the corresponding labeled target key point;

[0028] constructing a visibility loss term based on the prediction key point visibility of each prediction key point in the prediction key point information and the labeled key point visibility of the corresponding labeled target key point;

[0029] calculating a total loss term based on the position loss term and the visibility loss term, and training the initial target key point detection model based on the total loss term to obtain the target key point detection model.

[0030] The training method of the target key point detection model, wherein the detection information comprises a detection score; before the initial target key point detection model is trained based on the predicted key point information and the labeled key point information of the detection target in the training image to obtain the target key point detection model, the method further comprises:

[0031] determining a predicted bounding box of the detection target based on the predicted key point position information, and determining a labeled bounding box based on the labeled key point position information;

[0032] determining the target labeled bounding box corresponding to the predicted bounding box based on the detection score of the detection target, the intersection-over-union score of the predicted bounding box and the labeled bounding box, so as to determine the labeled key point corresponding to each predicted key point.

[0033] The second aspect of the application provides a target key point detection method using the target key point detection model trained by the training method of the target key point detection model, and the target key point detection method specifically comprises:

[0034] detecting a target in a to-be-detected image by using the trained target detection model to obtain detection information of the detection target in the to-be-detected image, a plurality of image features and a plurality of scale image coding features, wherein the detection information comprises target position information and a target category;

[0035] determining a target image feature based on the plurality of image features and the plurality of scale image coding features, and generating a learnable query vector group for the detection target based on the target category and the target position information of the detection target;

[0036] outputting target key point information of the detection target by using the target key point detection model based on the target image feature and the learnable query vector group.

[0037] The third aspect of the application provides a training device of a target key point detection model, and the training device of the target key point detection model specifically comprises:

[0038] a detection module, configured to detect a target in a training image in a preset training image set by using a trained target detection model to obtain the detection target in the training image, detection information of the detection target, a plurality of image features and a plurality of scale image coding features, wherein the detection information comprises target position information and a target category;

[0039] a determination module, configured to determine a target image feature based on the plurality of image features and the plurality of scale image coding features, and generate a learnable query vector group for the detection target based on the target category and the target position information of the detection target;

[0040] detecting a target image feature in the target image;

[0041] training a target key point detection model based on the predicted key point information and the labeled key point information of the detected target in the training image.

[0042] The fourth aspect of the present application provides a computer readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the training method of the target key point detection model and / or implement the steps in the target key point detection method.

[0043] The fifth aspect of the present application provides a terminal device, which comprises a processor and a memory.

[0044] The memory stores a computer readable program which can be executed by the processor.

[0045] The processor executes the computer readable program to implement the steps in the training method of the target key point detection model and / or implement the steps in the target key point detection method.

[0046] Advantages:

[0047] 1. The training process of open set detection and pose estimation is decoupled in the present application, which can be regarded as adding the ability of target key point detection on the basis of the open set detection network, simplifying the training process and saving the training cost on the basis of retaining the ability of open set detection.

[0048] 2. The multi-scale image features extracted by the open set detection network are reused in the target key point detection model in the present application, and the calculation amount required for feature extraction is omitted, so that the calculation amount of target key point detection can be reduced.

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0050] Figure 1 The flowchart of the training method of the target key point detection model provided by the embodiments of the present application.

[0051] Figure 2A principle flowchart of one specific example of a training method of a target key point detection model provided by an embodiment of the present application.

[0052] Figure 3 A principle block diagram of a training device of a target key point detection model provided by an embodiment of the present application.

[0053] Figure 4 A principle block diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] Embodiments of the present application provide a training method and a detection method of a target key point detection model, equipment and media. In order to make the purpose, technical solutions and effects of the present application clearer and more explicit, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0055] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the phrase "comprising" used in the specification of the present application means that the features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0056] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0057] It should be understood that the sequence numbers and sizes of the steps in the embodiments do not mean the order of execution, and the execution order of the processes is determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0058] The content of the application will be further described by describing the embodiments with reference to the accompanying drawings.

[0059] The embodiment provides a training method of a target key point detection model, as shown in Figure 1 and Figure 2 The training method of the target key point detection model specifically comprises the following steps.

[0060] S10, performing target detection on a training image in a preset training image set by using a trained target detection model to obtain a detected target in the training image, detection information of the detected target, a plurality of image features and a plurality of scale image coding features.

[0061] Specifically, the preset training image set includes a plurality of training images, and each training image in the plurality of training images carries labeled key point information, the labeled key point information includes a plurality of sets of labeled key point position information and labeled key point visibility, each set of labeled key point position information and labeled key point visibility corresponds to a target key point, the labeled key point position information is detection target position information of the target key point, and the labeled key point visibility is visibility of the target key point. For example, if the target category is a person, the human body key points are generally set to 17, and the labeled key point information of the human body in the training image includes 17 sets of labeled key point position information and labeled key point visibility. Of course, in actual application, the target category is different, the number of target key points is different, and the number of human body key points can be other, for example, 14, 21 and the like. Similarly, if the target category is a table, the key points can be set to 8; if the target category is clothing, the key points can be set to 5.

[0062] The target detection model is a trained open set network model (for example, YOLO-World and the like) for target detection of an image. The target detection model can detect the detected target in the training image and the detection information of the detected target, wherein the detection information includes the detection target position information and the target category.

[0063] It should be noted that, in actual application, the target detection model can also use other network models with target detection capability, which is not limited here. Meanwhile, the training image can include one detected target or a plurality of detected targets, and when the training image includes one detected target, the target detection model detects one detected target and the detection information of the detected target; when the training image includes a plurality of detected targets, the target detection model detects a plurality of detected targets and the detection information of each detected target.

[0064] The multiple image features and the multi-scale image coding features are obtained by the target detection model in a target detection process on the training image, wherein the multiple image features and the multi-scale image coding features can be obtained by performing feature extraction on the training image respectively, or the multiple image features are obtained by directly performing feature extraction on the training image, and the multi-scale image coding features are obtained by performing feature extraction again on all or part of the multiple image features. In addition, the number of image features contained in the multiple image features and the number of image features contained in the multi-scale image coding features can be the same or different, and the image scales of each image feature in the multiple image features and the image scales of each image coding feature in the multi-scale image coding features can also be set according to actual needs, which is not limited here.

[0065] For example, the target detection model comprises an image feature extraction module and a feature enhancement encoder, the image feature extraction module is used to perform feature extraction on the training image to obtain multiple image features, and the feature enhancement encoder is used to interact the image feature output by the last network layer of the image feature extraction module with a preset prompt feature, and obtain multi-scale image coding features through the interaction learning. That is, the multiple image features are obtained by each network layer of the image feature extraction module through feature extraction on the training image, wherein the image features sampled by each network layer are different. The multi-scale image coding features are obtained by interacting the image feature output by the last network layer of the image feature extraction module with the preset prompt feature in the feature enhancement encoder. For example, the image feature extraction module comprises 24 network layers, each network layer outputs a different image feature to obtain 24 image features, and the feature enhancement encoder comprises 6 network layers, each network layer outputs 4 image coding features of different scales.

[0066] Further, the target detection model is used for open set detection, such as Figure 2As shown, the target detection model can also include a text feature extraction module, which extracts features of the preset prompt text through the text question extraction module to obtain preset prompt features, wherein the preset prompt text can be a text prompt or a visual prompt. Meanwhile, the target detection model also includes a multi-modal decoder, which is configured to update the learnable query vector based on the multi-scale image coding features obtained by the feature enhancement encoder and the updated preset text features, to obtain an updated query vector, and finally determine the detection target of the training image and the detection information of the detection target based on the updated query vector. Of course, in actual application, the target detection model can also prompt the guided query vector initialization, which is configured to initialize the learnable query vector with the updated preset prompt features. In addition, the target detection model can also not include a text feature extraction module, and when performing open set detection, the pre-constructed preset prompt features (such as visual prompts) are directly input into the target detection module for interaction with the image features output by the last network layer of the image feature extraction module.

[0067] S20, determining a target image feature based on the multiple image features and the multi-scale image coding features, and generating a learnable query vector group for the detection target based on the target category and the detection target position information of the detection target.

[0068] Specifically, the target image feature is used to provide image features for the learning process of the learnable query vector group, wherein the feature data of the target image feature is the same as the second number of the multi-scale image coding features, that is, one target image feature will be determined based on each scale image coding feature.

[0069] For example, the determination of the target image feature based on the multiple image features and the multi-scale image coding features specifically includes:

[0070] selecting a second number of image features from the multiple image features, wherein the first number of the multiple image features is greater than the second number of the multi-scale image coding features;

[0071] selecting the image feature corresponding to each scale image coding feature from the selected image features, and determining the target image feature based on each scale image coding feature and its corresponding image feature.

[0072] Specifically, the second quantity of image features can be selected according to a preset selection rule, randomly selected, or determined through user interaction. In a typical implementation, the first quantity is 24, the second quantity is 4, the 24 image features correspond to 24 network layers of the image feature extraction module respectively, and the preset selection rule is to select the image features output by the sixth network layer, the twelfth network layer, the eighteenth network layer, and the twenty-fourth network layer of the image feature extraction module. Therefore, when the second quantity of image features is selected from the plurality of image features, the image features output by the sixth network layer, the twelfth network layer, the eighteenth network layer, and the twenty-fourth network layer of the image feature extraction module are directly selected. The multi-scale image coding feature is a multi-scale image feature obtained by any network layer of the feature enhancement encoder, or a multi-scale image feature obtained by a specified network layer. In this typical implementation, the multi-scale image coding feature is a multi-scale image feature obtained by any intermediate network layer (such as the second-to-last network layer) of the feature enhancement encoder.

[0073] After obtaining the second quantity of image features, each image feature is matched with each image coding feature to obtain an image coding feature corresponding to each image feature, and each image feature is spliced with the image coding feature corresponding thereto to obtain a target image feature.

[0074] For example, the selecting, from the selected image features, an image feature corresponding to each scale image coding feature specifically includes:

[0075] Determining a network layer of the image feature extraction module corresponding to each image feature in the selected image features in a layer sequence number of the image feature extraction module.

[0076] Based on the layer sequence number and the resolution of each scale image coding feature, an image feature corresponding to each scale image coding feature is determined.

[0077] Specifically, after selecting the second number of image features, the image features can be sorted according to the layer numbers of the respective corresponding network layers in the image feature extraction module, and the layer numbers are sorted in ascending order, and the image coding features are sorted in descending order of resolution, and the image coding features with the same sequence number as the image features are taken as the image coding features corresponding to the image features, that is, the sequence number of the image features is the same as the sequence number of the image coding features corresponding to the image features. That is, for any two scale image coding features, the layer number of the image feature corresponding to the image coding feature with larger resolution is smaller than the layer number of the image feature corresponding to the image coding feature with smaller resolution. Of course, in actual application, other matching methods can also be used, for example, the correspondence between the image features and the image coding features can be obtained through user interaction.

[0078] Further, when determining the target image feature based on the image coding features of each scale and the image features corresponding thereto, it is determined whether the image size of the image feature is the same as the image size of the image coding feature corresponding thereto, and if so, the image features are directly spliced, and if not, the size is aligned before splicing. That is, the image coding features of each scale and the image features corresponding thereto are aligned in size to obtain the image coding features of each scale after alignment; the image coding features of each scale after alignment and the image coding features of each scale and the image features corresponding thereto are spliced to obtain the target image feature, wherein the image coding features of each scale and the image features corresponding thereto are aligned in size by projecting the image coding features to align the image sizes of the image coding features of each scale and the image features corresponding thereto.

[0079] The number of learnable query vector groups is the same as the number of detection targets output by the target detection model, and each learnable query vector group includes a key point vector group and a position encoding vector, and the number of key point vectors in the key point vector group is the same as the number of target key points, for example, if the number of target key points is 17, then the key point vector group includes 17 key point vectors. In addition, the number of learnable query vector groups is the same as the number of detection targets included in the training image, for example, if the training image includes 3 detection targets, then the number of learnable query vector groups is 3, and each learnable query vector group corresponds to a detection target.

[0080] For example, the learnable query vector group generated based on the target category and the detection target position information of the detection target includes:

[0081] determining a key point vector group based on the target category;

[0082] Based on the detection target location information, a location encoding vector is determined, and the location encoding vector is used as the reference location point of the key point vector group to obtain a learnable query vector group.

[0083] Specifically, the keypoint vector group comprises several keypoint vectors, each of which can be randomly initialized, and the number of vectors in each keypoint vector group is the same as the number of target keypoints corresponding to the target category. Therefore, when determining the keypoint vector group, the number of target keypoints can be determined first based on the target category, and then several keypoint vectors can be randomly initialized to obtain the keypoint vector group.

[0084] The positional encoding vector is obtained by encoding the positional information of the detected target. For example, the target positional information includes the width and height of the bounding box and the coordinates of the center point of the bounding box, i.e., the target positional information is represented as {x,y,w,h}. Then, the positional encoding vector is a four-dimensional vector including x,y,w,h, that is, the positional encoding vector can be represented as [x,y,w,h]. Furthermore, after obtaining the positional encoding vector, it is concatenated with the keypoint vector set to determine the learnable query vector set. In the learnable query vector set, the positional encoding vector is used as a reference position point to guide the target keypoint detection model to focus on the image features of the local image region corresponding to the positional encoding vector.

[0085] S30. Based on the target image features and the learnable query vector group, the predicted key point information of the detected target is output through the initial target key point detection model.

[0086] Specifically, after obtaining the target image features and the learnable query vector set, the target image features and the learnable query vector set are input into the initial target key point detection model. The key point vector set in the learnable query vector set is updated through the initial target key point detection model to obtain the updated key point vector set. Then, the predicted key point information is determined based on the updated key point vector set.

[0087] For example, such as Figure 2 As shown, the initial target keypoint detection model includes a decoder, which comprises multiple decoding layers. The input to each decoding layer includes the updated keypoint vector set output from the previous encoding layer. The output of each encoding layer also includes the updated keypoint vector set. The input to the first encoding layer consists of a randomly initialized keypoint vector set. In other words, the updated keypoint vector set output from the previous encoding layer in two adjacent encoding layers is used as the keypoint vector set for the next encoding layer. Then, based on this keypoint vector set and the positional encoding vector, a learnable query vector set for the next encoding layer is constructed.

[0088] As shown in Figure 2 The decoding layer includes a self-attention layer, a cross-attention layer, and a feedforward network, the self-attention layer is connected to the cross-attention layer, and the cross-attention layer is connected to the feedforward network, wherein the input of the self-attention layer includes a set of learnable query vectors, the input of the cross-attention layer includes the output of the self-attention layer and target image features, and the input of the feedforward network is the output of the cross-attention layer.

[0089] S40, based on the predicted key point information and the labeled key point information of the detected target in the training image, the initial target key point detection model is trained to obtain a target key point detection model.

[0090] Specifically, after obtaining the predicted key point information, a loss term is constructed based on the predicted key point information and the labeled key point information, and then the initial target key point detection model is trained based on the loss term, wherein the predicted key point information includes predicted key point position information and predicted key point visibility, and the labeled key point information includes labeled key point position information and labeled key point visibility. Therefore, the loss term can include a position loss term and a visibility loss term, or only a position loss term, or only a visibility loss term.

[0091] Exemplarily, the training of the initial target key point detection model based on the predicted key point information and the labeled key point information of the detected target in the training image to obtain a target key point detection model specifically includes:

[0092] constructing a position loss term based on the predicted key point position information of each predicted key point in the predicted key point information and the labeled key point position information of the corresponding labeled target key point;

[0093] constructing a visibility loss term based on the predicted key point visibility of each predicted key point in the predicted key point information and the labeled key point visibility of the corresponding labeled target key point;

[0094] calculating a total loss term based on the position loss term and the visibility loss term, and training the initial target key point detection model based on the total loss term to obtain a target key point detection model.

[0095] Specifically, the position loss term is used to reflect the position difference between the predicted key points and the labeled key points, and the visibility loss term is used to reflect the visibility difference between the predicted key points and the labeled key points. The position loss term can adopt an L1 loss term, an OKS loss term, etc., and the visibility loss term can adopt an L1 loss term. In a specific implementation manner, the position loss term includes an L1 loss term and an OKS loss term, and the visibility loss term includes an L1 loss term, that is, the total loss term is the sum of the L1 loss term of the position, the L1 loss term of the visibility, and the OKS loss term, that is, the total loss term can be expressed as:

[0096]

[0097] wherein L represents the total loss term, represents the L1 loss term of the position, L OKS represents the OKS loss term, represents the L1 loss term of the visibility.

[0098] It should be noted that when calculating the total loss term, the predicted key point information determined by the updated key point vector group output by the last decoding layer in the decoder and the labeled key point information can be used to determine the total loss term corresponding to the initial target key point detection model; or the predicted key point information corresponding to each decoding layer can be determined by using the updated key point vector group output by each decoding layer in the decoder and the labeled key point information to construct the total loss term corresponding to each decoding layer, and then the total loss terms corresponding to each decoding layer are added to determine the total loss term corresponding to the initial target key point detection model. Of course, in actual application, the predicted key point information can also be determined by using the updated key point vector group output by part of the decoding layers in the decoder.

[0099] For the updated key point vector group output by each decoding layer, the updated key point vector group can be output through a linear layer to output the predicted key point information and the predicted visibility information.

[0100] In an implementation manner, the detection information includes a detection score; before the initial target key point detection model is trained based on the predicted key point information and the labeled key point information of the detection target in the training image to obtain the target key point detection model, the method further includes:

[0101] determining a predicted box of the detection target based on the predicted key point position information, and determining a labeled box based on the labeled key point position information;

[0102] determining the target labeled box corresponding to the predicted box based on the detection score of the detection target, the intersection over union score of the predicted box and the labeled box, to determine the labeled key point corresponding to each predicted key point.

[0103] Specifically, the prediction box is a prediction box of a detection target corresponding to the prediction key point position information, the prediction box includes all prediction target key points, and the annotation box is an annotation box of a detection target in a training image. The annotation key point position information in the training image can determine a plurality of annotation boxes, and each annotation box in the plurality of annotation boxes corresponds to a detection target. Therefore, after obtaining the prediction box of the detection target, the prediction box is matched with the plurality of annotation boxes in the training image to obtain the annotation box corresponding to the prediction box, and then the annotation key point corresponding to each prediction key point is determined.

[0104] Further, in the determination of the target annotation box corresponding to the prediction box, the detection score of the detection target, the intersection over union score of the prediction box and the annotation box can be used as a basis to match (such as Hungarian matching) the prediction box and the annotation box to obtain the target annotation box corresponding to the prediction box, so as to improve the accuracy of matching the prediction box and the annotation box. Of course, in actual application, the sum of the detection score of the detection target and the intersection over union score of the prediction box and the annotation box can be used as a matching basis, or the detection score of the detection target and the intersection over union score of the prediction box and the annotation box can be used as a matching basis, or only the intersection over union score of the prediction box and the annotation box can be used as a matching basis.

[0105] In summary, the embodiment provides a training method of a target key point detection model, which comprises: performing target detection on a training image in a preset training image set by a trained target detection model to obtain a detection target in the training image, detection information of the detection target, a plurality of image features, and multi-scale image coding features, wherein the detection information comprises target position information and a target category; determining a target image feature based on the plurality of image features and the multi-scale image coding features, and generating a learnable query vector group for the detection target based on the target category and the target position information of the detection target; outputting prediction key point information of the detection target by an initial target key point detection model based on the target image feature and the learnable query vector group; and training the initial target key point detection model based on the prediction key point information and annotation key point information of the detection target in the training image to obtain a target key point detection model. The training process of the open set detection and the pose detection is decoupled, which can be regarded as adding the ability of target key point detection on the basis of the open set detection network, simplifying the training process on the basis of retaining the ability of open set detection, and saving the training cost. At the same time, the multi-scale image features extracted by the open set detection network are reused in the target key point detection model, and the calculation amount required for feature extraction is omitted, so that the calculation amount of target key point detection can be reduced.

[0106] Based on the training method of the target key point detection model, the embodiment provides a training device of a target key point detection model, as shown in the following table. Figure 3 The training device of the target key point detection model specifically comprises:

[0107] The target detection module 100 is configured to perform target detection on a training image in a preset training image set by using the trained target detection model, to obtain a detection target in the training image, detection information of the detection target, a plurality of image features, and multi-scale image coding features, wherein the detection information comprises target position information and a target category.

[0108] The determination module 200 is configured to determine target image features based on the plurality of image features and the multi-scale image coding features, and generate a learnable query vector group for the detection target based on the target category and the target position information of the detection target.

[0109] The key point detection module 300 is configured to output predicted key point information of the detection target by using an initial target key point detection model based on the target image features and the learnable query vector group.

[0110] The training module 400 is configured to train the initial target key point detection model based on the predicted key point information and labeled key point information of the detection target in the training image, to obtain a target key point detection model.

[0111] Based on the training method of the target key point detection model, the embodiment provides a target key point detection method, which uses the target key point detection model trained by the training method of the target key point detection model.

[0112] The target detection module 100 is configured to perform target detection on a training image in a preset training image set by using the trained target detection model, to obtain a detection target in the training image, detection information of the detection target, a plurality of image features, and multi-scale image coding features, wherein the detection information comprises target position information and a target category.

[0113] The determination module 200 is configured to determine target image features based on the plurality of image features and the multi-scale image coding features, and generate a learnable query vector group for the detection target based on the target category and the target position information of the detection target.

[0114] The key point detection module 300 is configured to output predicted key point information of the detection target by using an initial target key point detection model based on the target image features and the learnable query vector group.

[0115] Based on the training method of the target key point detection model, the embodiment provides a training device of a target key point detection model, as shown in the following table. Figure 4As shown, it comprises at least one processor 20, a display screen 21, and a memory 22, and can further comprise a communications interface 23 and a bus 24. The processor 20, the display screen 21, the memory 22 and the communications interface 23 can communicate with each other through the bus 24. The display screen 21 is configured to display the preset user guide interface in the initial setting mode. The communications interface 23 can transmit information. The processor 20 can call the logical instructions in the memory 22 to execute the method in the above embodiments.

[0116] In addition, the logical instructions in the memory 22 described above can be realized in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0117] The memory 22, as a computer readable storage medium, can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the method in the embodiments of the present disclosure. The processor 20 executes the functions and data processing by running the software programs, instructions or modules stored in the memory 22, that is, realizes the method in the above embodiments.

[0118] The memory 22 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 22 can include a high-speed random access memory, and can further include a non-volatile memory. For example, a variety of media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc., can also be a transient storage medium.

[0119] In addition, the specific processes of the above storage medium and the terminal device in which the processor loads and executes a plurality of instructions have been described in detail in the above method, and will not be repeated here.

[0120] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a target key point detection model, comprising: The training method for the target key point detection model specifically includes: The trained target detection model performs target detection on training images in a preset training image set to obtain the detected targets and their detection information, multiple image features, and multi-scale image coding features in the training images. The detection information includes the location information and category of the detected targets, and the training images carry the annotation key point information of the detected targets. The target image features are determined based on the features of the multiple images and the multi-scale image coding features, and a learnable query vector set is generated for the detected target based on the target category and the detection target location information. Based on the target image features and the learnable query vector set, the predicted key point information of the detected target is output through the initial target key point detection model; Based on the predicted key point information and the labeled key point information of the detected target in the training image, the initial target key point detection model is trained to obtain the target key point detection model.

2. The method of claim 1, wherein, The target detection model includes an image feature extraction module and a feature enhancement encoder. The multiple image features are the image features obtained by each network layer in the image feature extraction module during the feature extraction process of the training image. The multi-scale image coding features are the image coding features obtained by a network layer in the feature enhancement encoder during the interactive enhancement process of the image features output by the last network layer in the image feature extraction module and the preset prompt features.

3. The method of claim 1 or 2, wherein, The first number of images with multiple image features is greater than the second number of multi-scale image coding features, and the determination of target image features based on the multiple image features and the multi-scale image coding features specifically includes: Select a second number of image features from the plurality of image features; From the selected image features, the image features corresponding to the image coding features at each scale are selected, and the target image features are determined based on the image coding features at each scale and their corresponding image features.

4. The training method of claim 3, wherein, The first quantity is 24, the second quantity is 4, and the selected 4 image features are the image features obtained from any 4 network layers in the image feature extraction module.

5. The method of claim 3, wherein the training of the target key point detection model comprises: The step of selecting the image features corresponding to the image coding features at each scale from the selected image features specifically includes: The network layer of the image feature extraction module corresponding to each selected image feature is determined to be the layer number of the image feature extraction module. Based on the layer number and the resolution of the image coding feature at each scale, the image feature corresponding to each scale image coding feature is determined. For any two scale image coding features, the layer number of the image feature corresponding to the image coding feature with larger resolution is smaller than the layer number of the image feature corresponding to the image coding feature with smaller resolution.

6. The method of claim 3, wherein the training of the target key point detection model comprises: The determination of target image features based on image coding features at each scale and their corresponding image features specifically includes: The image coding features at each scale are aligned with their corresponding image features to obtain the scale-aligned image coding features. The image feature of each scale after alignment is spliced with the image feature of each scale and the corresponding image feature to obtain a target image feature, wherein the number of the target image features is the same as the number of the multi-scale image coding features.

7. The method of claim 1, wherein the target key point detection model is trained by using a training dataset including a plurality of images and a plurality of key points corresponding to the plurality of images. The method for generating the learnable query vector set for the detection target based on the target category and the detection target position information of the detection target specifically comprises: determining a key point vector set based on the target category; determining a position encoding vector based on the detection target position information of the detection target, and taking the position encoding vector as a reference position point of the key point vector set to obtain the learnable query vector set.

8. The training method of claim 7, wherein, The method for determining the key point vector set based on the target category specifically comprises: determining a target key point number according to the target category, and randomly initializing a key point vector of the target key point number to obtain the key point vector set.

9. The method of claim 1, wherein, The prediction key point information comprises prediction key point position information and prediction key point visibility, and the annotation key point information comprises annotation key point position information and annotation key point visibility; the method for training the initial target key point detection model based on the prediction key point information and the annotation key point information of the detection target in the training image to obtain the target key point detection model specifically comprises: constructing a position loss term based on the prediction key point position information of each prediction key point in the prediction key point information and the annotation key point position information of the corresponding annotation target key point; constructing a visibility loss term based on the prediction key point visibility of each prediction key point in the prediction key point information and the annotation key point visibility of the corresponding annotation target key point; calculating a total loss term based on the position loss term and the visibility loss term, and training the initial target key point detection model based on the total loss term to obtain the target key point detection model.

10. The method of claim 1 or 9, wherein, The detection information comprises a detection score; before the method for training the initial target key point detection model based on the prediction key point information and the annotation key point information of the detection target in the training image to obtain the target key point detection model, the method further comprises: determining a prediction box of the detection target based on the prediction key point position information, and determining an annotation box based on the annotation key point position information; determining the target annotation box corresponding to the prediction box based on the detection score of the detection target, the intersection over union score of the prediction box and the annotation box, to determine the annotation key point corresponding to each prediction key point.

11. A method for detecting a target key point, characterized in that, The target key point detection model trained by the training method of the target key point detection model according to any one of claims 1-10, the target key point detection method specifically comprises: detecting a target in a to-be-detected image by the trained target detection model to obtain detection information of the detection target in the to-be-detected image, a plurality of image features and a plurality of multi-scale image coding features, wherein the detection information comprises detection target position information and a target category; determining a target image feature based on the plurality of image features and the plurality of multi-scale image coding features, and generating a learnable query vector set for the detection target based on the target category and the detection target position information of the detection target; The target key point information of the detection target is output by a target key point detection model based on the target image feature and the learnable query vector group.

12. An apparatus for training a target key point detection model, comprising: The training device of the target key point detection model specifically comprises: A target detection module is configured to perform target detection on training images in a preset training image set by a trained target detection model to obtain detection targets in the training images, detection information of the detection targets, multiple image features, and multiple scale image coding features, wherein the detection information comprises target position information and a target category; and the training images carry labeled key point information of the detection targets. A determination module is configured to determine a target image feature based on the multiple image features and the multiple scale image coding features, and generate a learnable query vector group for the detection target based on the target category and the target position information of the detection target. A key point detection module is configured to output predicted key point information of the detection target by an initial target key point detection model based on the target image feature and the learnable query vector group. A training module is configured to train the initial target key point detection model based on the predicted key point information and the labeled key point information of the detection target in the training images to obtain a target key point detection model.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores one or more programs that can be executed by one or more processors to implement the steps in the training method of the target key point detection model according to any one of claims 1-10, and / or to implement the steps in the detection method of the target key point according to claim 11.

14. A terminal device, comprising: comprises: a processor and a memory; the memory stores a computer readable program that can be executed by the processor; the processor executes the computer readable program to implement the steps in the training method of the target key point detection model according to any one of claims 1-10, and / or to implement the steps in the detection method of the target key point according to claim 11.

Images