Target key point detection model training method, target key point detection method, target key point detection equipment and medium

Abstract

The invention discloses a training method of a target key point detection model, a detection method, equipment and a medium. The method comprises the following steps: acquiring a detection target, detection information of the detection target, multiple image features and multi-scale image coding features; target image features are determined based on the multiple image features and the multi-scale image coding features, and a learnable query vector group is constructed for the detection target based on the position information of the detection target; outputting predicted key point information through an initial target key point detection model based on the target image features and the learnable query vector group; and training an initial target key point detection model based on the predicted key point information and the marked key point information. According to the method, the training process of open set detection and the training process of attitude detection are decoupled, the target key point detection capability is added on the basis of the open set detection network, the training process is simplified on the basis of keeping the model universality capability of open set detection, and the training cost is saved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, a detection method, a device and a medium for a target key point detection model. Background Art

[0002] The current open set detection network requires a large amount of data for training and contains a large number of mechanisms designed for detection. The open set detection itself is already very complex. Therefore, when the open set detection network is needed to estimate the pose of the target, it is necessary to make the open set detection network have the ability to detect both the target and the key points of the target, which makes the training process of the open set detection network very complicated and requires a lot of time. At the same time, the training process is also limited by the labeled data, which limits the ability of the open set detection.

[0003] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0004] The technical problem to be solved by the present application is to provide a training method, detection method, device and medium for a target key point detection model in view of the deficiencies in the prior art.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a training method for a target key point detection model, wherein the training method for the target key point detection model specifically includes:

[0006] Performing target detection on training images in a preset training image set by means of a trained target detection model, obtaining detection targets in the training images and detection information of the detection targets, multiple image features, and multi-scale image encoding features, wherein the detection information includes detection target position information and target category, and the training images carry annotated key point information of the detection targets;

[0007] Determining target image features based on the multiple image features and the multi-scale image encoding features, and generating a learnable query vector group for the detection target based on the target category and the detection target position information of the detection target;

[0008] Based on the target image features and the learnable query vector group, outputting predicted key point information of the detected target through an initial target key point detection model;

[0009] Based on the predicted key point information and the annotated key point information of the detection target in the training image, the initial target key point detection model is trained to obtain a target key point detection model.

[0010] The training method of the target key point detection model, wherein the target detection model includes an image feature extraction module and a feature enhancement encoder, the multiple image features are image features obtained by each network layer in the image feature extraction module during feature extraction of training images, and the multi-scale image coding features are image coding features obtained by a network layer in the feature enhancement encoder during interactive enhancement of image features output by the last network layer in the image feature extraction module and preset prompt features.

[0011] The training method of the target key point detection model, wherein the first number of the multiple image features is greater than the second number of the multi-scale image coding features, and the determining of the target image features based on the multiple image features and the multi-scale image coding features specifically includes:

[0012] Selecting a second number of image features from the plurality of image features;

[0013] The image feature corresponding to each scale image coding feature is selected from the selected image features, and the target image feature is determined based on each scale image coding feature and its corresponding image feature.

[0014] The training method of the target key point detection model, wherein the first number is 24, the second number is 4, and the 4 image features are image features obtained by any 4 network layers of the image feature extraction module.

[0015] The training method of the target key point detection model, wherein the step of selecting the image feature corresponding to each scale image coding feature from the selected image features specifically includes:

[0016] Determine the layer sequence number of the network layer of the image feature extraction module corresponding to each image feature in the selected image features in the image feature extraction module;

[0017] Based on the layer sequence number and the resolution of each scale image coding feature, the image feature corresponding to each scale image coding feature is determined, wherein, for any two scale image coding features, the layer sequence number of the image feature corresponding to the image coding feature with a larger resolution is smaller than the layer sequence number of the image feature corresponding to the image coding feature with a smaller resolution.

[0018] The training method of the target key point detection model, wherein the determining of the target image features based on the image coding features of each scale and their corresponding image features specifically includes:

[0019] Align the image coding features of each scale with their corresponding image features to obtain the aligned image coding features of each scale;

[0020] The image coding features after alignment at each scale are concatenated with the image coding features at each scale and their corresponding image features to obtain target image features, wherein the number of target image features is the same as the number of multi-scale image coding features.

[0021] The training method of the target key point detection model, wherein the generating of a learnable query vector group for the detection target based on the target category and the detection target position information of the detection target specifically comprises:

[0022] Determine a key point vector group based on the target category;

[0023] A position encoding vector is determined based on the detection target position information of the detection target, and the position encoding vector is used as a reference position point of the key point vector group to obtain a learnable query vector group.

[0024] The training method of the target key point detection model, wherein the determining of the key point vector group based on the target category specifically includes:

[0025] The number of target key points is determined according to the target category, and key point vectors of the number of target key points are randomly initialized to obtain the key point vector group.

[0026] The training method of the target key point detection model, wherein the predicted key point information includes predicted key point position information and predicted key point visibility, and the annotated key point information includes annotated key point position information and annotated key point visibility; the training of the initial target key point detection model based on the predicted key point information and the annotated key point information of the detection target in the training image to obtain the target key point detection model specifically includes:

[0027] Constructing a position loss term based on the predicted key point position information of each predicted key point in the predicted key point information and the labeled key point position information of the corresponding labeled target key point;

[0028] Constructing a visibility loss term based on the predicted key point visibility of each predicted key point in the predicted key point information and the labeled key point visibility of its corresponding labeled target key;

[0029] A total loss term is calculated based on the position loss term and the visibility loss term, and an initial target key point detection model is trained based on the total loss term to obtain a target key point detection model.

[0030] The training method of the target key point detection model, wherein the detection information includes a detection score; before the initial target key point detection model is trained based on the predicted key point information and the labeled key point information of the detection target in the training image to obtain the target key point detection model, the method further includes:

[0031] Determine the prediction box of the detection target based on the predicted key point position information, and determine the annotation box based on the annotation key point position information;

[0032] The target annotation frame corresponding to the prediction frame is determined based on the detection score of the detection target and the intersection-over-union ratio score of the prediction frame and the annotation frame, so as to determine the annotation key points corresponding to each prediction key point.

[0033] A second aspect of the present application provides a method for detecting target key points, using a target key point detection model trained by the training method for the target key point detection model as described above, wherein the method for detecting target key points specifically includes:

[0034] Performing target detection on the image to be detected by using the trained target detection model to obtain detection information of the detection target in the image to be detected, multiple image features and multi-scale image coding features, wherein the detection information includes detection target position information and target category;

[0035] Determining target image features based on the multiple image features and the multi-scale image encoding features, and generating a learnable query vector group for the detection target based on the target category and the detection target position information of the detection target;

[0036] Based on the target image features and the learnable query vector group, target key point information of the detected target is output through a target key point detection model.

[0037] The third aspect of the present application provides a training device for a target key point detection model, wherein the training device for the target key point detection model specifically includes:

[0038] A detection module, used to perform target detection on training images in a preset training image set through a trained target detection model, and obtain detection targets in the training images and detection information of the detection targets, multiple image features and multi-scale image coding features, wherein the detection information includes detection target position information and target category;

[0039] A determination module, configured to determine target image features based on the multiple image features and the multi-scale image encoding features, and generate a learnable query vector group for the detection target based on the target category and the detection target position information of the detection target;

[0040] A detection module, configured to output predicted key point information of the detected target through an initial target key point detection model based on the target image features and the learnable query vector group;

[0041] A training module is used to train the initial target key point detection model based on the predicted key point information and the labeled key point information of the detection target in the training image to obtain a target key point detection model.

[0042] A fourth aspect of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the training method of the target key point detection model as described above, and / or to implement the steps in the detection method of the target key point as described above.

[0043] A fifth aspect of the present application provides a terminal device, comprising: a processor and a memory;

[0044] The memory stores a computer-readable program executable by the processor;

[0045] When the processor executes the computer-readable program, it implements the steps in the training method of the target key point detection model as described above, and / or implements the steps in the detection method of the target key point as described above.

[0046] Beneficial effects:

[0047] 1. This application decouples the training process of open set detection and posture estimation, which can be regarded as adding the ability of target key point detection on the basis of the open set detection network. It simplifies the training process and saves training costs while retaining the ability of open set detection.

[0048] 2. This application reuses the multi-scale image features extracted by the open set detection network in the target key point detection model, omitting the computational effort required for feature extraction, thereby reducing the computational effort for target key point detection.

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 A flowchart of a method for training a target key point detection model provided in an embodiment of the present application.

[0051] Figure 2A principle flow chart of a specific example of a method for training a target key point detection model provided in an embodiment of the present application.

[0052] Figure 3 A functional block diagram of a training device for a target key point detection model provided in an embodiment of the present application.

[0053] Figure 4 A functional block diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The embodiments of the present application provide a training method, a detection method, a device and a medium for a target key point detection model. In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0056] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0057] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0058] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0059] This embodiment provides a method for training a target key point detection model. Figure 1 and Figure 2 As shown, the training method of the target key point detection model specifically includes:

[0060] S10. Performing target detection on training images in a preset training image set by using a trained target detection model to obtain detection targets in the training images and detection information of the detection targets, multiple image features, and multi-scale image coding features.

[0061] Specifically, the pre-trained image set includes several training images, and the training images in the several training images all carry annotated key point information, the annotated key point information includes several groups of annotated key point position information and annotated key point visibility, each group of annotated key point position information and annotated key point visibility corresponds to a target key point, the annotated key point position information is the detection target position information of the target key point, and the annotated key point visibility is the visibility of the target key point. For example, if the target category is human, the human body key points are generally set to 17, then the annotated key point information of the human body in the training image includes 17 groups of annotated key point position information and annotated key point visibility. Of course, in actual applications, different target categories have different numbers of target key points, and the number of human body key points can be other, for example, 14, 21, etc. Similarly, if the target category is a table, the key points can be set to 8; if the target category is clothing, the key points can be set to 5.

[0062] The target detection model is a trained open set network model (e.g., YOLO-Wor ld, etc.) for performing target detection on an image. The target detection model can detect the detection target in the training image and the detection information of the detection target, wherein the detection information includes the detection target position information and the target category.

[0063] It should be noted that in practical applications, the target detection model may also use other network models with target detection capabilities, and no specific restrictions are made here. At the same time, the training image may include one detection target or multiple detection targets, and when the training image includes one detection target, the target detection model will detect one detection target and detection information of the detection target; when the training image includes multiple detection targets, the target detection model will detect multiple detection targets and detection information of each detection target.

[0064] The multiple image features and multi-scale image coding features are obtained by the target detection model during the target detection process of the training image, wherein the multiple image features and the multi-scale image coding features can be obtained by extracting features from the training image respectively, or the multiple image features are obtained by directly extracting features from the training image, and the multi-scale image coding features are obtained by re-extracting features from all the multiple image features or part of the scaled image features in the multiple image features. In addition, the number of image features contained in the multiple image features and the number of image features contained in the multi-scale image coding features may be the same or different; and the image scale of each image feature in the multiple image features and the image scale of each image coding feature in the multi-scale image coding features can also be set according to actual needs, without specific restrictions here.

[0065] Exemplarily, the target detection model includes an image feature extraction module and a feature enhancement encoder, wherein the image feature extraction module is used to extract features from training images to obtain multiple image features, and the feature enhancement encoder is used to interact the image features output by the last network layer in the image feature extraction module with preset prompt features, and obtain multi-scale image coding features through the interactive learning. That is to say, the multiple image features are obtained by each network layer of the image feature extraction module through feature extraction of the training images, wherein the image features sampled by each network layer are different. The multi-scale image coding features are obtained by interacting the image features output by the last network layer of the image feature extraction module with the preset prompt features in the feature enhancement encoder. For example, the image feature extraction module includes 24 network layers, each network layer outputs a different image feature to obtain 24 image features, and the feature enhancement encoder includes 6 network layers, each network layer outputs 4 image coding features of different scales.

[0066] Furthermore, the target detection model is used to perform open set detection, such as Figure 2As shown, the target detection model can also include a text feature extraction module, and extract features from the preset prompt text through the text question extraction module to obtain preset prompt features, wherein the preset prompt text can be a text prompt or a visual prompt. At the same time, the target detection model also includes a multimodal decoder, which is used to update the learnable query vector based on the multi-scale image coding features obtained by the feature enhancement encoder and the updated preset text features to obtain an updated query vector, and finally determine the detection target of the training image and the detection information of the detection target based on the updated query vector. Of course, in practical applications, the target detection model can also be prompted to initialize the query vector guided by the prompt, which is used to initialize the learnable query vector with the updated preset prompt features. In addition, the target detection model may not include a text feature extraction module. When performing open set detection, the pre-built preset prompt features (such as visual prompts, etc.) are directly input into the target detection module for interacting with the image features output by the last network layer of the image feature extraction module.

[0067] S20, determining target image features based on the multiple image features and the multi-scale image encoding features, and generating a learnable query vector group for the detection target based on the target category and the detection target position information of the detection target.

[0068] Specifically, the target image feature is used to provide image features for the learning process of the learnable query vector group, wherein the feature data of the target image feature is the same as the second number of the multi-scale image coding features, that is, a target image feature is determined based on the image coding features of each scale.

[0069] Exemplarily, the determining the target image feature based on the multiple image features and the multi-scale image coding feature specifically includes:

[0070] Selecting a second number of image features from the plurality of image features, wherein a first number of images of the plurality of image features is greater than a second number of the multi-scale image coding features;

[0071] The image feature corresponding to each scale image coding feature is selected from the selected image features, and the target image feature is determined based on each scale image coding feature and its corresponding image feature.

[0072] Specifically, when selecting the second number of image features from the plurality of image features, the selection may be made according to a preset selection rule, or may be randomly selected, or may be determined by interaction with the user. In a typical implementation, the first number is 24, the second number is 4, and the 24 image features correspond to the 24 network layers of the image feature extraction module, respectively. The preset selection rule is to select the image features output by the sixth network layer, the twelfth network layer, the eighteenth network layer, and the twenty-fourth network layer of the image feature extraction module. To this end, when selecting the second number of image features from the plurality of image features, the image features output by the sixth network layer, the twelfth network layer, the eighteenth network layer, and the twenty-fourth network layer of the image feature extraction module are directly selected. The multi-scale image coding feature is a multi-scale image feature obtained by any network layer of the feature enhancement encoder, or a multi-scale image feature obtained by a specified network layer. In this typical implementation, the multi-scale image coding feature is a multi-scale image feature obtained by any intermediate network layer (such as the penultimate network layer, etc.) of the feature enhancement encoder.

[0073] After acquiring the second number of image features, each image feature is matched with each image coding feature to obtain an image coding feature corresponding to each image feature, and each image feature is concatenated with its corresponding image coding feature to obtain a target image feature.

[0074] Exemplarily, the selecting of the image features corresponding to each scale image coding feature from the selected image features specifically includes:

[0075] Determine the layer sequence number of the network layer of the image feature extraction module corresponding to each image feature in the selected image features in the image feature extraction module;

[0076] Based on the layer sequence number and the resolution of each scale image coding feature, an image feature corresponding to each scale image coding feature is determined.

[0077] Specifically, after selecting the second number of image features, each image feature can be sorted in ascending order according to the layer sequence number of each corresponding network layer in the image feature extraction module, and each image coding feature can be sorted in descending order according to the resolution, and the image coding feature with the same sequence number as the image feature is used as the image coding feature corresponding to the image feature, that is, the sequence number of the image feature is the same as the sequence number of the image coding feature corresponding to it. In other words, for any two scale image coding features, the layer sequence number of the image feature corresponding to the image coding feature with a larger resolution is smaller than the layer sequence number of the image feature corresponding to the image coding feature with a smaller resolution. Of course, in practical applications, other methods can also be used for matching, for example, obtaining the correspondence between image features and image coding features by interacting with users, etc.

[0078] Furthermore, when determining the target image feature based on the image coding features of each scale and their corresponding image features, it is first determined whether the image size of the image feature is the same as the image size of the image coding feature corresponding to it. If they are the same, they are directly spliced. If they are not the same, they are first size-aligned and then spliced. In other words, the image coding features of each scale and its corresponding image feature are size-aligned to obtain the image coding features after the alignment of each scale; the image coding features after the alignment of each scale are spliced ​​with the image coding features of each scale and its corresponding image features to obtain the target image feature, wherein the size alignment of the image coding features of each scale and its corresponding image feature can transform the image coding features by projection so that the image sizes of the scale image coding features and its corresponding image features are aligned.

[0079] The number of learnable query vector groups is the same as the number of detection targets output by the target detection model, and each learnable query vector group includes a key point vector group and a position encoding vector, and the number of vectors in the key point vector group is the same as the number of target key points. For example, if the number of target key points is 17, then the key point vector group includes 17 key point vectors. In addition, the number of learnable query vector groups is the same as the number of detection targets included in the training image. For example, if the training image includes 3 detection targets, then there are 3 learnable query vector groups, and each learnable query vector group corresponds to one detection target.

[0080] Exemplarily, the generating of a learnable query vector group for the detection target based on the target category and the detection target position information of the detection target specifically includes:

[0081] Determine a key point vector group based on the target category;

[0082] A position encoding vector is determined based on the detection target position information of the detection target, and the position encoding vector is used as a reference position point of the key point vector group to obtain a learnable query vector group.

[0083] Specifically, the key point vector group includes several key point vectors, each of which can be randomly initialized, and the number of vectors in each key point vector group is the same as the number of target key points corresponding to the target category. To this end, when determining the key point vector group, the number of target key points can be first determined according to the target category, and then the key point vectors of the target key point number can be randomly initialized to obtain the key point vector group.

[0084] The position encoding vector is obtained by position encoding the detection target position information of the detection target. For example, the detection target position information includes the width and height of the detection target's annotation box and the coordinates of the center point of the annotation box, that is, the detection target position information is expressed as {x, y, w, h}, then the position encoding vector is a four-dimensional vector including x, y, w, h, that is, the position encoding vector can be expressed as [x, y, w, h]. In addition, after obtaining the position encoding vector, the position encoding vector is concatenated with the key point vector group to determine the learnable query vector group, in which the position encoding vector is used as a reference position point to guide the target key point detection model to focus on the image features of the local image area corresponding to the position encoding vector.

[0085] S30: Based on the target image features and the learnable query vector group, output the predicted key point information of the detected target through an initial target key point detection model.

[0086] Specifically, after obtaining the target image features and the learnable query vector group, the target image features and the learnable query vector group are input into the initial target key point detection model, the key point vector group in the learnable query vector group is updated through the initial target key point detection model to obtain the updated key point vector group, and then the predicted key point information is determined based on the updated key point vector group.

[0087] For example, Figure 2 As shown, the initial target key point detection model includes a decoder, which includes multiple decoding layers, and the input item of each decoding layer includes the updated key point vector group output by the previous encoding layer. The output item of each encoding layer includes the updated key point vector group, and the key point vector group included in the input item of the first encoding layer is a randomly initialized key point vector group. In other words, the updated key point vector group output by the previous encoding layer in two adjacent encoding layers will be used as the key point vector group of the next encoding layer, and then the learnable query vector group of the next encoding layer is constructed based on the key point vector group and the position encoding vector.

[0088] For example, Figure 2 As shown, the decoding layer includes a self-attention layer, a cross-attention layer and a feedforward network, the self-attention layer is connected to the cross-attention layer, and the cross-attention layer is connected to the feedforward network, wherein the input item of the self-attention layer includes a learnable query vector group, the input item of the cross-attention layer includes the output item of the self-attention layer and the target image feature, and the input item of the feedforward network is the output item of the cross-attention layer.

[0089] S40: Based on the predicted key point information and the labeled key point information of the detection target in the training image, the initial target key point detection model is trained to obtain a target key point detection model.

[0090] Specifically, after obtaining the predicted key point information, a loss term is constructed based on the predicted key point information and the labeled key point information, and then the initial target key point detection model is trained based on the loss term, wherein the predicted key point information includes the predicted key point position information and the predicted key point visibility, and the labeled key point information includes the labeled key point position information and the labeled key point visibility. To this end, the loss term may include a position loss term and a visibility loss term, or may only include a position loss term, or may only include a visibility loss term.

[0091] Exemplarily, the training of the initial target key point detection model based on the predicted key point information and the labeled key point information of the detection target in the training image to obtain the target key point detection model specifically includes:

[0092] Constructing a position loss term based on the predicted key point position information of each predicted key point in the predicted key point information and the labeled key point position information of the corresponding labeled target key point;

[0093] Constructing a visibility loss term based on the predicted key point visibility of each predicted key point in the predicted key point information and the labeled key point visibility of its corresponding labeled target key;

[0094] A total loss term is calculated based on the position loss term and the visibility loss term, and an initial target key point detection model is trained based on the total loss term to obtain a target key point detection model.

[0095] Specifically, the position loss term is used to reflect the position difference between the predicted key point and the annotated key point, and the visibility loss term is used to reflect the visibility difference between the predicted key point and the annotated key point. Among them, the position loss term can adopt L1 loss term, OKS loss term, etc., and the visibility loss term can adopt L1 loss term. In a specific implementation, the position loss term shown includes L1 loss term and OKS loss term, and the visibility loss term includes L1 loss term, that is, the total loss term is the sum of L1 loss term of position, L1 loss term of visibility and OKS loss term, that is, the total loss term can be expressed as:

[0096]

[0097] Where L represents the total loss term, Represents the L1 loss term of the position, L OKS represents the OKS loss term, The L1 loss term representing visibility.

[0098] It should be noted that when calculating the total loss term, the predicted key point information and the labeled key point information determined by the updated key point vector group output by the last decoding layer in the decoder can be used to determine the total loss term corresponding to the initial target key point detection model; the predicted key point information and the labeled key point information corresponding to each decoding layer determined by the updated key point vector group output by each decoding layer in the decoder can also be used to construct the total loss term corresponding to each decoding layer, and then the total loss terms corresponding to each decoding layer are added to determine the total loss term corresponding to the initial target key point detection model. Of course, in practical applications, the updated key point vector group output by some decoding layers in the decoder can also be used to determine the predicted key point information to determine the total loss term.

[0099] For each updated key point vector group output by the decoding layer, the updated key point vector group may be passed through a linear layer, and the linear layer outputs predicted key point information and predicted visibility information.

[0100] In one implementation, the detection information includes a detection score; before training the initial target key point detection model based on the predicted key point information and the labeled key point information of the detection target in the training image to obtain the target key point detection model, the method further includes:

[0101] Determine the prediction box of the detection target based on the predicted key point position information, and determine the annotation box based on the annotation key point position information;

[0102] The target annotation frame corresponding to the prediction frame is determined based on the detection score of the detection target and the intersection-over-union ratio score of the prediction frame and the annotation frame, so as to determine the annotation key points corresponding to each prediction key point.

[0103] Specifically, the prediction box is the prediction box of the detection target corresponding to the prediction key point position information, and the prediction box includes all the prediction target key points. The annotation box is the annotation box of the detection target in the training image. The annotation key point position information in the training image can determine multiple annotation boxes, and each of the multiple annotation boxes corresponds to a detection target. To this end, after obtaining the prediction box of the detection target, the prediction box will be matched with multiple annotation boxes in the training image to obtain the annotation box corresponding to the prediction box, and then the annotation key points corresponding to each prediction key point will be determined.

[0104] Further, when determining the target annotation frame corresponding to the prediction frame, the prediction frame and the annotation frame can be matched (such as Hungarian matching) based on the detection score of the detection target and the intersection-and-union ratio score of the prediction frame and the annotation frame to obtain the target annotation frame corresponding to the prediction frame, so as to improve the accuracy of matching the prediction frame and the annotation frame. Of course, in practical applications, the sum of the detection score of the detection target and the intersection-and-union ratio score of the prediction frame and the annotation frame can be used as the matching basis, or the detection score of the detection target and the intersection-and-union ratio score of the prediction frame and the annotation frame can be used as the matching basis, or only the intersection-and-union ratio score of the prediction frame and the annotation frame can be used as the matching basis, etc.

[0105] In summary, the present embodiment provides a method for training a target key point detection model, the method comprising performing target detection on a training image in a preset training image set through a trained target detection model, obtaining a detection target in the training image and detection information of the detection target, multiple image features and multi-scale image coding features, wherein the detection information includes detection target position information and target category; determining target image features based on the multiple image features and the multi-scale image coding features, and generating a learnable query vector group for the detection target based on the target category and the detection target position information of the detection target; outputting the predicted key point information of the detection target through an initial target key point detection model based on the target image features and the learnable query vector group; training the initial target key point detection model based on the predicted key point information and the labeled key point information of the detection target in the training image to obtain a target key point detection model. The present application decouples the training process of open set detection and posture detection, which can be regarded as adding the ability of target key point detection on the basis of the open set detection network, simplifying the training process on the basis of retaining the ability of open set detection, and saving training costs. At the same time, the present application reuses the multi-scale image features extracted by the open set detection network in the target key point detection model, omitting the computational effort required for feature extraction, thereby reducing the computational effort for target key point detection.

[0106] Based on the above-mentioned training method of the target key point detection model, this embodiment provides a training device for the target key point detection model, such as Figure 3 As shown, the training device of the target key point detection model specifically includes:

[0107] The target detection module 100 is used to perform target detection on the training images in the preset training image set through the trained target detection model, and obtain the detection target in the training image and the detection information of the detection target, multiple image features and multi-scale image coding features, wherein the detection information includes the detection target position information and the target category;

[0108] A determination module 200, configured to determine target image features based on the multiple image features and the multi-scale image encoding features, and to generate a learnable query vector group for the detection target based on the target category and the detection target position information of the detection target;

[0109] A key point detection module 300, configured to output predicted key point information of the detected target through an initial target key point detection model based on the target image features and the learnable query vector group;

[0110] The training module 400 is used to train the initial target key point detection model based on the predicted key point information and the labeled key point information of the detection target in the training image to obtain the target key point detection model.

[0111] Based on the above-mentioned target key point detection model training method, this embodiment provides a target key point detection method, using the target key point detection model trained by the above-mentioned target key point detection model training method, the target key point detection method specifically includes:

[0112] Performing target detection on the image to be detected by using the trained target detection model to obtain detection information of the detection target in the image to be detected, multiple image features and multi-scale image coding features, wherein the detection information includes detection target position information and target category;

[0113] Determine target image features based on the multiple image features and the multi-scale image encoding features, and construct a learnable query vector group based on the target category and the detection target;

[0114] Based on the target image features and the learnable query vector group, target key point information of the detected target is output through a target key point detection model.

[0115] Based on the training method of the target key point detection model, the present application also provides a terminal device, such as Figure 4As shown, it includes at least one processor 20, a display screen 21, and a memory 22, and may also include a communication interface 23 and a bus 24. The processor 20, the display screen 21, the memory 22, and the communication interface 23 may communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communication interface 23 may transmit information. The processor 20 may call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0116] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0117] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.

[0118] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0119] In addition, the specific process of loading and executing the multiple instruction processors in the above-mentioned storage medium and the terminal device has been described in detail in the above-mentioned method, and will not be described one by one here.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A training method for a target key point detection model, characterized in that: The training method of the target key point detection model specifically includes: Performing target detection on training images in a preset training image set by means of a trained target detection model, obtaining detection targets in the training images and detection information of the detection targets, multiple image features, and multi-scale image encoding features, wherein the detection information includes detection target position information and target category, and the training images carry annotated key point information of the detection targets; Determining target image features based on the multiple image features and the multi-scale image encoding features, and generating a learnable query vector group for the detection target based on the target category and the detection target position information of the detection target; Based on the target image features and the learnable query vector group, outputting predicted key point information of the detected target through an initial target key point detection model; Based on the predicted key point information and the annotated key point information of the detection target in the training image, the initial target key point detection model is trained to obtain a target key point detection model.

2. The training method of the target key point detection model according to claim 1, characterized in that: The target detection model includes an image feature extraction module and a feature enhancement encoder. The multiple image features are image features obtained by each network layer in the image feature extraction module during feature extraction of training images. The multi-scale image coding features are image coding features obtained by a network layer in the feature enhancement encoder during interactive enhancement of image features output by the last network layer in the image feature extraction module and preset prompt features.

3. The training method of the target key point detection model according to claim 1 or 2, characterized in that: The first number of images of the plurality of image features is greater than the second number of the multi-scale image coding features, and the determining of the target image features based on the plurality of image features and the multi-scale image coding features specifically includes: Selecting a second number of image features from the plurality of image features; The image feature corresponding to each scale image coding feature is selected from the selected image features, and the target image feature is determined based on each scale image coding feature and its corresponding image feature.

4. The training method of the target key point detection model according to claim 3 is characterized in that: The first number is 24, the second number is 4, and the 4 image features are image features obtained by any 4 network layers in the image feature extraction module.

5. The training method of the target key point detection model according to claim 3 is characterized in that: The step of selecting the image feature corresponding to each scale image coding feature from the selected image features specifically includes: Determine the layer sequence number of the network layer of the image feature extraction module corresponding to each image feature in the selected image features in the image feature extraction module; Based on the layer sequence number and the resolution of each scale image coding feature, the image feature corresponding to each scale image coding feature is determined, wherein, for any two scale image coding features, the layer sequence number of the image feature corresponding to the image coding feature with a larger resolution is smaller than the layer sequence number of the image feature corresponding to the image coding feature with a smaller resolution.

6. The training method of the target key point detection model according to claim 3, characterized in that: The determining of the target image features based on the image coding features at each scale and the corresponding image features specifically includes: Align the image coding features of each scale with their corresponding image features to obtain the aligned image coding features of each scale; The image coding features after alignment at each scale are concatenated with the image coding features at each scale and their corresponding image features to obtain target image features, wherein the number of target image features is the same as the number of multi-scale image coding features.

7. The training method of the target key point detection model according to claim 1, characterized in that: The step of generating a learnable query vector group for the detection target based on the target category and the detection target position information of the detection target specifically includes: Determine a key point vector group based on the target category; A position encoding vector is determined based on the detection target position information of the detection target, and the position encoding vector is used as a reference position point of the key point vector group to obtain a learnable query vector group.

8. The training method of the target key point detection model according to claim 7, characterized in that: Determining the key point vector group based on the target category specifically includes: The number of target key points is determined according to the target category, and key point vectors of the number of target key points are randomly initialized to obtain the key point vector group.

9. The training method of the target key point detection model according to claim 1, characterized in that: The predicted key point information includes predicted key point position information and predicted key point visibility, and the annotated key point information includes annotated key point position information and annotated key point visibility; the training of the initial target key point detection model based on the predicted key point information and the annotated key point information of the detection target in the training image to obtain the target key point detection model specifically includes: Constructing a position loss term based on the predicted key point position information of each predicted key point in the predicted key point information and the labeled key point position information of the corresponding labeled target key point; Constructing a visibility loss term based on the predicted key point visibility of each predicted key point in the predicted key point information and the labeled key point visibility of its corresponding labeled target key; A total loss term is calculated based on the position loss term and the visibility loss term, and an initial target key point detection model is trained based on the total loss term to obtain a target key point detection model.

10. The training method of the target key point detection model according to claim 1 or 9, characterized in that: The detection information includes a detection score; before the initial target key point detection model is trained based on the predicted key point information and the labeled key point information of the detection target in the training image to obtain the target key point detection model, the method further includes: Determine the prediction box of the detection target based on the predicted key point position information, and determine the annotation box based on the annotation key point position information; The target annotation frame corresponding to the prediction frame is determined based on the detection score of the detection target and the intersection-over-union ratio score of the prediction frame and the annotation frame, so as to determine the annotation key points corresponding to each prediction key point.

11. A method for detecting target key points, characterized in that: The target key point detection model is obtained by training using the target key point detection model training method according to any one of claims 1 to 10, wherein the target key point detection method specifically comprises: Performing target detection on the image to be detected by using the trained target detection model to obtain detection information of the detection target in the image to be detected, multiple image features and multi-scale image coding features, wherein the detection information includes detection target position information and target category; Determining target image features based on the multiple image features and the multi-scale image encoding features, and generating a learnable query vector group for the detection target based on the target category and the detection target position information of the detection target; Based on the target image features and the learnable query vector group, target key point information of the detected target is output through a target key point detection model.

12. A training device for a target key point detection model, characterized in that: The training device of the target key point detection model specifically includes: The target detection module is used to perform target detection on the training images in the preset training image set through the trained target detection model, and obtain the detection target in the training image and the detection information of the detection target, multiple image features and multi-scale image coding features, wherein the detection information includes the detection target position information and the target category; A determination module, configured to determine target image features based on the multiple image features and the multi-scale image encoding features, and generate a learnable query vector group for the detection target based on the target category and the detection target position information of the detection target; A key point detection module, configured to output predicted key point information of the detected target through an initial target key point detection model based on the target image features and the learnable query vector group; A training module is used to train the initial target key point detection model based on the predicted key point information and the labeled key point information of the detection target in the training image to obtain a target key point detection model.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the training method of the target key point detection model as described in any one of claims 1-10, and / or to implement the steps in the target key point detection method as described in claim 11.

14. A terminal device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, it implements the steps in the training method of the target key point detection model as described in any one of claims 1-10, and / or implements the steps in the target key point detection method as described in claim 11.

Images