A method, device, electronic device and medium for detecting key points of a target

Abstract

The embodiments of the present application disclose a method, device, electronic device and medium for detecting target key points. The method includes: encoding the target key point type text to obtain key point type text encoding features; in the process of training the transform neural network to predict the target key points based on the sample image, adding the key point type text encoding features to the self-attention layer and cross-attention layer of the decoder to obtain the target key point prediction results of the sample image; adjusting and optimizing the transform neural network according to the target key point prediction results and annotation information to obtain a target key point detection model, and performing target key point detection on the image to be detected based on the target key point detection model. The above scheme adds key point type text encoding features, which enables the model to learn the common features of the same type of key points in the same part of the target, thereby improving the detection accuracy of the target key points and the convergence speed of the model.

Description

Technical Field

[0001] The present application relates to the field of image detection technology, and in particular to a method, device, electronic device and medium for detecting key points of a target. Background Art

[0002] ‌Keypoint detection technology is a computer vision technology that aims to locate the position of specific points on an object or human body. These points are usually anatomical or structural features of great significance, such as human joints (such as elbows, knees, shoulders, etc.), facial features (such as eyes, nose, mouth, etc.), or key points of objects (such as wheels, door handles, etc. of a car).

[0003] At present, the key point detection technology detects the entire image and detects the detection frames of different types of targets and the key points of different types of targets at the same time. However, this method needs to learn the characteristics between global key points. The commonalities and differences between key points are complex and diverse, and it is difficult to accurately and quickly learn the characteristics of key points, resulting in poor accuracy of key point detection and slow convergence. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device and medium for detecting target key points to improve the accuracy of target key point detection and increase the speed of model convergence.

[0005] According to one aspect of the present application, a method for detecting target key points is provided, the method comprising:

[0006] Encode the target key point type text to obtain a key point type text encoding feature containing the semantics of the target key point type text; wherein the target key point type text is used to reflect the target key point type in the sample image; the target key point type reflects the position of the target key point in the target;

[0007] In the process of training the transform neural network to predict the target key points based on the sample image, the key point type text encoding features are added to the self-attention layer and the cross-attention layer of the decoder to obtain the target key point prediction result of the sample image;

[0008] The transformation neural network is adjusted and optimized according to the target key point prediction results and the annotation information of the sample image to obtain a target key point detection model, so as to perform target key point detection on the image to be detected based on the target key point detection model.

[0009] According to one aspect of the present application, a target key point detection device is provided, the device comprising:

[0010] An encoding module is used to encode the target key point type text to obtain a key point type text encoding feature containing the semantics of the target key point type text; wherein the target key point type text is used to reflect the target key point type in the sample image; the target key point type reflects the position of the target key point in the target;

[0011] A prediction module, used for adding the key point type text encoding features in the self-attention layer and the cross-attention layer of the decoder during the process of training the transform neural network to predict the target key point based on the sample image, to obtain the target key point prediction result of the sample image;

[0012] The optimization module is used to adjust and optimize the transformation neural network according to the target key point prediction results and the annotation information of the sample image to obtain a target key point detection model, so as to perform target key point detection on the image to be detected based on the target key point detection model.

[0013] According to another aspect of the present application, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the target key point detection method of any embodiment of the present application.

[0017] According to another aspect of the present application, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a processor to implement the target key point detection method of any embodiment of the present application when executed.

[0018] The technical solution of the embodiment of the present application is to encode the target key point type text to obtain the key point type text encoding features containing the semantics of the target key point type text; wherein the target key point type text is used to reflect the target key point type in the sample image; the target key point type reflects the position of the target key point in the target; in the process of training the transform neural network to predict the target key point based on the sample image, the key point type text encoding features are added to the self-attention layer and the cross-attention layer of the decoder to obtain the target key point prediction result of the sample image; the transform neural network is adjusted and optimized according to the target key point prediction result and the annotation information of the sample image to obtain the target key point detection model, so as to perform target key point detection on the image to be detected based on the target key point detection model. The above scheme adds the key point type text encoding features, which can enable the model to learn the common features of the same type of key points in the same part of the target for the key points corresponding to the same key point type text encoding features, thereby improving the detection accuracy of the target key points and the convergence speed of the model.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flowchart of a target key point detection method provided in an embodiment of the present application;

[0022] Figure 2 A flowchart of a method for detecting target key points provided by another embodiment of the present application;

[0023] Figure 3 A flowchart of a target key point detection method provided in another embodiment of the present application;

[0024] Figure 4 A flowchart of a method for detecting target key points provided in another embodiment of the present application;

[0025] Figure 5 A schematic diagram of a network structure provided for another embodiment of the present application;

[0026] Figure 6A schematic diagram of the structure of a target key point detection device provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", "third", "fourth", "actual", "preset", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Figure 1 The flowchart of a target key point detection method provided in an embodiment of the present application is applicable to the case of detecting key points on a target. The method can be executed by a target key point detection device, which can be implemented in the form of hardware and / or software, and the target key point detection device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0031] S110. Encode the target key point type text to obtain a key point type text encoding feature containing the semantics of the target key point type text; wherein the target key point type text is used to reflect the target key point type in the sample image; the target key point type reflects the position of the target key point in the target.

[0032] Among them, the target key point type text may be a text describing the target key point type, and the target key point type is information reflecting the location of the target key point in the target. For example, when the target is a person, the target key point type may include head, nose, elbow, knee, shoulder, left hand, right hand, left foot, right foot, etc., and the corresponding target key point types may be "head", "nose", "elbow", "knee", "shoulder", "nose", "head", "left hand", "right hand", etc. When the target is a car, the target key point type may include front windshield, front door, rear door, door handle, wheel, "car door", "car window", "wheel", etc.

[0033] Exemplarily, for the target key point type text corresponding to each target key point type in the sample image, the target key point type text can be encoded to obtain the key point type text encoding feature. The key point type text encoding feature contains the semantic information of the target key point type text, thereby reflecting the target key point type. The key point type text encoding features corresponding to the same type of target key points have similarities, thereby achieving the effect of labeling the same target key points.

[0034] In an embodiment of the present application, the model used to encode the target key point type text can be determined according to the actual situation. The model can extract the semantic information in the target key point type text to reflect the commonality between the texts corresponding to the same target key point type. For example, there is similarity between the key point type text encoding features corresponding to the Chinese "left and right" and the English "left hand", reflecting the same semantics. The model used to encode the target key point type text can be, for example, the Clip model, the BGE model, the Jina model, etc.

[0035] S120. In the process of training the transform neural network to predict target key points based on the sample image, the key point type text encoding features are added to the self-attention layer and the cross-attention layer of the decoder to obtain the target key point prediction results of the sample image.

[0036] The sample image is an image that is acquired and annotated in advance. Transformer neural network is a transformer. The conventional network structure of the transformer neural network includes a backbone network for extracting features from the sample image to obtain a feature map, an encoder for enhancing the features, and a decoder for predicting the target detection frame, the detection frame type, the target key points, and the target key point type. The encoder includes a self-attention layer and a feedforward neural network, and the decoder includes a self-attention layer, a cross self-attention layer, and a feedforward neural network. Both the encoder and the decoder can be at least one.

[0037] In the embodiment of the present application, the function of the transformation neural network is to predict the target key points, so the process of predicting the target key points is also carried out during the training of the transformation neural network. The transformation neural network can be trained based on sample images. During the training process, key point type text encoding features are added to the self-attention layer and cross self-attention layer of the decoder of the transformation neural network, so that the model can learn common features for the same type of target key points with similar key point type text encoding features during the learning process, which is equivalent to adding prior knowledge to the transformation neural network for learning, thereby improving the convergence speed of the transformation neural network, quickly learning the commonality of the target key points, and being able to more clearly distinguish different types of target key points, thereby guiding the determination of the position of the target key points and improving the accuracy of the target key point detection.

[0038] Specifically, adding key point type text encoding features to the self-attention layer and the cross-attention layer of the decoder can be done by adding key point type text encoding features in the process of calculating the query matrix and the key matrix in the self-attention layer and the cross-attention layer, so as to add prior knowledge to the training process for training. For example, in general, the query matrix and the key matrix are obtained by multiplying the weight matrix on the basis of the input matrix. In an embodiment of the present application, the input matrix is ​​connected to the key point type text encoding feature, and the matrix formed after the connection is used as the input matrix, and multiplied by the weight matrix to obtain the query matrix and the key matrix.

[0039] S130, adjusting and optimizing the transformation neural network according to the target key point prediction result and the annotation information of the sample image to obtain a target key point detection model, so as to perform target key point detection on the image to be detected based on the target key point detection model.

[0040] Among them, the annotation information may include the detection frame, detection frame type, target key points and target key point type of each target. The target key point type can also be expressed in text, that is, the target key point type text. The detection frame of the target can be a rectangular frame containing the target, and the coordinates of the four vertices of the rectangular frame can reflect the position and size of the rectangular frame. The type of the detection frame can reflect the type of target contained in the detection frame, such as whether the target is a person or a car. The annotation of the target key point includes the coordinates of the target key point. The target key point type is information reflecting the location of the target key point on the target, such as the head, nose, eyes, left hand, right hand, etc.

[0041] For example, during the training process, the transformation neural network will predict the target key point prediction results based on the input sample image, that is, the predicted key points, including the predicted target detection frame, detection frame type, target key points and target key point type. The transformation neural network can be optimized and adjusted according to the difference between the target key point prediction results and the annotation information of the sample image, so that the transformation neural network can further learn the relationship between the sample image and the annotation information, and stop the iteration when the iteration condition is met to obtain the target key point detection model.

[0042] In subsequent applications, the target key point detection model can detect target key points on the input image to be detected, and obtain the detected target detection frame, detection frame type, target key points, and target key point type. The process executed by the target key point detection model in the application process is consistent with the process executed in the training process. Similarly, the key point type text encoding features will be added to the self-attention layer and cross-attention layer of the decoder to improve the accuracy of target key point detection.

[0043] The technical solution of the embodiment of the present application is to encode the target key point type text to obtain the key point type text encoding features containing the semantics of the target key point type text; wherein the target key point type text is used to reflect the target key point type in the sample image; the target key point type reflects the position of the target key point in the target; in the process of training the transform neural network to predict the target key point based on the sample image, the key point type text encoding features are added to the self-attention layer and the cross-attention layer of the decoder to obtain the target key point prediction result of the sample image; the transform neural network is adjusted and optimized according to the target key point prediction result and the annotation information of the sample image to obtain the target key point detection model, so as to perform target key point detection on the image to be detected based on the target key point detection model. The above scheme adds the key point type text encoding features, which can enable the model to learn the common features of the same type of key points in the same part of the target for the key points corresponding to the same key point type text encoding features, thereby improving the detection accuracy of the target key points and the convergence speed of the model.

[0044] Figure 2 This is a flow chart of a target key point detection method provided by another embodiment of the present application. The present embodiment is optimized based on the above embodiment. For solutions not described in detail in the present embodiment, please refer to the above embodiment. Figure 2 As shown, the method of the embodiment of the present application specifically includes the following steps:

[0045] S210. Encode the target key point type text to obtain a key point type text encoding feature containing the semantics of the target key point type text; wherein the target key point type text is used to reflect the target key point type in the sample image; the target key point type reflects the position of the target key point in the target.

[0046] S220. In the process of training the transform neural network to predict target key points based on the sample image, in the self-attention layer of the decoder, the query matrix and the key matrix in the self-attention layer are determined according to the target key point query features, the feature map grid position encoding features, the feature map hierarchical encoding features and the key point type text encoding features; wherein the target key point query features, the feature map grid position encoding features and the feature map hierarchical encoding features reflect the information that needs to be queried and obtained for the feature map of the sample image.

[0047] Among them, the target key point query feature can be a pre-initialized feature, which contains the information of the key points to be queried, such as the number of target key points, the type of target key points, the type of detection box, the coordinates of the detection box, and the coordinates of the target key points. The feature map grid position encoding feature and the feature map hierarchical encoding feature are similar, and the specific process is the same as the determination process in the conventional application of the transform neural network.

[0048] In an embodiment of the present application, in the process of determining the query matrix and the key matrix of the self-attention layer, the target key point query features, the feature map grid position encoding features, the feature map hierarchical encoding features and the key point type text encoding features are connected to jointly determine the query matrix and the key matrix, so as to associate each target key point based on the key point type text encoding features. When the key point type text encoding features are the same or similar, they can reflect the same type of target key points, so that the model learns the common characteristics of the same type of target key points to accelerate convergence.

[0049] Specifically, the target key point query feature, the feature map grid position coding feature, the feature map hierarchical coding feature and the key point type text coding feature can be connected, and the matrix formed after the connection is multiplied by a weight matrix to obtain a query matrix, and multiplied by another weight matrix to obtain a key matrix. The determination process of the weight matrix is ​​consistent with that in conventional applications.

[0050] S230, determining a value matrix according to the target key point query feature.

[0051] Exemplarily, the value matrix is ​​determined according to the target key point query feature. Specifically, assuming that the target key point query feature is , the feature map grid position encoding feature is , the feature layer level encoding feature is , the key point type text encoding feature is The input matrix is ​​Cat ( ). Where Cat refers to the matrix connection. Multiply the input matrix by the weight matrix , get the query matrix Q, multiply the input matrix by the weight matrix Get the key matrix K. Multiply by the weight matrix Get the value matrix V.

[0052] S240. Calculate a self-attention matrix according to the query matrix, the key matrix and the value matrix.

[0053] based on Compute the self-attention matrix. To calculate the self-attention matrix, Indicates normalization. is the dimension of the key matrix.

[0054] S250, adding the key point type text encoding features in the cross attention layer.

[0055] S260, obtaining a target key point prediction result for the sample image, adjusting and optimizing the transformation neural network according to the target key point prediction result and the annotation information of the sample image, and obtaining a target key point detection model, so as to perform target key point detection on the image to be detected based on the target key point detection model.

[0056] The embodiment of the present application provides a method for detecting target key points. In the self-attention layer of the decoder, according to the target key point query feature, the feature map grid position coding feature, the feature map hierarchical coding feature and the key point type text coding feature, the query matrix and the key matrix in the self-attention layer are determined; wherein the target key point query feature, the feature map grid position coding feature and the feature map hierarchical coding feature reflect the information that needs to be queried and obtained for the feature map of the sample image; the value matrix is ​​determined according to the target key point query feature; the self-attention matrix is ​​calculated according to the query matrix, the key matrix and the value matrix. The above scheme can add the key point type text coding feature to the self-attention layer of the decoder to calculate the query matrix and the key matrix, so that the model can learn the common characteristics of the target key points of the same target key point type in different targets, give the model prior knowledge, and improve the convergence speed of the model. And it can clearly distinguish the target key points with different key point type text coding features, guide the generation of coordinates of the same type of target key points, and improve the accuracy of target key point detection.

[0057] Figure 3This is a flowchart of a target key point detection method provided by another embodiment of the present application. The present embodiment is optimized based on the above embodiment. For solutions not described in detail in the present embodiment, please refer to the above embodiment. Figure 3 As shown, the method of the embodiment of the present application specifically includes the following steps:

[0058] S310. Encode the target key point type text to obtain a key point type text encoding feature containing the semantics of the target key point type text; wherein the target key point type text is used to reflect the target key point type in the sample image; the target key point type reflects the position of the target key point in the target.

[0059] S320, in the process of training the transform neural network to predict the target key points based on the sample images, adding the key point type text encoding features to the self-attention layer of the decoder.

[0060] S330. In the cross-attention layer of the decoder, a query matrix in the cross-attention layer is determined according to the target key point query features, the feature map grid position encoding features, the feature map hierarchical encoding features and the key point type text encoding features; wherein the target key point query features, the feature map grid position encoding features and the feature map hierarchical encoding features reflect the information that needs to be queried and obtained for the feature map of the sample image.

[0061] Exemplarily, similarly to the self-attention layer, in order to add prior knowledge so that the model can learn the common characteristics between target key points of the same type more quickly and accurately, the query matrix in the cross-attention layer can be determined according to the target key point query features, feature map grid position encoding features, feature map hierarchical encoding features, and the key point type text encoding features. Specifically, the target key point query features, feature map grid position encoding features, feature map hierarchical encoding features, and the key point type text encoding features can be connected to generate an input matrix, and the input matrix is ​​multiplied by the weight matrix to obtain a query matrix. The same is true for Q in the above embodiment.

[0062] S340, obtaining reference point position query features for reference points in the feature map through feedforward neural network and normalization processing; wherein the feature map is obtained by extracting features from the sample image; and the reference point is selected from other target key points around each target key point in the feature map.

[0063] Exemplarily, in the process of calculating cross-attention, in order to further reduce unnecessary calculations, the dimension of the cross-attention calculation can be adaptively reduced. Normally, for each target key point, the similarity with other target key points is calculated, but in an embodiment of the present application, a preset number of reference points with the greatest correlation with the target key point can be selected from other target key points, and the similarity is calculated only based on the reference point and the target key point. Therefore, it is necessary to obtain the reference point position query feature only for the reference point through a feedforward neural network and normalization processing, thereby reducing the number of target key points that need to be calculated, reducing the matrix dimension, and effectively reducing the amount of calculation.

[0064] S350. Determine a key matrix of a cross attention layer according to the reference point position query feature and the key point type text encoding feature.

[0065] Exemplarily, the reference point location query feature is connected with the key point type text encoding feature to determine the key matrix of the cross attention layer. Specifically, the input matrix of the key matrix can be , Query the feature for the reference point location. Multiply the input matrix by the weight matrix , and get the key matrix K of the cross attention layer.

[0066] S360. Determine a value matrix according to an output matrix of an encoder in the transformation neural network.

[0067] Exemplarily, the output matrix of the encoder in the transformation neural network is multiplied by the weight matrix to obtain the value matrix V.

[0068] S370. Calculate a cross attention matrix based on the query matrix, the key matrix and the value matrix.

[0069] based on Compute the criss-cross attention matrix. To calculate the cross attention matrix, Indicates normalization. is the dimension of the key matrix.

[0070] S380, obtaining a target key point prediction result for the sample image, adjusting and optimizing the transformation neural network according to the target key point prediction result and the annotation information of the sample image, and obtaining a target key point detection model, so as to perform target key point detection on the image to be detected based on the target key point detection model.

[0071] An embodiment of the present application provides a method for detecting target key points. In the cross-attention layer of the decoder, a query matrix in the cross-attention layer is determined according to target key point query features, feature map grid position encoding features, feature map hierarchical encoding features, and the key point type text encoding features; wherein the target key point query features, feature map grid position encoding features, and feature map hierarchical encoding features reflect the information that needs to be queried and obtained for the feature map of the sample image; a reference point position query feature is obtained by a feedforward neural network and normalization processing for the reference point in the feature map; wherein the feature map is obtained by feature extraction of the sample image; a reference point is selected from other target key points around the target key point in the feature map for each target key point; a key matrix of the cross-attention layer is determined according to the reference point position query features and the key point type text encoding features; a value matrix is ​​determined according to the output matrix of the encoder in the transform neural network; and a cross-attention matrix is ​​calculated according to the query matrix, the key matrix, and the value matrix. The above scheme introduces key point type text encoding features in the process of generating the key matrix, and jointly queries the detection box of the target and the position of the target key points with the reference point position feature query, which is equivalent to introducing prior knowledge in the target key point prediction task, effectively improving the convergence speed of the model.

[0072] Figure 4 This is a flowchart of a target key point detection method provided in another embodiment of the present application. The present embodiment is optimized based on the above embodiment. For solutions not described in detail in the present embodiment, please refer to the above embodiment. Figure 4 As shown, the method of the embodiment of the present application specifically includes the following steps:

[0073] S410. Encode the target key point type text to obtain a key point type text encoding feature containing the semantics of the target key point type text; wherein the target key point type text is used to reflect the target key point type in the sample image; the target key point type reflects the position of the target key point in the target.

[0074] S420, in the process of training the transform neural network to predict the target key points based on the sample images, the key point type text encoding features are added to the self-attention layer and the cross-attention layer of the decoder.

[0075] S430. For the target key points predicted by the decoder, determine the offset of each target key point relative to the first target key point through the target key point correction module.

[0076] In the embodiment of the present application, after the decoder of the transformation neural network, a target key point correction module is also included. Exemplarily, the model structure is as follows Figure 5As shown. Among them, backbone is the backbone network, which is used to extract features from sample images to obtain feature maps. Image Encode encoder is used to enhance features, and Decoder decoder is used to predict target detection box, detection box type, target key points and target key point type. Box is the output detection box, Class is the output detection box type, and Kpt contains the output target key points and target key point types. ReRank Decoder is the target key point correction module. "1.nose 2.head 3.left hand 4.right hand" is the target key point type text, Text Encoder is the text encoder, the obtained Text embed is the key point type text encoding feature, and query is the key point type text encoding feature used as the query feature. Contrastive loss is the loss function, and Kpt class embed is the target key point type encoding feature. dist score weight is the weight.

[0077] like Figure 5 As shown, the target key point Kpt predicted by the decoder first undergoes coordinate transformation, and the coordinates of each target key point are transformed into relative coordinates relative to the first target key point. After the coordinate transformation, the coordinates are input into the target key point correction module to determine the offset of each target key point relative to the first target key point, that is, the relative coordinates relative to the first target key point. The relative coordinates of each target key point relative to the first target key point output by the target key point correction module may be different from the relative coordinates of each target key point relative to the first target key point input to the target key point correction module. When the relative coordinates of each target key point relative to the first target key point input to the target key point correction module are inaccurate, the target key point correction module performs correction and offset, thereby outputting relatively accurate relative coordinates of each target key point relative to the first target key point.

[0078] S440, determining the corrected coordinates of each target key point according to the offset of each target key point relative to the first target key point and the coordinates of the first target key point, and outputting the target key point category corresponding to each target key point through the target key point correction module.

[0079] Exemplarily, based on the offset of each target key point relative to the first target key point and the coordinates of the first target key point, the corrected coordinates of each target key point can be determined, and the relative coordinates of each target key point relative to the first target key point can be converted into global absolute coordinates, so that the position of the target key point can be further corrected based on the target key point coordinates predicted by the decoder, thereby improving the accuracy of the target key point position.

[0080] In an embodiment of the present application, the training process of the target key point correction module includes:

[0081] According to the annotation information in the sample image, some of the target key points are subjected to error processing; wherein the error processing includes at least one of random masking, disrupting the order, and adding noise;

[0082] For each target key point, the origin coordinates are used as the transformation coordinates of the first target key point, and the transformation coordinates of other target key points relative to the first target key point are calculated;

[0083] For the transformed coordinates of each target key point, the target key point position coding features are extracted, and according to the target key point position coding features and the target key point type query features, the input matrix of the self-attention layer is determined, and the self-attention matrix is ​​calculated; wherein the target key point type query features are features used to query the type and coordinates of the target key point;

[0084] Determine a query matrix of cross self-attention according to the self-attention matrix and the target key point position encoding feature, determine a key matrix according to the target key point type query feature, determine a value matrix according to the self-attention matrix, and calculate a cross self-attention matrix;

[0085] The cross self-attention matrix is ​​processed by a feedforward neural network to predict the target key point category and the offset of each target key point coordinate relative to the first target key point coordinate.

[0086] Exemplarily, during the training process of the target key point correction module, some of the target key points are processed as errors according to the annotation information in the sample image. For example, assuming that the sample image contains 13 target key points, at least one of random masking, scrambling, and adding noise can be performed on 6 target key points to make the 6 target key points offset from the accurate position, or make the corresponding target key point category at this position wrong. The coordinates of the first target key point among the 13 target key points are set as the origin coordinates as the transformation coordinates of the first target key point, and the transformation coordinates of other target key points relative to the first target key point are calculated. According to the transformation coordinates of each target key point, the position coding features of the target key point are extracted, and according to the position coding features of the target key point and the target key point type query features, the input matrix of the self-attention layer is determined, and the self-attention matrix is ​​calculated. That is, the position coding features of the target key points and the target key point type query features are connected, and the connected matrix is ​​multiplied by the weight matrix to obtain the query matrix, key matrix and value matrix of the self-attention layer. Based on Calculate the self-attention matrix. In the cross-attention layer, connect the self-attention matrix output by the self-attention layer with the target key point position encoding feature, multiply the connected matrix by the weight matrix to determine the cross-self-attention query matrix, multiply the target key point type query feature by the weight matrix to determine the key matrix, and multiply the self-attention matrix by the weight matrix to determine the value matrix. Calculate the cross self-attention matrix. The cross self-attention matrix is ​​processed by a feedforward neural network to predict the target key point category and the offset of each target key point coordinate relative to the first target key point coordinate.

[0087] S450, adjusting and optimizing the transformation neural network according to the target key point prediction result and the annotation information of the sample image to obtain a target key point detection model, so as to perform target key point detection on the image to be detected based on the target key point detection model.

[0088] In an embodiment of the present application, the method further includes:

[0089] Determine target key point type features of the predicted target key point type for the predicted target key point corrected by the target key point correction module;

[0090] For a real target key point corresponding to the predicted target key point, determining a target key point type text feature of the real target key point;

[0091] A loss function is determined according to the target key point type feature and the target key point type text feature, so as to optimize the transformation neural network and the target key point correction module based on the loss function.

[0092] Exemplarily, in the process of calculating the loss function, the predicted target key point type and the real key point type text can be mapped to a similar space, thereby constraining the optimization direction of the target key point type. Specifically, the predicted target key point obtained after the target key point correction module can be obtained, and the predicted key point type can be encoded to obtain the target key point type feature. Determine the real target key point corresponding to the target key point, and obtain the target key point type text feature obtained by encoding the target key point type text corresponding to the real target key point. Determine the loss function based on the target key point type feature and the target key point type text feature. , based on the loss function, the transformation neural network and the target key point correction module are optimized. The beneficial effect of the above scheme is that, since the real key point type text contains semantic information, it can more accurately reflect the target key point type. Compared with the conventional encoding (such as 0, 1) representation of the target key point type, it can better reflect the type characteristics of the target key point, thereby making it clearer to optimize the defense line and improving the optimization accuracy and optimization speed.

[0093] In an embodiment of the present application, determining a loss function according to the target key point type feature and the target key point type text feature includes:

[0094] For the predicted target key points corrected by the target key point correction module, the distance between the predicted target key points and the real target key points is calculated;

[0095] Normalizing the distance to determine a weight value corresponding to the target key point type of the target key point;

[0096] A loss function is determined according to the target key point type features of the predicted target key point, the target key point type text features of the actual target key point, and the weight value corresponding to the target key point type.

[0097] Exemplarily, in the case of constraining the optimization direction of the target key point type, taking into account the regression of the target key point, the distance factor between the predicted target key point and the real target key is added to the loss function. Specifically, the distance between the predicted target key point and the real target key point is calculated, and the distance is normalized to determine the weight value corresponding to the target key point type. The greater the distance between the predicted target key point and the real target key point, the higher the proportion of participating in the optimization, and the greater the corresponding weight of participating in the loss function calculation. Exemplarily, the distance between the predicted target key point and the real target key point is expressed as , n is the target key point type, correspond( ) is the predicted coordinate of the key point of the n-th target, correspond( ) is the true coordinate of the key point of the n-th target. Weight , Norm means standardization. The loss function after adding weights is The beneficial effect of the above scheme is that the difference of the target key point coordinates can be added for optimization, which can not only achieve the optimization of the category, but also the optimization of the target key point coordinates. For the same position, even if the target key point category optimization is completed, if the target key point coordinates are very different, the optimization degree of the position coordinates can be increased by a larger weight, thereby improving the optimization effect, and then improving the accuracy of target key point detection.

[0098] The embodiment of the present application provides a method for detecting target key points. For the target key points predicted by the decoder, the offset of each target key point relative to the first target key point is determined by the target key point correction module; the corrected coordinates of each target key point are determined according to the offset of each target key point relative to the first target key point and the coordinates of the first target key point, and the target key point correction module outputs the target key point category corresponding to each target key point. By introducing the target key point correction module, it is possible to correct the situation where there is a deviation in the target key point coordinates output by the decoder, or the target key point type does not correspond to the coordinates, thereby further improving the accuracy of target key point detection.

[0099] Figure 6 This is a schematic diagram of the structure of a target key point detection device provided in an embodiment of the present application. The device can execute the target key point detection method provided in any embodiment of the present application and has the corresponding functional modules and beneficial effects of the execution method. Figure 6 As shown, the device comprises:

[0100] The encoding module 510 is used to encode the target key point type text to obtain the key point type text encoding features containing the semantics of the target key point type text; wherein the target key point type text is used to reflect the target key point type in the sample image; the target key point type reflects the position of the target key point in the target;

[0101] Prediction module 520, used for adding the key point type text encoding features in the self-attention layer and the cross-attention layer of the decoder during the process of training the transform neural network to predict the target key point based on the sample image, to obtain the target key point prediction result of the sample image;

[0102] The optimization module 530 is used to adjust and optimize the transformation neural network according to the target key point prediction result and the annotation information of the sample image to obtain a target key point detection model, so as to perform target key point detection on the image to be detected based on the target key point detection model.

[0103] In the embodiment of the present application, the prediction module 520 adds the target key point type text encoding in the self-attention layer of the decoder, including:

[0104] In the self-attention layer of the decoder, a query matrix and a key matrix in the self-attention layer are determined according to the target key point query feature, the feature map grid position encoding feature, the feature map hierarchical encoding feature, and the key point type text encoding feature; wherein the target key point query feature, the feature map grid position encoding feature, and the feature map hierarchical encoding feature reflect the information that needs to be queried and obtained for the feature map of the sample image;

[0105] Determine the value matrix based on the target key point query feature;

[0106] A self-attention matrix is ​​calculated based on the query matrix, the key matrix, and the value matrix.

[0107] In the embodiment of the present application, the prediction module 520 adds the target key point type text encoding in the cross attention layer of the decoder, including:

[0108] In the cross attention layer of the decoder, a query matrix in the cross attention layer is determined according to the target key point query feature, the feature map grid position encoding feature, the feature map hierarchical encoding feature and the key point type text encoding feature; wherein the target key point query feature, the feature map grid position encoding feature and the feature map hierarchical encoding feature reflect the information that needs to be queried and obtained for the feature map of the sample image;

[0109] The reference point position query feature is obtained by subjecting the reference point in the feature map to feedforward neural network and normalization processing; wherein the feature map is obtained by extracting features from the sample image; the reference point is selected from other target key points around each target key point in the feature map;

[0110] Determine a key matrix of a cross attention layer according to the reference point position query feature and the key point type text encoding feature;

[0111] Determining a value matrix based on an output matrix of an encoder in the transform neural network;

[0112] A cross attention matrix is ​​calculated based on the query matrix, the key matrix, and the value matrix.

[0113] In the embodiment of the present application, after the decoder of the transformation neural network, a target key point correction module is also included; the device also includes:

[0114] An offset determination module, for determining, for the target key points predicted by the decoder, an offset of each target key point relative to the first target key point through the target key point correction module;

[0115] The correction module is used to determine the correction coordinates of each target key point according to the offset of each target key point relative to the first target key point and the coordinates of the first target key point, and output the target key point category corresponding to each target key point through the target key point correction module.

[0116] In the embodiment of the present application, the device further includes:

[0117] An error processing module, used for performing error processing on some target key points in the target key points according to the annotation information in the sample image; wherein the error processing includes at least one of random masking, disrupting the order, and adding noise;

[0118] A conversion module, for each target key point, using the origin coordinates as the conversion coordinates of the first target key point, and calculating the conversion coordinates of other target key points relative to the first target key point;

[0119] A self-attention calculation module is used to extract the target key point position coding features for the transformed coordinates of each target key point, determine the input matrix of the self-attention layer according to the target key point position coding features and the target key point type query features, and calculate the self-attention matrix; wherein the target key point type query features are features used to query the type and coordinates of the target key point;

[0120] A cross-attention calculation module is used to determine a query matrix of cross-self-attention according to the self-attention matrix and the target key point position encoding feature, determine a key matrix according to the target key point type query feature, determine a value matrix according to the self-attention matrix, and calculate the cross-self-attention matrix;

[0121] The offset prediction module is used to process the cross self-attention matrix through a feedforward neural network to predict the target key point category and the offset of each target key point coordinate relative to the first target key point coordinate.

[0122] In the embodiment of the present application, the device further includes:

[0123] A target key point type feature determination module, used to determine the target key point type feature of the predicted target key point type according to the predicted target key point corrected by the target key point correction module;

[0124] A target key point type text feature determination module is used to determine the target key point type text feature of the real target key point corresponding to the predicted target key point;

[0125] The target key point optimization module is used to determine the loss function according to the target key point type characteristics and the target key point type text characteristics, so as to optimize the transformation neural network and the target key point correction module based on the loss function.

[0126] In an embodiment of the present application, the target key point optimization module determines a loss function according to the target key point type feature and the target key point type text feature, including:

[0127] For the predicted target key points corrected by the target key point correction module, the distance between the predicted target key points and the real target key points is calculated;

[0128] Normalizing the distance to determine a weight value corresponding to the target key point type of the target key point;

[0129] A loss function is determined according to the target key point type features of the predicted target key point, the target key point type text features of the actual target key point, and the weight value corresponding to the target key point type.

[0130] A target key point detection device provided in an embodiment of the present application can execute a target key point detection method provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method.

[0131] Figure 7 A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0132] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0133] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless target key point detection transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0134] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a target key point detection method.

[0135] In some embodiments, the target key point detection method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the target key point detection method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the target key point detection method in any other appropriate manner (for example, by means of firmware).

[0136] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0137] The computer program for implementing the method of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable target key point detection device, so that the computer program, when executed by the processor, enables the functions / operations specified in the flow chart and / or the block diagram to be implemented. The computer program can be executed completely on the machine, partially on the machine, as an independent software package, partially on the machine and partially on a remote machine, or completely on a remote machine or server.

[0138] In the context of the present application, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0139] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0140] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0141] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0142] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the information expected by the technical solution of this application can be achieved, and this document is not limited here.

[0143] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A target key point detection method, characterized in that: The method comprises: Encode the target key point type text to obtain a key point type text encoding feature containing the semantics of the target key point type text; wherein the target key point type text is used to reflect the target key point type in the sample image; the target key point type reflects the position of the target key point in the target; In the process of training the transform neural network to predict the target key points based on the sample image, the key point type text encoding features are added to the self-attention layer and the cross-attention layer of the decoder to obtain the target key point prediction result of the sample image; Adjusting and optimizing the transformation neural network according to the target key point prediction result and the annotation information of the sample image to obtain a target key point detection model, so as to perform target key point detection on the image to be detected based on the target key point detection model; Adding the target key point type text encoding to the self-attention layer of the decoder includes: In the self-attention layer of the decoder, a query matrix and a key matrix in the self-attention layer are determined according to the target key point query feature, the feature map grid position encoding feature, the feature map hierarchical encoding feature, and the key point type text encoding feature; wherein the target key point query feature, the feature map grid position encoding feature, and the feature map hierarchical encoding feature reflect the information that needs to be queried and obtained for the feature map of the sample image; Determine the value matrix based on the target key point query feature; A self-attention matrix is ​​calculated based on the query matrix, the key matrix, and the value matrix.

2. The method according to claim 1, characterized in that Add the target key point type text encoding in the cross attention layer of the decoder, including: In the cross attention layer of the decoder, a query matrix in the cross attention layer is determined according to the target key point query feature, the feature map grid position encoding feature, the feature map hierarchical encoding feature and the key point type text encoding feature; wherein the target key point query feature, the feature map grid position encoding feature and the feature map hierarchical encoding feature reflect the information that needs to be queried and obtained for the feature map of the sample image; The reference point position query feature is obtained by subjecting the reference point in the feature map to feedforward neural network and normalization processing; wherein the feature map is obtained by extracting features from the sample image; the reference point is selected from other target key points around each target key point in the feature map; Determine a key matrix of a cross attention layer according to the reference point position query feature and the key point type text encoding feature; Determining a value matrix based on an output matrix of an encoder in the transform neural network; A cross attention matrix is ​​calculated based on the query matrix, the key matrix, and the value matrix.

3. The method according to claim 1, characterized in that After the decoder of the transformation neural network, a target key point correction module is also included; the method also includes: For the target key points predicted by the decoder, determining the offset of each target key point relative to the first target key point by the target key point correction module; According to the offset of each target key point relative to the first target key point and the coordinates of the first target key point, the corrected coordinates of each target key point are determined, and the target key point category corresponding to each target key point is outputted through the target key point correction module.

4. The method according to claim 3, characterized in that The training process of the target key point correction module includes: According to the annotation information in the sample image, some of the target key points are subjected to error processing; wherein the error processing includes at least one of random masking, disrupting the order, and adding noise; For each target key point, the origin coordinates are used as the transformation coordinates of the first target key point, and the transformation coordinates of other target key points relative to the first target key point are calculated; For the transformed coordinates of each target key point, the target key point position coding features are extracted, and according to the target key point position coding features and the target key point type query features, the input matrix of the self-attention layer is determined, and the self-attention matrix is ​​calculated; wherein the target key point type query features are features used to query the type and coordinates of the target key point; Determine a query matrix of cross self-attention according to the self-attention matrix and the target key point position encoding feature, determine a key matrix according to the target key point type query feature, determine a value matrix according to the self-attention matrix, and calculate a cross self-attention matrix; The cross self-attention matrix is ​​processed by a feedforward neural network to predict the target key point category and the offset of each target key point coordinate relative to the first target key point coordinate.

5. The method according to claim 3, characterized in that: The method further comprises: Determine target key point type features of the predicted target key point type for the predicted target key point corrected by the target key point correction module; For a real target key point corresponding to the predicted target key point, determining a target key point type text feature of the real target key point; A loss function is determined according to the target key point type feature and the target key point type text feature, so as to optimize the transformation neural network and the target key point correction module based on the loss function.

6. The method according to claim 5, characterized in that Determining a loss function according to the target key point type feature and the target key point type text feature includes: For the predicted target key points corrected by the target key point correction module, the distance between the predicted target key points and the real target key points is calculated; Normalizing the distance to determine a weight value corresponding to the target key point type of the target key point; A loss function is determined according to the target key point type features of the predicted target key point, the target key point type text features of the actual target key point, and the weight value corresponding to the target key point type.

7. A target key point detection device, characterized in that: The device comprises: An encoding module is used to encode the target key point type text to obtain a key point type text encoding feature containing the semantics of the target key point type text; wherein the target key point type text is used to reflect the target key point type in the sample image; the target key point type reflects the position of the target key point in the target; A prediction module, used for adding the key point type text encoding features in the self-attention layer and the cross-attention layer of the decoder during the process of training the transform neural network to predict the target key point based on the sample image, to obtain the target key point prediction result of the sample image; An optimization module, used to adjust and optimize the transformation neural network according to the target key point prediction result and the annotation information of the sample image, to obtain a target key point detection model, so as to perform target key point detection on the image to be detected based on the target key point detection model; The prediction module adds the target key point type text encoding in the self-attention layer of the decoder, including: In the self-attention layer of the decoder, a query matrix and a key matrix in the self-attention layer are determined according to the target key point query feature, the feature map grid position encoding feature, the feature map hierarchical encoding feature, and the key point type text encoding feature; wherein the target key point query feature, the feature map grid position encoding feature, and the feature map hierarchical encoding feature reflect the information that needs to be queried and obtained for the feature map of the sample image; Query the features based on the target key points to determine the value matrix; A self-attention matrix is ​​calculated based on the query matrix, the key matrix, and the value matrix.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the target key point detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the target key point detection method according to any one of claims 1 to 6 when executed.

Images