Class-independent attitude estimation method based on learning transferable key points

Abstract

The invention provides a category-independent attitude estimation method based on learning migratable key points, and the method comprises the steps: carrying out the data enhancement of an obtained basic category class image, and obtaining a data-enhanced image; constructing attitude improvement layers, and forming an attitude improvement framework by a plurality of attitude improvement layers; inputting the image after data enhancement into a backbone network to obtain a multi-scale feature map; based on the key point embedding vector, the key point coordinates and the multi-scale feature map, decoding through a structure sensing decoder to obtain improved key point coordinates and improved key point visibility; based on the improved key point embedding vector, obtaining an improved key point link existence degree through a position sensing decoder, and further obtaining a prediction attitude; and inputting the image into the trained model to obtain a prediction result. The invention provides a posture improvement framework based on coordination matching supervision, and the framework can jointly learn transferable key points and links to estimate the posture of a category-agnostic object.

Description

Technical Field

[0001] The invention relates to the technical field of machine computer vision and attitude estimation, and in particular to a category-independent attitude estimation method based on learning transferable key points. Background Art

[0002] Pose estimation is a fundamental computer vision task with widespread real-world applications, which aims to extract the object structure of a target, such as keypoints, links (skeleton). Although existing methods have achieved great success, most of them are trained using data from specific categories, and thus fail to estimate the pose of objects that are independent of the training category, especially when the objects have different structures. However, humans have an inherent ability to perceive the basic geometric information of category-agnostic objects, including position, area, and structure. In view of this, object detection and segmentation of category-agnostic objects have attracted great attention and have been well studied, such as bounding box and mask proposal learning. However, the exploration of category-agnostic pose estimation (CAPE) is relatively insufficient.

[0003] Recently, some works have introduced the task of CAPE, but it actually belongs to few-shot keypoint detection, which predicts the target keypoints based on a small number of annotated images of the same category. In such a CAPE setting, the category of each test image must be known, and the correct few samples are used as reference to detect keypoints, which is inconsistent with the category-agnostic intention. In addition, such CAPE can make predictions without support and reference in the task of detecting corresponding parts and segmentation tasks. The first stage of MetaPoint can predict category-agnostic keypoints without support, but how to predict the links of keypoints remains an open problem. Although existing CAPE methods have achieved great success, they are far from completing the real CAPE task (RealCAPE). Summary of the invention

[0004] In view of the above situation, the main purpose of the present invention is to propose a category-independent posture estimation method based on learning transferable key points to solve the above technical problems.

[0005] The present invention proposes a class-independent posture estimation method based on learning transferable key points, the method comprising the following steps: Step 1: Perform data enhancement on the acquired basic category image to obtain a data enhanced image; Step 2: construct a posture improvement layer with a structure-aware decoder and a position-aware decoder as the main body, and then form a posture improvement framework composed of multiple posture improvement layers; Step 3: Input the data-enhanced image into the backbone network to obtain a multi-scale feature map; Update the key point embedding vector through the key point embedding vector, the key point coordinates and the multi-scale feature map to obtain an updated key point embedding vector; Using key point links to improve the updated key point embedding vector, to obtain an improved key point embedding vector; The improved key point embedding vector is then decoded by a structure-aware decoder to obtain improved key point coordinates and improved key point visibility respectively; Step 4: Add each pair of improved key point embedding vectors by enumeration to obtain the preliminary link embedding vector; Based on the preliminary link embedding vector, the completed link embedding vector is obtained through the deformable attention mechanism; The completed link embedding vector is then decoded into the improved key point link existence through the position-aware decoder; The predicted pose is obtained by combining the improved key point coordinates, the improved key point visibility and the improved key point link existence; Step 5: Use the predicted pose and the true label pose to update the weight of the model through back propagation training to obtain an updated model; Input the base category image into the trained model to get the prediction result.

[0006] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention introduces a truly class-agnostic pose estimation task (RealCAPE). At the same time, the model proposed in the present invention can estimate the key points and key point links of any object with structural adaptability and semantic consistency; 2. This paper proposes a posture improvement framework based on coordinated matching supervision, which can jointly learn transferable key points and links to estimate the posture of category-agnostic objects; 3. The present invention uses the query embedding vector as a learning parameter to uniformly characterize various features, learn category-general information, and estimate general postures for any category. It can not only directly unify the posture estimation of labeled categories, but also predict a wide range of unlabeled categories based on this. 4. The present invention is based on the learned general postures that meet actual needs, mines the rich information in sufficient unlabeled images, and uses this to structurally refine the general postures, thereby obtaining accurate postures of a wide range of categories. It can not only uniformly learn new categories that are widely unlabeled, but also help to keep the predicted postures in line with actual needs, which is more economical and improves the practicality of posture estimation.

[0007] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A flow chart of the class-independent posture estimation method based on learning transferable key points proposed by the present invention; Figure 2 This is an overall framework diagram of the training phase of the category-independent pose estimation method based on learning transferable key points proposed in the present invention. DETAILED DESCRIPTION

[0009] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0010] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0011] See also Figure 1 , an embodiment of the present invention proposes a class-independent posture estimation method based on learning transferable key points, the method comprising the following steps: Step 1: Perform data enhancement on the acquired basic category image to obtain a data enhanced image; Specifically, the specific steps for data enhancement of basic category images are as follows: Scale the base category images to a uniform size of 256 256; Normalize the scaled image.

[0012] Furthermore, during initialization, the initial pose is initialized with a constant of 0.5 and the key point embedding vectors are randomly initialized.

[0013] Step 2: construct a posture improvement layer with a structure-aware decoder and a position-aware decoder as the main body, and then form a posture improvement framework composed of multiple posture improvement layers; Specifically, in this step, a posture improvement layer is constructed with the structure-aware decoder and the position-aware decoder as the main body. The framework consists of multiple improvement layers. The improvement layer will continue to improve the learnable posture and key point embedding vector improved by the previous layer as input until the final improved posture is obtained.

[0014] Step 3: Input the data-enhanced image into the backbone network to obtain a multi-scale feature map; Update the key point embedding vector through the key point embedding vector, the key point coordinates and the multi-scale feature map to obtain an updated key point embedding vector; Using key point links to improve the updated key point embedding vector, to obtain an improved key point embedding vector; The improved key point embedding vector is then decoded by a structure-aware decoder to obtain improved key point coordinates and improved key point visibility respectively; In step 3, the key point embedding vector is updated by the key point embedding vector, the key point coordinates and the multi-scale feature map to obtain the updated key point embedding vector. The relationship between the corresponding process is: ; in, represents the updated key point embedding vector, represents the deformable attention mechanism, Indicates The key point embedding vector of the layer, Indicates The key point coordinates of the layer, represents a multi-scale feature map, represents the total number of layers of multi-scale feature maps, Indicates the number of layers of the feature map; The updated key point embedding vector is improved by using the key point link to obtain the improved key point embedding vector. The relationship between the corresponding process is: ; in, represents the improved key point embedding vector, Indicates key points The updated keypoint embedding vector, and Both represent the index of key points. represents the learnable weight of the corresponding attention head, represents the operation of normalizing the sum of attention weights, represents the initial attention weight, represents the index of the attention head, Indicated in Key points in layer and key points The link strength between represents the learnable weight of the corresponding attention head after being processed by the deformable attention mechanism, Represents the key points after the initial update The corresponding embedding vector; The improved key point embedding vector is decoded by the structure-aware decoder to obtain the improved key point coordinates and the improved key point visibility respectively. The corresponding process can be expressed by the following relationship: ; in, represents the key point coordinates, represents the key point visibility, Indicates The keypoint coordinates and keypoint visibility of the layer, represents the sigmoid function, represents the inverse sigmoid function, Indicates The keypoint coordinates and keypoint visibility of the layer, Indicates that it has been processed by a lightweight prediction head.

[0015] Step 4: Add each pair of improved key point embedding vectors by enumeration to obtain the preliminary link embedding vector; Based on the preliminary link embedding vector, the completed link embedding vector is obtained through the deformable attention mechanism; The completed link embedding vector is then decoded into the improved key point link existence through the position-aware decoder; The predicted pose is obtained by combining the improved key point coordinates, the improved key point visibility and the improved key point link existence; In step 4, each pair of improved key point embedding vectors is enumerated and added to obtain the preliminary link embedding vector. The relationship between the corresponding process is: ; in, Indicates key points and key points The preliminary link embedding vector between Indicates key points The improved key point embedding vector, Indicates key points The improved key point embedding vector of Based on the preliminary link embedding vector, the completed link embedding vector is obtained through the deformable attention mechanism. The relationship between the corresponding process is: ; in, represents the completed link embedding vector, represents the preliminary link embedding vector, Indicates the midpoint; The completed link embedding vector is decoded into the improved key point link existence through the position-aware decoder. The corresponding process can be expressed by the following relationship: ; in, Indicates The key point link existence of the layer, represents the visibility of key point pairs, Indicates The key point link existence of the layer, Indicates that it has been processed by a unary existence classifier.

[0016] Step 5: Use the predicted pose and the true label pose to update the weight of the model through back propagation training to obtain an updated model; Input the base category image into the trained model to get the prediction result; See also Figure 2 In step 5, the predicted pose and the true label pose are used to update the model weights through back propagation training to obtain the updated model. During the training process, the collaborative matching cost is calculated based on the predicted pose and the true label pose. The relationship between the corresponding process is: ; in, Indicates prediction key points to The co-matching cost of the real label key points is represents the total number of base category images in each group, The total number of layers indicating that the pose is too tight. represents the absolute value error loss function, represents the second-hospital cross entropy loss function, Indicates The base category image is passed through The first The coordinates of the predicted key points, Indicates The first image in the base category The coordinates of the real label key points, represents the hyperparameter used to balance the keypoint regression loss, Indicates The base category image is passed through The first The link existence of the predicted key points, Indicates The first image in the base category The link existence of the real label key points; The predicted posture and the true label posture are used to update the weight of the model through back propagation training to obtain the updated model. During the training process, the parameters of the model are adjusted through back propagation. The loss function expression in the corresponding process is: ; in, represents the total loss function, represents the total number of true label key points, represents the operation of solving the optimal collaborative matching, and Both represent the key point index after optimal collaborative matching. Expressing the The predicted key point coordinates obtained after the optimal collaborative matching of the real label key points, Indicates The key point coordinates of the true labels, and Both represent weight coefficients used to balance the key point regression loss. Expressing the The visibility of the predicted key points obtained after the optimal collaborative matching of the real label key points, Indicates The visibility of the key points of the true labels, represents the existence of predicted key point links after optimal collaborative matching, represents the key point index after optimal collaborative matching, Represents the index of the key point of the true label; Input the basic category image into the trained model to obtain the prediction result. The relationship between the corresponding process is: ; in, represents the matching cost matrix associated with the evaluation metric, Indicates that it has been processed by the indicator function. Indicates that based on The first The key point coordinates, represents the distance threshold, Indicates that based on The first The existence of key point links, Indicates the score threshold for calculating the average precision; Furthermore, in this step, the co-matching cost between the predicted pose and the true label pose is calculated to assist the training of the model, wherein the predicted key points correspond one-to-one to the true label key points; Specifically, in this step, for each small batch containing multiple groups, and each group containing the same type of basic category images, the optimal key point co-matching cost of matching the predicted key points to the true labels is first calculated; Then, for each base category image in the same group, the same matching relationship is used to calculate the difference between the predicted key point and the true key point label, the predicted visibility, and the cross entropy between the key point link and the true label to return the loss; Finally, back propagation is performed to make the predicted posture distribution of the sample as close to the true distribution as possible.

[0017] Furthermore, in this step, after the back-propagation training is completed, the network parameters are fixed, and the new category image is input into the model framework, and the output result of the last layer of the framework improvement layer is used as the predicted posture; In the experimental setting of RealCAPE, each test image comes from a new category of images and is evaluated using the last predicted pose. This method has wide compatibility because any pose estimation can be filled with visibility in the form of confidence or constraints. The average precision (AP) is selected as the basic evaluation metric because it can balance precision and recall well. Considering that the pose consists of key points and links, the method uses the average key point AP and link AP to obtain the pose AP. The main evaluation metric of the invention is the pose mean average precision (mAP), which is the average of the pose AP of all categories; Evaluation indicators need to focus on the structural adaptability and semantic consistency of the predicted pose, because RealCAPE's model has never been exposed to the real annotations of new categories, so the predicted pose must be matched with the real pose before the evaluation indicators are calculated. Matching-based evaluation methods are very suitable in this case, because predictions that are not matched with real annotations will reduce the recall rate and thus the mean average precision. At the same time, the matching correspondence between all test images of each category and the real label is calculated, which means that the estimated pose should be consistent and well matched in different images.

[0018] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0019] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0020] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A class-independent pose estimation method based on learning transferable key points, characterized in that The method comprises the following steps: Step 1: Perform data enhancement on the acquired basic category image to obtain a data enhanced image; Step 2: construct a posture improvement layer with a structure-aware decoder and a position-aware decoder as the main body, and then form a posture improvement framework composed of multiple posture improvement layers; Step 3: Input the data-enhanced image into the backbone network to obtain a multi-scale feature map; Update the key point embedding vector through the key point embedding vector, the key point coordinates and the multi-scale feature map to obtain an updated key point embedding vector; Using key point links to improve the updated key point embedding vector, to obtain an improved key point embedding vector; The improved key point embedding vector is then decoded by a structure-aware decoder to obtain improved key point coordinates and improved key point visibility respectively; Step 4: Add each pair of improved key point embedding vectors by enumeration to obtain the preliminary link embedding vector; Based on the preliminary link embedding vector, the completed link embedding vector is obtained through the deformable attention mechanism; The completed link embedding vector is then decoded into the improved key point link existence through the position-aware decoder; The predicted pose is obtained by combining the improved key point coordinates, the improved key point visibility and the improved key point link existence; Step 5: Use the predicted pose and the true label pose to update the weight of the model through back propagation training to obtain an updated model; Input the base category image into the trained model to get the prediction result.

2. The method for class-independent posture estimation based on learning transferable key points according to claim 1, characterized in that: In step 3, the key point embedding vector is updated by the key point embedding vector, the key point coordinates and the multi-scale feature map to obtain the updated key point embedding vector. The relationship between the corresponding process is: ； in, represents the updated key point embedding vector, represents the deformable attention mechanism, Indicates The key point embedding vector of the layer, Indicates The key point coordinates of the layer, represents a multi-scale feature map, represents the total number of layers of multi-scale feature maps, Indicates the number of layers of the feature map.

3. The method for class-independent posture estimation based on learning transferable key points according to claim 2, characterized in that: In step 3, the updated key point embedding vector is improved by using the key point link to obtain the improved key point embedding vector. The relationship between the corresponding process is: ； in, represents the improved key point embedding vector, Indicates key points The updated keypoint embedding vector, and Both represent the index of key points. represents the learnable weight of the corresponding attention head, represents the operation of normalizing the sum of attention weights, represents the initial attention weight, represents the index of the attention head, Indicated in Key points in layer and key points The link strength between represents the learnable weight of the corresponding attention head after being processed by the deformable attention mechanism, Represents the key points after the initial update The corresponding embedding vector.

4. The method for class-independent posture estimation based on learning transferable key points according to claim 3, characterized in that: In step 3, the improved key point embedding vector is decoded by a structure-aware decoder to obtain improved key point coordinates and improved key point visibility, respectively. The corresponding process can be expressed by the following relationship: ； in, represents the key point coordinates, represents the key point visibility, Indicates The keypoint coordinates and keypoint visibility of the layer, represents the sigmoid function, represents the inverse sigmoid function, Indicates The keypoint coordinates and keypoint visibility of the layer, Indicates that it has been processed by a lightweight prediction head.

5. The method for class-independent posture estimation based on learning transferable key points according to claim 4, characterized in that: In step 4, each pair of improved key point embedding vectors is enumerated and added to obtain a preliminary link embedding vector. The relationship between the corresponding process is: ； in, Indicates key points and key points The preliminary link embedding vector between Indicates key points The improved key point embedding vector, Indicates key points The improved keypoint embedding vector.

6. The method for class-independent posture estimation based on learning transferable key points according to claim 5, characterized in that: In step 4, based on the preliminary link embedding vector, the completed link embedding vector is obtained through the deformable attention mechanism. The relationship between the corresponding process is: ； in, represents the completed link embedding vector, represents the preliminary link embedding vector, Indicates the midpoint.

7. The method for class-independent posture estimation based on learning transferable key points according to claim 6, characterized in that: In step 4, the completed link embedding vector is decoded into the improved key point link existence by the position-aware decoder. The corresponding process can be expressed by the following relation: ； in, Indicates The key point link existence of the layer, represents the visibility of key point pairs, Indicates The key point link existence of the layer, Indicates that it has been processed by a unary existence classifier.

8. The method for class-independent posture estimation based on learning transferable key points according to claim 7, characterized in that: In step 5, the predicted pose and the true label pose are used to update the weight of the model through back propagation training to obtain an updated model. During the training process, the collaborative matching cost is calculated based on the predicted pose and the true label pose. The relationship between the corresponding process is: ； in, Indicates prediction key points to The co-matching cost of the real label key points is represents the total number of base category images in each group, The total number of layers indicating that the pose is too tight. represents the absolute value error loss function, represents the second-hospital cross entropy loss function, Indicates The base category image is passed through The first The coordinates of the predicted key points, Indicates The first image in the base category The coordinates of the real label key points, represents the hyperparameter used to balance the keypoint regression loss, Indicates The base category image is passed through The first The link existence of the predicted key points, Indicates The first image in the base category The link existence of the real label key points.

9. The method for class-independent posture estimation based on learning transferable key points according to claim 8, characterized in that: In step 5, the predicted posture and the true label posture are used to update the weight of the model through back propagation training to obtain an updated model. During the training process, the parameters of the model are adjusted by back propagation. The loss function expression in the corresponding process is: ； in, represents the total loss function, represents the total number of true label key points, represents the operation of solving the optimal collaborative matching, and Both represent the key point index after optimal collaborative matching. Indicates The predicted key point coordinates obtained after the optimal collaborative matching of the real label key points, Indicates The coordinates of the real label key points, and Both represent weight coefficients used to balance the key point regression loss. Indicates The visibility of the predicted key points obtained after the optimal collaborative matching of the real label key points, Indicates The visibility of the real label key points, represents the existence of predicted key point links after optimal collaborative matching, represents the key point index after optimal collaborative matching, Indicates the index of the ground-truth label keypoint.

10. The method for class-independent posture estimation based on learning transferable key points according to claim 9, characterized in that: In step 5, the basic category image is input into the trained model to obtain the prediction result. The relationship between the corresponding process is: ； in, represents the matching cost matrix associated with the evaluation metric, Indicates that it has been processed by the indicator function. Indicates that based on The first The coordinates of the key points, represents the distance threshold, Indicates that based on The first The link existence of key points, Indicates the score threshold for calculating the average precision.

Images