A class-agnostic pose estimation method and system based on collaborative matching supervision
By constructing a pose estimation model with collaborative matching supervision, the pose of objects of unknown categories can be generated directly from a single image. This solves the problem of existing methods relying on external support information, achieves zero-shot generalization and efficient pose estimation, and improves the application performance in open-world scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JIANGXI UNIVERSITY OF FINANCE AND ECONOMICS
- Filing Date
- 2026-01-23
- Publication Date
- 2026-05-01
AI Technical Summary
Existing pose estimation methods cannot achieve "out-of-the-box" class-independent perception capabilities in open-world scenarios, rely on external support information, are costly, and have unstable performance.
By constructing a pose estimation model based on collaborative matching supervision, and utilizing a feature extraction module, a general prior module, a pose optimization module, and a collaborative matching supervision module, the model learns and solidifies general structural knowledge, directly generating the pose of unknown category objects from a single image. Through multi-round iterative optimization and collaborative matching supervision mechanism, zero-shot generalization is achieved.
It enables direct estimation of the pose of unknown object categories without the need for external support information, reducing deployment costs, improving generalization ability and performance, and featuring a compact and efficient system design with strong knowledge transfer capabilities and robustness.
Smart Images
Figure CN121564449B_ABST
Abstract
Description
A Class-Independent Pose Estimation Method and System Based on Cooperative Matching Supervision Technical Field
[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a class-independent pose estimation method and system based on cooperative matching supervision. Background Technology
[0002] Pose estimation is a core task in computer vision, aiming to locate semantic keypoints and their connections (i.e., skeletons) of predefined objects in an image. Although deep learning-based methods have achieved significant success in specific categories such as humans and vehicles, these models heavily rely on large amounts of labeled data for specific categories, making it difficult to directly generalize to new categories not seen during training. This greatly limits their application in open-world scenarios such as home service robots and general object manipulation.
[0003] To extend to more categories, existing research often employs few-shot or zero-shot learning paradigms. However, these methods still rely on external supporting information during inference (e.g., using a small number of labeled samples or semantic descriptions to estimate the pose of new objects through keypoint matching or feature alignment), and suffer from three inherent drawbacks: First, annotation and computational costs are high, and supporting data must be prepared for each new category; second, performance depends on the correct pre-classification of the test image, and misclassification will lead to pose estimation failure; third, the supporting information and the query image may mismatch in terms of viewpoint, occlusion, or appearance, affecting the reliability of the estimation. These problems make it difficult for existing methods to achieve "out-of-the-box" category-agnostic capabilities in practical deployments.
[0004] Inspired by the human visual system's ability to infer the approximate skeleton of unfamiliar objects, "pose proposal" has been proposed as a more fundamental task—that is, the model should be able to generate possible keypoints and connections for any object in an image directly, without any external support, just like generating bounding boxes or segmentation masks. However, how to construct a pose proposal framework that can learn transferable structural knowledge from known categories and generate semantically consistent and structurally sound pose proposals for unknown categories remains a highly challenging research problem. Summary of the Invention
[0005] To address the technical shortcomings of existing pose estimation methods, such as their reliance on external support information during testing and their difficulty in achieving "out-of-the-box" class-independent perception capabilities, this invention proposes a novel pose estimation paradigm. By training a model capable of learning and solidifying general structural knowledge, it can directly generate complete structured poses for any unknown category object from a single image without any prior category information or example support, significantly improving its practicality and generalization ability in open-world scenarios.
[0006] The first aspect of this invention provides a class-independent pose estimation method based on cooperative matching supervision, comprising:
[0007] S1. Obtain an image dataset containing multiple basic categories and divide the dataset into a training set and a test set; each training image is labeled with the coordinates of real keypoints. Visibility and connection relationship ;
[0008] S2. Construct the attitude estimation model and input it. The training images are of the same category; the pose estimation model includes a feature extraction module, a learnable general prior module, a pose optimization module, and a collaborative matching supervision module enabled during the training phase; wherein, the general prior module stores learnable keypoint embedding vectors and initial pose parameters;
[0009] S3, Feature Extraction Module Feature extraction is performed on each of the training images to output multi-scale image features. ;
[0010] S4, the pose optimization module is based on multi-scale image features of each training image. For a set of learnable keypoint embedding vectors from a general prior module With initial attitude parameters Perform multiple rounds of iterative optimization and output the coordinates, visibility, and connectivity of predicted keypoints for each training image;
[0011] S5. Call the collaborative matching supervision module to... The consistency loss is calculated for training images of the same category, and the pose estimation model parameters are optimized through backpropagation so that the model learns structured knowledge that can be transferred to new categories.
[0012] S6. Input the new category images to be estimated in the test set into the trained pose estimation model, and directly output the pose estimation results of the objects in the new category images through forward propagation; wherein, the pose estimation results include key point coordinates, visibility and connectivity.
[0013] Furthermore, the attitude optimization module consists of It consists of a series of sequentially stacked optimization layers. is an integer greater than 1; where the calculation process of the t-th optimization layer is defined as: In the formula, and Let these represent the output pose and keypoint embedding vector of the (t-1)th layer, respectively. Representing multi-scale image features, and Let represent the output pose and keypoint embedding vector of the t-th layer, respectively.
[0014] Furthermore, each optimization layer in the attitude optimization module sequentially includes a structure-aware keypoint decoder and a position-aware connectivity decoder; the operation process of each optimization layer is as follows:
[0015] Structure-aware keypoint decoding: Receives pose parameters output from the previous optimization layer. and keypoint embedding vector using key point coordinates Using a reference point, a deformable attention mechanism is used to extract multi-scale image features. Extract appearance information; utilize a multi-head attention mechanism to link the connections of the previous optimization layer. Attention weights are introduced as structural biases to calculate the updated keypoint embedding vectors, and the keypoint coordinates of the current layer are decoded. With visibility The previous optimization layer is layer t-1, and the current layer is layer t.
[0016] Position-aware connection decoding: Receives keypoint coordinates output from the structure-aware keypoint decoder. Calculate the geometric midpoint of all keypoint pairs, and using the midpoint as a reference point, apply a deformable attention mechanism to... The system aggregates visual features along the connection path and then predicts and outputs the connection relationships of all key points in the current layer. ;
[0017] The current layer ultimately outputs the refined attitude parameters. and keypoint embedding vector This serves as the input for the next optimization layer or the final pose prediction result.
[0018] Furthermore, the collaborative matching supervision module described in step S5... The process of calculating the consistency loss for training images of the same category includes:
[0019] Select the same category from the training set Zhang training images To form a supervisory group ≥2;
[0020] Calculate the collaborative matching cost matrix of the supervision group , Indicates the number of predicted key points. This represents the number of real keypoints; where the cost matrix is... elements in The calculation formula is In the formula, This represents the cost of matching the nth predicted keypoint to the kth real keypoint. This indicates that the optimization layer t is applied to the i-th training image x. i The predicted coordinates of the nth key point Indicates L1 loss, This represents the binary cross-entropy loss. This represents the k-th real key point. This indicates that the optimization layer t is applied to the i-th training image x. i The visibility of the predicted nth keypoint Indicates the k-th real visibility;
[0021] Based on the cost matrix of cooperative matching The bipartite graph matching algorithm is used to find a globally optimal matching mapping between predicted keypoints and real keypoints for the supervision group. ; wherein, the matching mapping Assign a unique predicted keypoint index to each real keypoint k within the supervision group. Furthermore, the mapping relationship is shared among all images within the group to ensure semantic consistency;
[0022] Based on global matching mapping For each training image in the supervised group, a consistency loss is calculated for matched keypoints, while a classification loss is applied to the visibility and connectivity of unmatched keypoints. The consistency loss is forced to approach zero to avoid generating invalid and redundant predictions; the consistency loss includes key point localization loss, visibility classification loss, and connection relationship classification loss.
[0023] The expression for calculating the matching loss for each training image is as follows: In the formula, This represents the set of indices of matched keypoints in each training image. Denotes the hyperparameters used to balance the visibility classification loss and , Denotes the hyperparameters used to balance the connection classification loss and , This represents the predicted keypoint corresponding to the k-th true keypoint after optimal matching. Indicates the actual number of key points. Indicates a real connection relationship. This represents the predicted connection relationship after matching.
[0024] Furthermore, the learnable keypoint embedding vectors and initial pose parameters in the general prior module are optimized by the collaborative matching supervision module during training to learn cross-class general structural and geometric constraint priors, and are solidified into fixed parameters for inference after training is completed.
[0025] A second aspect of the present invention provides a class-independent pose estimation system based on cooperative matching supervision, comprising:
[0026] Feature extraction module: Used to extract multi-scale features from the input image and output multi-scale image features. ;
[0027] General prior module: stores a set of learnable keypoint embedding vectors and initial pose parameters;
[0028] Pose optimization module: Connects the feature extraction module and the general prior module, used for multi-scale image feature optimization. The learnable keypoint embedding vector and the initial pose parameters are iteratively optimized to output the predicted pose parameters; wherein, the pose optimization module consists of multiple sequentially connected optimization layers, each of which includes a structure-aware keypoint decoder and a position-aware connection decoder.
[0029] Collaborative matching supervision module: Connected to the pose optimization module during the training phase, it groups the prediction results of multiple training images of the same category and solves the globally optimal matching relationship between predicted keypoints and ground truth keypoints. The system parameters are optimized by calculating the consistency loss based on the matching relationship, so that the key point embedding vectors in the general prior module learn structured knowledge that can be transferred to new categories.
[0030] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor as described above in a class-independent pose estimation method based on cooperative matching supervision.
[0031] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the class-independent pose estimation method based on cooperative matching supervision as described above.
[0032] Compared with the prior art, the present invention has the following beneficial effects:
[0033] 1) Achieve true zero-shot generalization. After the model is trained, it can directly estimate the pose of unknown objects without any external support information, which greatly reduces deployment costs and application threshold.
[0034] 2) A new paradigm for efficient posture proposal is proposed, and an end-to-end posture optimization framework is designed to transform the task from "identification and matching" to "direct structure generation". The accurate estimation from coarse to fine is achieved through iterative optimization layers.
[0035] 3) Through an innovative collaborative matching supervision mechanism, the model is forced to learn consistent part representations and geometric relationships across objects from the basic categories, which enables the model to acquire strong knowledge transfer capabilities.
[0036] 4) The model structure is compact and efficient. The general prior module solidifies the learned knowledge into a set of transferable query embeddings, making the reasoning process fast and simple. The system design is both advanced and practical.
[0037] 5) Validation of superior performance: In large-scale benchmark tests, this method achieved performance comparable to or better than state-of-the-art methods that rely on external information without the need for test support, demonstrating its effectiveness and robustness. Attached Figure Description
[0038] Figure 1 is a flowchart of the steps of a class-independent pose estimation method based on cooperative matching supervision provided in an embodiment of the present invention.
[0039] Figure 2 is a detailed flowchart of data processing within a single optimization layer in the attitude estimation model. Detailed Implementation
[0040] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.
[0041] Please refer to Figure 1, which is a flowchart of a class-independent pose estimation method based on cooperative matching supervision provided by an embodiment of the present invention, including:
[0042] S1. Obtain an image dataset containing multiple basic categories and divide the dataset into a training set and a test set; each training image is labeled with the coordinates of real keypoints. Visibility and connection relationship ;
[0043] S2. Construct the attitude estimation model and input it. The training images are of the same category; the pose estimation model includes a feature extraction module, a learnable general prior module, a pose optimization module, and a collaborative matching supervision module enabled during the training phase; wherein, the general prior module stores learnable keypoint embedding vectors and initial pose parameters;
[0044] S3, Feature Extraction Module Feature extraction is performed on each of the training images to output multi-scale image features. ;
[0045] S4, the pose optimization module is based on multi-scale image features of each training image. For a set of learnable keypoint embedding vectors from a general prior module With initial attitude parameters Perform multiple rounds of iterative optimization and output the coordinates, visibility, and connectivity of predicted keypoints for each training image;
[0046] S5. Call the collaborative matching supervision module to... The consistency loss is calculated for training images of the same category, and the pose estimation model parameters are optimized through backpropagation so that the model learns structured knowledge that can be transferred to new categories.
[0047] S6. Input the new category images to be estimated in the test set into the trained pose estimation model, and directly output the pose estimation results of the objects in the new category images through forward propagation; wherein, the pose estimation results include key point coordinates, visibility and connectivity.
[0048] In one specific implementation of this embodiment, the process of acquiring an image dataset containing multiple basic categories and dividing the dataset into a training set and a test set includes:
[0049] This invention employs the largest relevant benchmark dataset currently available (MP-100 dataset) for experimental analysis. The MP-100 dataset covers 100 categories across 8 superclasses, containing approximately 18,000 images and 20,000 labeled instances. The number of keypoints in different categories ranges from 8 to 68. Considering that some categories in the original dataset lack connection annotations, this invention supplements the missing connection annotations based on relevant research. In the dataset partitioning process, this invention follows the original partitioning method of the MP-100 dataset, dividing all categories into non-overlapping training, validation, and test sets in a 70:10:20 ratio. Since the default keypoint order in the MP-100 dataset relies on cross-category association indices (e.g., the first keypoint always corresponds to "nose"), this design requires pre-defining semantic correspondences between categories, thus severely limiting the model's scalability for truly unknown categories. Therefore, this invention uses an independent order obtained by randomly shuffling the keypoint order (5 random partitions). This setting does not rely on any cross-category association priors and is closer to the needs of real-world open scenarios.
[0050] This invention uses the independent order by default and analyzes the impact of the keypoint order through experiments. Table 1 summarizes the pose mAP results of various methods using related and independent orders under five dataset partitions.
[0051] Table 1. Pose mAP results of different methods on five partitioned datasets using related or independent keypoint order.
[0052]
[0053] As shown in Table 1, regardless of the keypoint order, our proposed method significantly outperforms existing comparable baselines. Secondly, while baseline methods lacking a matching supervision mechanism can leverage semantic priors from the associated order for transfer learning, their performance deteriorates significantly under more realistic and challenging independent order settings. Overall, our proposed method demonstrates robust and superior performance across different data partitioning and keypoint order types.
[0054] In one specific implementation of this embodiment, the attitude estimation model consists of the following components and data processing steps:
[0055] (1) Overall model architecture and module composition:
[0056] As shown in Figure 2, the pose estimation model constructed in this invention is an end-to-end deep learning system, mainly composed of four core modules working together:
[0057] Feature extraction module: The visual front end of the model, responsible for extracting multi-scale features with rich semantics from the input image.
[0058] General prior module: A structured knowledge base for the model, storing and maintaining a set of learnable keypoint embedding vectors. and initial attitude parameters These parameters are designed to encode common components and connection priors across object categories.
[0059] The pose optimization module is the core processor of the model, consisting of T layers (insufficient optimization layers cannot accurately estimate the pose, while too many layers would introduce redundancy and unnecessary burden; therefore, the default number of layers in this embodiment is T). The optimization layers are stacked sequentially with identical structures and are responsible for performing iterative pose refinement; each optimization layer contains a structure-aware keypoint decoder and a position-aware connection decoder.
[0060] Collaborative matching supervision module: The "supervision engine" during the training phase, which is only enabled during model training, is responsible for generating consistency loss signals that drive the model to learn transferable knowledge.
[0061] (2) Data processing flow:
[0062] The model data processing flow constructed in this invention is mainly divided into a training phase and an inference phase. Since the number of keypoints in different types of images may vary, this invention fills in keypoints through visibility during implementation, uniformly representing all poses as keypoint coordinates. , and connection relationships Due to smaller keypoint embedding This could lead to a single embedding being forced to capture irrelevant keypoint categories, while a larger one... This may result in the relevant key point categories being too scattered; therefore, the default value in this embodiment of the invention is... The specific process of model data processing is as follows:
[0063] ① Input preprocessing and feature extraction
[0064] The input image x is first scaled to 256×256 pixels through size normalization. Then, a feature extraction module extracts features from the processed image. In this embodiment, a ResNet-50 pre-trained on the ImageNet dataset is used as the backbone network to extract feature maps at multiple levels. These feature maps are then projected onto a uniform feature dimension (D=256) through 1×1 convolutions, ultimately outputting a set of multi-scale image features. (s=1,2,…,S).
[0065] ② General Prior Injection and Iterative Optimization
[0066] A set of learnable keypoint embedding vectors and initial pose parameters from a general prior module are input into the pose optimization module, which uses multi-scale features. As a condition, an iterative optimization process is initiated. During the initiation of the iterative optimization process, this embodiment of the invention maintains a set of learnable keypoint embedding vectors. As input to the first layer, and during the model initialization phase, the initial pose is used. Initialize to a constant of 0.5, and then randomly initialize. Taking the iterative optimization of the t-th optimization layer as an example, the expression for the calculation process of the t-th optimization layer is shown in equation (1):
[0067] (1)
[0068] In the formula, and Let these represent the output pose and keypoint embedding vector of the (t-1)th layer, respectively. Representing multi-scale image features, and Let represent the pose and keypoint embedding vectors of the output of layer t, respectively.
[0069] In addition, during the iterative optimization process, each optimization layer performs two core operations in a fixed order, as follows:
[0070] Structure-aware keypoint decoding:
[0071] Receive attitude parameters from the output of the previous optimization layer and keypoint embedding vector using key point coordinates Using a reference point, a deformable attention mechanism is used to adaptively extract multi-scale image features. Mining fine-grained features, the expression is:
[0072] (2)
[0073] Then, a multi-head attention mechanism is used to reconnect the relationships from the previous optimization layer. Attention weights are introduced as a structural bias to further update the keypoint embedding vector. Keypoint embedding vector The expression for updating the embedding vector of the q-th keypoint is:
[0074] (3)
[0075] In the formula, q and n represent the indices of the keypoint embedding, and m represents the attention head index. Indicates learnable weights, Represents the original attention weights. This represents the connection strength between the q-th keypoint and the n-th keypoint. This means normalizing the sum of attention weights to one.
[0076] Finally, the optimized keypoints are embedded into the vector E using equation (4). t Decode into keypoint coordinates of the current layer With visibility The expression is:
[0077] (4)
[0078] In the formula, and These represent the sigmoid function and its inverse function, respectively. This represents a lightweight prediction head.
[0079] Location-aware connectivity decoding:
[0080] Enumerate all predicted keypoint pairs (p, q), add their embedding vectors together to generate a coarse-grained connection embedding vector. The expression is:
[0081] (5)
[0082] Receive the key point coordinates P output by the structure-aware key point decoder t Calculate the geometric midpoint of each keypoint pair (p, q). And using the midpoint as a reference point, a deformable attention mechanism is used to... The visual features along the connection path are aggregated to supplement the coarse-grained connection embedding, thereby improving the coarse-grained connection embedding vector. Refined into fine-grained connection embedding vectors rich in contextual evidence The expression is:
[0083] (6)
[0084] Finally, the refined connection embedding vector is... Feed into a lightweight classifier Predict and output the connection relationships of all key points in the current layer. The expression is:
[0085] (7)
[0086] In the formula, Represent a univariate existence classifier by enumerating and averaging. Get keypoint visibility This is used to filter connections because a connection cannot exist if the key is not visible.
[0087] The current layer ultimately outputs the refined attitude parameters. and keypoint embedding vector This serves as the input for the next optimization layer or the final pose prediction result. After... After this optimization layer, the pose optimization module outputs the final predicted pose. .
[0088] ③ Collaborative matching supervision and parameter update
[0089] During the training phase, the collaborative matching supervision module begins its work, responsible for generating core supervision signals that drive the model to learn transferable knowledge. For a given i-th training image... The estimated A posture and its actual posture after filling. If a query-based model is used with matching-based supervision, inconsistent matching results may occur for images of the same category. Therefore, this invention proposes joint optimization of a set of similar images to achieve better semantic consistency. The specific process is as follows:
[0090] During training iterations, classes belonging to the same category are dynamically selected from the current batch of training sets. Zhang training images (default This constitutes a supervisory group;
[0091] Calculate the collaborative matching cost matrix of the supervisory group using the formula. , Indicates the number of predicted key points. Represents the number of real keypoints; using the cost matrix Middle elements For example, calculation This represents the cost of matching the nth predicted keypoint to the kth real keypoint, calculated using the following formula:
[0092] (8)
[0093] In the formula, This indicates that the optimization layer t is applied to the i-th training image x. i The predicted coordinates of the nth key point Indicates L1 loss, This represents the binary cross-entropy loss. This represents the k-th real key point. This indicates that the optimization layer t is applied to the i-th training image x. i The visibility of the predicted nth keypoint Indicates the k-th real visibility;
[0094] Based on the cost matrix of cooperative matching The bipartite graph matching algorithm is used to find a globally optimal matching mapping between predicted keypoints and real keypoints for the supervision group. This mapping assigns a unique predicted keypoint index to each true keypoint k. This matching relationship is shared among all images within the group, ensuring semantic consistency.
[0095] Based on global matching relationships For each image in the supervision group, a fine-grained consistency loss is calculated for the keypoints that have been successfully matched. This includes keypoint localization loss that makes the predicted coordinates of the matched points approximate the true coordinates, visibility classification loss that makes the predicted visibility confidence approximate the true label, and connection relationship classification loss that makes the predicted connection probability approximate the true skeleton connection.
[0096] The matching loss for each image is calculated using the following expression:
[0097] (9)
[0098] In the formula, This represents the set of indices of matched keypoints in each training image. Denotes the hyperparameters used to balance the visibility classification loss and , Denotes the hyperparameters used to balance the connection classification loss and , This represents the predicted keypoint corresponding to the k-th true keypoint after optimal matching. Indicates the actual number of key points. Indicates a real connection system. This represents the predicted connection relationship after matching.
[0099] Meanwhile, for unmatched functions For the selected prediction key points, an additional penalty loss is applied. This forces the visibility of the predictions and the probability of all involved connections to approach zero, ensuring the compactness of the model output and avoiding invalid and redundant predictions.
[0100] Based on the above calculation of the supervision group loss, the model parameters are updated through the following optimization process:
[0101] The consistency loss of all supervisory groups within a batch (including) and The summation of these values yields the final total backpropagation loss. Automatic differentiation is then used to calculate the gradient of this loss with respect to all learnable parameters of the model (feature extraction network, general prior module (containing keypoint embedding vectors), and pose optimization module). Based on this, training is performed using the Adam optimizer for 100 epochs with a batch size of 4, and linear warm-up with a learning rate (initially 1000 iterations starting from 1e). -8 linearly increasing to 1e -5 The model employs a step decay strategy (multiplying the learning rate by 0.1 in the 60th and 80th epochs) to ensure training stability and enable fine-tuning in later stages. This optimization process ultimately drives the learning of keypoint embedding vectors in the model and solidifies them into transferable cross-class structured knowledge.
[0102] ④ Direct prediction of new categories
[0103] During the inference phase, given a new class image not seen during training, the model outputs the complete pose estimate of the object directly through only one forward pass. No external support is required. Furthermore, during the model inference phase, the collaborative matching supervision module is not enabled, and the trained model parameters (especially the keypoint embedding vectors that have been solidified into transferable knowledge) are fixed.
[0104] In one specific embodiment of this example, a class-independent pose estimation system based on cooperative matching supervision is provided, comprising:
[0105] Feature extraction module: Used to extract multi-scale features from the input image and output multi-scale image features. ;
[0106] General prior module: stores a set of learnable keypoint embedding vectors and initial pose parameters;
[0107] Pose optimization module: Connects the feature extraction module and the general prior module, used for multi-scale image feature optimization. The learnable keypoint embedding vector and the initial pose parameters are iteratively optimized to output the predicted pose parameters; wherein, the pose optimization module consists of multiple sequentially connected optimization layers, each of which includes a structure-aware keypoint decoder and a position-aware connection decoder.
[0108] Collaborative matching supervision module: Connected to the pose optimization module during the training phase, it groups the prediction results of multiple training images of the same category and solves the globally optimal matching relationship between predicted keypoints and ground truth keypoints. The system parameters are optimized by calculating the consistency loss based on the matching relationship, so that the key point embedding vectors in the general prior module learn structured knowledge that can be transferred to new categories.
[0109] This invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor of steps of a class-independent pose estimation method based on cooperative matching supervision as described in any of the above embodiments.
[0110] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a class-independent pose estimation method based on cooperative matching supervision as described in any of the above embodiments.
[0111] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions conceived without inventive effort should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims.
[0112] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A class-independent pose estimation method based on cooperative matching supervision, characterized in that, include: S1. Obtain an image dataset containing multiple basic categories and divide the dataset into a training set and a test set; each training image is labeled with the coordinates of real keypoints. Visibility and connection relationship S2. Construct the attitude estimation model and input it. The training images are of the same category; the pose estimation model includes a feature extraction module, a learnable general prior module, a pose optimization module, and a collaborative matching supervision module enabled during the training phase; wherein, the general prior module stores learnable keypoint embedding vectors and initial pose parameters; the pose optimization module consists of... It consists of a series of sequentially stacked optimization layers. For each optimization layer, the integer is greater than 1. Each optimization layer consists of a structure-aware keypoint decoder and a position-aware connectivity decoder. The computation process of the t-th optimization layer is defined as follows: In the formula, and Let these represent the output pose and keypoint embedding vector of the (t-1)th layer, respectively. Representing multi-scale image features, and S3 and S4 represent the output pose and keypoint embedding vector of the t-th layer, respectively; the feature extraction module... Feature extraction is performed on each of the training images to output multi-scale image features. S4, the pose optimization module is based on multi-scale image features of each training image. For a set of learnable keypoint embedding vectors from a general prior module With initial attitude parameters Perform multiple rounds of iterative optimization and output the coordinates, visibility, and connectivity of predicted keypoints for each training image; S5, call the collaborative matching supervision module to... S6. Calculate the consistency loss for training images of the same category, and optimize the pose estimation model parameters through backpropagation so that the model learns structured knowledge that can be transferred to the new category; S7. Input the new category images to be estimated in the test set into the trained pose estimation model, and directly output the pose estimation results of the objects in the new category images through forward propagation; wherein, the pose estimation results include key point coordinates, visibility and connectivity.
2. The method according to claim 1, characterized in that, The operation process of each optimization layer in the attitude optimization module is as follows: Structure-aware key point decoding: receiving attitude parameters output from the previous optimization layer. and keypoint embedding vector using key point coordinates Using a reference point, a deformable attention mechanism is used to extract multi-scale image features. Extract appearance information; Utilizing a multi-head attention mechanism to link the connections of the previous optimization layer Attention weights are introduced as structural biases to calculate the updated keypoint embedding vectors, and the keypoint coordinates of the current layer are decoded. With visibility The previous optimization layer is layer t-1, and the current layer is layer t. Position-aware connection decoding: receives the keypoint coordinates output by the structure-aware keypoint decoder. Calculate the geometric midpoint of all keypoint pairs, and using the midpoint as a reference point, apply a deformable attention mechanism to... The system aggregates visual features along the connection path and then predicts and outputs the connection relationships of all key points in the current layer. The current layer ultimately outputs the refined attitude parameters. and keypoint embedding vector This serves as the input for the next optimization layer or the final pose prediction result.
3. The method according to claim 1, characterized in that, The collaborative matching supervision module described in step S5... The process of calculating the consistency loss for training images of the same category includes: selecting images of the same category from the training set. Zhang training images To form a supervisory group ≥2; Calculate the collaborative matching cost matrix of the supervision group. ， Indicates the number of predicted key points. This represents the number of real keypoints; where the cost matrix is... elements in The calculation formula is: In the formula, This represents the cost of matching the nth predicted keypoint to the kth real keypoint. This indicates that the optimization layer t is applied to the i-th training image x. i The predicted coordinates of the nth key point Indicates L1 loss, This represents the binary cross-entropy loss. This represents the k-th real key point. This indicates that the optimization layer t is applied to the i-th training image x. i The visibility of the predicted nth keypoint Represents the k-th true visibility; based on the collaborative matching cost matrix The bipartite graph matching algorithm is used to find a globally optimal matching mapping between predicted keypoints and real keypoints for the supervision group. ; wherein, the matching mapping Assign a unique predicted keypoint index to each real keypoint k within the supervision group. Furthermore, the mapping relationship is shared among all images within the group to ensure semantic consistency; based on global matching mapping... For each training image in the supervised group, a consistency loss is calculated for matched keypoints, while a classification loss is applied to the visibility and connectivity of unmatched keypoints. This forces the value to approach zero to avoid generating invalid and redundant predictions; the consistency loss includes keypoint localization loss, visibility classification loss, and connectivity classification loss; wherein, the matching loss for each training image is calculated using the following expression: In the formula, This represents the set of indices of matched keypoints in each training image. Denotes the hyperparameters used to balance the visibility classification loss and ， Denotes the hyperparameters used to balance the connection classification loss and ， This represents the predicted keypoint corresponding to the k-th true keypoint after optimal matching. Indicates the actual number of key points. Indicates a real connection relationship. This represents the predicted connection relationship after matching.
4. The method according to claim 1, characterized in that, The learnable keypoint embedding vectors and initial pose parameters in the general prior module are optimized by the collaborative matching supervision module during training to learn cross-class general structural and geometric constraint priors, and are solidified into fixed parameters for inference after training is completed.
5. A class-independent pose estimation system based on cooperative matching supervision, used to implement the method according to any one of claims 1-4, characterized in that, include: Feature extraction module: Used to extract multi-scale features from the input image and output multi-scale image features. General prior module: stores a set of learnable keypoint embedding vectors and initial pose parameters; Pose optimization module: Connects the feature extraction module and the general prior module, used for multi-scale image feature optimization. The learnable keypoint embedding vectors and initial pose parameters are iteratively optimized to output predicted pose parameters. The pose optimization module consists of multiple sequentially connected optimization layers, each including a structure-aware keypoint decoder and a position-aware connection decoder. A collaborative matching supervision module, connected to the pose optimization module during training, groups the prediction results of multiple training images of the same category and solves for the globally optimal matching relationship between predicted and ground truth keypoints. The system parameters are optimized by calculating the consistency loss based on the matching relationship, so that the key point embedding vectors in the general prior module learn structured knowledge that can be transferred to new categories.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of a class-independent pose estimation method based on cooperative matching supervision as described in any one of claims 1-4.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of a class-independent pose estimation method based on cooperative matching supervision as described in any one of claims 1-4.