A Multi-Class Pose Estimation Method and System Based on Shared Keypoint Adaptive Matching

Abstract

This application provides a multi-class pose estimation method and system based on shared keypoint adaptive matching, comprising: constructing a large-scale multi-class pose dataset for model training and evaluation; processing the dataset to obtain a training image subset and a test image subset; inputting the training image subset into a query model containing an image feature extraction module, a structural prototype classification module, a shared keypoint prediction module, an adaptive matching module, and a model optimization module; training the query model and optimizing its parameters through the collaboration of these modules; and applying the optimized query model to the test images to obtain the structural prototype category prediction results and keypoint estimation results for the test images. This invention achieves unified and high-precision pose estimation for massive categories and structurally variable objects by learning cross-class general features through shared keypoint embedding and utilizing a prototype-specific matching mechanism based on momentum updates.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a multi-class pose estimation method and system based on adaptive matching of shared keypoints. Background Technology

[0002] Object pose estimation is a fundamental task in computer vision, aiming to predict the location of predefined key points of an object from a single image. This technology provides an indispensable visual perception foundation for advanced applications such as robot grasping, autonomous driving, augmented reality, and industrial quality inspection.

[0003] However, current pose estimation techniques still have significant limitations. Firstly, the task scope is highly specific. Most existing mature methods rely on pre-defined, invariant keypoint topologies designed for fixed structural categories such as humans and vehicles, resulting in a strong binding of model architecture and output dimensions to specific categories and a lack of generalization ability.

[0004] Secondly, there is a lack of universal data benchmarks. Although image classification and object detection have achieved general modeling for tens of thousands of categories with the help of datasets such as ImageNet, the field of pose estimation still lacks large-scale, high-quality datasets with broad category coverage and unified annotation standards. Existing multi-category datasets are mostly simple concatenations with inconsistent annotation standards, making it difficult to support the training and reliable evaluation of universal models.

[0005] Finally, the rigid model architecture is a major issue. Traditional methods, whether based on heatmaps or coordinate regression, have output channels that are strictly fixed to the semantics of specific keypoints. This rigid one-to-one correspondence cannot adapt to the vast differences in the number, type, and topological structure of keypoints between different categories. While few-shot or zero-shot learning attempts to expand the category range, its focus is on rapid adaptation with a small number of samples, rather than building a fully trained, unified model capable of directly and accurately handling massive numbers of categories.

[0006] Therefore, current technology faces a prominent contradiction: upper-level applications urgently need a universal pose perception capability that can uniformly understand the position, contour, and structure of various objects, much like human vision, while existing technologies can only provide a series of scattered and specialized solutions. How to build a unified pose estimation framework that can adapt to massive categories and flexibly handle changes in the intrinsic structure of objects has become a key bottleneck in the field's development. Summary of the Invention

[0007] To address the problems of task specificity, lack of data benchmarks, and rigid model architecture in existing pose estimation techniques, this invention proposes a multi-class pose estimation method and system based on adaptive matching of shared keypoints. By constructing a universal pose estimation framework that can handle a massive number of object categories and adapt to the diversity of their internal structures, a "structural prototype" is introduced as the basic unit for describing geometric structure. Combined with a collaborative model mechanism of "shared keypoint query" and "adaptive matching," this achieves a leap from a "dedicated" paradigm for limited categories to a "universal" paradigm for the open world.

[0008] The first aspect of this invention provides a multi-class pose estimation method based on adaptive matching of shared keypoints, comprising:

[0009] S1. Construct a multi-class pose dataset, process the dataset, and obtain training images and test images;

[0010] S2. Design a query-based model that incorporates matching loss; the query-based model includes an image feature extraction module, a structural prototype classification module, a shared keypoint prediction module, an adaptive matching module, and a model optimization module;

[0011] S3. Input the training image into the query model, and extract features from the input training image through the image feature extraction module to obtain multi-scale image features;

[0012] S4. The structural prototype classification module performs structural prototype classification based on multi-scale image features to obtain the prototype category prediction result y.

[0013] S5. The shared keypoint prediction module interacts with a set of learnable keypoint query vectors and multi-scale image features to decode and generate coordinate prediction P and visibility prediction V for M keypoints.

[0014] S6. The adaptive matching module is based on the prototype category prediction result y or the real prototype category label from step S4. Obtain the corresponding matching cost matrix, and based on the matching cost matrix, perform adaptive optimal matching between the M keypoint predictions in step S5 and the K target keypoints of the category to which the training image belongs, and output the matching mapping relationship; wherein, the structural prototype of each image category in the dataset contains at most K real keypoints, and K≤M;

[0015] S7. Based on the matching mapping relationship, calculate the prototype classification loss and the regression loss of the matched key points, and optimize the model parameters based on the comprehensive loss.

[0016] S8. Input the test image into the optimized query model to obtain the structural prototype category prediction results and key point estimation results of the test image.

[0017] Furthermore, the construction of a multi-class pose dataset, and the processing of the dataset to obtain training and test images, includes:

[0018] Image filtering: From images in a large-scale classification dataset, images of object categories with clear geometric structures are filtered out to form an initial image set; wherein, the large-scale classification dataset includes, but is not limited to, the ImageNet dataset and the UniKPT dataset;

[0019] Structural prototype annotation: For each semantic category in the initial image set, a structural prototype annotation operation is performed; wherein, the structural prototype annotation operation is based on the geometric alignability criterion, which clusters object images with the same geometric structure into the same structural prototype;

[0020] Attitude annotation: For each structural prototype divided by structural prototype annotation, the attitude annotation operation is performed, wherein the attitude annotation operation is based on a unified topology to annotate the key point coordinates and visibility of all object instances under each structural prototype;

[0021] External data integration: Mapping and integrating labeled data from external pose datasets into the structural prototype system established by the structural prototype annotations;

[0022] Dataset partitioning: Within each structural prototype, object instances are randomly divided into training image subsets and test image subsets according to a preset ratio to form a multi-class pose dataset for model training and evaluation.

[0023] Furthermore, the image feature extraction module in step S3 performs multi-level feature extraction on the input image based on the backbone network of a deep convolutional neural network, and outputs multi-scale image feature maps containing different semantic levels and spatial resolutions. The multi-scale image feature maps are used for subsequent structural prototype classification and key point estimation. The backbone network includes, but is not limited to, ResNet-50, ResNet-101, and DenseNet-201.

[0024] Further, the structural prototype classification module in step S4 includes a global feature extraction unit and a classification prediction unit; the structural prototype classification module performs structural prototype classification based on multi-scale image features to obtain the prototype category prediction result y, including:

[0025] A global feature extraction unit is used to perform global average pooling on the deepest feature map of the multi-scale image features output by the image feature extraction module to generate a global feature vector.

[0026] The global feature vector is mapped to the log odds value of the structural prototype category by the classification prediction unit, and the prototype category prediction result y is obtained.

[0027] Further, the shared keypoint prediction module in step S5 includes a keypoint query vector library, a deformable Transformer decoder, a coordinate regression head, and a visibility classification head; the shared keypoint prediction module interacts with a set of learnable keypoint query vectors and multi-scale image features to decode and generate coordinate predictions P and visibility predictions V for M keypoints, including:

[0028] A fixed number of learnable keypoint query vectors E are initialized from the keypoint query vector library; wherein, the keypoint query vector library stores M learnable D-dimensional keypoint query vectors. Each keypoint query vector is designed to learn a universal keypoint feature shared across structural prototype categories through training;

[0029] The keypoint query vector E and multi-scale image features are input into the deformable Transformer decoder. A deformable attention mechanism allows the keypoint query vector E to interact with the multi-scale image features, adaptively focusing on the local image regions corresponding to each keypoint to extract fine-grained features. A self-attention mechanism enables interaction between the keypoint query vectors to model the spatial dependencies between keypoints. The state of the query vectors is updated to obtain a keypoint query vector E' that integrates specific image information features. ;

[0030] Based on the updated keypoint query vector E', the coordinate regression head maps the features of each query vector to 2D normalized coordinates through a multilayer perceptron or linear layer, obtaining the coordinate predictions of M keypoints. P ,in, ;

[0031] Based on the updated keypoint query vector E', the visibility classification head uses a lightweight binary classifier to predict the visibility confidence of each keypoint, obtaining the visibility predictions for M keypoints. V ,in, ;

[0032] Finally, the coordinates are predicted. P and visibility prediction V Combine the results to output an unordered prediction set containing M keypoints.

[0033] Furthermore, the adaptive matching module in step S6 includes a matching cost matrix storage unit, a matrix retrieval unit, an instant cost calculation unit, a momentum update unit, and an optimal matching solution unit; the adaptive matching module is based on the prototype category prediction result y or the true prototype category label in step S4. Obtain the corresponding matching cost matrix, and based on the matching cost matrix, perform adaptive optimal matching between the M keypoint predictions in step S5 and the K target keypoints of the category to which the training image belongs, outputting the matching mapping relationship, including:

[0034] The adaptive matching module receives category identification data corresponding to training image samples and a set of M key points to be matched; wherein, the category identification data includes the prototype category prediction result y and the true prototype category label. ;

[0035] The instant cost calculation unit calculates the real-prototype annotation data of the currently input training image samples. and Given the model prediction data P and V, calculate the instant cost matrix C', which reflects the matching cost of M keypoints in the training image; the instant cost matrix of the m-th keypoint in the instant cost matrix C' is calculated as follows: ,in, This represents the instantaneous cost matrix for matching the m-th predicted keypoint to the k-th target keypoint. α This represents the balancing hyperparameters, where P and V represent the coordinates and visibility confidence of the m-th predicted keypoint, respectively. and Let || represent the true coordinates and visibility label of the k-th target keypoint, respectively; || represents the distance metric. Represents the binary cross-entropy loss function;

[0036] The matrix retrieval unit is based on the true prototype category label of the current sample. The corresponding matching cost matrix is ​​retrieved from the matching cost matrix storage unit, wherein the matching cost matrix is ​​initialized to a zero matrix during the first training and is updated through momentum accumulation in each training iteration;

[0037] The momentum update unit, based on the instantaneous cost matrix C', updates the corresponding matching cost matrix in the matching cost matrix storage unit using the momentum update formula. The matching cost matrix is ​​updated to obtain a more statistically stable and generalizable matching cost matrix C. new Save back to the matching cost matrix storage unit; where the momentum update expression for the matching cost matrix is: , where λ represents the momentum coefficient;

[0038] The optimal matching solution unit is based on the updated matching cost matrix C. new The optimal bipartite graph matching is solved using the Hungarian algorithm, and the matching mapping relationship between key points and the matching status indicator are output.

[0039] Further, step S7, which involves calculating the prototype classification loss and the regression loss of the matched keypoints based on the matching mapping relationship, and optimizing the model parameters based on the comprehensive loss, includes:

[0040] Based on the matching mapping relationship output by the adaptive matching module, distinguish between matched keypoint predictions and unmatched keypoint predictions;

[0041] Prototype classification loss calculation: Compare the model's predicted prototype category y with the true prototype category label. Substituting the values ​​into the cross-entropy loss function, we obtain the prototype classification loss for the training image samples. ;

[0042] Calculation of regression loss for matched keypoints: For each matched keypoint, calculate the error between the predicted coordinates and the actual coordinates, as well as the classification error between the predicted visibility and the actual visibility. The regression loss for the k-th target keypoint is calculated as follows: ,in, and Let || represent the true coordinates and visibility label of the k-th target keypoint, respectively, and || represent the distance metric. Indicates the matching mapping relationship. L ce Represents the binary cross-entropy loss function;

[0043] Comprehensive Loss Calculation: The prototype classification loss and the regression loss of matched keypoints are weighted and summed according to preset weights to obtain the comprehensive loss. The formula for calculating the comprehensive loss is as follows: , where y and These represent the predicted prototype category and the actual prototype category label, respectively. Represents the prototype classification loss. Let represent the coordinates of the k-th matched predicted keypoint and the visibility regression loss, and β represent the balancing hyperparameter;

[0044] Based on the comprehensive loss, all trainable parameters in the query model are jointly optimized using the backpropagation algorithm; wherein the trainable parameters include the parameters of the image feature extraction module, the parameters of the structural prototype classification module, and the parameters of the deformable Transformer decoder.

[0045] Further, step S8, which involves inputting the test image into the optimized query model to obtain the structural prototype category prediction result and keypoint estimation result of the test image, includes:

[0046] The test image is input into the optimized query model, and the structural prototype category prediction result y corresponding to the test image, as well as the coordinate prediction and visibility prediction of M key points, are obtained through forward propagation.

[0047] Based on the structural prototype category prediction result y, determine the final structural prototype category of the object in the test image;

[0048] Based on the final structural prototype category, retrieve the corresponding target matching cost matrix from the pre-stored matching cost matrix storage unit;

[0049] Based on the target matching cost matrix, the predictions of M key points are matched and filtered to obtain K ordered key point estimation results corresponding to the final structural prototype category.

[0050] Output the final structural prototype category and the estimated results of K ordered key points.

[0051] A second aspect of the present invention provides a multi-class pose estimation system based on adaptive matching of shared keypoints, comprising:

[0052] Image feature extraction module: used to extract multi-scale features from the input image and output multi-scale image features;

[0053] Structural Prototype Classification Module: Connected to the image feature extraction module, it is used to predict the structural prototype category of objects in the input image based on the multi-scale image features;

[0054] Shared keypoint prediction module: Connected to the image feature extraction module, the module mainly contains a set of learnable keypoint query vectors and a deformable Transformer decoder, which is used to enable the keypoint query vectors to interact with multi-scale image features through the deformable Transformer decoder, so as to decode and output the coordinate prediction and visibility prediction of M keypoints.

[0055] Adaptive matching module: connected to the structural prototype classification module and the shared key point prediction module respectively; the module stores multiple matching cost matrices corresponding to different structural prototype categories, which are used to retrieve the corresponding matching cost matrix according to the input category information, and perform optimal matching between the M key point predictions and the K target key points of the current category based on the matching cost matrix, and output the matching mapping relationship.

[0056] Model optimization module: Connected to the structural prototype classification module, shared keypoint prediction module, and adaptive matching module; used to calculate the prototype classification loss and the regression loss of matched keypoints based on the matching mapping relationship, and to jointly optimize the parameters of all trainable modules in the system based on the comprehensive loss.

[0057] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor as described above in a multi-class pose estimation method based on adaptive matching of shared keypoints.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] 1) It breaks through the category limitations of traditional pose estimation methods. By introducing the concept of "structural prototype" and a unified modeling framework, it can perform unified pose estimation for massive object categories at the ImageNet level, enabling the task to move from the "dedicated" stage for specific categories to the "general" stage in the open world.

[0060] 2) Through the "shared key point" query mechanism and the "adaptive matching" supervision strategy, the model can learn general structural features across categories and flexibly adapt to the number of key points and topological changes of different objects. When facing new categories or intra-class structural variants, it can maintain excellent performance without structural adjustment, and its generalization ability is significantly enhanced.

[0061] 3) Through the design of visibility-aware matching costs and momentum update mechanisms, the system can robustly learn the optimal matching pattern across samples during the training phase and achieve high-precision keypoint localization and filtering during the inference phase. This enables the model to achieve higher accuracy on standard evaluation metrics and maintain the stability and rationality of the output results under complex imaging conditions such as occlusion and deformation.

[0062] 4) An innovative end-to-end technical system was constructed, from "structural prototype" data annotation to unified model training and efficient inference. The established large-scale, high-quality pose dataset provides the first benchmark in the field covering a massive number of categories, promoting the expansion of research paradigms; the proposed unified model eliminates the dependence on complex multi-stage pipelines or manual rules, significantly improving the overall efficiency and engineering deployability of the system.

[0063] 5) This technology provides a unified and high-precision posture understanding capability for fields that rely on fine geometric understanding of objects, such as robot grasping, autonomous driving, augmented reality, and industrial quality inspection. Its core value lies in its ability to effectively address the core challenge of diverse object categories and structures in open scenarios, providing a crucial visual perception foundation for the intelligent upgrading of related industries, and possessing broad application prospects and commercial value. Attached Figure Description

[0064] Figure 1 This is a flowchart of the steps for a multi-class pose estimation method based on adaptive matching of shared keypoints.

[0065] Figure 2 Statistical information of the pose dataset constructed for embodiments of the present invention.

[0066] Figure 3 A schematic diagram illustrating the operation of a query-based model designed for a multi-class pose estimation method based on adaptive matching of shared keypoints. Detailed Implementation

[0067] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0068] Please see Figure 1 , Figure 1 This is a flowchart illustrating the steps of a multi-class pose estimation method based on adaptive matching of shared keypoints. The specific steps include:

[0069] S1. Construct a multi-class pose dataset, process the dataset, and obtain training images and test images;

[0070] S2. Design a query-based model that incorporates matching loss; the query-based model includes an image feature extraction module, a structural prototype classification module, a shared keypoint prediction module, an adaptive matching module, and a model optimization module;

[0071] S3. Input the training image into the query model, and extract features from the input training image through the image feature extraction module to obtain multi-scale image features;

[0072] S4. The structural prototype classification module performs structural prototype classification based on multi-scale image features to obtain the prototype category prediction result y.

[0073] S5. The shared keypoint prediction module interacts with a set of learnable keypoint query vectors and multi-scale image features to decode and generate coordinate prediction P and visibility prediction V for M keypoints.

[0074] S6. The adaptive matching module is based on the prototype category prediction result y or the real prototype category label from step S4. Obtain the corresponding matching cost matrix, and based on the matching cost matrix, perform adaptive optimal matching between the M keypoint predictions in step S5 and the K target keypoints of the category to which the training image belongs, and output the matching mapping relationship; wherein, the structural prototype of each image category in the dataset contains at most K real keypoints, and K≤M;

[0075] S7. Based on the matching mapping relationship, calculate the prototype classification loss and the regression loss of the matched key points, and optimize the model parameters based on the comprehensive loss.

[0076] S8. Input the test image into the optimized query model to obtain the structural prototype category prediction results and key point estimation results of the test image.

[0077] In one specific implementation of this embodiment, the process of constructing a multi-class pose dataset, processing the dataset to obtain training images and test images includes:

[0078] Image Selection: This embodiment of the invention uses the ImageNet dataset, which contains over a million images and covers thousands of semantic categories, as the primary image source, and the UniKPT dataset, which contains a large amount of labeled keypoint data for people, animals, vehicles, etc., as an auxiliary and supplementary source. First, images of object categories with clear geometric structures are selected from the ImageNet dataset, while 62 categories lacking obvious structures, such as coastlines, cliffs, and hot pots, are removed, forming an initial image set. Considering sample balance, up to 200 pose samples for each category are randomly selected from the UniKPT dataset and added to the image dataset to supplement the sample count for certain categories in ImageNet.

[0079] Structural prototype annotation: During the structural prototype annotation process, for each semantic category in the dataset, an initially empty set of structural prototypes P={x i} where the i-th image in the set represents the i-th prototype. During annotation, annotators adhere to a strict "geometric alignability" criterion—that is, using the annotation's key points as control points, achieving good alignment through spatial transformations (such as thin-plate spline transformations)—to determine the images to be annotated, assigning them to existing corresponding prototypes or creating new prototypes. Images of objects with unclear structures due to occlusion, density, or other reasons are marked as discarded samples. Through this process, a hierarchical system of "semantic category - structural prototype" is ultimately formed.

[0080] Attitude annotation: Attitude annotation is based on structural prototype division. A unified keypoint topological definition is established for each prototype, and the precise coordinates and visibility status of each keypoint in all object instances under that prototype are systematically annotated according to this definition. To ensure the consistency of annotation, a quality control mechanism based on reference alignment is introduced during the annotation process: spatial alignment methods such as thin plate spline (TPS) transformation are used to align different instances within the same prototype on a common main structure. This serves as an objective basis for verifying and constraining the consistency of annotation, thereby providing high-quality ground truth data with clear structure and standardized annotation for model training.

[0081] External Data Integration: Annotated data from external pose datasets are systematically integrated into the structural prototype system established in this invention through semantic alignment, keypoint mapping, and quality verification. This expands the data scale and unifies annotation standards, thereby constructing an enhanced multi-class pose dataset. Finally, the pose dataset constructed in this invention covers 2821 prototypes across 720 categories, annotating 900665 keypoints for 85823 objects. Statistical information for the pose dataset can be found... Figure 2 ,exist Figure 2 In (a), we can observe the number of subclasses contained in each superclass in the dataset. It can be found that the "device" superclass has the most subclasses (e.g., computer keyboard, vertical device, etc.). Figure 2 In (b), (c), and (d), we can observe the number of prototypes in each category, the number of samples in each category, and the number of key points in each prototype, respectively.

[0082] Dataset partitioning: Within each structural prototype, the object instances are randomly divided into training and testing subsets according to a preset ratio on the final integrated dataset, thereby constructing a standard multi-class pose dataset that can be directly used for model training and performance evaluation. In this embodiment of the invention, the ratio of training set samples to test set samples is set to 7:3.

[0083] In one specific implementation of this embodiment, a query-based model incorporating matching loss is used (see...). Figure 3 ),include:

[0084] Image feature extraction module: Extracts features of images at different semantic levels and spatial resolutions through deep convolutional networks, providing rich visual representations for subsequent classification and key point localization.

[0085] Structural Prototype Classification Module: Performs high-level semantic understanding of objects. It categorizes image features into a predefined structural prototype category, providing guidance for subsequent pose estimation.

[0086] Shared keypoint prediction module: Receives multi-scale image features extracted from the image feature extraction module, and generates a set of updatable keypoint query vectors. Each query vector interacts with the image features through a deformable Transformer decoder, and finally outputs a set containing M keypoint predictions, each prediction including coordinates and visibility.

[0087] Adaptive matching module: During model training, the matching cost matrix is ​​updated based on the momentum retrieved according to the category. Using the Hungarian algorithm, the model dynamically performs optimal one-to-one matching between the M unordered key points predicted by the model and the K target key points of the current category to obtain the matching mapping relationship.

[0088] Model optimization module: Receives the "scheduling scheme" (matching mapping relationship) from the adaptive matching module, transforms it into accurate supervision signals, and drives and coordinates the updates and optimizations of all learnable parts in the system through the backpropagation algorithm.

[0089] In one specific implementation of this embodiment, the training image is input into the query model, and the image feature extraction module extracts features from the input training image to obtain multi-scale image features. This process includes:

[0090] In this embodiment of the invention, the image feature extraction module employs different types of deep convolutional networks to extract features from the input training images. The resulting multi-scale feature images are used for subsequent prototype classification and pose estimation. Taking the prototype classification branch as an example, in the prototype classification process, this embodiment of the invention compares and analyzes the accuracy of deep convolutional networks such as ResNet-18, ResNet-50, ResNet-101, DenseNet-121, DenseNet-129, and DenseNet-201 in prototype classification using different evaluation metrics (Top-1 accuracy, Top-3 accuracy, and Top-5 accuracy); the results are shown in Table 1.

[0091] Table 1. Prototype classification accuracy of convolutional networks with different depths on the dataset.

[0092]

[0093] As shown in Table 1, ResNet-50, ResNet-101, and DenseNet-201 all achieve high accuracy in prototype classification. In the training and testing of the models, this embodiment of the invention uses the ResNet-50 backbone network for image feature extraction.

[0094] In one specific implementation of this embodiment, the structural prototype classification module performs structural prototype classification based on multi-scale image features to obtain a prototype category prediction result y, including:

[0095] The structural prototype classification module used in this embodiment of the invention includes a global feature extraction unit and a classification prediction unit. When performing prototype classification on the input training image, the global feature extraction unit performs a global average pooling operation on the deepest feature map in the multi-scale image features output by the image feature extraction module to generate a global feature vector. ;

[0096] The classification prediction unit maps the global feature vector to the log odds value of the structural prototype category, thus obtaining the prototype category prediction result. , where N is the number of structural prototypes.

[0097] In one specific implementation of this embodiment, the shared keypoint prediction module interacts with a set of learnable keypoint query vectors and multi-scale image features to decode and generate coordinate prediction P and visibility prediction V for M keypoints, including:

[0098] The shared keypoint prediction module provided in this embodiment of the invention comprises a keypoint query vector library, a deformable Transformer decoder, a coordinate regression head, and a visibility classification head.

[0099] To achieve keypoint estimation, the embodiment sets up M≥K learnable keypoint (query) embedding vectors. These embedding vectors will learn keypoint shared information across categories and prototypes. Simultaneously, to extract keypoint-related features from the image, a deformable attention mechanism is employed in the Transformer decoder to achieve keypoint-image interaction, and a self-attention mechanism is used to complete keypoint-keypoint interaction. The functions of each component in the module are as follows:

[0100] The module's keypoint query vector library stores M (in this embodiment, M=300) learnable D-dimensional keypoint query vectors. Each keypoint query vector is designed to learn a universal keypoint feature shared across structural prototype categories through training. At the start of training, a fixed set of learnable keypoint query vectors E is initialized from the keypoint query vector library.

[0101] The keypoint query vector E and multi-scale image features are input into a deformable Transformer decoder. The Transformer decoder used in this embodiment comprises three decoder layers, each using keypoint coordinates as reference points. A deformable attention mechanism enables the keypoint query vector E to interact with the multi-scale image features, adaptively focusing on the local image regions corresponding to each keypoint to extract fine-grained features. Simultaneously, a self-attention mechanism enables interaction between keypoint query vectors to model spatial dependencies between keypoints. The state of the query vectors is updated through both the deformable attention mechanism and the self-attention mechanism to obtain a keypoint query vector E' that integrates specific image information features. ;

[0102] Based on the updated keypoint query vector E', the coordinate regression head maps the features of each query vector to 2D normalized coordinates through a multilayer perceptron or linear layer, obtaining the coordinate predictions of M keypoints. P ,in, ;

[0103] Based on the updated keypoint query vector E'The visibility classification head uses a lightweight binary classifier to predict the visibility confidence of each keypoint, resulting in visibility predictions for M keypoints. V ,in, ;

[0104] Finally, the coordinates are predicted. P and visibility prediction V Combine the results to output an unordered prediction set containing M keypoints.

[0105] In one specific implementation of this embodiment, the adaptive matching module is based on the prototype category prediction result y or the actual prototype category label. Obtain the corresponding matching cost matrix, and based on the matching cost matrix, perform adaptive optimal matching between the predicted M keypoints and the K target keypoints of the category to which the training image belongs, outputting the matching mapping relationship, including:

[0106] The adaptive matching module provided in this embodiment of the invention comprises a matching cost matrix storage unit, a matrix retrieval unit, an instant cost calculation unit, a momentum update unit, and an optimal matching solution unit; the functions of each component are as follows:

[0107] During model training, the adaptive matching module receives category label data (prototype category prediction result y and true prototype category label) corresponding to the training image samples. The prototype category prediction result y contains coordinate prediction P and visibility prediction V, and the true prototype category label. Includes real prototype coordinate annotations Visibility annotations of real prototypes .

[0108] The instant cost calculation unit calculates the real prototype annotation data corresponding to the current input training image sample. and And the model prediction data P and V, calculate the instant cost matrix C' that reflects the matching cost of M key points in the training image; taking the calculation of the instant cost matrix of the m-th key point as an example, the calculation method of the instant cost matrix of the m-th key point is shown in equation (1):

[0109] (1)

[0110] In the formula, This represents the instantaneous cost matrix for matching the m-th predicted keypoint to the k-th target keypoint. α This represents the balancing hyperparameters, where P and V represent the coordinates and visibility confidence of the m-th predicted keypoint, respectively. and Let || represent the true coordinates and visibility label of the k-th target keypoint, respectively, and || represent the distance metric. Represents the binary cross-entropy loss function;

[0111] The matrix retrieval unit is based on the true prototype category label of the current sample. The corresponding matching cost matrix is ​​retrieved from the matching cost matrix storage unit. The matching cost matrix is ​​initialized to a zero matrix during the initial training of the model and updated through momentum accumulation in each training iteration.

[0112] The momentum update unit, based on the instantaneous cost matrix C', updates the corresponding matching cost matrix in the matching cost matrix storage unit using the momentum update formula. The matching cost matrix is ​​updated to obtain a more statistically stable and generalizable matching cost matrix C. new Save back to the matching cost matrix storage unit; where the momentum update expression for the matching cost matrix is ​​shown in equation (2):

[0113] (2)

[0114] In the formula, λ represents the momentum coefficient, and C' represents the instantaneous cost matrix;

[0115] The optimal matching solution unit is based on the updated matching cost matrix C. new The Hungarian algorithm is used to solve the problem and output the matching mapping relationship between key points (the k-th target key point will be matched to the k-th target key point). δ (k) predicted keypoints and matching status indicators. Since the number of predicted keypoints (M) is not less than the number of true keypoints (K), the matching status indicators output by the model will have M... K unmatched prediction keypoints.

[0116] In addition, during the key point matching process, a threshold is set to control the key point matching process. Only when the condition of equation (3) is met will the predicted key point corresponding to the kth real key point be determined as valid.

[0117] (3)

[0118] In the formula, T p This represents the distance threshold corresponding to the key point accuracy rate metric. T v This represents the visibility / confidence threshold corresponding to the mean accuracy metric. In this embodiment of the invention, a threshold is set. T p =0.2, and uses 9 visibility thresholds from 0.1 to 0.9.

[0119] In one specific implementation of this embodiment, based on the matching mapping relationship, the prototype classification loss and the regression loss of the matched keypoints are calculated, and the model parameters are optimized based on the comprehensive loss, including:

[0120] This invention optimizes the model based on the Adam optimizer, with the following optimization conditions: 100 training epochs, batch size of 64, and initial learning rate of 1×10⁻⁶. 5 The coefficient decays to one-tenth of its original value at the 60th and 80th rounds. A linear warm-up strategy is used for the first 1000 iterations of training, with a warm-up coefficient set to 0.001.

[0121] The specific process of model training and optimization is as follows:

[0122] Based on the matching mapping relationship output by the adaptive matching module, distinguish between the predicted K matched key points and the predicted MK unmatched key points.

[0123] Prototype classification loss calculation: Compare the model's predicted prototype category y with the true prototype category label. Substituting the values ​​into the cross-entropy loss function, we obtain the prototype classification loss for the training image samples. ;

[0124] Calculation of regression loss for matched key points: For each matched key point, calculate the error between the predicted coordinates and the actual coordinates, as well as the classification error between the predicted visibility and the actual visibility. The regression loss calculation method for the k-th target key point is shown in equation (4):

[0125] (4)

[0126] In the formula, and Let |k| represent the true coordinates and visibility label of the k-th target keypoint, respectively, and |||| represent the distance metric. Indicates the matching mapping relationship. α Indicates the balance hyperparameters, L ce Represents the binary cross-entropy loss function;

[0127] Comprehensive loss calculation: The prototype classification loss and the regression loss of matched key points are weighted and summed according to preset weights to obtain the comprehensive loss. The formula for calculating the comprehensive loss is shown in equation (5):

[0128] (5)

[0129] In the formula, y and These represent the predicted prototype category and the actual prototype category label, respectively. Represents the prototype classification loss. This represents the coordinates of the k-th matched predicted keypoint and the visibility regression loss; β Indicates the equilibrium hyperparameters;

[0130] Based on the calculated comprehensive loss, all trainable parameters in the query model are jointly optimized using the backpropagation algorithm; where all trainable parameters in the model include the parameters of the image feature extraction module, the parameters of the structural prototype classification module, and the parameters of the deformable Transformer decoder.

[0131] In one specific implementation of this embodiment, the test image is input into the optimized query model to obtain the structural prototype category prediction result and keypoint estimation result of the test image, including:

[0132] The test image is input into the query model optimized by the training set, and the structural prototype category prediction result y corresponding to the test image and the coordinate prediction and visibility prediction of M key points are obtained through forward propagation.

[0133] Based on the structural prototype category prediction result y, determine the final structural prototype category of the object in the test image;

[0134] Based on the final structural prototype category, retrieve the corresponding target matching cost matrix from the pre-stored matching cost matrix storage unit;

[0135] Based on the target matching cost matrix, the predictions of M key points are matched and filtered to obtain K ordered key point estimation results corresponding to the final structural prototype category.

[0136] Output the final required structural prototype category and the estimated results of K ordered key points.

[0137] In one specific embodiment of this example, a multi-class pose estimation system based on shared keypoint adaptive matching is provided, comprising:

[0138] Image feature extraction module: used to extract multi-scale features from the input image and output multi-scale image features;

[0139] Structural Prototype Classification Module: Connected to the image feature extraction module, it is used to predict the structural prototype category of objects in the input image based on the multi-scale image features;

[0140] Shared keypoint prediction module: Connected to the image feature extraction module, the module mainly contains a set of learnable keypoint query vectors and a deformable Transformer decoder, which is used to enable the keypoint query vectors to interact with multi-scale image features through the deformable Transformer decoder, so as to decode and output the coordinate prediction and visibility prediction of M keypoints.

[0141] Adaptive matching module: connected to the structural prototype classification module and the shared key point prediction module respectively; the module stores multiple matching cost matrices corresponding to different structural prototype categories, which are used to retrieve the corresponding matching cost matrix according to the input category information, and perform optimal matching between the M key point predictions and the K target key points of the current category based on the matching cost matrix, and output the matching mapping relationship.

[0142] Model optimization module: Connected to the structural prototype classification module, shared keypoint prediction module, and adaptive matching module; used to calculate the prototype classification loss and the regression loss of matched keypoints based on the matching mapping relationship, and to jointly optimize the parameters of all trainable modules in the system based on the comprehensive loss.

[0143] In one specific embodiment of this example, a computer-readable storage medium is also provided, on which a computer program is stored, which is executed by a processor as the steps of the multi-class pose estimation method based on shared keypoint adaptive matching as described in any of the above embodiments.

[0144] To further illustrate the effectiveness of the baseline model designed in this invention, several existing representative attitude estimation methods were adjusted to form an experimental baseline model and compared with the baseline model designed in this invention for attitude recognition. Other attitude estimation methods included SimBase, QueryPose, ViTPose, PCT, EDPose, GroupPose, NerPE, and RTMO. Specifically, for heatmap-based attitude estimation methods, the maximum value of each heatmap was used as the visibility of keypoints. For regression-based attitude estimation methods, an additional dimension was added to each keypoint to predict visibility. During implementation, mAP@0.2 was used as the main evaluation index for each attitude estimation method. The performance of different methods on each superclass and the entire class is shown in Table 2.

[0145] Table 2. Performance of different methods on each superclass and the entire class (mAP@0.2)

[0146]

[0147] As shown in Table 2, the baseline model designed in this invention achieves optimal performance compared to other baseline models. For example, compared to the RTMO method, the method used in this invention improves mAP@0.2 by 8.8 percentage points; for wearable items supercategories, compared to the GroupPose method, the method used in this invention improves mAP@0.2 by 16.1 percentage points. These comparative results demonstrate the effectiveness of the proposed adaptive matching query method for learning cross-keypoint shared information.

[0148] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multi-class pose estimation method based on adaptive matching of shared keypoints, characterized in that, include: S1. Construct a multi-class pose dataset, process the dataset, and obtain training images and test images; S2. Design a query-based model that incorporates matching loss; the query-based model includes an image feature extraction module, a structural prototype classification module, a shared keypoint prediction module, an adaptive matching module, and a model optimization module; S3. Input the training image into the query model, and extract features from the input training image through the image feature extraction module to obtain multi-scale image features; S4. The structural prototype classification module performs structural prototype classification based on multi-scale image features to obtain the prototype category prediction result y. The structural prototype classification module in step S4 includes a global feature extraction unit and a classification prediction unit. The global feature extraction unit performs global average pooling on the deepest feature map of the multi-scale image features output by the image feature extraction module to generate a global feature vector. The classification prediction unit maps the global feature vector to the log odds value of the structural prototype category to obtain the prototype category prediction result y. S5. The shared keypoint prediction module interacts with a set of learnable keypoint query vectors and multi-scale image features to decode and generate coordinate prediction P and visibility prediction V for M keypoints. S6. The adaptive matching module is based on the prototype category prediction result y or the real prototype category label from step S4. Obtain the corresponding matching cost matrix, and based on the matching cost matrix, perform adaptive optimal matching between the M keypoint predictions in step S5 and the K target keypoints of the category to which the training image belongs, and output the matching mapping relationship; wherein, the structural prototype of each image category in the dataset contains at most K real keypoints, and K≤M; S7. Based on the matching mapping relationship, calculate the prototype classification loss and the regression loss of the matched key points, and optimize the model parameters based on the comprehensive loss; S8. Input the test image into the optimized query model to obtain the structural prototype category prediction results and key point estimation results of the test image.

2. The method according to claim 1, characterized in that, The construction of a multi-class pose dataset, and the processing of the dataset to obtain training and test images, includes: Image filtering: From images in a large-scale classification dataset, images of object categories with clear geometric structures are filtered out to form an initial image set; wherein, the large-scale classification dataset includes, but is not limited to, the ImageNet dataset and the UniKPT dataset; Structural prototype annotation: For each semantic category in the initial image set, a structural prototype annotation operation is performed; wherein, the structural prototype annotation operation is based on the geometric alignability criterion, which clusters object images with the same geometric structure into the same structural prototype; Attitude annotation: For each structural prototype divided by structural prototype annotation, the attitude annotation operation is performed, wherein the attitude annotation operation is based on a unified topology to annotate the key point coordinates and visibility of all object instances under each structural prototype; External data integration: Mapping and integrating labeled data from external pose datasets into the structural prototype system established by the structural prototype annotations; Dataset partitioning: Within each structural prototype, object instances are randomly divided into training image subsets and test image subsets according to a preset ratio to form a multi-class pose dataset for model training and evaluation.

3. The method according to claim 1, characterized in that, The image feature extraction module in step S3 performs multi-level feature extraction on the input image based on the backbone network of a deep convolutional neural network, and outputs a multi-scale image feature map containing different semantic levels and spatial resolutions. The multi-scale image feature map is used for subsequent structural prototype classification and key point estimation.

4. The method according to claim 1, characterized in that, The shared keypoint prediction module in step S5 includes a keypoint query vector library, a deformable Transformer decoder, a coordinate regression head, and a visibility classification head. The shared keypoint prediction module interacts with a set of learnable keypoint query vectors and multi-scale image features to decode and generate coordinate predictions P and visibility predictions V for M keypoints, including: A fixed number of learnable keypoint query vectors E are initialized from the keypoint query vector library; wherein, the keypoint query vector library stores M learnable D-dimensional keypoint query vectors. Each keypoint query vector is designed to learn a universal keypoint feature shared across structural prototype categories through training; The keypoint query vector E and multi-scale image features are input into the deformable Transformer decoder. A deformable attention mechanism allows the keypoint query vector E to interact with the multi-scale image features, adaptively focusing on the local image regions corresponding to each keypoint to extract fine-grained features. A self-attention mechanism enables interaction between the keypoint query vectors to model the spatial dependencies between keypoints. The state of the query vectors is updated to obtain a keypoint query vector E' that integrates specific image information features. ; Based on the updated keypoint query vector E', the coordinate regression head maps the features of each query vector to 2D normalized coordinates through a multilayer perceptron or linear layer, obtaining the coordinate prediction P of M keypoints, where... ; Based on the updated keypoint query vector E', the visibility classification head uses a lightweight binary classifier to predict the visibility confidence of each keypoint, resulting in visibility predictions V for M keypoints. ; Coordinate prediction P Combined with visibility prediction V, it outputs an unordered prediction set containing M keypoints.

5. The method according to claim 1, characterized in that, The adaptive matching module in step S6 includes a matching cost matrix storage unit, a matrix retrieval unit, an instant cost calculation unit, a momentum update unit, and an optimal matching solution unit; the adaptive matching module is based on the prototype category prediction result y or the true prototype category label in step S4. Obtain the corresponding matching cost matrix, and based on the matching cost matrix, perform adaptive optimal matching between the M keypoint predictions in step S5 and the K target keypoints of the category to which the training image belongs, outputting the matching mapping relationship, including: The adaptive matching module receives category identification data corresponding to training image samples and a set of M key points to be matched; wherein, the category identification data includes the prototype category prediction result y and the true prototype category label. ; The instant cost calculation unit calculates the real-prototype annotation data of the currently input training image samples. and Given the model prediction data P and V, calculate the instantaneous cost matrix C', which reflects the matching cost of M keypoints in the training image; the m-th keypoint in the instantaneous cost matrix C' is calculated as follows: ,in, This represents the instantaneous cost matrix for matching the m-th predicted keypoint to the k-th target keypoint. Indicates the balance hyperparameters, and Let these represent the coordinates and visibility confidence of the m-th predicted keypoint, respectively. and These represent the true coordinates and visibility label of the k-th target keypoint, respectively. Represents distance metric, Represents the binary cross-entropy loss function; The matrix retrieval unit is based on the true prototype category label of the current sample. The corresponding matching cost matrix is ​​retrieved from the matching cost matrix storage unit, wherein the matching cost matrix is ​​initialized to a zero matrix during the first training and is updated through momentum accumulation in each training iteration; The momentum update unit, based on the instantaneous cost matrix C', updates the corresponding matching cost matrix in the matching cost matrix storage unit using the momentum update formula. The matching cost matrix is ​​updated to obtain the updated matching cost matrix C. new Save back to the matching cost matrix storage unit; where the momentum update expression for the matching cost matrix is: ,in, λ Indicates the momentum coefficient; The optimal matching solution unit is based on the updated matching cost matrix C. new The optimal bipartite graph matching is solved using the Hungarian algorithm, and the matching mapping relationship between key points and the matching status indicator are output.

6. The method according to claim 1, characterized in that, Step S7, which involves calculating the prototype classification loss and the regression loss of the matched keypoints based on the matching mapping relationship, and optimizing the model parameters based on the comprehensive loss, includes: Based on the matching mapping relationship output by the adaptive matching module, distinguish between matched keypoint predictions and unmatched keypoint predictions; Prototype classification loss calculation: Compare the model's predicted prototype category y with the true prototype category label. Substituting into the cross-entropy loss function, we obtain the prototype classification loss for the training image samples. ; Calculation of regression loss for matched keypoints: For each matched keypoint, calculate the error between the predicted coordinates and the actual coordinates, as well as the classification error between the predicted visibility and the actual visibility. The regression loss for the k-th target keypoint is calculated as follows: ,in, and These represent the true coordinates and visibility label of the k-th target keypoint, respectively. Represents distance metric, Indicates the matching mapping relationship. L ce This represents the binary cross-entropy loss function. Indicates the equilibrium hyperparameters; Comprehensive Loss Calculation: The prototype classification loss and the regression loss of matched keypoints are weighted and summed according to preset weights to obtain the comprehensive loss. The formula for calculating the comprehensive loss is as follows: ,in, Represents the prototype classification loss. This represents the coordinates of the k-th matched predicted keypoint and the visibility regression loss; β Indicates the equilibrium hyperparameters; Based on the comprehensive loss, all trainable parameters in the query model are jointly optimized using the backpropagation algorithm; wherein the trainable parameters include the parameters of the image feature extraction module, the parameters of the structural prototype classification module, and the parameters of the deformable Transformer decoder.

7. The method according to claim 1, characterized in that, Step S8 involves inputting the test image into the optimized query model to obtain the structural prototype category prediction result and keypoint estimation result of the test image, including: The test image is input into the optimized query model, and the structural prototype category prediction result y corresponding to the test image, as well as the coordinate prediction and visibility prediction of M key points, are obtained through forward propagation. Based on the structural prototype category prediction result y, determine the final structural prototype category of the object in the test image; Based on the final structural prototype category, retrieve the corresponding target matching cost matrix from the pre-stored matching cost matrix storage unit; Based on the target matching cost matrix, the predictions of M key points are matched and filtered to obtain K ordered key point estimation results corresponding to the final structural prototype category. Output the final structural prototype category and the estimated results of K ordered key points.

8. A multi-class pose estimation system based on shared keypoint adaptive matching, used to perform the method described in any one of claims 1-7, characterized in that, include: Image feature extraction module: used to extract multi-scale features from the input image and output multi-scale image features; Structural Prototype Classification Module: Connected to the image feature extraction module, it predicts the structural prototype category of objects in the input image based on multi-scale image features; Shared keypoint prediction module: Connected to the image feature extraction module, the module mainly contains a set of learnable keypoint query vectors and a deformable Transformer decoder, which is used to enable the keypoint query vectors to interact with multi-scale image features through the deformable Transformer decoder, so as to decode and output the coordinate prediction and visibility prediction of M keypoints. Adaptive matching module: connected to the structural prototype classification module and the shared keypoint prediction module respectively; The module stores multiple matching cost matrices corresponding to different structural prototype categories. It is used to retrieve the corresponding matching cost matrix based on the input category information, and perform optimal matching between the M key points prediction and the K target key points of the current category based on the matching cost matrix, and output the matching mapping relationship. Model optimization module: Connects to the structural prototype classification module, the shared keypoint prediction module, and the adaptive matching module; It is used to calculate the prototype classification loss and the regression loss of the matched key points based on the matching mapping relationship, and to jointly optimize the parameters of all trainable modules in the system based on the comprehensive loss.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-class pose estimation method based on adaptive matching of shared keypoints as described in any one of claims 1-7.

Images