An Unsupervised Keypoint Detection Method Based on Graph Attention Networks

By employing an unsupervised keypoint detection method based on graph attention networks, this method leverages the topological structure of graphs for reasoning, addressing the issues of high manual annotation costs and inaccurate detection in complex scenarios in existing technologies. This approach achieves efficient and accurate unsupervised keypoint detection.

CN118229991BActive Publication Date: 2026-11-10INST OF OPTICS & ELECTRONICS CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410397169.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-03
Publication Date
2026-11-10
Estimated Expiration
2044-04-03

AI Technical Summary

Technical Problem

Existing keypoint detection methods require a large amount of manually labeled data and are inaccurate under occlusion and large pose changes, making it difficult to meet the needs of practical applications.

Method used

An unsupervised keypoint detection method based on graph attention networks is adopted. A graph network is built by utilizing the semantic consistency of feature maps. Reasoning is performed through the topological structure of the graph, and the relationship between keypoints is adaptively learned. No manual data annotation is required, and attention weights are dynamically allocated to achieve accurate localization of keypoints.

Benefits of technology

It enables key point detection without manual data annotation, improving the accuracy and robustness of detection, and is suitable for precise positioning in complex scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118229991B_ABST
    Figure CN118229991B_ABST
Patent Text Reader

Abstract

This invention discloses an unsupervised keypoint detection method based on graph attention networks. In complex scenes, such as when the target is partially occluded, deformed, or undergoes large pose changes, it is difficult to accurately detect keypoints. Therefore, this invention introduces graph attention networks into the unsupervised keypoint detection algorithm. On one hand, graph attention networks can adaptively learn the relationships between keypoints. By establishing a graph network with keypoints as nodes, the location of occluded keypoints can be inferred from the graph's topology in complex scenes such as target occlusion, thereby significantly improving the accuracy and robustness of the keypoint detection system. On the other hand, this invention adopts an unsupervised learning approach, which not only greatly reduces the dependence on manually labeled data but is also not limited by target category, making it widely applicable to the localization of various targets and their key parts.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of image processing and key point detection, specifically to an unsupervised key point detection method based on graph attention networks. Background Technology

[0002] Keypoint detection, as an important task in the field of image processing, has significant applications in many fields such as face recognition, autonomous driving, medical imaging, and 3D reconstruction, and has broad application prospects and important research significance.

[0003] Keypoints are essentially iconic feature points in an image, such as eyes and mouth on a face, or corners and edges of an object. The goal of keypoint detection is to automatically locate these significant feature points in an image, thereby aiding in the understanding and analysis of image content and providing more accurate information support for subsequent applications. Traditional keypoint detection methods typically rely on manually designed features and rules, which are only effective in specific, simple scenarios and difficult to generalize to complex ones, thus failing to meet the needs of current practical applications. With the development of deep learning technology, there are now many keypoint detection methods based on supervised learning. These methods can automatically learn feature representations from large amounts of labeled data to locate keypoints. However, these methods require a large amount of manually labeled data, and the labeling of keypoints usually requires the labeler to have professional knowledge and experience, as well as manual checking and correction, resulting in high labeling costs and limiting their widespread application. Compared with supervised learning methods, existing unsupervised learning-based keypoint detection methods have broader application prospects, but also face more challenges and limitations. For example, in cases of target occlusion, large pose changes, or drastic changes in lighting, existing unsupervised keypoint detection methods often struggle to accurately locate the position of target keypoints. Therefore, an unsupervised keypoint detection method is needed to solve the above problems.

[0004] Keypoints are not isolated points; they and their connections can actually be described using a graph in non-Euclidean space. Graph networks can effectively process and analyze graph-structured data, capturing complex structural dependencies between entities, and possess strong flexibility and generalization capabilities. Graph attention networks, in particular, can adaptively learn the feature representations of targets and the relationships between them and their context, eliminating the need for manually designed features. This meets the needs of keypoint detection tasks, and by utilizing the topological structure of the graph built by the network for reasoning, keypoint detection methods can maintain high accuracy and stability even in complex situations such as occlusion and large pose changes. Summary of the Invention

[0005] The purpose of this invention is to provide an unsupervised keypoint detection method based on graph attention networks. This method proposes to use the semantic consistency of feature maps to build a graph network for keypoints, which can be trained without manually labeled data. This method can make full use of context and structural information, and perform reasoning based on the mapping relationship between feature space and physical space and the topological structure of the graph, thereby effectively solving problems such as occlusion and large pose changes of the target, and achieving accurate localization of keypoints.

[0006] The technical solution adopted in this invention is: an unsupervised keypoint detection method based on graph attention networks, comprising the following steps: Step 1: For the input image First, perform a thin plate spline transformation on it to obtain the transformed image. ; Step 2: Use the UNet network to process the images respectively. and Candidate keypoints are extracted as initial keypoint coordinates. First, the UNet network is used to extract the target feature map. Then, upsampling is applied to convert the extracted feature map to high resolution. The network's final output is a multi-channel heatmap, with each channel corresponding to a keypoint. The coordinates of each keypoint are calculated using Soft-argmax. The specific calculation process is as follows: (1) The symbols in formula (1) have the following meanings: The heatmap output by the network Okay, number The value of the column; H: Height of the heatmap; W: Width of the heatmap; Heatmap No. Okay, number The normalized probability value of the column; Predicted keypoint coordinates, including coordinates and coordinate; : Components In the Okay, number The column weight matrix includes Components and Quantity; : Predicted key point coordinates; Step 3: Introduce an attention mechanism module into the encoder part of the UNet network, use the feature map of each channel as an attention node, perform attention calculation, then weight the calculated attention map with the original feature map to obtain the attention-processed feature map, and finally concatenate the attention-processed feature map with the shape information, i.e., the key point coordinates, as the node feature. Step 4: Build a graph model based on a graph attention network for the key points, defining the graph model as follows: ,in Indicates detected One key point, Indicate key points and The connection between them, when the key point and When there is a connection between them, And for all , . Indicates the first The node feature representations of key points include their appearance and shape information. These node features are input into a graph attention network (GAN), which uses them as nodes. The GAN learns the degree of association between nodes and dynamically assigns attention weights based on their similarity. An adjacency matrix is ​​then determined based on these attention weights, thereby enabling the processing of images. and Construct a graph model with key points as nodes; Step 5: Train the model by maximizing the consistency between the two graph models to obtain the final graph model.

[0007] The advantages of this invention compared to the prior art are: (1) This invention does not require a large amount of costly manual annotation data, saving manpower and time costs. It is also a categoryless, universal target key point detection, with a wide range of applications and strong generalization ability.

[0008] (2) The present invention utilizes graph attention network to dynamically learn the relationship between key points, which can make full use of contextual information and structural information to improve the accuracy of key point detection.

[0009] (3) This invention introduces graph networks into the unsupervised key point detection task. It can use the topology of the graph to infer the occluded key points based on the existing key points, thereby effectively solving the problem of inaccurate key point detection when the target is occluded or has a large pose change. This improves the accuracy and robustness of key point detection and enables precise positioning of key points. Attached Figure Description

[0010] Figure 1This is a flowchart of the unsupervised keypoint detection method based on graph attention networks of the present invention; Figure 2 This is a schematic diagram illustrating the method for updating the features of each key point in a graph network. Figure 3 This is a schematic diagram of the established graphical model. Detailed Implementation

[0011] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0012] Figure 1 The flowchart below shows the unsupervised keypoint detection method based on graph attention networks. The specific steps of the unsupervised keypoint detection method based on graph attention networks described in this invention are as follows: Step 1: For the input image First, a thin-plate spline transformation is performed on the image to obtain a transformed image that matches the original image. ; Step 2: Use a convolutional neural network to process the images respectively. and Candidate keypoints are extracted as initial keypoint coordinates. To extract the initial keypoint coordinates, the UNet network is first used to extract the target feature map. Then, upsampling is applied to convert the extracted feature map to high resolution. The network's final output is a multi-channel heatmap, with each channel corresponding to a keypoint. The coordinates of each keypoint are calculated using Soft-argmax. The specific calculation process is as follows: (1) The symbols in formula (1) have the following meanings: The heatmap output by the network Okay, number The value of the column; H: Height of the heatmap; W: Width of the heatmap; Heatmap No. Okay, number The normalized probability value of the column; Predicted keypoint coordinates, including coordinates and coordinate; : Components In the Okay, number The column weight matrix includes Components and Quantity; : Predicted key point coordinates; Step 3: Extract key points and their surrounding appearance information using an appearance encoder. Specifically, the appearance encoder employs a UNet network incorporating an attention mechanism. After the input image is processed by the UNet encoder to extract feature maps, each channel's feature map is used as an attention node for attention calculation. The calculated attention map is then weighted and summed with the original feature map to obtain a attention-processed feature map. This allows for adaptive selection of relevant regions around key points to extract appearance information. Finally, this is concatenated with shape information (the key point location information extracted in Step 2) as node features. The attention calculation formula between the feature maps of each channel is as follows: (2) The symbols in formula (2) have the following meanings: : Feature map of the query in attention; : Feature map as key; : Feature map as value; : The dimension; Step 4: Build a graph model based on a graph attention network for the key points, defining the graph model as follows: ,in Indicates detected One key point, Indicate key points and The connection between them, when the key point and When there is a connection between them, And for all , . express The node feature representation of each keypoint includes its appearance and shape information. The node features F are input into the graph attention network as nodes, such as... Figure 2 The diagram illustrates the method for updating the features of each node in a graph attention network. The graph attention network learns the degree of association between nodes and dynamically assigns attention weights based on the similarity between nodes. The adjacency matrix is ​​determined based on the attention weights, thereby processing the image... and A graph model with keypoints as nodes is constructed. The update formula for the features of each node in the graph attention network is as follows: (3) The symbols in formula (3) have the following meanings: : No. Initial node feature representation of each key point;

[0013] Weight matrix; : No. The key point and the first Attention values ​​representing the feature representations of key points; Weight vector; : No. All neighboring nodes of each key point; : No. The key point and the first The degree of correlation between the key points; splicing operation; There are 3 attention points for each graph. : Non-linear activation function; The first learned by the graph network Node feature representation of each key point; Step 5: Train the graph model by maximizing the consistency between the two graph models to obtain the final graph model. Specifically, KL divergence is used to measure the consistency between the graph models, and the calculation formula is as follows: (4) The symbols in formula (4) have the following meanings: :image The Middle The key point and the first The adjacency probability between key points; :image The Middle The key point and the first The adjacency probability between key points; The consistency loss of a graph model is used to measure the difference in adjacency relationships between two graph models. Figure 3The figure shows a schematic diagram of a graph network built using the human body as an example. As shown, the relationships between key points of the target learned by this method are not limited to the human torso, but include the relationships between key points that do not have physical connections. This method can make full use of the structural dependencies between key points. The graph network built in this way can provide more robust structural information when the target is occluded or undergoes severe deformation, thereby achieving accurate positioning of the target and key parts.

[0014] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of the present invention. Contents not described in detail in this specification are prior art known to those skilled in the art.

Claims

1. An unsupervised keypoint detection method based on graph attention networks, characterized in that, The method includes the following steps: Step 1: For the input image First, perform a thin-plate spline transformation on it to obtain the transformed image. ; Step 2: Use a convolutional neural network to process the images respectively. and Extract candidate keypoints as the initial keypoint coordinates; Step 3: Process the images separately. and Encode the appearance information of key points and fuse it with shape information, i.e., key point coordinates, as node features; Step 4: Process the images separately. and The key point is to establish a graph model based on a graph attention network. The node features from step three are input into the graph attention network as nodes. The graph attention network learns the degree of association between nodes and dynamically assigns attention weights based on the similarity between nodes. The adjacency matrix is ​​determined based on the attention weights, thereby processing the image data. and Construct a graph model with key points as nodes; Step 5: Train by maximizing the consistency between the two graph models to determine the final graph model; In step four, the graphical model is defined as follows: ,in Indicates detected One key point, Indicate key points and The connection between them, when the key point and When there is a connection between them, And for all , , Indicates the first Node feature representation of key points, node features Includes the appearance and shape information of key points; node features Inputting data into a graph attention network (GAN) as nodes allows the network to learn the relationships between nodes and dynamically assign attention weights based on node similarity. The adjacency matrix is ​​then determined based on these attention weights, thus enabling the network to process data from the image. and Construct a graph model with key points as nodes; In step five, the formula for calculating the consistency between the two graphical models is as follows: ,(2) The symbols in formula (2) have the following meanings: :image Key points and The adjacency probability between them; :image Key points and The adjacency probability between them; Consistency loss in graph models is used to measure the difference in adjacency relationships between two graph models.

2. The unsupervised keypoint detection method based on graph attention networks according to claim 1, characterized in that: In step one, the thin-plate spline transformation is an interpolation method that performs specific deformations on the input image to obtain an image similar to the original image. Matching images .

3. The unsupervised keypoint detection method based on graph attention networks according to claim 2, characterized in that: Step two uses the UNet network to extract target features, and then upsampling is used to convert the extracted feature map into a high resolution. The final output of the UNet network is a multi-channel heatmap, with each channel corresponding to a key point. The coordinates of each key point are calculated using Soft-argmax. The specific calculation process is as follows: ,(1) The symbols in formula (1) have the following meanings: The heatmap output by the network Okay, number The value of the column; H: Height of the heatmap; W: Width of the heatmap; Heatmap No. Okay, number The normalized probability value of the column; Predicted keypoint coordinates, including coordinates and coordinate; : Components In the Okay, number The column weight matrix includes Components and Quantity; : Coordinates of the predicted key points.

4. The unsupervised keypoint detection method based on graph attention networks according to claim 3, characterized in that: In step three, an attention mechanism module is introduced into the encoder part of the UNet network. The feature map of each channel is used as an attention node for attention calculation. Then, the calculated attention map is weighted with the original feature map to obtain the attention-processed feature map. Finally, the attention-processed feature map is concatenated with the shape information, i.e., the key point coordinates, as the node feature.