Hinged object function visibility feature recognition method based on K-means clustering feature prompt
By using a method based on K-means clustering feature hints, the problems of ambiguity and high computational complexity in the identification of functional visibility features of articulated objects are solved, and efficient and accurate functional visibility feature identification is achieved in embodied intelligent systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUILIN UNIV OF ELECTRONIC TECH
- Filing Date
- 2026-01-13
- Publication Date
- 2026-04-17
AI Technical Summary
Existing technologies for recognizing the functional visibility features of articulated objects based on single images suffer from problems such as ambiguous recognition results, lack of local semantic consistency modeling, and high computational complexity, making them difficult to deploy effectively in practical embodied intelligence systems.
A method based on K-means clustering feature cues is adopted. Global foreground features are extracted by RepViT image encoder, and point cue embedding vectors are generated by combining position encoding. A coarse heatmap is generated by functional visibility feature decoder and multilayer perceptron. After K-means clustering and adaptive feature masking, two-stage refinement is achieved, and finally a fine functional visibility feature heatmap is generated.
It improves the accuracy and robustness of identifying the functional visibility features of articulated objects, reduces the number of model parameters and computational complexity, and is suitable for deployment in embodied intelligence systems.
Smart Images
Figure CN121883897A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to image recognition technology, specifically to a method for recognizing the functional visibility features of articulated objects based on K-means clustering feature cues. Background Technology
[0002] In daily life, humans can naturally interact with a large number of objects around them and intuitively understand the potential interaction methods inherent in these objects. To enable intelligent agents to possess similar capabilities, they need to be endowed with the ability to perceive the functional visibility features of objects from visual information. Among these, the functional visibility feature heatmap of articulated objects is used to characterize the potential interaction areas of objects in space that can be directly perceived and executed, revealing the intrinsic connection between perception and action, and serving as a crucial foundation for achieving autonomous operation and decision-making by intelligent agents.
[0003] From a two-dimensional image perspective, early research focused primarily on specific types of articulated objects. For example, the 3DAND method identifies the interaction methods of articulated objects through video streams. However, this relies on multi-view interaction processes, has limited input conditions, and lacks fine-grained analysis of functional regions within a single frame image. Subsequently, some studies attempted to directly identify the operable parts of articulated objects from RGB images. For instance, OPD and OPDMulti proposed an openable part detection task to identify multiple openable regions of articulated objects and their motion parameters. However, these methods only focus on operable parts at the structural level and do not model the functional visibility feature heatmap of the object, thus limiting their guidance for agent operation strategies.
[0004] With the development of large-scale visual foundational models, the 3DOI method introduces a pre-trained segmentation model, SAM, to identify the 3D position, physical properties, and functional visibility feature heatmap of objects through a query point mechanism, achieving some progress in task completeness. However, this method is prone to semantic ambiguity and regional redundancy during the recognition of functional visibility feature heatmaps, resulting in a scattered distribution of functional regions and significant recognition ambiguity, affecting the agent's accurate judgment of key interaction regions.
[0005] In addition, some studies have expanded the applicability of object types and interaction scenarios by introducing language descriptions or cross-perspective knowledge transfer mechanisms. However, such methods often focus on high-level semantic generalization and ignore the local semantic consistency of functional visibility features in the visual feature space. This can easily lead to overgeneralization of functional heatmaps and make it difficult to accurately locate the real interactive areas.
[0006] In summary, existing technologies for recognizing the functional visibility features of articulated objects based on single images still have the following shortcomings: First, the results of functional visibility feature heatmap recognition are ambiguous, making it difficult to accurately focus on key interaction areas; second, there is a lack of effective modeling of the local semantic consistency of functional visibility features, resulting in low efficiency in utilizing prompting information; and third, other methods have a large number of model parameters and high computational complexity, which is not conducive to deployment in practical embodied intelligent systems. Therefore, there is an urgent need for a method for recognizing the functional visibility features of articulated objects that can effectively guide and optimize the functional visibility feature heatmap using local semantic features while ensuring computational efficiency, in order to improve the accuracy and robustness of functional region localization. Summary of the Invention
[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for recognizing the functional visibility features of articulated objects based on K-means clustering feature cues. This method improves the accuracy of recognizing the functional visibility features of articulated objects, and its model parameter count and required floating-point computation are both superior to existing methods.
[0008] The technical solution to achieve the objective of this invention is:
[0009] A method for recognizing the functional visibility features of articulated objects based on K-means clustering feature cues includes the following steps:
[0010] 1) First, obtain an indoor scene image containing articulated objects, and select the query point [x,y] as the articulated object whose functional visibility features need to be viewed by simulating a mouse click;
[0011] 2) Extract global foreground feature representations of images using reparameterized depthwise separable convolutional blocks of the RepViT image encoder. The input is a single image, and the output is the global foreground features of the image.
[0012] 3) The query point [x,y] is positionally encoded to generate a point hint embedding vector with spatial awareness. This vector serves as a visibility feature decoder for the hint input function, guiding image features to focus on the region related to the query point and extracting the corresponding local semantic features.
[0013] 4) Obtain the feature query of the functional visibility of articulated objects through the functional visibility feature decoder. The input is the global foreground features of the image and the point cue embedding vector. The output is the feature query of the functional visibility of articulated objects corresponding to each query point.
[0014] 5) Feature query for functional visibility of articulated objects: A coarse functional visibility feature heatmap of articulated objects is obtained through a multilayer perceptron. ;
[0015] 6) Construct an adaptive functional visibility feature prompting module, first by using the global foreground features of the image. And a heatmap of the functional visibility features of a coarse articulated object based on a 2D Gaussian convex point probability distribution. As input; a heatmap of the functional visibility features of a rough, articulated object. Binarization thresholding After feature masking:
[0016] (1),
[0017] In formula (1) It is the foreground feature of the heatmap showing the rough functional visibility characteristics of the articulated object; It is the global foreground feature of the image; It is a heat map of the functional visibility characteristics of a rough, articulated object; It is the binarization threshold;
[0018] 7) Iteratively extract the foreground features of the rough functional visibility feature heatmap of the articulated object using the K-means clustering method. Feature space divided into Each clustered region constitutes a set of component-level functional visibility features. ; Calculate the functional visibility characteristics of each component level With global foreground features of the image Cosine similarity between From this, select the similarity graph index with the highest similarity. In the similarity graph Select the pixel coordinates with the highest similarity. Constitutes a unique positive sample point. :
[0019] (2),
[0020] (3),
[0021] (4),
[0022] (5),
[0023] Formula (2)-Formula (5) Visibility features for each function With global foreground features of the image Cosine similarity between them; It is a component-level functional visibility feature; It is the global foreground feature of the image; The index of the most similar graphs; It refers to the index corresponding to the largest value among N numbers. ; Similarity graph Select the two-dimensional pixel coordinates with the highest similarity. The value represents the maximum similarity among all pixels in the image. Time refers to the maximum similarity map The similarity value of all pixels in the array; It refers to positive sample point hints on a two-dimensional image. The prompt point is defined by two-dimensional coordinates and the prompt's label type ( This indicates that the time is set. (Indicates a negative point warning).
[0024] 10) Through a two-stage functional visibility feature refinement method, the global foreground features of the image are progressively refined and semantically enhanced, ultimately obtaining a refined heatmap of the functional visibility features of the articulated object. ;
[0025] (6),
[0026] (7),
[0027] (8),
[0028] (9),
[0029] Formula (6)-Formula (9), Represents the sparse cue embedding vector and the dense cue embedding vector in the first stage; A point-of-time encoder with point-of-time input; It refers to positive sample point hints on a two-dimensional image; The first phase of refined functional visibility feature heatmap; Decoder representing functional visibility features; Represents the global foreground features of the image; Represents the sparse cue embedding vector and the dense cue embedding vector in the second stage; This refers to the second-stage refined functional visibility feature heatmap; It represents the sigmoid function;
[0030] 12) Training: Training was performed on the NVIDIA A100 graphics processing unit, using parameter training with a learning rate. The AdamW optimizer was trained for 200 epochs with an input / output resolution of 768 / 1024, and the optimal checkpoint was selected based on performance evaluation.
[0031] 13) Apply the trained model to the task of recognizing functional visibility features in indoor scenes containing articulated objects. Given the input image and rough point cues, output a heatmap of the functional visibility features of the articulated objects.
[0032] This technical solution is applied to the functional visibility feature recognition of articulated objects in indoor scenes, and has the following characteristics:
[0033] A functional visibility feature recognition model for articulated objects based on K-means clustering feature cues is provided, aiming to address the difficulty of functional visibility feature recognition for articulated objects in indoor scenes, specifically in the following aspects:
[0034] 1) The results of the functional visibility feature heatmap identification are ambiguous, making it difficult to accurately focus on key functional visibility feature regions;
[0035] 2) The lack of effective modeling of the local semantic consistency of functional visibility features results in low efficiency in the utilization of cue information;
[0036] 3) Other methods have a large number of model parameters and high computational complexity, which is not conducive to deployment in actual embodied intelligence systems.
[0037] This method can improve local spatial understanding, effectively alleviate ambiguity in the identification of functional visibility features, and achieve accurate identification of the functional visibility features of articulated objects. Attached Figure Description
[0038] Figure 1 This is a schematic diagram of the method flow for an embodiment. Detailed Implementation
[0039] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.
[0040] Example:
[0041] 1) First, obtain an indoor scene image containing articulated objects, and select the query point [x,y] as the articulated object whose functional visibility features need to be viewed by simulating a mouse click;
[0042] 2) Extract global foreground feature representations of images using reparameterized depthwise separable convolutional blocks of the RepViT image encoder. The input is a single image, and the output is the global foreground features of the image.
[0043] 3) The query point [x,y] is positionally encoded to generate a point hint embedding vector with spatial awareness. This vector serves as a visibility feature decoder for the hint input function, guiding image features to focus on the region related to the query point and extracting the corresponding local semantic features.
[0044] 4) Obtain the feature query of the functional visibility of articulated objects through the functional visibility feature decoder. The input is the global foreground features of the image and the point cue embedding vector. The output is the feature query of the functional visibility of articulated objects corresponding to each query point.
[0045] 5) Feature query for functional visibility of articulated objects: A coarse functional visibility feature heatmap of articulated objects is obtained through a multilayer perceptron. ;
[0046] 6) Construct an adaptive functional visibility feature prompting module, first by using the global foreground features of the image. And a heatmap of the functional visibility features of a coarse articulated object based on a 2D Gaussian convex point probability distribution. As input; a heatmap of the functional visibility features of a rough, articulated object. Binarization thresholding After feature masking:
[0047] (1)
[0048] In formula (1) It is the foreground feature of the heatmap showing the rough functional visibility characteristics of the articulated object; It is the global foreground feature of the image; It is a heat map of the functional visibility characteristics of a rough, articulated object; It is the binarization threshold;
[0049] 7) Iteratively extract the foreground features of the rough functional visibility feature heatmap of the articulated object using the K-means clustering method. Feature space divided into Each clustered region constitutes a set of component-level functional visibility features. ; Calculate the functional visibility characteristics of each component level With global foreground features of the image Cosine similarity between From this, select the similarity graph index with the highest similarity. In the similarity graph Select the pixel coordinates with the highest similarity. Constitutes a unique positive sample point. :
[0050] (2)
[0051] (3)
[0052] (4)
[0053] (5)
[0054] Formula (2)-Formula (5) Visibility features for each function With global foreground features of the image Cosine similarity between them; It is a component-level functional visibility feature; It is the global foreground feature of the image; The index of the most similar graphs; It refers to the index corresponding to the largest value among N numbers. ; Similarity graph Select the two-dimensional pixel coordinates with the highest similarity. The value represents the maximum similarity among all pixels in the image. Time refers to the maximum similarity map The similarity value of all pixels in the array; It refers to positive sample point hints on a two-dimensional image. The prompt point is defined by two-dimensional coordinates and the prompt's label type. This indicates that the time is set. This indicates a negative point warning;
[0055] 10) Through a two-stage functional visibility feature refinement method, the global foreground features of the image are progressively refined, ultimately obtaining a refined heatmap of the functional visibility features of the articulated object. ;
[0056] (6)
[0057] (7)
[0058] (8)
[0059] (9)
[0060] Formula (6)-Formula (9), Represents the sparse cue embedding vector and the dense cue embedding vector in the first stage; A point-of-time encoder with point-of-time input; It refers to positive sample point hints on a two-dimensional image; The first phase of refined functional visibility feature heatmap; Decoder representing functional visibility features; Represents the global foreground features of the image; Represents the sparse cue embedding vector and the dense cue embedding vector in the second stage; This refers to the second-stage refined functional visibility feature heatmap; It represents the sigmoid function;
[0061] 12) Training: Training was performed on the NVIDIA A100 graphics processing unit, using parameter training with a learning rate. The AdamW optimizer was trained for 200 epochs with an input / output resolution of 768 / 1024, and the optimal checkpoint was selected based on performance evaluation.
[0062] 13) Apply the trained model to the task of recognizing functional visibility features in indoor scenes containing articulated objects. Given the input image and rough point cues, output a heatmap of the functional visibility features of the articulated objects.
[0063] This example focuses on addressing the identification ambiguity problem in optimizing functional visibility maps. The adaptive functional visibility component-level prompting module uses the foreground feature with the highest similarity in the functional visibility heatmap clustering features as a prompt, which improves the computational efficiency of the model and enhances the recognition performance of the functional visibility feature heatmap of articulated objects. This provides assistance for applications such as 3D scene understanding, robot operation, and embodied intelligence.
Claims
1. A method for recognizing the functional visibility features of articulated objects based on K-means clustering feature cues, characterized in that, Includes the following steps: 1) First, obtain an indoor scene image containing articulated objects, and select the query point [x,y] as the articulated object whose functional visibility features need to be viewed by simulating a mouse click; 2) Extract global foreground feature representations of images using reparameterized depthwise separable convolutional blocks of the RepViT image encoder. The input is a single image, and the output is the global foreground features of the image. 3) Encode the query point [x,y] to generate a point cue embedding vector with spatial awareness. This vector serves as a visibility feature decoder for the cue input function, guiding image features to focus on the region related to the query point and extracting the corresponding local semantic features. 4) Obtain the feature query of the functional visibility of articulated objects through the functional visibility feature decoder. The input is the global foreground features of the image and the point cue embedding vector. The output is the feature query of the functional visibility of articulated objects corresponding to each query point. 5) Feature query for functional visibility of articulated objects: A coarse functional visibility feature heatmap of articulated objects is obtained through a multilayer perceptron. ; 6) Construct an adaptive functional visibility feature prompting module, first by using the global foreground features of the image. And a heatmap of the functional visibility features of a coarse articulated object based on a 2D Gaussian convex point probability distribution. As input; a heatmap of the functional visibility features of a rough, articulated object. Binarization thresholding After feature masking: （1）， In formula (1) It is the foreground feature of the heatmap showing the rough functional visibility characteristics of the articulated object; It is the global foreground feature of the image; It is a heat map of the functional visibility characteristics of a rough, articulated object; It is the binarization threshold; 7) Iteratively extract the foreground features of the rough functional visibility feature heatmap of the articulated object using the K-means clustering method. Feature space divided into Each clustered region constitutes a set of component-level functional visibility features. ; Calculate the functional visibility features at each component level. Global foreground features of the image Cosine similarity between From this, select the similarity graph index with the highest similarity. In the similarity graph Select the pixel coordinates with the highest similarity. Constitutes a unique positive sample point. : （2）， （3）， （4）， （5）， Formula (2)-Formula (5) Visibility features for each function Global foreground features of the image Cosine similarity between them; It is a component-level functional visibility feature; It is the global foreground feature of the image; The index of the most similar graphs; It refers to the index corresponding to the largest value among N numbers. ; Similarity graph Select the two-dimensional pixel coordinates with the highest similarity. The value represents the maximum similarity among all pixels in the image. Time refers to the maximum similarity map The similarity value of all pixels in the array; It refers to positive sample point hints on a two-dimensional image. The prompt point is defined by two-dimensional coordinates and the prompt's label type ( This indicates that the time is set. (Indicates a negative point warning). 10) Through a two-stage functional visibility feature refinement method, the global foreground features of the image are progressively refined and semantically enhanced, ultimately obtaining a refined heatmap of the functional visibility features of the articulated object. ; （6）， （7）， （8）， （9）， Formula (6)-Formula (9), Represents the sparse cue embedding vector and the dense cue embedding vector in the first stage; A point-of-time encoder with point-of-time input; It refers to positive sample point hints on a two-dimensional image; The first phase of refined functional visibility feature heatmap; Decoder representing functional visibility features; Represents the global foreground features of the image; Represents the sparse cue embedding vector and the dense cue embedding vector in the second stage; This refers to the second-stage refined functional visibility feature heatmap; It represents the sigmoid function; 12) Training: Training was performed on the NVIDIA A100 graphics processing unit, using parameter training with a learning rate. The AdamW optimizer was trained for 200 epochs with an input / output resolution of 768 / 1024, and the optimal checkpoint was selected based on performance evaluation. 13) Apply the trained model to the task of recognizing functional visibility features in indoor scenes containing articulated objects. Given the input image and rough point cues, output a heatmap of the functional visibility features of the articulated objects.