A single sample component segmentation method based on adaptive feature selection

Abstract

The application discloses a single sample part segmentation method based on adaptive feature selection, which realizes part segmentation by establishing the corresponding relationship between the context reference sample and the test image by using the feature representation extracted from DINOv2 and SD. The application contains a new adaptive channel selection module ACS to fuse the features of DINOv2 and SD. ACS can enhance the distinguishability of the feature representation without destroying the original generalization performance of DINOv2 and SD. The application does not need to train or fine-tune the model, has the characteristics of strong adaptability and generalization ability and high data efficiency, improves the segmentation performance, and promotes the development of related applications.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a single-sample component segmentation method based on adaptive feature selection. Background Technology

[0002] In the field of computer vision, image semantic segmentation is a crucial task, where each pixel in an image is assigned a class label, facilitating various applications such as medical imaging, image editing, and environmental monitoring. With the development of semantic segmentation, a more granular category of image segmentation has been proposed, called part segmentation. Part segmentation aims to describe individual components of an object, rather than simply segmenting the object as a whole. It plays a vital role in fields such as 3D reconstruction, autonomous driving, medical image analysis, and scene resolution. In industrial design and manufacturing, part segmentation enables the precise segmentation of different parts of an object from scanned or image data, which is essential for reverse engineering, rapid prototyping, and quality control of parts. On automated production lines, accurately identifying and locating different parts of components helps robots perform precise grasping, assembly, and disassembly. In medical imaging such as CT and MRI images, part segmentation helps doctors and researchers identify and measure individual parts of organs, lesion areas, or other anatomical structures. The task of part segmentation is highly complex, primarily due to the different definitions of parts for different object categories and the varying granularity of parts defined for different purposes. Furthermore, obtaining a sufficient amount of labeled data for this task requires significant annotation costs; therefore, it is crucial to research a general and data-efficient method for component segmentation.

[0003] The ongoing development of Visual Foundation Models (VFMs) has revolutionized many computer vision tasks, demonstrating remarkable capabilities across a wide range of tasks. VFMs exhibit significant generalization ability in context learning, making them well-suited for adapting to downstream tasks. For example, SegGPT is a context-based semantic segmentation model that allows reasoning about new objects using only a single segmented example as a reference. However, these methods still heavily rely on labeled data for training. Current work also focuses on exploring the part segmentation capabilities of specific VFMs. For instance, SLiMe utilizes a Stable Diffusion model (SD) to localize partial regions by learning cue embeddings for each part from one or a few annotated examples. Nevertheless, these methods still rely on training from a single labeled example, which can lead to overfitting and compromise the model's generalization ability. Therefore, SLiMe struggles with test samples where there are significant appearance and viewpoint differences between the training and training samples, as well as test samples where objects are partially visible. These challenges have prompted researchers to explore a training-free paradigm that relies on a single contextual segmentation example for accurate part segmentation with superior generalization capabilities. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a One-shot In-context Part Segmentation (OIParts) method based on adaptive feature selection. This method achieves part segmentation by establishing a correspondence between contextual reference examples and test images using feature representations extracted from DINOv2 (Dual-Stage Implicit Object-Oriented Network) and SD. This invention includes a novel Adaptive Channel Selection (ACS) module to fuse features from DINOv2 and SD. ACS enhances the discriminability of feature representations without compromising the original generalization performance of DINOv2 and SD. This invention requires no model training or fine-tuning, exhibits strong adaptability and generalization capabilities, and high data efficiency, while simultaneously improving segmentation performance and promoting the development of related applications.

[0005] The technical solution adopted by this invention to solve its technical problem is as follows:

[0006] Step 1: Construct the Adaptive Channel Selection (ACS) module and extract semantic features from the reference and test images using DINOv2. and Semantic features of the reference and test images were extracted using SD. and

[0007] Step 2: and Input to ACS to get the selected channel L dino and L sd ;

[0008] Step 3: Put and According to L dino and L sd Select channels and splice them to obtain F r and F t ;

[0009] Step 4: Calculate F r and F t The similarity between pixels is analyzed and labels are merged. Then, the mask is refined using a Fast Two-Sided Solver (FBS). The entire process is shown in the following equation:

[0010]

[0011] Where <·> is the dot product symbol, and ‖·‖ is the modulo symbol, s i,j This represents the cosine similarity between pixel i in the test image and pixel j in the reference image, where γ represents the scaling hyperparameter. s i The p largest values ​​in the range, where softmax is the normalization function. The normalized similarity weights, The reference image serves as a segmentation mask on component c, Dilate is the image dilation function, and FBS is the Fast Bilateral Solver. Step 5: For each segmented part, according to the prediction mask of the test image on part c. The foreground pixel areas are sorted in descending order, and then the prediction masks with larger pixel areas are successively covered by the prediction masks with smaller pixel areas from the upper layers, thus obtaining the final prediction mask M. t .

[0012] Preferably, the adaptive channel selection module ACS is specifically as follows:

[0013] Step 1: Calculation and The mean variance of the foreground and background regions on each segmented component and

[0014] Step 2: For and Sort in ascending order, then select. and The new feature is formed by the channel feature maps corresponding to the first k minimum values. and

[0015] Step 3: Calculate features and Average feature vector in the foreground and background regions and Cluster centers were used as foreground and background clusters, and cosine similarity was used to cluster the features to obtain the clustering results.

[0016] Step 4: Obtain the optimal number of channels using the following formula:

[0017]

[0018] Step 5: Center front The channel index corresponding to the minimum value in the original feature is the channel selection result when segmenting component c. Similarly, we can obtain

[0019] The beneficial effects of this invention are as follows:

[0020] The method of this invention can adaptively fuse semantic features extracted by DINOv2 and SD based on different segmentation components, and enhances the discriminability of feature representations without compromising the original generalization performance of DINOv2 and SD. Furthermore, this method requires no model training or fine-tuning; inference can be performed using only a reference image, offering advantages such as strong adaptability, high generalization ability, and high data efficiency. Attached Figure Description

[0021] Figure 1 This is an overall structural diagram of the present invention.

[0022] Figure 2 This is a structural diagram of the adaptive channel selection module (ACS) of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] This invention aims to address some of the problems existing in related technologies by proposing a single-sample part segmentation method based on adaptive feature selection. This invention aims to unleash the full potential of Visual Feature Models (VFMs) in part segmentation tasks by establishing a correspondence between contextual reference samples and test images using feature representations extracted from VFMs. To enhance the fine-grained representation of each component of the segmented object, this invention utilizes the complementary advantages of two different VFMs: DINOv2 and Stable Diffusion Model (SD). DINOv2 effectively captures dense visual feature representations, which is crucial for accurate part association, while SD enables the perception of global structural information.

[0025] The Adaptive Channel Selection (ACS) module designed in this invention adaptively selects channels that improve the compactness within part classes based on a given context reference example, creating a unique feature representation for each segmented part class. This enhances the discriminability of the feature representation without compromising the original generalization performance of DINOv2 and SD. During the inference phase, this invention can calculate pixel-level similarity between the provided context segmentation example and the test image based on the fused features obtained by ACS to perform precise fine-grained part segmentation. This process does not require any training or fine-tuning of the model. Furthermore, ACS can flexibly and effectively select the most relevant features of the part to be segmented from different context examples, providing adaptability for different segmentation objects. Therefore, this invention features strong adaptability and generalization ability, as well as high data efficiency.

[0026] The overall process is as shown in the instruction manual. Figure 1 As shown in the figure, the reference image is first processed by DINOv2 and the StableDiffusion model (SD) to extract features. and If we perform concatenation along the direct channel dimension at this point, we can obtain F. r ∈R H′×W′×D After channel selection using Adaptive Channel Selection (ACS), the selected fusion features can be obtained. L in the picture c This is the channel selection result of ACS for component c. (x in the figure) c and y represents the foreground feature vector set and background feature vector set of segmentation component c, respectively. c and Let F represent the foreground feature vector set and background feature vector set of the segmented component c after channel selection, respectively. It can be seen that the intraclass features are more compact after feature selection. During the test inference phase, both the reference and test images were used for feature extraction using DINOv2 and SD, and channel selection was performed using L obtained from ACS, resulting in the selected fused features F. r and F t For each segmented component, feature F is calculated separately. r and F t The similarity, and then for F t Each pixel i in F r Find the top p pixels with the highest similarity and merge their corresponding labels to obtain the predicted label of i.

[0027] Adaptive Channel Selection Module (ACS)

[0028] The process of Adaptive Channel Selection (ACS) is as follows: Figure 2 As shown, the reference image is first processed by feature extractors (DINOv2 and SD). Then, each segmentation component c is sorted according to the variance of the feature map on each channel. The first k channels with the smallest variance after sorting are selected to form new features. Then, the average feature vector of each pixel in the foreground and background regions is calculated as the cluster center and clustering is performed to obtain the desired features. Maximize by the following formula and The final channel selection result is obtained by comparing the intersection and union of the two channels.

[0029]

[0030] Example:

[0031] First, we need to use ACS to obtain the channel selection results of each segmentation component based on the reference image and the reference segmentation mask. For ease of description, we will take component c as an example below. Other components will be selected in the same way.

[0032] Step 1: Place the reference image I r ∈R H×W×3 Input into DINOv2, use DINOv2 to extract features and perform L2 regularization to obtain... Reference image I r In the input SD, features are extracted using SD and then L2 regularized to obtain...

[0033] Step 1-1: Apply the segmentation mask to segmentation component c. Modify the shape using bilinear interpolation. Remember the characteristics The feature map on channel d is G. d ∈R H′×W′ Calculate G d Chinese correspondence Variance of all values ​​in the foreground location region Calculate G d Chinese correspondence Variance of all values ​​in the background location region Describe the mean variance Calculation features In the mask Mean variance across all channels under the action Similarly, the features can be obtained. In the mask Mean variance across all channels under the action

[0034] Steps 1-2: First, for and Sort in ascending order, then select. The first k minimum values ​​corresponding to The channel feature maps form new features Calculation features exist The average eigenvector of the foreground location region Calculation features exist Average feature vector of the background location region Then put and Cluster centers were used as foreground and background, and cosine similarity was used to evaluate the features. Perform clustering to obtain clustering results Then calculate the optimal value of k according to the following formula.

[0035]

[0036] Where IoU represents the crossover-union ratio, and δ is the minimum selection rate, which is set to 0.85 in practice. Center front The minimum value corresponds to The index of the channel feature map is the channel selected on the DINOv2 feature when segmenting component c. Similarly, the channel selected on the SD feature when segmenting component c can be determined.

[0037] Step 2: After obtaining the channel selection results for each component, for each segmented component, fuse the DINOv2 features and SD features of the reference image and the test image respectively. Note that the channel selection of the test image and the reference image are exactly the same. For ease of description, the following takes segmented component c as an example. The specific process is as follows:

[0038] Step 2-1: First, test image I t ∈R H×W×3 Input into DINOv2, use DINOv2 to extract features and perform L2 regularization to obtain... Test image I t In the input SD, features are extracted using SD and then L2 regularized to obtain...

[0039] Step 2-2: Channel selection for image features. First, select the channels for each feature... and Selected from The selected features are obtained from the corresponding channel feature maps. and Then, respectively in the features and Selected from The selected features are obtained from the corresponding channel feature maps. and Then, the DINOv2 and SD features of the reference image and the test image are fused according to the following formula.

[0040]

[0041] Concat is a concatenation function that concatenates the selected DINOv2 and SD features along the channel dimension to obtain fused features for the reference image of segmented component c. Fusion features of the test image

[0042] Step 3: Fuse features F based on the reference image r Fusing features with test images t and reference segmentation mask M r To obtain the segmentation results of the test image for each component, taking the segmentation of component c as an example, the specific process is as follows:

[0043] Step 3-1: First, create a segmentation mask for component c. Modify the feature map size using bilinear interpolation. Cosine similarity was then used to measure features. and The semantic similarity between pixels is calculated using the following formula. The feature vector of each pixel and The feature vector of each pixel in Cosine similarity between them:

[0044]

[0045] Where s i,j express and Cosine similarity between them.

[0046] Step 3-2: For features Each pixel in In features Select the p pixels with the highest cosine similarity from the given image, and denote their similarity values ​​as p. Calculate according to the following formula In features Similarity score between the top p pixels with the highest cosine similarity.

[0047]

[0048] γ represents the scaling hyperparameter, which is set to 0.1 during implementation, and p is set to 9 during implementation.

[0049] Step 3-3: According to the following formula, The corresponding labels are merged to predict each pixel in the test image. This yields the predicted segmentation mask for the test image.

[0050]

[0051] Steps 3-4: Apply the prediction mask Use bilinear interpolation to modify the size to the input image size. Then to Perform a dilation operation, setting the size of the dilation kernel to ε (6 in practice), and then apply the dilated prediction mask. The segmentation mask is refined using the Fast Bilateral Solver (FBS), as shown in the following equation.

[0052]

[0053] Step 4: After obtaining the prediction mask for each segmentation component, calculate the foreground pixel area of ​​each prediction mask and sort them in descending order. The lower the segmentation mask, the larger the foreground pixel area. Then, start from the upper layer prediction mask with smaller pixel area and cover the lower layer prediction mask with larger pixel area. In this way, merge the prediction masks of all segmentation components onto one mask to obtain the final prediction mask M. t ∈R H×W .

Claims

1. A single-sample component segmentation method based on adaptive feature selection, characterized in that, Includes the following steps: Step 1: Construct the Adaptive Channel Selection (ACS) module and extract semantic features from the reference and test images using DINOv2. and Semantic features of the reference and test images are extracted using SD. and ; Step 2: and Input into ACS to get the selected channel and ; Step 3: Put , and according to and Select channels and stitch them together to obtain and ; Step 4: Calculation and The similarity between pixels is analyzed and labels are merged. Then, the mask is refined using a Fast Two-Sided Solver (FBS). The entire process is shown in the following equation: in, The dot product symbol. To find the modulus sign, Represents the number of pixels in the test image. Cosine similarity between them Represents the scaling hyperparameter. express The largest One value, For normalization function, The normalized similarity weights, For reference image in component The segmentation mask on top, The image dilation function is... It is a Fast Bilateral Solver. For test images on parts Predictive mask on; Step 5: For each segmented component, according to The foreground pixel areas are sorted in descending order, and then the prediction masks with larger pixel areas are successively covered by the prediction masks with smaller pixel areas from the upper layers, thus obtaining the final prediction mask. ; The adaptive channel selection module (ACS) is as follows: Step 1: Calculation and The mean variance of the foreground and background regions on each segmented component and ; Step 2: For and Sort in ascending order, then select. and Center front The channel feature maps corresponding to the minimum values ​​form a new feature. and ; Step 3: Calculate features and Average feature vector in the foreground and background regions and Cluster centers were used as foreground and background clusters, and cosine similarity was used to cluster the features to obtain the clustering results. , ; Step 4: Obtain the optimal number of channels using the following formula: ； Step 5: Center front The channel index corresponding to the minimum value in the original feature is the segmentation component. Channel selection results at time Similarly, we can obtain .

Images