Unsupervised screw image classification and segmentation method based on target saliency mask
By adopting an unsupervised segmentation framework and significance detection technology in screw image processing, combined with detection converter and clustering algorithm, the problem of low screw image segmentation and classification accuracy in the prior art is solved, and efficient and accurate screw image processing is achieved, which is suitable for industrial automation applications.
Patent Information
- Application Number
- CN202411882835.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-19
- Publication Date
- 2025-03-25
AI Technical Summary
The prior art is difficult to accurately segment screw images, resulting in inaccurate classification and relying on a large amount of labeled data, increasing labor costs.
The unsupervised screw image classification and segmentation method based on the target significance mask is used to extract the significance mask through the unsupervised segmentation frame, and feature extraction and segmentation are performed by combining the detection transformer with improved noise-de-anchor frame. Finally, the defect positioning image is mapped to the segmented image for classification.
It realizes high-precision screw image segmentation and classification, reduces dependence on manual annotation and intervention, saves human resources and time costs, and is suitable for real-time detection needs of industrial automation.
Smart Images
Figure CN119693716A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of image processing, and in particular relates to an unsupervised screw image classification and segmentation method based on target saliency mask. Background Art
[0002] In the manufacturing industry, screws are almost the basic elements of mechanical connection of parts of industrial products (including automotive and aviation related products, construction and electronic components, etc.). Therefore, the quality of screws directly affects the development of the manufacturing industry; at the same time, the quality of screws is not only related to the production process, but also to the quality inspection process; since quality inspection requires high precision, quality inspection accounts for a large proportion of the factory labor cost. In order to save costs and improve the production capacity of screws, the intelligentization of industrial quality inspection has become an inevitable trend. For this reason, defect detection plays a key role in the industrial manufacturing of screws. In order to ensure the quality of screws, various defect detection models have been proposed in recent years; however, these models only realize defect detection and defect location to determine whether there are defects in the image; in practice, how to determine the location of defects is very critical in the quality inspection process; the manufacture of screws can be divided into three steps: forging, tapping and plating. The determination of defect location can provide a basis for optimizing subsequent production processes or parameters and reduce production scrap rate.
[0003] Therefore, it is necessary to develop effective methods that can effectively determine the location of defects. Compared with defect detection, defect diagnosis requires further refinement of the semantic information of the image. Image segmentation, as an important computer vision task, plays a vital role in the interpretation of visual content at a fine-grained level. However, involving semantic information usually requires a large amount of pixel-level labeled image data to train the model, which increases the labor cost. To solve this problem, unsupervised image segmentation is one of the most effective methods. Unsupervised semantic segmentation methods aim to use the intrinsic structure of the data, image features, and clustering to achieve pixel classification without relying on labeled data or relying on only a small amount of labeled data. Despite its great progress, most of the existing methods are applicable to natural scenes and not suitable for screws. Due to the large difference in data distribution between natural scene images and industrial product images, natural scene images may contain a lot of background information and complex scenes, while screw images focus more on the product itself, usually with a relatively simple background, but similar features between screw parts. This difference in data distribution makes it difficult for existing segmentation models to accurately segment spiral images. At the same time, spiral images usually require high accuracy, and natural scene segmentation models cannot provide sufficient accuracy. Summary of the invention
[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides an unsupervised screw image classification and segmentation method based on target saliency mask, which solves the problems of low accuracy, poor fitness, difficulty in accurately segmenting screw images and inaccurate classification in processing screw images by existing classification and segmentation methods.
[0005] In order to achieve the above objectives, the technical solution adopted by the present invention is: an unsupervised screw image classification and segmentation method based on target saliency mask, comprising the following steps:
[0006] S1. Obtaining the original screw image, dividing the original screw image into a training set and a test set, converting the screw image in the training set into a grayscale image, and extracting the saliency mask of the screw image in the training set based on an unsupervised segmentation framework to obtain a saliency image;
[0007] S2. Use the detection transformer with improved denoising anchor frame to extract features from the screw image in the test set to obtain test features, and pass the saliency image through the first layer of the detection transformer with improved denoising anchor frame to obtain intermediate features. Use the greedy algorithm and K-means algorithm to process the intermediate features to obtain centroid features.
[0008] S3, calculating the cosine similarity of the test feature and the centroid feature to obtain a preliminary segmentation map, and using the interpolation process to restore the preliminary segmentation map to the original screw image size, and using saliency detection to process the interpolated segmentation map to obtain a segmentation image, thereby completing the segmentation of the screw component;
[0009] S4, using the defect detection model to process the original screw image to obtain a defect location image;
[0010] S5. Map the defect location image to the segmented image to obtain a classified image, and output a component label corresponding to the color of the defective component in the classified image. Use the component label as the name of the defective component to complete the classification of the defective component.
[0011] The beneficial effects of the present invention are as follows: the present invention uses an unsupervised segmentation framework as a benchmark, combines saliency detection and clustering, extracts features and clusters to calculate the centroid and similarity of the centroid comparison features through a saliency detection subject and a detection transformer with an improved denoising anchor frame, and performs segmentation, and maps the defect location image to the segmented image for classification, thereby realizing unsupervised screw image classification and segmentation based on a target saliency mask, effectively dividing the image contour into meaningful fragments or regions, and solving the problems of low accuracy, poor fitness, difficulty in accurately segmenting screw images, and inaccurate classification in processing screw images by existing classification and segmentation methods; the present invention reduces dependence on manual labeling and intervention, saves human resources and time costs, and can more accurately identify and segment the contours of the screws through saliency and feature extraction technology, thereby reducing the situation of missed segmentation and misclassification, and at the same time, the present invention does not require training time, can achieve fast image processing, and meet the real-time requirements of industrial applications.
[0012] Furthermore, the S1 comprises the following steps:
[0013] S101, obtaining an original screw image, dividing the original screw image into a training set and a test set, and converting the screw image in the training set into a grayscale image;
[0014] S102, based on an unsupervised segmentation framework, using an open source computer vision library, mapping the pixel value range of the screw image of the training set to an integer value of a preset interval, returning pixels that do not reach the right endpoint of the preset interval to zero, and performing pixel eight connectivity analysis;
[0015] S103. According to the result of the pixel eight-connectivity analysis, the background of the screw image in the training set is set to black and the main pixels are retained to obtain a saliency image.
[0016] The beneficial effects of the above further scheme are as follows: the present invention introduces traditional segmentation to extract the saliency mask of the screw image, realizes saliency detection, reduces the influence of shadow interference and computational complexity, realizes more accurate identification of target objects, and improves the overall detection accuracy.
[0017] Furthermore, the S2 comprises the following steps:
[0018] S201, inputting the test set screw image into the detection transformer with the improved denoising anchor frame, segmenting the test set screw image to obtain multiple image blocks;
[0019] S202, using a fully connected layer, projecting multiple image blocks into a high-dimensional space through linear projection to obtain a sequence;
[0020] S203, using the transformer model to process the sequence through the self-attention mechanism to obtain test features;
[0021] S204, passing the saliency image through the first layer of the detection transformer with the improved denoising anchor frame to obtain intermediate features;
[0022] S205, performing greedy algorithm sampling on the intermediate features to obtain sampled features, and using the sampled features to construct a memory library;
[0023] S206. Cluster the sampling features in the memory bank using the K-means algorithm to obtain initial centroid features, and perform ordered clustering on the initial centroid features to obtain centroid features.
[0024] The beneficial effects of the above further scheme are as follows: the present invention extracts test features of screw images by using a detection transformer with an improved denoising anchor frame, thereby achieving rapid extraction of intermediate features in real-time detection scenarios without the need for training time, and meeting the needs of industrial automation; utilizing intermediate features, combined with greedy algorithms, K-means clustering, and ordered clustering, can better capture the potential structure of the data, reduce the amount of memory for output storage, and make the clustering results more representative and consistent.
[0025] Furthermore, the S3 comprises the following steps:
[0026] S301, calculating the cosine similarity of the test feature and the centroid feature according to the test feature and the centroid feature to obtain a preliminary segmentation map;
[0027] S302, interpolating the preliminary segmentation map to the size of the original screw image to obtain an interpolated segmentation map;
[0028] S303, performing saliency detection on the interpolated segmentation map to obtain a saliency image of the segmentation map;
[0029] S304, retaining the main segmentation result of the segmentation map according to the saliency image of the segmentation map, and uniformly setting the background part of the saliency image of the segmentation map to the pixel color corresponding to the background label to obtain a segmentation image, thereby completing the segmentation of the screw component.
[0030] The beneficial effect of the above further scheme is: the present invention reduces the information loss and estimation error caused by the interpolation method by calculating the cosine similarity of the test feature and the centroid feature, and obtains the segmentation map by the interpolation method, and performs a second significance processing, thereby realizing unsupervised screw image classification, increasing the model's attention to important features, and further improving the classification effect.
[0031] Furthermore, the S4 comprises the following steps:
[0032] S401, training a defect detection model using a defect-free image, and freezing parameters of the defect detection model to obtain a trained defect detection model;
[0033] S402: Use the trained defect detection model to process the original screw image, and use color to distinguish the severity of the defects to obtain a defect location image.
[0034] Furthermore, the S5 comprises the following steps:
[0035] S501, acquiring a corresponding segmented image according to a defect location image of a current screw image;
[0036] S502, mapping the part with serious defects in the defect location image of the current screw image to the corresponding segmented image to obtain a classified image;
[0037] S503: Output the component label corresponding to the color of the defective component in the classified image, use the component label as the name of the defective component, and complete the classification of the defective component.
[0038] The beneficial effects of the above further scheme are as follows: the present invention generates accurate defect location images through a trained defect detection model, and maps the visual defect location images to the segmented images to complete the classification of defective parts, thereby reducing the need for manual intervention, improving production efficiency, simplifying the data processing process, and making the system easier to deploy and use; at the same time, the accurate classification of defective parts provides an intuitive basis for subsequent quality analysis and decision-making, helping engineers to better understand the causes of defects and make targeted improvements, thereby reducing the scrap rate. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 The figure is a flow chart of the method of the present invention.
[0040] Figure 2 This is the overall flow chart of unsupervised segmentation in this embodiment.
[0041] Figure 3 This is the overall flow chart of unsupervised classification in this embodiment. DETAILED DESCRIPTION
[0042] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.
[0043] Before describing this embodiment, the following terms are explained:
[0044] DINO: Detection Transformer with Improved Denoising Anchor Boxes.
[0045] opencv - open source computer vision library;
[0046] convertScaleAbs: A function in opencv used to linearly transform the pixel values of an image and take the absolute value;
[0047] ScaleAbs processing: linearly transform the pixel values of the image and take the absolute value;
[0048] K-means: K-means algorithm;
[0049] Encoder: Encoder.
[0050] Example
[0051] like Figure 1 As shown, the present invention provides an unsupervised screw image classification and segmentation method based on target saliency mask, and its implementation method is as follows:
[0052] S1. Obtain the original screw image, divide the original screw image into a training set and a test set, convert the screw image in the training set into a grayscale image, and extract the saliency mask of the screw image in the training set based on the unsupervised segmentation framework to obtain a saliency image. The specific steps are as follows:
[0053] S101, obtaining an original screw image, dividing the original screw image into a training set and a test set, and converting the screw image in the training set into a grayscale image;
[0054] S102, based on an unsupervised segmentation framework, using an open source computer vision library, mapping the pixel value range of the screw image of the training set to an integer value of a preset interval, returning pixels that do not reach the right endpoint of the preset interval to zero, and performing pixel eight connectivity analysis;
[0055] S103. According to the result of the pixel eight-connectivity analysis, the background of the screw image in the training set is set to black and the main pixels are retained to obtain a saliency image.
[0056] In this embodiment, based on the unsupervised segmentation framework, a traditional segmentation method is used to extract a saliency mask of a training image. Since industrial product images usually do not require detailed segmentation of the background, but due to the shooting light, shadow interference is inevitable, so traditional segmentation is introduced to realize saliency detection and solve the problem of shadow interference; the grayscale screw images are divided into a training set and a test set, and the screw images have obvious contrast, so the screw images in the training set are converted into grayscale images to enhance their light and dark contrast; then, the training set screw images are ScaleAbs processed, the pixel value range of the image is mapped from an arbitrary range to an integer value between [0,255], and the pixels that are not 255 are uniformly zeroed, and pixel eight connectivity analysis is performed; according to the results generated after the analysis, the background pixels of the original image are set to black, the main pixels are retained, and a saliency image is obtained to complete the segmentation preprocessing.
[0057] S2. Use the detection transformer with improved denoising anchor frame to extract features from the screw image in the test set to obtain test features, and pass the saliency image through the first layer of the detection transformer with improved denoising anchor frame to obtain intermediate features. Use the greedy algorithm and K-means algorithm to process the intermediate features to obtain the centroid features. The specific steps are as follows:
[0058] S201, inputting the test set screw image into the detection transformer with the improved denoising anchor frame, segmenting the test set screw image to obtain multiple image blocks;
[0059] S202, using a fully connected layer, projecting multiple image blocks into a high-dimensional space through linear projection to obtain a sequence;
[0060] S203, using the transformer model to process the sequence through the self-attention mechanism to obtain test features;
[0061] S204, passing the saliency image through the first layer of the detection transformer with the improved denoising anchor frame to obtain intermediate features;
[0062] S205, performing greedy algorithm sampling on the intermediate features to obtain sampled features, and using the sampled features to construct a memory library;
[0063] S206. Cluster the sampling features in the memory bank using the K-means algorithm to obtain initial centroid features, and perform ordered clustering on the initial centroid features to obtain centroid features.
[0064] In this embodiment, Figure 2As shown in the figure, DINO (Detection Transformer with Improved Denoising Anchor Box) is used to extract the main features of the training images; compared with the diversity of images in natural scenes, the distribution of industrial images is relatively fixed, which reduces the requirements for the generalization ability of the model; DINO divides the test set screw images into a series of blocks, linearly projects the blocks into a high-dimensional space through a fully connected layer, and inputs them into the Transformer model (transformer model) as a sequence; the Transformer model processes these sequences through the self-attention mechanism to learn the test features; the salient images obtained from the training set are passed through the first layer of pre-trained DINO to obtain intermediate features; each intermediate feature is sampled by a greedy algorithm to remove redundant features, reduce storage costs, and obtain sampled features; because the dimension of the feature vector is not reduced during the processing, which will affect the final result, the sampled features are clustered; a memory library is constructed using all the sampled features, and the sampled feature vectors in the feature library are clustered using K-means to obtain 3 clusters and centroid features, and the centroid features are clustered in order, that is, sorted, to obtain centroid features.
[0065] S3. Calculate the cosine similarity of the test feature and the centroid feature to obtain a preliminary segmentation map, and use the interpolation process to restore the preliminary segmentation map to the original screw image size, and use saliency detection to process the interpolated segmentation map to obtain a segmentation image, and complete the segmentation of the screw component. The specific steps are as follows:
[0066] S301, calculating the cosine similarity of the test feature and the centroid feature according to the test feature and the centroid feature to obtain a preliminary segmentation map;
[0067] S302, interpolating the preliminary segmentation map to the size of the original screw image to obtain an interpolated segmentation map;
[0068] S303, performing saliency detection on the interpolated segmentation map to obtain a saliency image of the segmentation map;
[0069] S304, retaining the main segmentation result of the segmentation map according to the saliency image of the segmentation map, and uniformly setting the background part of the saliency image of the segmentation map to the pixel color corresponding to the background label to obtain a segmentation image, thereby completing the segmentation of the screw component.
[0070] In this embodiment, the main features of the training image are post-segmented, the cosine similarity between the centroid feature and the test feature is calculated to obtain a preliminary segmentation map, and the preliminary segmentation map is interpolated to the size of the original screw image; because the interpolation process is essentially to fill in the existing pixel values, which may cause the loss and blurring of image details, a saliency detection method is used for post-processing to obtain a saliency image of the segmentation map, the main segmentation result is retained, and the background part is uniformly set to the pixel color corresponding to the background label, thereby obtaining the final segmented image.
[0071] S4. Use the defect detection model to process the original screw image to obtain a defect location image. The specific steps are as follows:
[0072] S401, training a defect detection model using a defect-free image, and freezing parameters of the defect detection model to obtain a trained defect detection model;
[0073] S402: Use the trained defect detection model to process the original screw image, and use color to distinguish the severity of the defects to obtain a defect location image.
[0074] In this embodiment, a preset defect detection model Encoder is used, the defect detection model Encoder is trained using a defect-free image, and the parameters of the defect detection model Encoder are frozen to obtain a trained defect detection model Encoder; the grayscale screw image is processed using the trained defect detection model Encoder, and the severity of the defects is distinguished by color, with the parts with more serious defects being red and the parts with fewer defects or no defects being blue, to obtain a defect location image, that is, a visual image of the defect.
[0075] S5. Map the defect location image to the segmented image to obtain a classified image, and output the component label corresponding to the color of the defective component in the classified image. Use the component label as the name of the defective component to complete the classification of the defective component. The specific steps are as follows:
[0076] S501, acquiring a corresponding segmented image according to a defect location image of a current screw image;
[0077] S502, mapping the part with serious defects in the defect location image of the current screw image to the corresponding segmented image to obtain a classified image;
[0078] S503: Output the component label corresponding to the color of the defective component in the classified image, use the component label as the name of the defective component, and complete the classification of the defective component.
[0079] In this embodiment, Figure 3As shown, according to the defect location image of the current screw image, the corresponding segmented image is obtained, which is mainly segmented into three parts: background, screw head and thread, and each part is defined by different colors and labels; the defect location image of the current screw image is mapped to the corresponding segmented image, and the red area is mapped to a certain part of the segmented image; and the component label is corresponding to the color of the component, and the component label is used as the name of the defective component and output to complete the classification of the defective component.
[0080] In this embodiment, for the task scenario, an unsupervised screw image classification and segmentation method based on target saliency mask is proposed; based on the unsupervised segmentation framework, combined with saliency detection and clustering, the saliency detection subject, the benchmark model extracts features and clusters to calculate the centroid and similarity comparison features and the centroid for segmentation, and the defect location image is mapped to the segmented image for classification;
[0081] The present invention mainly classifies defective parts of industrial manufacturing product images. Since defective sample images of industrial screws are currently scarce and the defect types are unknown, the present invention uses unsupervised learning to construct a model for segmenting screw parts and classifying defective parts.
[0082] The method of the present invention utilizes saliency detection and unsupervised segmentation, and includes five steps: 1. Using a traditional segmentation method to extract a saliency mask of a training image; 2. Using DINO to extract the main features of the training image; 3. Storing the training features in a memory bank for ordered clustering, and assigning a visual color and a label to each centroid; 4. Extracting the features and centroid of a test image, and calculating the similarity thereof, and performing secondary saliency processing at the same time to achieve segmentation; 5. Passing the same test image through a defect detection model to obtain a defect location model, and obtaining the name of the defective component according to the color.
Claims
1. An unsupervised screw image classification and segmentation method based on target saliency mask, characterized by: The following steps are involved: S1. Obtain original screw images, divide the original screw images into a training set and a test set, convert the screw images in the training set into grayscale images, and extract the saliency masks of the screw images in the training set based on an unsupervised segmentation framework to obtain a saliency image; S2. Use the detection transformer with improved denoising anchor frame to extract features from the screw images in the test set to obtain test features. Then pass the saliency image through the first layer of the detection transformer with improved denoising anchor frame to obtain intermediate features. Use the greedy algorithm and K-means algorithm to process the intermediate features to obtain the centroid features. S3. Calculate the cosine similarity between the test feature and the centroid feature to obtain a preliminary segmentation map, and use the interpolation process to restore the preliminary segmentation map to the original screw image size. Then, use saliency detection to process the interpolated segmentation map to obtain a segmented image, thereby completing the segmentation of the screw component. S4, using the defect detection model to process the original screw image to obtain a defect location image; S5. Map the defect location image to the segmented image to obtain a classified image, and output the component label corresponding to the color of the defective component in the classified image. Use the component label as the name of the defective component to complete the classification of the defective component.
2. The unsupervised screw image classification and segmentation method based on target saliency mask according to claim 1 is characterized in that: Said S1 comprises the following steps: S101, obtaining original screw images, dividing the original screw images into a training set and a test set, and converting the screw images in the training set into grayscale images; S102, based on an unsupervised segmentation framework and using an open source computer vision library, mapping the pixel value range of the screw images in the training set to integer values in a preset interval, returning pixels that do not reach the right endpoint of the preset interval to zero, and performing pixel eight connectivity analysis; S103. According to the result of the pixel eight-connectivity analysis, the background of the screw image in the training set is set to black and the main pixels are retained to obtain a saliency image.
3. The unsupervised screw image classification and segmentation method based on target saliency mask according to claim 1 is characterized in that: The S2 comprises the following steps: S201, inputting the test set screw image into the detection transformer with the improved denoising anchor frame, segmenting the test set screw image to obtain multiple image blocks; S202, using a fully connected layer, projecting multiple image blocks into a high-dimensional space through linear projection to obtain a sequence; S203. Using the transformer model, the sequence is processed through the self-attention mechanism to obtain test features. S204, passing the saliency image through the first layer of the detection transformer with the improved denoising anchor box to obtain intermediate features; S205, performing greedy algorithm sampling on the intermediate features to obtain sampled features, and using the sampled features to construct a memory bank; S206. Cluster the sampling features in the memory bank using the K-means algorithm to obtain initial centroid features, and perform ordered clustering on the initial centroid features to obtain centroid features.
4. The unsupervised screw image classification and segmentation method based on target saliency mask according to claim 1 is characterized in that: The S3 includes the following steps: S301, calculating the cosine similarity of the test feature and the centroid feature based on the test feature and the centroid feature to obtain a preliminary segmentation map; S302, interpolating the preliminary segmentation map to the size of the original screw image to obtain an interpolated segmentation map; S303, performing saliency detection on the interpolated segmentation map to obtain a saliency image of the segmentation map; S304 , retaining the main segmentation result of the segmentation image according to the saliency image of the segmentation image, and uniformly setting the background part of the saliency image of the segmentation image to the pixel color corresponding to the background label to obtain a segmentation image, thereby completing the segmentation of the screw component.
5. The unsupervised screw image classification and segmentation method based on target saliency mask according to claim 1 is characterized in that: The S4 comprises the following steps: S401, training a defect detection model using a defect-free image, and freezing parameters of the defect detection model to obtain a trained defect detection model; S402: Use the trained defect detection model to process the original screw image, and use color to distinguish the severity of the defects to obtain a defect location image.
6. The unsupervised screw image classification and segmentation method based on target saliency mask according to claim 1 is characterized in that: The S5 comprises the following steps: S501, obtaining a corresponding segmented image based on a defect location image of a current screw image; S502, mapping the severely defective portion of the defect location image of the current screw image to the corresponding segmented image to obtain a classified image; S503: Output the component label corresponding to the color of the defective component in the classified image, use the component label as the name of the defective component, and complete the classification of the defective component.