Vision-based defect unsupervised identification and segmentation method for intelligent manufacturing

Abstract

The invention relates to the technical field of computer vision, and discloses a vision-based defect unsupervised recognition and segmentation method for intelligent manufacturing, which comprises the following steps of: firstly, acquiring a cut image of a segmented cut in an assembly line; then, the collected image is coded, and the specific code is in an embedding vector form; then shadow detection is carried out, and the colors of the processed image and the original image are calculated by adopting CBENet; performing unsupervised clustering model training on the coded training set image; in the training process, after aggregation of an image processing module, a centroid image is obtained, then a binary threshold of OpenCV is used for dividing a centroid into a foreground containing defects and a background containing residual images, and the centroid of the foreground containing the defects is used as a prompt prompt of a basic model; and segmenting the defect. Compared with most existing defect segmentation schemes based on supervision, high-precision defect segmentation in the industrial field is achieved, and real-time detection can be conducted on a product assembly line.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a vision-based unsupervised defect recognition and segmentation method for intelligent manufacturing. Background Art

[0002] In the field of intelligent manufacturing, accurate and real-time defect segmentation is very important to ensure product quality. Especially in the manufacturing fields such as aerospace and automobile, quality control is crucial for safety. Accurately identifying and locating deviations through defect segmentation is an indispensable part of maintenance and error correction in the production process, and is a key step in ensuring high quality.

[0003] Unlike the medical field where computer vision technology is widely used, defects in industrial environments come in various forms and most scenarios lack sufficient labeled samples. This complexity requires detection methods to be more adaptable.

[0004] With the rapid development of artificial intelligence technology, early solutions based on support vector product (SVM) and random forest (RandomForest) are increasingly being replaced by solutions based on deep learning. Recently, based on computer vision, the U-Net-based model is currently a widely used solution for defect segmentation in the industrial field. However, in real-world scenarios, the labeled data in the industrial manufacturing field is often insufficient or even missing, which often makes it impossible to apply the traditional supervised learning training method of U-Net and its variants.

[0005] The machine learning-based solution has the following disadvantages: 1. The generalization is not strong, and it is difficult for models to be universal between different products; Second, many methods require manual setting of abnormal thresholds; 3. The accuracy is lower than that of deep learning-based solutions.

[0006] Supervised solutions such as U-Net have the following disadvantages: 1. It is highly dependent on labeled data, and a lot of labeled samples are often needed, otherwise it is easy to overfit; Second, some models are very complex, with low detection efficiency and poor real-time performance; Therefore, in order to address the above problems, a vision-based unsupervised defect recognition and segmentation method and system for intelligent manufacturing is needed to achieve high-precision detection under the premise of insufficient label data. Summary of the invention

[0007] The purpose of the present invention is to provide a visual-based unsupervised defect recognition and segmentation method for intelligent manufacturing. Compared with most of the current supervised defect segmentation solutions, the present invention achieves high-precision industrial defect segmentation based on the basic model SAM and contextual prompts in scenarios where labeled data is scarce. And with the help of the SAM lightweight decoder, the product assembly line can be detected in real time.

[0008] The present invention is achieved in that: The present invention provides a vision-based unsupervised defect recognition and segmentation method for intelligent manufacturing, which is specifically performed in the following steps: S1: First, in the pipeline, the incision image is collected for the segmented incision; S2: The collected image is then encoded, specifically into an embedding vector form; In step S2, the following steps are specifically performed: S2.1: First, a conv convolution is performed to compress the original feature dimension and obtain the patch embedding, as shown in formula (1): Formula (1); in, I is the input image or patch, K is the weight of the convolution kernel that is gradually updated as the gradient decreases, I mn is the image at position m , n The pixel value of K ijmn is the convolution kernel at position i , j With image position m , n The multiplied value, ij It is the convolution operation at position i , j Result; S2.2: Flatten the feature to get a one-dimensional vector, as shown in formula (2): Formula (2); S2.3: Perform patch embedding and concatenate position information to obtain position embedding, as shown in formula (3): Formula (3); in, PE is a learnable embedding vector that learns the encoding of each position during training. The dimensions and same; S2.4: Position embedding is learned through 12 window partition-based transformer blocks and 4 global attention blocks to learn image features; as follows: ; Among them, Transformer Blocker represents the output after processing by K local attention blocks, and LayerNorm is the layer normalization operation; S2.5: Finally, after two layers of conv convolution, the final image embbeding is obtained, as shown in formula (4): Formula (4); in, Indicates that after two Convolution for dimensionality reduction.

[0009] S3: Then perform shadow detection, and then use CBENet to calculate the color of the processed image and the original image; specifically, perform the following steps: S3.1: Use BGShadowNet to generate a rough shadow removal map for the original image, and then use CBENet to calculate the color of the processed image and the original image, as shown in formula (5): Formula (5); Where A and B are two embedding vectors, ||A|| and ||B|| are the L2 generalizations of the vectors respectively; S3.2: After calculating the similarity between the data in step S3.1 through equations (6) and (7), delete the images with obvious deviations from the training image dataset; Formula (6); Formula (7); in, is the image to be deleted, , is the average feature of the training set images, is the preset threshold.

[0010] S4: performing unsupervised clustering model training on the encoded training set images; Please follow the steps below: S4.1: First, initialize the centroid, select a random training sample from the encoding as the centroid, calculate the distance from each sample encoding vector to the centroid, represented by D(x), and then calculate the probability of each sample being selected as the next centroid. After the probability calculation of all samples is completed, divide the probability into intervals and randomly generate a number between 0 and 1 as the new centroid, as shown in formula (8): Formula (8); S4.2: Assign clusters and update centroids. Calculate the Euclidean distance from each sample encoding vector to all centroids and assign it to the nearest centroid, as shown in formula (9): Formula (9); Among them, x represents the current centroid, y represents the current sample, and d(x,y) represents the distance from x to y; S4.3: Recalculate the centroid of each cluster, as shown in formula (10): Formula (10); Among them, u j is the centroid of the jth cluster, C j is the set of all samples belonging to the jth cluster; S4.4: Repeat the above steps until the centroid encoding vector no longer changes.

[0011] S5: During the training process, after being aggregated by the image processing module, a centroid image is obtained, and then the centroid is divided into a foreground containing defects and a background containing the remaining image using the binary threshold of OpenCV, and these foreground centroids containing defects are used as prompts for the basic model; S6: Segment the defects; the mask of the test set in the training image will be similar to the mask generated by SAM to evaluate the accuracy of the segmentation, as shown in formula (5): Formula (5); Among them, A and B are two embedding vectors, ||A|| and ||B|| are the L2 generalizations of the vectors respectively.

[0012] Furthermore, the present invention provides a vision-based unsupervised defect recognition and segmentation system for intelligent manufacturing, comprising an image processing module, wherein the image processing module is used to encode images; and during the model training process, images containing shadow noise are removed; during the model training process, images in the training set are clustered unsupervisedly; A context prompt information generation module, which is used to generate prompt information for tuning the basic model. During the training process, after being aggregated by the image processing module, a centroid image is obtained, and then the centroid is divided into a foreground containing defects and a background containing the remaining image using the binary threshold of OpenCV, and these foreground centroids containing defects are used as prompts for the basic model; The defect segmentation module generates valid segmentation masks for any given input and hint, which are usually subsets, parts or the entire image with different segmentation accuracy.

[0013] Furthermore, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of a vision-based unsupervised defect recognition and segmentation method for intelligent manufacturing as described in any one of the above are implemented.

[0014] Furthermore, the present invention provides a computer-storable medium, which includes an embedded processing system and a stored program, and when the embedded system control program is running, the flow disturbance generation method based on packet feature inversion is controlled to execute any one of the above-mentioned visual-based unsupervised defect recognition and segmentation methods for intelligent manufacturing.

[0015] Compared with the prior art, the present invention has the following beneficial effects: Compared with most of the current supervised defect segmentation solutions, this invention achieves high-precision industrial defect segmentation based on the basic model SAM and contextual prompts in scenarios where labeled data is scarce. And with the help of SAM lightweight decoder, it can perform real-time detection of product assembly lines. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It is understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a SAM framework diagram based on unsupervised prompting of the present invention; Figure 3 It is a data processing structure diagram of the intelligent manufacturing defect segmentation pipeline of the present invention; Figure 4It is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but is only for selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0019] See also Figure 1-Figure 4 The present invention provides a vision-based unsupervised defect recognition and segmentation method for intelligent manufacturing, which is specifically performed in the following steps: S1: First, in the pipeline, the incision after segmentation is collected. Specifically, the image is collected by a camera, and if necessary, supplementary shooting is performed by lighting to obtain a high-definition image of the collected incision; S2: Then encode the collected image, specifically into the form of embedding vector; specifically, perform the following steps: S2.1: First, after a conv convolution, the original feature dimension is compressed to obtain the patch embedding, as shown in formula (1): Formula (1); in, I is the input image or patch, K is the weight of the convolution kernel that is gradually updated as the gradient decreases, I mn is the image at position m , n The pixel value of K ijmn is the convolution kernel at position i , j With image position m , n The multiplied value, ij It is the convolution operation at position i , j Result; S2.2: Flatten the feature to get a one-dimensional vector, as shown in formula (2): Formula (2); S2.3: Perform patch embedding and concatenate position information to obtain position embedding, as shown in formula (3): Formula (3); in, PE is a learnable embedding vector that learns the encoding of each position during training. The dimensions and same; S2.4: Position embedding is learned through 12 window partition-based transformer blocks and 4 global attention blocks to learn image features; as follows: ; Among them, Transformer Blocker represents the output after processing by K local attention blocks, and LayerNorm is the layer normalization operation; S2.5: Finally, after two layers of conv convolution, the final image embbeding is obtained, as shown in formula (4): Formula (4); in, Indicates dimensionality reduction after two conv convolutions.

[0020] S3: Then perform shadow detection, and then use CBENet to calculate the color of the processed image and the original image; specifically, perform the following steps: S3.1: Use BGShadowNet to generate a rough shadow removal map for the original image, and then use CBENet to calculate the color of the processed image and the original image, as shown in formula (5): Formula (5); Where A and B are two embedding vectors, ||A|| and ||B|| are the L2 generalizations of the vectors respectively; S3.2: After calculating the similarity between the data in step S3.1 through equations (6) and (7), delete the images with obvious deviations from the training image dataset: Formula (6); Formula (7); in, is the image to be deleted, , is the average feature of the training set images, is the preset threshold.

[0021] S4: Perform unsupervised clustering model training on the encoded training set images; specifically, perform the following steps: S4.1: First, initialize the centroid, select a random training sample from the encoding as the centroid, calculate the distance from each sample encoding vector to the centroid, represented by D(x), and then calculate the probability of each sample being selected as the next centroid. After the probability calculation of all samples is completed, divide the probability into intervals and randomly generate a number between 0 and 1 as the new centroid, as shown in formula (8): Formula (8); S4.2: Assign clusters and update centroids. Calculate the Euclidean distance from each sample encoding vector to all centroids and assign it to the nearest centroid, as shown in formula (9): Formula (9); Among them, x represents the current centroid, y represents the current sample, and d(x,y) represents the distance from x to y; S4.3: Recalculate the centroid of each cluster, as shown in formula (10): Formula (10); Among them, u j is the centroid of the jth cluster, C j is the set of all samples belonging to the jth cluster; S4.4: Repeat the above steps until the centroid encoding vector no longer changes.

[0022] S5: During the training process, after being aggregated by the image processing module, a centroid image is obtained, and then the centroid is divided into a foreground containing defects and a background containing the remaining image using the binary threshold of OpenCV, and these foreground centroids containing defects are used as prompts for the basic model; ; Using the centroid as the prompt hint can reduce the extra overhead of selecting the prompt area from each image and does not require any labeled data; S6: Segment the defects; the mask of the test set in the training image will be similar to the mask generated by SAM to evaluate the accuracy of the segmentation, as shown in formula (5): Formula (5); Among them, A and B are two embedding vectors, ||A|| and ||B|| are the L2 generalizations of the vectors respectively.

[0023] In this embodiment, the present invention provides a vision-based unsupervised defect recognition and segmentation system for intelligent manufacturing, including an image processing module, wherein the image processing module is used to encode images; and during the model training process, images containing shadow noise are removed; during the model training process, images in the training set are unsupervisedly clustered; A context prompt information generation module, which is used to generate prompt information for tuning the basic model. During the training process, after being aggregated by the image processing module, a centroid image is obtained, and then the centroid is divided into a foreground containing defects and a background containing the remaining image using the binary threshold of OpenCV, and these foreground centroids containing defects are used as prompts for the basic model; The defect segmentation module generates valid segmentation masks for any given input and hint, which are usually subsets, parts or the entire image with different segmentation accuracy.

[0024] In this embodiment, the present invention provides a second embodiment. The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a vision-based unsupervised defect recognition and segmentation method for intelligent manufacturing as described in any one of the above are implemented.

[0025] In this embodiment, the present invention provides a third embodiment, which provides a computer-storable medium, wherein the computer-readable storage medium includes an embedded processing system and a stored program, and when the embedded system control program is running, the flow disturbance generation method based on packet feature inversion is controlled to execute any one of the above-mentioned visual-based unsupervised defect recognition and segmentation methods for intelligent manufacturing.

[0026] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention has various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A vision-based unsupervised defect recognition and segmentation method for intelligent manufacturing, characterized by: Please follow the steps below: S1: First, in the pipeline, the incision image is collected for the segmented incision; S2: Then encode the collected image, specifically into the form of embedding vector; specifically, perform the following steps: S2.1: First, go through a Convolution compresses its original feature dimension and obtains , as shown in formula (1); Formula (1); in, is the input image or , is the weight of the convolution kernel that is gradually updated as the gradient decreases, is the image at position The pixel value of is the convolution kernel at position With image position The multiplied value, ij Is the convolution operation at position Result; S2.2: Flatten the features Get a one-dimensional vector, as shown in formula (2); Formula (2); S2.3: Conduct Splicing Information obtained , as shown in formula (3); Formula (3); in, is a learnable embedding vector that learns the encoding of each position during training. The dimensions and same; S2.4: After 12 years of , and 4 global Learn image features; S2.5: Finally, after two layers of conv convolution, the final image embbeding is obtained, as shown in formula (4); Formula (4); in, Indicates that after two Convolution dimensionality reduction; S3: Then perform shadow detection, and then compare the processed image with the original image using Calculate its color; specifically follow the steps below: S3.1: Adoption Generate a rough shadow removal map for the original image, and then compare the processed image with the original image using Calculate its color, as shown in formula (5); Formula (5); in, and There are two vector, and are vectors universal number; S3.2: After calculating the similarity between the data in step S3.1 through equations (6) and (7), delete the images with obvious deviations from the training image dataset; Formula (6); Formula (7); in, is the image to be deleted, , is the average feature of the training set images, is the preset threshold; S4: performing unsupervised clustering model training on the encoded training set images; S5: During the training process, after being aggregated by the image processing module, the centroid image is obtained, and then used The centroids are divided into the foreground containing defects and the background containing the remaining image by using the binary threshold of hint; S6: Segment the defects.

2. According to claim 1, a method for unsupervised defect recognition and segmentation based on vision for intelligent manufacturing, characterized in that: In step S4, the following steps are specifically performed: S4.1: First, initialize the centroid, select one of the encodings of the random training sample as the centroid, calculate the distance from each sample encoding vector to the centroid, and use Indicates that the probability of each sample being selected as the next centroid is then calculated. After the probability calculation of all samples is completed, the probability is divided into intervals and a random The number between is taken as the new centroid, as shown in formula (8): Formula (8); S4.2: Assign clusters and update centroids. Calculate the Euclidean distance from each sample encoding vector to all centroids and assign it to the nearest centroid, as shown in formula (9): Formula (9); in, represents the current center of mass, Represents the current sample, express arrive distance; S4.3: Recalculate the centroid of each cluster, as shown in formula (10): Formula (10); in, It is The centroid of the cluster, It belongs to The set of all samples of a cluster; S4.4: Repeat the above steps until the centroid encoding vector no longer changes.

3. The method for unsupervised defect recognition and segmentation based on vision for intelligent manufacturing according to claim 1, characterized in that: In step S6, the masks of the test set in the training image will be compared with The generated mask is used for similarity calculation to evaluate the accuracy of segmentation, as shown in formula (5): Formula (5); in, and There are two vector, and are vectors General number.

Images