SAM-based few-shot industrial anomaly segmentation deep learning method and device and terminal

Abstract

The application discloses a few-sample industrial anomaly segmentation deep learning method and device based on SAM and a terminal, and the method comprises the following steps: acquiring a query image and a support image, and extracting multi-scale features of the query image and the support image; calculating cosine similarity of the multi-scale features of the query image and the multi-scale features of the support image, and obtaining an L2 distance map according to the cosine similarity; acquiring a prototype support, combining the prototype support with the L2 distance map, and obtaining a rough segmentation hint; inputting the rough segmentation hint into a hint encoder, and generating a segmentation result of the query image; and the few-sample industrial anomaly segmentation deep learning method based on SAM solves the technical problem that the existing industrial image anomaly segmentation method cannot segment an industrial image anomaly area under a few-sample condition.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing technology, and in particular to a few-shot industrial anomaly segmentation deep learning method, apparatus and terminal based on SAM. Background Technology

[0002] In industrial product manufacturing, surface defects are often a significant factor affecting product quality and performance. By timely and accurately detecting and identifying surface defects and taking appropriate measures to repair or eliminate defective products, product quality can be improved, production costs reduced, and customer satisfaction enhanced.

[0003] Manual defect detection was once the mainstream method, but it is inefficient and has been gradually replaced by visual detection algorithms. Although deep learning has achieved great success in computer vision in recent years, the application of the model requires a large amount of labeled data to train the model. However, data labeling is a time-consuming and laborious task, and defect samples are rare in industrial production. This has brought great difficulties to the application of deep learning in industrial anomaly segmentation.

[0004] Therefore, existing industrial image anomaly segmentation methods still have the technical problem of being unable to segment abnormal regions in industrial images with a small number of samples. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that, in view of the defects of the prior art, the present invention provides a deep learning method, device and terminal for industrial anomaly segmentation based on SAM with few samples, so as to solve the technical problem that existing industrial image anomaly segmentation methods are unable to segment abnormal regions of industrial images with few samples.

[0006] The technical solution adopted by this invention to solve the technical problem is as follows:

[0007] In a first aspect, the present invention provides a few-shot industrial anomaly segmentation deep learning method based on SAM, comprising:

[0008] Obtain the query image and support images, and extract the multi-scale features corresponding to the query image and support images respectively;

[0009] Calculate the cosine similarity between the multi-scale features of the query image and the multi-scale features of the support image, and obtain the L2 distance map based on the cosine similarity;

[0010] Obtain prototype support and combine it with the L2 distance map to obtain coarse segmentation hints;

[0011] The coarse segmentation prompt is input into the prompt encoder to generate the segmentation result of the query image.

[0012] In one implementation, the step of acquiring the query image and support images, and extracting multi-scale features corresponding to the query image and support images respectively, includes:

[0013] Obtain the query image and the supporting image;

[0014] The query image and the support image are input into a convolutional neural network, and convolutions at different scales are performed on the same feature layer to extract multi-scale features corresponding to the query image and the support image, respectively.

[0015] In one implementation, calculating the cosine similarity between the multi-scale features of the query image and the multi-scale features of the support image, and obtaining an L2 distance map based on the cosine similarity, includes:

[0016] The multi-scale features of the query image and the multi-scale features of the support image are normalized.

[0017] Calculate the cosine similarity between features of the same scale between the query image and the supporting images;

[0018] The corresponding point in the query image is replaced with the image patch in the support image with the highest cosine similarity, resulting in a new support image.

[0019] In one implementation, the step of replacing the corresponding point in the query image with the image patch in the support image that has the highest cosine similarity to obtain a new support image includes:

[0020] Calculate the L2 distance between the feature points of the query image and the feature points of the new supporting image;

[0021] The L2 distance is normalized to obtain multiple L2 distance maps of the query image and the support image;

[0022] The obtained L2 distance maps are merged, and features at different scales are fused and decoded through a hierarchical decoding structure to obtain L2 distance maps at different scales.

[0023] In one implementation, merging the obtained multiple L2 distance maps includes:

[0024] For L2 distance maps of the same scale in different feature layers, the number of channels is superimposed by splicing, and the channels are merged by a convolutional network;

[0025] For L2 distance maps of the same scale in different support maps, average fusion is performed, and then a deeper layer of decoding is performed through a convolutional neural network.

[0026] In one implementation, obtaining prototype support and combining the prototype support with the L2 distance map to obtain coarse segmentation hints includes:

[0027] Multi-layer image features are extracted from the query image using an image encoder;

[0028] The image features are input into the prototype generator to obtain the prototype support;

[0029] The prototype support is combined with L2 distance maps of different scales and decoded to obtain the coarse segmentation hint.

[0030] In one implementation, the step of inputting the coarse segmentation cue into the cue encoder to generate the segmentation result of the query image includes:

[0031] The coarse segmentation prompt is input into the prompt encoder of SAM;

[0032] The SAM segments the abnormal regions of the query image based on the coarse segmentation prompts and generates the segmentation result of the query image.

[0033] Secondly, the present invention provides a few-shot industrial anomaly segmentation deep learning device based on SAM, comprising:

[0034] The acquisition module is used to acquire query images and support images, and extract multi-scale features corresponding to the query images and support images respectively;

[0035] The calculation module is used to calculate the cosine similarity between the multi-scale features of the query image and the multi-scale features of the support image, and to obtain an L2 distance map based on the cosine similarity.

[0036] The module is used to obtain prototype support, and the prototype support is combined with the L2 distance map to obtain coarse segmentation hints;

[0037] The segmentation module is used to input the coarse segmentation prompts into the prompt encoder to generate the segmentation result of the query image.

[0038] Thirdly, the present invention also provides a terminal, comprising: a processor and a memory, wherein the memory stores a SAM-based few-shot industrial anomaly segmentation deep learning program, wherein the SAM-based few-shot industrial anomaly segmentation deep learning program, when executed by the processor, is used to implement the operation of the SAM-based few-shot industrial anomaly segmentation deep learning method as described in the first aspect.

[0039] Fourthly, the present invention also provides a computer-readable storage medium storing a SAM-based few-shot industrial anomaly segmentation deep learning program, which, when executed by a processor, is used to implement the operation of the SAM-based few-shot industrial anomaly segmentation deep learning method as described in the first aspect.

[0040] The present invention, by employing the above technical solution, has the following effects:

[0041] This invention leverages the powerful feature extraction capabilities of the SAM backbone network, combines cosine similarity to calculate the similarity between abnormal and normal product images, and utilizes the calculated L2 distance feature map. This distance feature map is then combined with abnormal image prototype support obtained from a prototype generation network to generate a coarse segmentation mask. This coarse segmentation mask guides SAM in segmenting abnormal regions of abnormal product images. Defects are segmented by comparing positive and negative samples. The invention demonstrates good generalization ability even on new products for which the model has not been trained. This addresses the technical problem of existing industrial image anomaly segmentation methods being unable to segment abnormal regions of industrial images with limited samples. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0043] Figure 1 This is a flowchart of a few-shot industrial anomaly segmentation deep learning method based on SAM in one implementation of the present invention.

[0044] Figure 2 This is a schematic diagram comparing the original SAM and the SAM-based few-sample segmentation in industrial defect segmentation in one implementation of the present invention.

[0045] Figure 3 This is a schematic diagram of a few-shot industrial anomaly segmentation deep learning network based on SAM in one implementation of the present invention.

[0046] Figure 4 This is a schematic diagram illustrating the feature extraction process of the feature extraction module in one implementation of the present invention.

[0047] Figure 5 This is a schematic diagram of the multi-scale L2 distance map fusion process in one implementation of the present invention.

[0048] Figure 6 This is a schematic diagram of the prototype generator extracting prototype support process in one implementation of the present invention.

[0049] Figure 7 This is a schematic diagram of a multi-scale L2 distance map and supporting prototype feature fusion in one implementation of the present invention.

[0050] Figure 8 This is a schematic diagram of the multi-scale feature decoding process in one implementation of the present invention.

[0051] Figure 9 This is a functional schematic diagram of the terminal in one implementation of the present invention.

[0052] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] This invention provides a few-shot industrial anomaly segmentation deep learning method, apparatus, and terminal based on SAM. To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0054] Exemplary methods

[0055] In industrial product manufacturing, surface defects are often a significant factor affecting product quality and performance. Timely and accurate detection and identification of surface defects, along with appropriate measures to repair or eliminate defective products, can improve product quality, reduce production costs, and enhance customer satisfaction. Manual defect detection was once the mainstream method, but its inefficiency has led to its gradual replacement by visual inspection algorithms. While deep learning has achieved great success in computer vision in recent years, its application requires a large amount of labeled data for training. Data labeling is a time-consuming and labor-intensive task, and defect samples are rare in industrial production, posing significant challenges to the application of deep learning in industrial anomaly segmentation.

[0056] To address the aforementioned technical problems, this invention provides a deep learning method for few-shot industrial anomaly segmentation based on SAM. Leveraging the powerful feature extraction capabilities of the SAM backbone network, it combines cosine similarity to calculate the similarity between abnormal and normal industrial images. Utilizing the calculated L2 distance feature map, it combines the distance feature map with anomaly image prototype support obtained from a prototype generation network to generate a coarse segmentation mask. This coarse segmentation mask guides SAM to segment abnormal regions in abnormal industrial images. By comparing positive and negative samples, defects are segmented. The method demonstrates good generalization ability even on new products for which the model has not been trained, thus solving the technical problem that existing industrial image anomaly segmentation methods cannot segment abnormal regions in industrial images with limited samples.

[0057] like Figure 1 As shown, this embodiment of the invention provides a few-shot industrial anomaly segmentation deep learning method based on SAM, including the following steps:

[0058] Step S100: Obtain the query image and the support image, and extract the multi-scale features corresponding to the query image and the support image respectively.

[0059] In this embodiment, the SAM-based few-shot industrial anomaly segmentation deep learning method is applied to a terminal, which includes, but is not limited to, devices such as computers and mobile terminals; the terminal is equipped with a training platform for the SAM-based few-shot industrial anomaly segmentation deep learning method.

[0060] In this embodiment, as Figure 2 As shown, (a) shows the original SAM model in industrial defect segmentation, which requires the addition of relevant prompts to function, and (b) shows a model that can automatically segment industrial defects, generated by the present invention based on the SAM model.

[0061] Specifically, such as Figure 3As shown, the powerful pre-trained image encoder based on SAM (Segment Anything Model) extracts rich multi-scale and multi-level features from the query image and support images through a specific feature extraction network structure. Then, the similarity calculation module calculates the similarity between each image patch in the query image and the image patch in the support image. The image patch in the support image that is most similar to the query image is replaced in the query image to form a new support image that is approximately defect-free. L2 distance calculation is performed to obtain multi-scale distance maps. By combining these distance maps with the prototype support provided by the prototype generator, a coarse prediction mask is decoded with an approximate U-net structure and sent to the prompt encoder of SAM. By utilizing the original powerful segmentation capability of SAM, SAM is guided to segment industrial anomalies and defects.

[0062] Specifically, in one implementation of this invention, step S100 includes the following steps:

[0063] Step S101: Obtain the query image and the supporting image;

[0064] Step S102: Input the query image and the support image into the convolutional neural network, perform convolutions at different scales on the feature layers of the same layer, and extract the multi-scale features corresponding to the query image and the support image respectively.

[0065] In this embodiment, the few-shot segmentation process includes a training set D. train and test set D test The training set D train From set C train The product of the test set D is composed of the product of the test set D. test From set C new The product in C consists of... train ∩C new =Φ; in D train Train the segmentation model on D, and in D test The trained model is then tested. The model is trained and tested using a meta-learning method, where in each episode (a subset of a training iteration), K normal samples and one defective sample from a product class are randomly selected as the support set S = x1, x2, ..., x... K and query set Q = (x q m q ), where m q H×W ∈{0,1} is the defect image x qThe binary mask is used; the segmentation model first learns the relationship between the query image and the support images in the high-dimensional space, and then performs semantic segmentation on the query image based on these relationships; each episode is a new task, and the network is trained iteratively to make the trained model have good generalization ability; after obtaining the segmentation model on the training set, it is verified on the test set whether it has the ability to detect defects in new products.

[0066] In this embodiment, a pair of query images and supporting images are randomly obtained. For example, in the SAM-based Vit-h model (a Transformer-based image segmentation model), in a 32-layer network, the values ​​of l are 1, 8, 16, 24, and 32 respectively. This is because in the feature extraction of the Vit backbone network, layers of different depths contain different semantic information. In shallower layers, more texture details of the image features are preserved, while in deeper layers, the encoded information contains more supplementary information to the semantics of the image. Features in intermediate layers preserve both of these aspects.

[0067] In this embodiment, unlike existing methods that use ResNet to obtain corresponding scale features for different layers (i.e., obtaining larger scale features in shallower layers and smaller scale features in deeper layers), this invention uses a convolutional neural network, such as... Figure 4 As shown, convolutions of the same layer at different scales are performed to obtain feature maps of different scales. This operation is performed on multiple feature layers to extract multiple size features from the query image and the support image. In each extracted network layer, feature maps of three different scales, 64, 32, and 16, are obtained. The support image and the query image are divided into image blocks of different scales to calculate the similarity, so as to fully consider the global and local similarity.

[0068] In this embodiment, multi-scale features of the query image and support image are obtained by performing convolutions at different scales on the same layer. The rich features of the intermediate layer block of the SAM feature extraction network based on MAE mask pre-training are used to capture the multi-level semantic and geometric pattern similarity between the support image and the query image.

[0069] like Figure 1 As shown, this embodiment of the invention provides a few-shot industrial anomaly segmentation deep learning method based on SAM, which further includes the following steps:

[0070] Step S200: Calculate the cosine similarity between the multi-scale features of the query image and the multi-scale features of the support image, and obtain the L2 distance map based on the cosine similarity.

[0071] In this embodiment, the similarity between the query image and the support image is calculated by combining the image coding features of different layers. Multi-scale features of the query image and the support image are extracted through the feature extraction network structure. Then, the similarity between each image block of the query image and the image block of the support image is calculated through the similarity calculation module. The image block in the support image that is most similar to the query image is replaced in the query image to form a new support image that is approximately without defects. The multi-scale distance map is obtained by calculating the L2 distance.

[0072] Specifically, in one implementation of this invention, step S200 includes the following steps:

[0073] Step S201: Normalize the multi-scale features of the query image and the multi-scale features of the support image;

[0074] Step S202: Calculate the cosine similarity between the same scale features of the query image and the supporting image;

[0075] Step S203: Replace the corresponding point in the query image with the image patch with the highest cosine similarity in the support image to obtain a new support image;

[0076] Step S204: Calculate the L2 distance between the feature points of the query image and the feature points of the new supporting image;

[0077] Step S205: Normalize the L2 distance to obtain multiple L2 distance maps of the query image and the support image;

[0078] Step S206: Merge the multiple obtained L2 distance maps, and fuse and decode features of different scales through a hierarchical decoding structure to obtain L2 distance maps of different scales;

[0079] Step S207: For L2 distance maps of the same scale in different feature layers, the number of channels is superimposed by splicing, and the channels are merged by a convolutional network.

[0080] Step S208: Average and fuse the L2 distance maps of the same scale in different support maps, and perform a deeper decoding through a convolutional neural network.

[0081] In this embodiment, the multi-scale features of the extracted query image are... and support for multi-scale features of images Where k represents the scales obtained by the Feature Extraction Module in this layer (64, 32, 16); c l h represents the feature dimension of the l-th layer. bw b The width and height of the feature map.

[0082] In this embodiment, in order to reduce the feature scale differences between different samples, improve the discriminativeness of features and increase the generalization ability of the network, as shown in Equation (1), the multi-scale features of the query image and the support image are normalized.

[0083]

[0084] Among them, F k,l,i express and The eigenvectors in ||F k,l,i ||2 represents the L2 norm normalization of these feature vectors, as shown in Equation (2). The cosine similarity between each feature map point vector between the query image and the support image at the same scale is calculated:

[0085]

[0086] in, and They are and The point vector.

[0087] In this embodiment, as shown in equation (3), in order to evaluate the location of the defect in the query image through L2 distance, the corresponding point of the query feature map is replaced with the image block with the highest cosine similarity between the query image and the support image to generate a new support image.

[0088]

[0089] Here, [] is the index selection function, s represents support, q represents query, and argmax() returns the index j of the vector of the most similar point to .

[0090] In this embodiment, as shown in equation (4), the location of the defect is estimated by calculating the L2 distance between the query image feature points and the new supporting image feature points.

[0091]

[0092] Where j in i∈{1,2,…j} represents the number of channels of a point in the feature map.

[0093] In this embodiment, similar to the normalization processing of the query image and the support image, in order to reduce the feature scale difference between different samples, improve the discriminativeness of the features and increase the generalization ability of the network, the L2 distance is normalized as shown in Equation (5).

[0094]

[0095] In this embodiment, the L2 distance between feature maps of the same size on the same feature layer of the query image and the support image is obtained through the above calculation. In addition, it is also necessary to merge multiple L2 distance maps obtained at different scales on the same feature layer, at different scales on different feature layers, and when multiple support images are given.

[0096] In this embodiment, as Figure 5 As shown, when extended to different layers, scales, and shots, the specific method for merging different distance maps is as follows: First, merge distance maps of the same scale from different layers. Then, merge multiple distance maps of the same scale from different support maps. For merging features from different layers, we use concatenation to stack the number of channels, and then use a convolutional network for channel compression. Combining multi-layer semantic and texture information helps to obtain more refined decoding features. When merging features of multiple scales from different support maps, we average and fuse features of the same scale from multiple layers, and then use a convolutional neural network for a deeper layer of decoding to obtain more robust feature difference maps. Finally, we use a layered decoding structure similar to U-Net to fuse and decode features of different scales from small scale to large scale, and fuse the final distance maps of different scales with the prototype support generated by the Prototype Generator to guide the network to decode more refinedly.

[0097] like Figure 1 As shown, this embodiment of the invention provides a few-shot industrial anomaly segmentation deep learning method based on SAM, which further includes the following steps:

[0098] Step S300: Obtain prototype support, combine the prototype support with the L2 distance map to obtain coarse segmentation hints.

[0099] In this embodiment, prototype support is obtained through a prototype generator, and the different scale distance maps obtained above are combined with the prototype support. Coarse segmentation hints are obtained through the structure decoding of an approximate U-net to guide SAM in segmenting industrial anomalies and defects.

[0100] Specifically, in one implementation of this invention, step S200 includes the following steps:

[0101] Step S301: Extract multi-layer image features of the query image using an image encoder;

[0102] Step S302: Input the image features into the prototype generator to obtain the prototype support;

[0103] Step S303: Combine the prototype support with L2 distance maps of different scales and decode them to obtain the coarse segmentation hint.

[0104] In this embodiment, it was found that calculating the features between the query image and the support images using multi-layer similarity and then estimating the location of the defect in the query image using L2 distance inevitably leads to some regions of the defect being similar to normal regions. This makes it impossible to accurately determine the location of the defect through similarity calculation, resulting in inaccurate segmentation. Therefore, as... Figure 6 As shown, this invention proposes a Prototype Generator module, which extracts prototype support through the prototype generator.

[0105] In this embodiment, as Figure 7 As shown, this invention extracts query image features from the front, middle and back layers of the SAM image encoder, obtains prototype support through a designed prototype generator, and combines the prototype support with distance maps at different scales to supplement the shortcomings of the L2 distance method for defect location estimation.

[0106] Specifically, such as Figure 8 As shown, features from layers 8, 16, 24, and 32 of the basic segmentation model SAM are extracted respectively. The features from these different layers are convolved to obtain feature maps with scales of 16, 32, 64, and 64. A network architecture similar to U-net is applied to combine these feature maps of different scales with the distance maps of different scales obtained above, and then decoded to generate a coarse segmentation mask for the corresponding query image. Furthermore, ablation experiments show that the above method effectively improves the accuracy of segmentation.

[0107] like Figure 1 As shown, this embodiment of the invention provides a few-shot industrial anomaly segmentation deep learning method based on SAM, which further includes the following steps:

[0108] Step S400: Input the coarse segmentation prompt into the prompt encoder to generate the segmentation result of the query image.

[0109] In this embodiment, after obtaining the coarse segmentation prompt through the above steps, the coarse segmentation prompt is input into the promptencoder of SAM to guide SAM to segment the abnormal regions of the query image and generate the segmentation result of the query image.

[0110] Specifically, in one implementation of this invention, step S400 includes the following steps:

[0111] Step S401: Input the coarse segmentation prompt into the prompt encoder of SAM;

[0112] In step S402, the SAM segments the abnormal regions of the query image according to the coarse segmentation prompt and generates the segmentation result of the query image.

[0113] In this embodiment, since the SAM model itself can segment the query image through prompts and has very high performance in segmentation, based on the original segmentation capability of SAM, the generated coarse mask is given to the prompt entry of SAM, and finally the segmentation result of the query image is generated.

[0114] Specifically, the obtained coarse segmentation prompts are input into the prompt encoder of SAM. During network training, both the prompt encoder and mask decoder are unfrozen and fine-tuned to guide SAM in segmenting abnormal regions of the query image.

[0115] In this embodiment, the generated coarse segmentation hints and the final segmentation result are supervised respectively. For the two generated masks, cross-entropy loss is used for supervision. In the experiment, α = 0.5 and β = 1 are taken respectively. The formula for calculating the loss is shown in Equation (6):

[0116] Loss=α×CELoss(RoughMask,GT)+β×CELoss(Prediction,GT) (6)

[0117] RoughMak is a coarse mask derived from the Multi-Similarity Module and Prototype Generator. This mask is sent to the Mask Decoder of SAM as a hint to guide SAM in segmenting the defective parts of the query image.

[0118] This embodiment achieves the following technical effects through the above technical solution:

[0119] This invention leverages the powerful feature extraction capabilities of the SAM backbone network, combines cosine similarity to calculate the similarity between abnormal and normal product images, and utilizes the calculated L2 distance feature map. This distance feature map is then combined with abnormal image prototype support obtained from a prototype generation network to generate a coarse segmentation mask. This coarse segmentation mask guides SAM in segmenting abnormal regions of abnormal product images. Defects are segmented by comparing positive and negative samples. The invention demonstrates good generalization ability even on new products for which the model has not been trained. This addresses the technical problem of existing industrial image anomaly segmentation methods being unable to segment abnormal regions of industrial images with limited samples.

[0120] Exemplary device

[0121] Based on the above embodiments, the present invention also provides a few-shot industrial anomaly segmentation deep learning device based on SAM, comprising:

[0122] The acquisition module is used to acquire query images and support images, and extract multi-scale features corresponding to the query images and support images respectively;

[0123] The calculation module is used to calculate the cosine similarity between the multi-scale features of the query image and the multi-scale features of the support image, and to obtain an L2 distance map based on the cosine similarity.

[0124] The module is used to obtain prototype support, and the prototype support is combined with the L2 distance map to obtain coarse segmentation hints;

[0125] The segmentation module is used to input the coarse segmentation prompts into the prompt encoder to generate the segmentation result of the query image.

[0126] Based on the above embodiments, the present invention also provides a terminal, comprising: a processor, a memory, an interface, a display screen, and a communication module connected via a system bus; wherein, the processor is used to provide computing and control capabilities; the memory includes a storage medium and internal memory; the storage medium stores an operating system and application programs; the internal memory provides an environment for the operation of the operating system and application programs in the storage medium; the interface is used to connect to external devices, such as mobile terminals and computers; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or a mobile terminal.

[0127] When the application is executed by the processor, it is used to implement the operation of a few-shot industrial anomaly segmentation deep learning method based on SAM.

[0128] It will be understood by those skilled in the art that Figure 9 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0129] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing a SAM-based few-shot industrial anomaly segmentation deep learning program, which, when executed by the processor, is used to implement the operation of a SAM-based few-shot industrial anomaly segmentation deep learning method as described above.

[0130] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a SAM-based few-shot industrial anomaly segmentation deep learning program, which, when executed by the processor, is used to implement the operation of a SAM-based few-shot industrial anomaly segmentation deep learning method as described above.

[0131] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by an application program instructing related hardware. The application program can be stored in a non-volatile storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory.

[0132] In summary, this invention provides a few-shot industrial anomaly segmentation deep learning method, terminal, and storage medium based on SAM. The method includes: acquiring a query image and support images; extracting multi-scale features from the query image and support images; calculating the cosine similarity between the multi-scale features of the query image and the multi-scale features of the support images; obtaining an L2 distance map based on the cosine similarity; acquiring prototype support; combining the prototype support with the L2 distance map to obtain a coarse segmentation hint; and inputting the coarse segmentation hint into a hint encoder to generate a segmentation result for the query image. This invention's few-shot industrial anomaly segmentation deep learning method based on SAM solves the technical problem that existing industrial image anomaly segmentation methods cannot segment abnormal regions in industrial images with few samples.

[0133] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A few-shot industrial anomaly segmentation deep learning method based on SAM, characterized in that, The SAM-based few-shot industrial anomaly segmentation deep learning method includes the following steps: Obtain the query image and support images, and extract the multi-scale features corresponding to the query image and support images respectively; Calculate the cosine similarity between the multi-scale features of the query image and the multi-scale features of the support image, and obtain the L2 distance map based on the cosine similarity; Obtain prototype support and combine it with the L2 distance map to obtain coarse segmentation hints; The coarse segmentation prompt is input into the prompt encoder to generate the segmentation result of the query image; The process of obtaining prototype support, combining the prototype support with the L2 distance map to obtain coarse segmentation hints, includes: Multi-layer image features are extracted from the query image using an image encoder; The image features are input into the prototype generator to obtain the prototype support; The prototype support is combined with L2 distance maps of different scales and decoded to obtain the coarse segmentation hint.

2. The SAM-based deep learning method for few-shot industrial anomaly segmentation according to claim 1, characterized in that, The process of acquiring the query image and support images, and extracting multi-scale features corresponding to the query image and support images respectively, includes: Obtain the query image and the supporting image; The query image and the support image are input into a convolutional neural network, and convolutions at different scales are performed on the same feature layer to extract multi-scale features corresponding to the query image and the support image, respectively.

3. The few-shot industrial anomaly segmentation deep learning method based on SAM according to claim 1, characterized in that, The calculation of the cosine similarity between the multi-scale features of the query image and the multi-scale features of the support image, and the generation of an L2 distance map based on the cosine similarity, includes: The multi-scale features of the query image and the multi-scale features of the support image are normalized. Calculate the cosine similarity between features of the same scale between the query image and the supporting images; The corresponding point in the query image is replaced with the image patch in the support image with the highest cosine similarity, resulting in a new support image.

4. The few-shot industrial anomaly segmentation deep learning method based on SAM according to claim 3, characterized in that, The step of replacing the corresponding point in the query image with the image patch in the support image that has the highest cosine similarity to obtain a new support image includes: Calculate the L2 distance between the feature points of the query image and the feature points of the new supporting image; The L2 distance is normalized to obtain multiple L2 distance maps of the query image and the support image; The obtained L2 distance maps are merged, and features at different scales are fused and decoded through a hierarchical decoding structure to obtain L2 distance maps at different scales.

5. The SAM-based deep learning method for few-shot industrial anomaly segmentation according to claim 4, characterized in that, The step of merging the multiple obtained L2 distance maps includes: For L2 distance maps of the same scale in different feature layers, the number of channels is superimposed by splicing, and the channels are merged by a convolutional network; For L2 distance maps of the same scale in different support maps, average fusion is performed, and then a deeper layer of decoding is performed through a convolutional neural network.

6. The few-shot industrial anomaly segmentation deep learning method based on SAM according to claim 1, characterized in that, The step of inputting the coarse segmentation hints into the hint encoder to generate the segmentation result of the query image includes: The coarse segmentation prompt is input into the prompt encoder of SAM; The SAM segments the abnormal regions of the query image based on the coarse segmentation prompts and generates the segmentation result of the query image.

7. A few-shot industrial anomaly segmentation deep learning device based on SAM, used to implement the few-shot industrial anomaly segmentation deep learning method based on SAM as described in any one of claims 1-6, characterized in that, include: The acquisition module is used to acquire query images and support images, and extract multi-scale features corresponding to the query images and support images respectively; The calculation module is used to calculate the cosine similarity between the multi-scale features of the query image and the multi-scale features of the support image, and to obtain an L2 distance map based on the cosine similarity. The module is used to obtain prototype support, and the prototype support is combined with the L2 distance map to obtain coarse segmentation hints; The segmentation module is used to input the coarse segmentation prompts into the prompt encoder to generate the segmentation result of the query image.

8. A terminal, characterized in that, include: The processor and memory, wherein the memory stores a SAM-based few-shot industrial anomaly segmentation deep learning program, which, when executed by the processor, is used to implement the SAM-based few-shot industrial anomaly segmentation deep learning method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a SAM-based few-shot industrial anomaly segmentation deep learning program, which, when executed by a processor, is used to implement the operation of the SAM-based few-shot industrial anomaly segmentation deep learning method as described in any one of claims 1-6.

Images