General apparent defect detection method based on visual large model

Abstract

The invention relates to a general appearance defect detection method based on a visual large model in the field of industrial computer vision, and the method comprises the following steps: collecting and preprocessing a picture with defects, obtaining a defect picture feature T1, obtaining a target region feature # imgabs0 # and a target embedding feature T2 through coding calculation, storing # imgabs1 # and T2 in a memory bank, and obtaining a target embedded feature T1; a defect detection end-to-end neural network model is constructed and deployed, a to-be-detected picture is preprocessed and input into the defect detection end-to-end neural network model to obtain a detection result, and through combination of defect detection and a visual large model and dependence on a complex data set pre-trained by the large model, the training cost is reduced, and the detection precision is improved. Specific defect pictures are used as neural network prompts, the generalization of the method and the flexibility of model deployment are enhanced, the method adapts to different defect detection items, and customer requirements are met.

Description

Technical Field

[0001] The present invention relates to the field of industrial computer vision, and in particular to a general appearance defect detection method based on a large visual model. Background Art

[0002] With the development of deep neural networks, some existing deep learning industrial defect detection technologies have greatly surpassed traditional industrial vision methods in terms of accuracy. However, in practical applications, defect detection technologies still have the following defects:

[0003] (1) Because most industrial data have simple background structures but complex defect patterns, existing industrial defect detection methods have poor generalization and are prone to overfitting a certain industrial part image.

[0004] (2) Existing industrial defect detection methods require high training costs and manual labeling costs.

[0005] (3) Existing industrial defect detection methods rely on a single model. Different models are required to detect different defects, which is time-consuming and labor-intensive and cannot effectively support the project completion cycle.

[0006] Therefore, the present invention proposes a universal appearance defect detection method based on a visual large model to solve the above problems. Summary of the invention

[0007] In order to solve the above technical problems, the present invention discloses a general appearance defect detection method based on a visual large model, and the method comprises the following steps:

[0008] S1, collect pictures with defects as prompts on the automated defect detection machine, adjust the size of the defect pictures to H*W, and perform color normalization, where H is height and W is width;

[0009] S2, manually annotate the defects in the defect image, generate defect image labels through the first encoder and encode and extract defect image features T1;

[0010] S3, the defect image feature T1 is encoded and calculated to obtain the target area feature Target area characteristics The target embedding feature T2 is obtained through normalized block coding calculation;

[0011] S4, the target area features The target embedding feature T2 is saved to the memory library, where the memory library is instantiated by Python and the target region feature is loaded by GPU. The target embedding features T2 are stored in the GPU memory, and the instantiation parameters are called regardless of the number of times;

[0012] S5, build an end-to-end neural network model for defect detection, initialize model parameters and deploy it to the automated defect detection machine;

[0013] Preferably, the first encoder and the second encoder are TinyVit encoders; the visual large model decoder is a SAM decoder;

[0014] The first encoder is obtained by performing neural network distillation with the visual large model SAM encoder, the second encoder structure is the same as the first encoder, and the cosine similarity block is used to quantify the target area features output by the first encoder and the degree of similarity between the to-be-detected picture feature S1 output by the second encoder.

[0015] Preferably, in step S2, the defects in the defective image are generated into defective image labels through the first encoder, and the output dimension of the defective image feature T1 is B×H / 32×W / 32×C, where B is the image input batch, generally 1, and C is the output channel, which is set to 256.

[0016] Preferably, the target area feature in step S3 The encoding calculation process is:

[0017]

[0018] Among them, Mask is the defect image label after the binarization operation.

[0019] The encoding calculation process of the normalized block of the target embedded feature T2 is:

[0020]

[0021] Among them, mean means to find the average value of the input tensor in the first dimension, and the norm operation is the Frobenius norm, which finds the norm of the matrix and represents matrix multiplication.

[0022] S6, adjusting the size of the image to be detected to H*W, and performing color normalization, where H is height and W is width;

[0023] S7, manually annotating defects in the image to be detected, generating defect image labels through a second encoder and encoding and extracting defect image features S1;

[0024] Preferably, the defects in the image to be detected are used to generate defect image labels through a second encoder, and the output dimension of the defect image feature S1 is B×H / 32×W / 32×C, where B is the image input batch, generally 1, and C is the output channel, which is set to 256.

[0025] S8, compares the image feature S1 to be detected with the target area feature in the memory library The target-oriented attention map S2 is obtained by calculating the cosine similarity block coding, and the threshold card control is performed according to the cosine similarity value to obtain the position prompt information T3;

[0026] Preferably, the encoding calculation process of the target-oriented attention graph S2 is as follows:

[0027]

[0028] The norm operation is the Frobenius norm of the matrix, which represents matrix multiplication.

[0029] Preferably, the cosine similarity value is quantified by the TopK algorithm, and a threshold is controlled according to the cosine similarity value. If it is lower than the threshold, a pass prompt is returned on the automated defect detection machine, indicating that the image has no defects. If it is higher than the threshold, the TopK algorithm returns two prompt point coordinate information, which are the most similar and the least similar, respectively, corresponding to the position prompt interface of the visual large model decoder. After the prompt point coordinate information is filled, the position prompt information T3 is obtained by stacking the channel of the mask embedding tensor.

[0030] S9, input the target embedding feature T2, the target-oriented attention map S2, the position prompt information T3, and the feature S1 of the image to be detected into the visual large model decoder to obtain the feature tensor S3;

[0031] S10, the feature tensor S3 is upsampled and block encoded to obtain a predicted image P, which is then output to an automated defect detection machine;

[0032] Preferably, the up-sampling block adopts a bilinear interpolation method, and the output image size is the original image size.

[0033] The advantages of the present invention are as follows:

[0034] (1) In steps S1, S2 and S3 of the present invention, only one or more pictures with specified defects and annotations are required, which can effectively reduce the annotation time. Since only feature extraction is performed on the defective picture, the iteration of the training part and the neural network back propagation calculation in other inventions are eliminated, so computing power can be effectively saved.

[0035] (2) In step S4 of the present invention, the target area features are stored in a memory library. The target embedding feature T2 is saved and the memory library is directly called in step S8 to effectively save computing power, training cost and labor cost.

[0036] (3) In step S1 of the present invention, the user can determine the defect type detected by the model by providing different defect images. Assuming that there are M types of defects in total, for a specified number of defects and a specified defect type, N images to be detected are input in the reasoning part, and the predicted probability Y of the specified number of defects and the specified defect type is calculated by the softmax function. Compared with the prior art, the detection of new defects requires the formulation of a new data set, and then the model is trained by re-labeling the new data set. However, the present invention can obtain a new defect detection model without iteration, which can effectively adapt to a shorter project cycle.

[0037] (4) In the present invention, the first TinyVit encoder constructed is obtained by performing neural network distillation with the visual large model SAM encoder, which not only retains the generalization ability and accuracy of the visual large model learning big data, but also effectively reduces the size of the model parameters. The parameter amount is only 21M, and the number of multiplication and addition accumulation operations on the Tesla V100 industrial graphics card is only 4.3G, which can be effectively deployed in the automated defect detection platform service.

[0038] (5) In step S8 of the present invention, Figure 2 As shown in the figure, the cosine similarity block can directly return the result by quantizing the cosine similarity. Since the current industrial appearance data mainly consists of samples without defects, this function can effectively save detection time and improve detection speed.

[0039] (6) The present invention combines defect detection with a large visual model, relying on a complex data set pre-trained by the large model, reducing training costs and improving detection accuracy. By using specific defect images as neural network prompts, the generalization of the method and the flexibility of model deployment are enhanced to adapt to different defect detection projects and meet customer needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only one embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 It is a flowchart of a general appearance defect detection method based on a visual large model;

[0042] Figure 2 Flowchart for the inference calculation of the cosine similarity block. DETAILED DESCRIPTION

[0043] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention and the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by technicians in the technical field to which the present invention belongs; the terms used in the specific implementation methods are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the terms "including" and "having" in the specification and claims of the present invention and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0045] In the description of the specific implementation methods of the present invention, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present invention, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0046] Reference to "embodiments" in the present invention means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present invention may be combined with other embodiments.

[0047] In the description of the embodiments of the present invention, the term "and / or" is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.

[0048] It should be noted that, for the convenience of description, in the following embodiments, all identical technical features are marked with the same symbols.

[0049] The present invention discloses a general appearance defect detection method based on a large visual model. The method is implemented by relying on an end-to-end neural network model for defect detection. Figure 1 The specific steps are as follows:

[0050] On the automated defect detection machine, collect pictures with defects to be detected as defect picture prompts. Specifically, collect at least one picture with defects to be detected on site, with a total of M types of defects, including scratches, foreign matter, missing parts, oxidation and other industrial defects. Select pictures with a specified number of defects and a specified defect type, adjust the defect image size to H*W. In this embodiment, H and W are both 448, and perform color normalization on the defect image.

[0051] The defect images are manually annotated, and the corresponding mask image labels are generated by the first TinyVit encoder based on the manual annotations, that is, Figure 1 The defect image labels in the manual annotation only mark the defects that need to be detected. The defect foreground is obtained according to the defect image annotation and encoded to extract the defect image feature T1. Specifically, the output dimension of the defect image feature T1 is B×H / 32×W / 32×C, where B is the image input batch, which is generally 1, and C is the output channel, which is 256.

[0052] The defect image feature T1 is encoded and calculated to obtain the target area feature Target area characteristics The target embedding feature T2 is obtained by normalized block coding. Specifically, Among them, Mask is the defect image label after the binarization operation. Among them, mean means to find the average value of the input tensor in the first dimension, and the norm operation is the Frobennus norm (Frobennusnorm), which finds the norm of the matrix and represents matrix multiplication.

[0053] The target area features The target region feature and target embedding feature T2 are stored in the memory bank, where the memory bank is instantiated by Python. The target region feature and target embedding feature T2 are stored in the GPU memory by loading the GPU, and the instantiation parameters are called regardless of the number of times.

[0054] Build an end-to-end neural network model for defect detection, initialize model parameters, and deploy the model to the automated defect detection machine service.

[0055] The end-to-end neural network model for defect detection consists of a defect target feature generator and a backbone network. The defect target feature generator includes the first TinyVit encoder and a normalization block in sequence, and the backbone network consists of the second TinyVit encoder, a cosine similarity block, a large visual model decoder, and an upsampling block.

[0056] In this embodiment, the first TinyVit encoder is obtained by performing neural network distillation through the visual large model SAM encoder, wherein the visual large model SAM encoder is derived from the paper "Segment Anything Model arXiv: 2304.02643" (Alexander Kirillov, Eric Mintun, Nikhila Ravi1, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead Alexander C. Berg Wan-Yen Lo Piotr Dollar, Ross Girshick), the second TinyVit encoder structure is the same as the first TinyVit encoder structure, the SAM encoder is the teacher model, the first TinyVit encoder and the second TinyVit encoder are student models. Compared with the student model, the teacher model has a complex structure and takes a long time to train and infer, but has strong learning ability. The teacher model is used to assist the training of the student model to increase the generalization ability of the student model.

[0057] Manually mark the defects in the image to be detected, generate defect image labels through the second TinyVit encoder and encode and extract defect image features S1, specifically:

[0058] The defects in the image to be detected are used to generate defect image labels through the second TinyVit encoder. The output dimension of the defect image feature S1 is B×H / 32×W / 32×C, where B is the image input batch, generally 1, and C is the output channel, which is 256.

[0059] The feature S1 of the image to be detected and the feature of the target area in the memory library are calculated by cosine similarity block coding to obtain the target-oriented attention map S2, and the threshold is controlled according to the cosine similarity value to obtain the position prompt information T3.

[0060] For the cosine similarity block, the cosine similarity block does not include a neural network structure, and only performs cosine similarity calculation, where the cosine similarity block is used to quantify the similarity between two feature maps. The target-oriented attention map S2 encoding calculation process is: The norm operation is the Frobennus norm, which takes the norm of the matrix and represents matrix multiplication.

[0061] At the same time, in the cosine similarity block, the cosine similarity value is quantified according to the TopK algorithm, the index of the most similar area is found, and the threshold card control is performed according to the cosine similarity value. If it is lower than the threshold, it will directly return to the automatic defect detection machine to prompt the word "pass", indicating that the image has no defects, avoiding repeated decoding operations and improving the reasoning speed. If it is higher than the threshold, the TopK algorithm returns two prompt point coordinate information, the most similar and the least similar, corresponding to the position prompt interface of the visual large model decoder. After the prompt point coordinate information is filled, it is embedded in the tensor channel stacking operation and the position prompt information T3 is obtained.

[0062] The target embedding feature T2, target-oriented attention map S2, location hint information T3, and the image feature to be detected S1 are input into the visual large model decoder to load the original pre-trained weights to obtain the feature tensor S3. The visual large model decoder comes from the paper "SegmentAnything Model arXiv: 2304.02643"

[0063] (Alexander Kirillov, Eric Mintun, Nikhila Ravi1, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. BergWan-Yen LoPiotr Dollar, Ross Girshick).

[0064] The feature tensor S3 is encoded by the upsampling block to obtain the predicted image P and output it to the automatic defect detection machine. The upsampling block adopts bilinear interpolation method, and the output image size is the original image size. The detection results are visualized by opencv to the automatic defect detection machine.

[0065] It should be pointed out that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A general appearance defect detection method based on visual large model, It is characterized in that The steps include: S1, collect pictures with defects as prompts on the automated defect detection machine, adjust the size of the defect pictures to H*W, and perform color normalization, where H is height and W is width; S2, manually annotate the defects in the defect image, generate defect image labels through the first encoder and encode and extract defect image features T1; S3, the defect image feature T1 is encoded and calculated to obtain the target area feature Target area characteristics The target embedding feature T2 is obtained through normalized block coding calculation; S4, the target area features And the target embedding feature T2 is saved to the memory bank; S5, build an end-to-end neural network model for defect detection, initialize model parameters and deploy it to the automated defect detection machine; S6, adjusting the size of the image to be detected to H*W, and performing color normalization, where H is height and W is width; S7, manually annotating defects in the image to be detected, generating defect image labels through a second encoder and encoding and extracting defect image features S1; S8, compares the image feature S1 to be detected with the target area feature in the memory library The target-oriented attention map S2 is obtained by calculating the cosine similarity block coding, and the threshold card control is performed according to the cosine similarity value to obtain the position prompt information T3; S9, input the target embedding feature T2, the target-oriented attention map S2, the position prompt information T3, and the feature S1 of the image to be detected into the visual large model decoder to obtain the feature tensor S3; S10, the feature tensor S3 is upsampled and block encoded to obtain a predicted image P, which is then output to an automated defect detection machine; In step S5, the end-to-end neural network model for defect detection includes the first encoder, the second encoder, the normalization block, the cosine similarity block, the visual large model decoder and the upsampling block.

2. According to claim 1, a method for general appearance defect detection based on a visual large model, It is characterized in that The first encoder and the second encoder are TinyVit encoders; The visual large model decoder is a SAM decoder.

3. According to claim 2, a general appearance defect detection method based on a visual large model, It is characterized in that The first encoder is obtained by performing neural network distillation with the visual large model SAM encoder, the second encoder structure is the same as the first encoder, and the cosine similarity block is used to quantify the target area features output by the first encoder and the degree of similarity between the image feature S1 to be detected output by the second encoder.

4. According to claim 1, a general appearance defect detection method based on a visual large model, It is characterized in that In step S2, the defects in the defective image are generated into defective image labels through the first encoder, and the output dimension of the defective image feature T1 is B×H / 32×W / 32×C, where B is the image input batch and C is the output channel.

5. According to claim 4, a general appearance defect detection method based on a visual large model, It is characterized in that The target area features in step S3 The encoding calculation process is: Among them, Mask is the defect image label after the binarization operation. The encoding calculation process of the normalized block of the target embedded feature T2 is: Among them, mean means to find the average value of the input tensor in the first dimension, and the norm operation is the Frobenius norm, which finds the norm of the matrix and represents matrix multiplication.

6. The general appearance defect detection method based on visual large model according to claim 1, It is characterized in that In the step S7, the defects in the image to be detected are used to generate defect image labels through the second encoder, and the output dimension of the defect image feature S1 is B×H / 32×W / 32×C, where B is the image input batch and C is the output channel.

7. A general appearance defect detection method based on a visual large model according to claim 6, It is characterized in that In step S8, the encoding calculation process of the target-oriented attention graph S2 is as follows: The norm operation is the Frobenius norm of the matrix, which represents matrix multiplication.

8. The general appearance defect detection method based on visual large model according to claim 1, It is characterized in that In step S8, the cosine similarity value is quantified by the TopK algorithm, and a threshold is controlled according to the cosine similarity value. If it is lower than the threshold, a prompt of passing is returned on the automated defect detection machine, indicating that the image has no defects. If it is higher than the threshold, the TopK algorithm returns two prompt point coordinate information, which are the most similar and the least similar, respectively, corresponding to the position prompt interface of the visual large model decoder. After the prompt point coordinate information is filled, the position prompt information T3 is obtained by stacking the channel of the mask embedding tensor.

9. The general appearance defect detection method based on visual big model according to claim 1, It is characterized in that In the step S10, the up-sampling block processes the image using a bilinear interpolation method, and the size of the image output by the up-sampling block is the size of the original image.

Images