Defect category detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a defect category detection method and device, electronic equipment and a storage medium. The defect category detection method comprises the following steps: acquiring a to-be-detected image of a target object; the to-be-detected image is input into a trained classification model to obtain a classification result of the to-be-detected image for defects of a target object in the to-be-detected image, and the classification model comprises a feature extraction module, a convolution block attention module and a post-processing module which are connected in sequence; the feature extraction module is used for extracting a first feature of the to-be-detected image; the convolution block attention module is used for performing channel attention calculation and / or space attention calculation on the received first feature and determining a second feature according to a calculation result; and the post-processing module is used for classifying the second features to obtain a classification result. The defects of the target object can be accurately classified, misjudgment is avoided, and the similar defect classification precision is improved.

Description

Technical Field

[0001] The present application relates to the field of industrial inspection, and more specifically to a defect category inspection method, a defect category inspection device, an electronic device, a storage medium, and a computer program product. Background Art

[0002] In the industrial field, with the continuous development of production technology, people have put forward higher requirements for the production quality and production efficiency of industrial products. With the development of image processing, more and more product inspections are based on images. Product defects can be identified through images and the defect categories can be determined. Then, based on the defect categories, the product production process can be optimized to improve production quality and efficiency.

[0003] For example, in the semiconductor industry, for the classification of wafer defects, automated optical inspection (AOI) equipment is usually used to perform appearance inspection on the wafer to determine the type of defects on the wafer. Wafer images usually have high resolution, relatively small defects and similarities. In related technologies, deep learning methods are used to extract image features and classify defects, which is prone to misjudgment and has low accuracy. Summary of the invention

[0004] The present invention has been made in view of the above-mentioned problems.

[0005] According to one aspect of the present invention, a method for detecting defect categories is provided, comprising:

[0006] Acquire an image to be detected of a target object; and input the image to be detected into a trained classification model to obtain a classification result of the image to be detected for defects of the target object therein, wherein the classification model includes a feature extraction module, a convolutional block attention module and a post-processing module connected in sequence; the feature extraction module is used to extract a first feature of the image to be detected; the convolutional block attention module is used to perform channel attention calculation and / or spatial attention calculation on the received first feature, and determine a second feature based on the calculation result; the post-processing module is used to classify the second feature to obtain a classification result.

[0007] The convolutional block attention module can improve the ability to extract local features of the image to be detected through channel attention and spatial attention, and enhance the perception of diverse features. As a result, the defects of the target object can be accurately classified, misjudgment can be avoided, and the accuracy of classification of similar defects can be improved.

[0008] Exemplarily, the feature extraction module is implemented using a hierarchical visual self-attention model based on a moving window, and the hierarchical visual self-attention model based on a moving window includes multiple hierarchical units, the multiple hierarchical units are connected in series, and the multiple hierarchical units include at least one first hierarchical unit, and each first hierarchical unit is respectively connected to a convolutional block attention module. The combination of the hierarchical visual self-attention model based on a moving window and the convolutional block attention module can extract more global features of the image to be detected, and can also extract more local features of the image to be detected. As a result, the accuracy of the defect classification results of the target object can be improved.

[0009] Exemplarily, the first hierarchical unit includes the last hierarchical unit in the plurality of hierarchical units, thereby extracting more abundant features from the image to be detected, further improving the accuracy of detecting the defect category of the image to be detected.

[0010] Exemplarily, the first-level units include N, where N is a positive integer greater than 1; the convolutional block attention module determines the second feature according to the calculation results, including the following operations: downsampling or upsampling the calculation results of at least N-1 convolutional block attention modules respectively to obtain respective sampling results, wherein all sampling results have the same size, and in the case of unsampled calculation results, all sampling results have the same size as the unsampled calculation results; all sampling results are spliced, and in the case of unsampled calculation results, all sampling results are spliced ​​with unsampled calculation results to obtain the second feature. In this way, the output sizes of different first-level units are guaranteed to be consistent, and the spliced ​​second features have features of different levels, thereby including more features in the image to be detected, thereby improving the accuracy of defect category detection with less computation.

[0011] Exemplarily, the multiple hierarchical units are all first-level units. Thus, the second feature may include features of all levels, further improving the focus on local details of the image to be detected, and improving the accuracy of defect category detection.

[0012] Exemplarily, the post-processing module classifies the second feature to obtain a classification result, including performing the following operations: performing a convolution operation on the second feature to obtain a third feature; performing global average pooling on the third feature to obtain a fourth feature; and calculating the confidence that the defects of the target object belong to different categories based on the fourth feature to obtain a classification result. In this way, the amount of calculation can be reduced and the speed of defect category detection can be improved. In addition, the confidence can help users determine whether the classification result is reliable and improve user experience.

[0013] Exemplarily, the detection method further includes: training a classification model using labeled sample images based on a cross entropy loss function and a center loss function as loss functions, which can improve the classification accuracy of the trained classification model, and further, can improve the classification accuracy of similar defects.

[0014] According to another aspect of the present invention, there is also provided a defect category detection device, which is characterized in that it includes: a first acquisition module and a classification module. The first acquisition module is used to acquire an image to be detected of a target object; the classification module is used to input the image to be detected into a trained classification model to obtain a classification result of the image to be detected for the defects of the target object therein, wherein the classification model includes a feature extraction module, a convolutional block attention module and a post-processing module connected in sequence; the feature extraction module is used to extract the first feature of the image to be detected; the convolutional block attention module is used to perform channel attention calculation and / or spatial attention calculation on the received first feature, and determine the second feature based on the calculation result; the post-processing module is used to classify the second feature to obtain a classification result

[0015] According to another aspect of the present invention, there is also provided an electronic device, comprising: a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used to execute the above-mentioned defect category detection method when the processor is executed.

[0016] According to yet another aspect of the present invention, there is further provided a storage medium on which program instructions are stored, and the program instructions are used to execute the defect category detection method as described above when running.

[0017] According to another aspect of the present invention, there is also provided a computer program product, comprising computer program instructions, wherein the computer program instructions are used to execute the defect category detection method as described above when running.

[0018] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention.

[0020] Figure 1A schematic flow chart of a defect category detection method according to an embodiment of the present invention is shown;

[0021] Figure 2 A schematic diagram showing a defect category detection method according to an embodiment of the present invention;

[0022] Figure 3 A schematic block diagram of a defect category detection device according to an embodiment of the present invention is shown;

[0023] Figure 4 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention more obvious, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention.

[0025] In the image-based defect detection related technology, the local features of the image are not extracted sufficiently, and it is easy to make misjudgments when detecting defect categories, and the accuracy is low. In order to solve the above technical problems, the present invention provides a defect category detection method. Figure 1 FIG. 1 is a schematic flow chart of a defect category detection method according to an embodiment of the present invention. Figure 1 As shown, the method includes step S110 and step S120.

[0026] In step S110, an image to be detected of the target object is acquired.

[0027] The target object may be any object, such as a wafer, a liquid crystal panel, etc. The image to be detected may be obtained by photographing the target object with a camera. The photographing device may include any device suitable for photographing the image to be detected of the target object. For example, it may include a linear array camera, an area array camera, or a short-wave infrared camera. The target object may or may not have defects. Taking the target object as a wafer as an example, the defects may include tin slag, dust, foreign matter, etc.

[0028] In some embodiments, existing or future developed defect recognition technology may be used to determine whether the target object corresponding to the image to be detected has defects. For example, through a template matching method, an image of a target object without defects is used as a template to match all images to be detected to determine an image to be detected of a target object with defects. The image to be detected of the target object without defects may be eliminated, and the image to be detected of the target object with defects may be retained. In these embodiments, the image to be detected acquired in step S110 may be an image to be detected of a target object with defects.

[0029] The image to be detected of the target object can be any suitable image. Exemplarily, the image to be detected of the target object can be a color image or a grayscale image. The image to be detected of the target object can be an image of any suitable size and suitable resolution. The image to be detected can be the original image directly collected by the imaging device, or it can be an image after the original image is preprocessed. The preprocessing operation can include all operations to improve the visual effect of the image, improve its clarity, or highlight certain features in the image. Exemplarily and non-restrictively, the preprocessing operation can include operations such as geometric transformation, normalization, filtering, etc. of the original image. The image to be detected can also be a synthesized image without affecting subsequent image processing.

[0030] In step S120, the image to be detected is input into the trained classification model to obtain the classification result of the defects of the target object in the image to be detected. The classification model includes a feature extraction module, a convolution block attention module and a post-processing module connected in sequence. The feature extraction module is used to extract the first feature of the image to be detected. The convolution block attention module is used to perform channel attention calculation and / or spatial attention calculation on the received first feature, and determine the second feature according to the calculation result. The post-processing module is used to classify the second feature to obtain a classification result.

[0031] The trained classification model may be obtained by training an initial classification model using training data, and the training data may include an image of a target object for which a classification result of a defect has been determined and corresponding annotation data.

[0032] The feature extraction module can adopt an existing or future developed model or method suitable for extracting features of the image to be detected. In some embodiments, the feature extraction module can use deep neural networks such as residual neural network (ResNet), visual geometry group network (VGG), visual transformer model (VIT) as the basic architecture. The feature extraction module receives the image to be detected and can extract the first feature of the image to be detected. The first feature may include key features such as edges, textures, defects, etc. of the image to be detected. The first feature can be a feature map of the image to be detected after being processed by the feature extraction module.

[0033] The convolutional block attention module (CBAM) may include a channel attention module and / or a spatial attention module. The channel attention module may capture the relationship between channels of the first feature to enhance the first feature. The spatial attention module may enhance the first feature by capturing the spatially significant region of the first feature.

[0034] According to some embodiments of the present invention, the first feature may be input into the convolution block attention module, and the channel attention module may perform channel attention calculation on the first feature. The channel attention calculation may be performed according to the following formula:

[0035] F aυg =AυgPool(F) and F max =MaxPool(F) 1)

[0036] Among them, F represents the first feature; AvgPool represents the global average pooling; F avg represents the first feature after global average pooling; MaxPool represents global maximum pooling; F max Represents the first feature after global maximum pooling. The first feature can be subjected to global average pooling and global maximum pooling respectively to obtain the first feature after global average pooling and the first feature after global maximum pooling.

[0037] M aυg =MLP(F aυg ) and M max =MLP(F max ) 2)

[0038] Among them, MLP represents multi-layer perceptron; M avg represents the activation value of the first feature after global average pooling; M max Represents the activation value of the first feature after global maximum pooling. The first feature after global average pooling and the first feature after global maximum pooling are passed through a shared multi-layer perceptron to obtain the corresponding activation values.

[0039] M channel =σ(M aυg +M max ) 3)

[0040] Where σ represents the sigmoid function; M channel Represents the channel attention map. The activation value of the first feature after global maximum pooling and the activation value of the first feature after global average pooling are added, and the channel attention map is obtained through the sigmoid function.

[0041] Before using the spatial attention module to perform spatial attention calculation on the first feature, the first feature may be weighted according to the channel attention map to obtain the first feature map weighted according to the channel attention map. The weighting may be performed according to the following formula:

[0042] F′=M channel ⊙F 4)

[0043] Among them, F represents the first feature; M channelrepresents the channel attention map; F' represents the first feature map weighted according to the channel attention map. The channel attention map and the elements at the corresponding positions in the first feature are multiplied to obtain the first feature weighted according to the channel attention map.

[0044] Then, the channel attention module can be used to calculate the first feature according to the following formula:

[0045]

[0046] Among them, Mean represents average pooling; Max represents maximum pooling; represents the average pooling feature map; Here, the first feature map weighted by the channel attention map is subjected to average pooling and maximum pooling to obtain an average pooling feature map and a maximum pooling feature map, respectively.

[0047]

[0048] Among them, Fconca t Represents a two-dimensional feature map. Here, the average pooling feature map and the maximum pooling feature map are concatenated to obtain a two-dimensional feature map.

[0049] M spatial =σ(Conυ(F concat )) 7)

[0050] M spatial represents the spatial attention map; σ represents the sigmoid function; Conv represents the convolutional layer. Here, the two-dimensional feature map F concat After the convolution operation through the convolution layer, the spatial attention map M is obtained through the sigmoid function spatial .

[0051] The first feature after the weighting of the channel attention map and the spatial attention map can be weighted and calculated, and the second feature can be determined according to the calculation result. Exemplarily, the calculation can be performed according to the following formula:

[0052] F″=M spatial ⊙F′ 8)

[0053] Among them, F" represents the second feature; M spatial represents the spatial attention map; F' represents the first feature map weighted according to the channel attention map. Here, the elements at corresponding positions in the spatial attention map and the first feature map weighted according to the channel attention map are multiplied to determine the second feature.

[0054] It can be understood that in the above embodiment, the convolutional attention module includes both the channel attention module and the spatial attention module. It can be understood that the convolutional attention module can also include only one of them, thereby only performing channel attention calculation or spatial attention calculation on the first feature.

[0055] The post-processing module may classify the second feature to obtain a classification result corresponding to the second feature. The classification result may represent a classification result of a defect of a target object of the image to be detected corresponding to the second feature. The post-processing module may include a fully connected layer or other classifiers. The post-processing layer may also include a normalized exponential function to convert the classification result into a probability distribution for better interpretation of the classification result.

[0056] The above technical solution extracts the first feature of the image to be detected based on the feature extraction module; uses the convolution block attention module to process the first feature to determine the second feature; and uses the post-processing module to classify the second feature to obtain the classification result of the defects of the target object of the image to be detected. As mentioned above, in target objects such as wafers, defects are usually located in their local areas. The convolution block attention module can improve the ability to extract local features of the image to be detected through channel attention and spatial attention, and enhance the perception of diverse features. In this way, the defects of the target object can be accurately classified, misjudgment can be avoided, and the accuracy of classifying similar defects can be improved.

[0057] Exemplarily, the feature extraction module is implemented using a hierarchical visual self-attention model (Swin-Transformer) based on a moving window. The hierarchical visual self-attention model based on a moving window includes a plurality of hierarchical units. The plurality of hierarchical units are connected in series, and the plurality of hierarchical units include at least one first hierarchical unit, and each first hierarchical unit is respectively connected to a convolutional block attention module.

[0058] The hierarchical visual self-attention model based on the moving window uses a multi-head attention mechanism to extract features from the target area, which can better capture the features in the image. The feature extraction module can use the hierarchical visual self-attention model based on the moving window to extract features from the image to be detected. The hierarchical visual self-attention model based on the moving window usually includes multiple hierarchical units. For example, it can include four hierarchical units, namely hierarchical unit 1, hierarchical unit 2, hierarchical unit 3 and hierarchical unit 4. Hierarchical unit 1, hierarchical unit 2, hierarchical unit 3 and hierarchical unit 4 are connected in series in sequence. The input of hierarchical unit 1 can include the image to be detected. The output of each hierarchical unit can be used as the first feature. The output of the previous hierarchical unit can be used as the input of the next hierarchical unit. For example, the output of hierarchical unit 3 can be used as the input of hierarchical unit 4. At least one hierarchical unit among the multiple hierarchical units can be determined as a first hierarchical unit. Each first hierarchical unit is connected to a convolutional block attention module respectively. In other words, the first hierarchical unit and the convolutional block attention module correspond one to one. The first feature of the output of each first-level unit can be used not only as the input of the next-level unit, but also as the input of the convolutional block attention module connected to it. For example, the first-level unit and the second-level unit can be connected to a convolutional block attention module as the first-level unit, and the first features of their outputs can be input into the corresponding convolutional block attention module. The second features output by the two convolutional block attention modules can be mapped to the same dimension and concatenated into one second feature.

[0059] In the above technical solution, the feature extraction module is implemented using a hierarchical visual self-attention model based on a moving window. In addition, the hierarchical visual self-attention model based on a moving window includes multiple hierarchical units, at least one hierarchical unit being a first hierarchical unit. Each first hierarchical unit is respectively connected to a convolutional block attention module. The combination of the hierarchical visual self-attention model based on a moving window and the convolutional block attention module can extract more global features of the image to be detected, and can also extract more local features of the image to be detected. As a result, the accuracy of the defect classification results of the target object can be improved.

[0060] Exemplarily, the first level unit includes the last level unit of the multiple level units. In other words, the last level unit of the multiple level units is connected to a convolutional block attention module. The last level unit of the multiple level units of the moving window-based hierarchical visual self-attention model (Swin-Transformer) can usually include more channels relative to the previous level units, and can extract richer features. Taking the above model as an example including four level units, namely level unit No. 1, level unit No. 2, level unit No. 3 and level unit No. 4, the first level unit may include level unit No. 4, that is, the last level unit of the multiple level units.

[0061] In the above technical solution, the first level unit connected to the convolutional block attention module includes the last level unit among the multiple level units of the hierarchical visual self-attention model based on the moving window. This can extract richer features in the image to be detected, and further improve the accuracy of detecting the defect category of the image to be detected.

[0062] Exemplarily, the first level units include N, where N is a positive integer greater than 1. In step S120, the convolutional block attention module determines the second feature according to the calculation result, including executing the following steps S121 and S122.

[0063] In step S121, the calculation results of at least N-1 convolution block attention modules are downsampled or upsampled to obtain respective sampling results, wherein all sampling results have the same size, and in the case where there are unsampled calculation results, all sampling results have the same size as the unsampled calculation results.

[0064] Each convolution block attention module is connected to a first-level unit. The size of the first feature output by each first-level unit may be different from each other. After each first feature is input to the corresponding convolution block attention module, the calculation result obtained is related to the size of the input first feature. In other words, the calculation result output by each convolution block attention module is of different sizes. For example, a hierarchical visual self-attention model based on a moving window may include four hierarchical units, namely hierarchical unit 1, hierarchical unit 2, hierarchical unit 3 and hierarchical unit 4. The first hierarchical unit may include hierarchical unit 1, hierarchical unit 2 and hierarchical unit 3. The size of the calculation result of the corresponding convolution block attention module may be gradually reduced. The size of the output result of the convolution block attention module corresponding to the hierarchical unit 1 in each dimension may be twice the size of the output result of the convolution block attention module corresponding to the hierarchical unit 2 in the corresponding dimension. The size of the output result of the convolution block attention module corresponding to the hierarchical unit 2 in each dimension may be twice the size of the output result of the convolution block attention module corresponding to the hierarchical unit 3 in the corresponding dimension. In other words, the size of the output result of the convolution block attention module corresponding to the level unit No. 1 in each dimension may be four times the size of the output result of the convolution block attention module corresponding to the level unit No. 3 in the corresponding dimension. Exemplarily, in order to ensure that the calculation results of all convolution block attention modules have the same size, the calculation results of some or all of the convolution block attention modules may be appropriately downsampled or upsampled. It may include downsampling or upsampling the calculation results of each of the N-1 convolution block attention modules to obtain respective sampling results, and the size of the sampling results is the same. It can be understood that upsampling and downsampling can be upsampled or downsampled according to a certain target size annotation to downsample or upsample the size of the calculation result to the size annotation. In other words, the size of the unsampled calculation result may be the sampling standard. In some embodiments, the sampling standard may be set by the user. It may be determined that the size of a calculation result of a convolution block attention module is the sampling standard, and the calculation result does not need to be sampled. The sampling standard may also be a size other than the calculation results of all convolution block attention modules, and all convolution block attention module calculation results need to be sampled to reach the size of the sampling standard. Downsampling can reduce the resolution of the calculation result. Upsampling can increase the resolution of the calculation result. Upsampling and downsampling can use any existing or future developed downsampling and upsampling methods, which are not limited here. For example, the nearest neighbor algorithm, linear interpolation, global average pooling, maximum pooling and other methods can be used for upsampling or downsampling.

[0065] Still taking the example that the first hierarchical unit may include hierarchical unit No. 1, hierarchical unit No. 2, and hierarchical unit No. 3, the calculation result of the convolutional block attention module corresponding to hierarchical unit No. 1 may be downsampled by 4 times, and the calculation result of the convolutional block attention module corresponding to hierarchical unit No. 2 may be downsampled by 2 times, so that the size of the calculation result of the convolutional block attention module corresponding to hierarchical unit No. 1 and hierarchical unit No. 2 is the same as the size of the calculation result of the convolutional block attention module corresponding to hierarchical unit No. 3. For example, the size of the calculation result of the convolutional block attention module corresponding to hierarchical unit No. 1 may be 56*56*4, and the size may be 14*14*1 after four times downsampling, the size of the calculation result of the convolutional block attention module corresponding to hierarchical unit No. 2 may be 28*28*2, and the size may be 14*14*1 after two times downsampling, and the size of the calculation result of the convolutional block attention module corresponding to hierarchical unit No. 3 may be 14*14*1, thereby ensuring that the sizes of all sampled results are the same, and in the case where there are unsampled calculation results, the sizes of all sampled results are the same as the unsampled calculation results. Alternatively, the calculation result of the convolution block attention module corresponding to the level unit No. 1 may be downsampled by two times, and the size of the calculation result may be changed from 56*56*4 to 28*28*2; the calculation result of the convolution block attention module corresponding to the level unit No. 3 may be upsampled by two times, and the size of the calculation result may be changed from 14*14*1 to 28*28*2. This can also ensure that the sizes of all sampled results are the same, and in the case where there are unsampled calculation results, the sizes of all sampled results are the same as those of the unsampled calculation results.

[0066] In step S122, all sampling results are concatenated, and if there are unsampled calculation results, all sampling results are concatenated with the unsampled calculation results to obtain the second feature.

[0067] All sampling results have the same size, and in the case where there are unsampled calculation results, the unsampled calculation results and the sampling results have the same size. In some embodiments, there are no unsampled calculation results, in other words, all calculation results have been sampled. Therefore, all sampling results can be spliced ​​to obtain the second feature. In other embodiments, there are unsampled calculation results. All sampling results can be spliced ​​with the unsampled calculation results to obtain the second feature. Splicing can be to merge different sampling results and unsampled calculation results according to channels. In other words, the data of each channel of the second feature includes the data of the channel of all sampling results and unsampled calculation results.

[0068] In the above technical solution, the first-level unit includes N convolutional block attention modules (N is a positive integer greater than 1). The calculation results of at least N-1 modules are downsampled or upsampled so that all sampled results have the same size, and the unsampled calculation results are also consistent with the size. Finally, all sampled results and possible unsampled calculation results are spliced ​​to form a second feature. In this way, the output sizes of different first-level units are guaranteed to be consistent, and the spliced ​​second features have features of different levels, thereby including more features in the image to be detected, thereby improving the accuracy of defect category detection with less computational effort.

[0069] Exemplarily, multiple hierarchical units are all first-level units. All hierarchical units of the hierarchical visual self-attention model based on the moving window are first-level units. Each first-level unit is connected to a convolutional block attention module. In other words, all hierarchical units of the hierarchical visual self-attention model based on the moving window are respectively connected to a convolutional block attention module. For example, the multiple hierarchical units may include four hierarchical units, hierarchical unit No. 1, hierarchical unit No. 2, hierarchical unit No. 3 and hierarchical unit No. 4, which are respectively connected to a convolutional block attention module.

[0070] In the above technical solution, all hierarchical units of the hierarchical visual self-attention model based on the moving window are respectively connected to a convolutional block attention module. As a result, the second feature can include features of all levels, further improving the attention to local details of the image to be detected, and can improve the accuracy of defect category detection.

[0071] Exemplarily, in step S120 , the post-processing module classifies the second feature to obtain a classification result, including step S123 , step S124 , and step S125 .

[0072] In step S123, a convolution operation is performed on the second feature to obtain a third feature.

[0073] The second feature may include multi-channel features, and the number of channels in the second feature may be reduced through a convolution operation to obtain a third feature. For example, a 1×1 convolution is performed on the second feature. The second feature after the 1×1 convolution can be used as the third feature.

[0074] In step S124, global average pooling is performed on the third feature to obtain a fourth feature.

[0075] Global average pooling can add all feature values ​​of each classification category to calculate the mean, and use the mean to identify the feature value of the classification category. The fourth feature can be obtained by global average pooling of the third feature, reducing the subsequent calculation amount and reducing overfitting.

[0076] In step S125, the confidence that the defects of the target object belong to different categories is calculated based on the fourth feature to obtain a classification result.

[0077] The fourth feature may include the mean of the feature values ​​of each classification category, and the confidence of each classification category may be calculated according to the mean of the feature values ​​of each classification category. The classification category with the largest confidence is used as the classification result of the defect of the target object. For example, the fourth feature may be calculated using a normalized exponential (Softmax) function to determine the classification result of the defect of the target object corresponding to the image to be detected.

[0078] The above technical solution obtains the fourth feature by convolving the second feature and performing global average pooling. The confidence that the defects of the target object belong to different categories is calculated based on the fourth feature to obtain the classification result. In this way, the amount of calculation can be reduced and the speed of defect category detection can be improved. In addition, the confidence can help users judge whether the classification result is reliable and improve the user experience.

[0079] Exemplarily, the above defect category detection method also includes: based on the cross entropy loss function and the center loss function as the loss function, using the labeled sample images to train the classification model.

[0080] The labeled sample images may include images of target objects labeled with defect categories. The sample images may be flipped, cropped, and other operations may be performed to increase the robustness of the trained classification model. The cross entropy loss function may describe the difference between the classification result of the classification model for the image to be tested and the true result. The center loss function may focus on the compactness of the defect image to be tested with the same classification result in the feature space, as well as the separation of different classification results. The classification model may be trained based on the cross entropy loss function and the center loss function as loss functions. The loss function may be calculated according to the following formula:

[0081] L=L CE +λL Center

[0082] Where L represents the loss function; L ce represents the cross entropy loss function; L Center represents the center loss function; λ represents a hyperparameter. λ can be set by the user to balance the weights of the cross entropy loss function and the center loss function.

[0083] The above technical solution uses the cross entropy loss function and the center loss function as loss functions to train the classification model, which can improve the classification accuracy of the trained classification model, and further improve the classification accuracy of similar defects.

[0084] For example, Figure 2FIG. 1 is a schematic diagram showing a defect category detection method according to an embodiment of the present invention. Figure 2 As shown in the figure, before the image to be detected (Image) is input into the hierarchical visual self-attention model based on the moving window, the image to be detected can be processed by the patch embedding module (Patch Embedding) to facilitate subsequent model processing. The hierarchical visual self-attention model (Swin Transformer) based on the moving window includes four hierarchical units, hierarchical unit 1 (stage1), hierarchical unit 2 (stage2), hierarchical unit 3 (stage3) and hierarchical unit 4 (stage4), which are connected in series in sequence, and each hierarchical unit is connected to a convolutional block attention module (CBAM). The calculation result of the convolutional block attention module connected to the hierarchical unit 1 is downsampled by 8 times; the calculation result of the convolutional block attention module connected to the hierarchical unit 2 is downsampled by 4 times; the calculation result of the convolutional block attention module connected to the hierarchical unit 3 is downsampled by 2 times. All sampling results and the calculation result of the convolutional block attention module connected to the hierarchical unit 4 are concatenated (CAT). The concatenated results are subjected to global average pooling (GAP). The classification result is further obtained by the normalized exponential function. Specifically, the activation function softmax can be used to convert the output into a probability representation to represent the probability that the target object belongs to each different category in its defect classification result.

[0085] Illustratively, according to yet another aspect of the present invention, a device for detecting defect categories is also provided. Figure 3 FIG. 3 is a schematic block diagram of a defect category detection device 300 according to an embodiment of the present invention. Figure 3 As shown, the defect category detection device 300 includes: a first acquisition module 310 and a classification module 320 .

[0086] The first acquisition module 310 is used to acquire the image to be detected of the target object. The classification module 320 is used to input the image to be detected into the trained classification model to obtain the classification result of the defects of the target object in the image to be detected, wherein the classification model includes a feature extraction module, a convolution block attention module and a post-processing module connected in sequence; the feature extraction module is used to extract the first feature of the image to be detected; the convolution block attention module is used to perform channel attention calculation and / or spatial attention calculation on the received first feature, and determine the second feature according to the calculation result; the post-processing module is used to classify the second feature to obtain the classification result.

[0087] Exemplarily, the feature extraction module is implemented using a hierarchical visual self-attention model based on a moving window, which includes a plurality of hierarchical units, the plurality of hierarchical units are connected in series, and the plurality of hierarchical units include at least one first-level unit, and each first-level unit is respectively connected to a convolutional block attention module.

[0088] Exemplarily, the first hierarchical unit includes the last hierarchical unit among the plurality of hierarchical units.

[0089] Exemplarily, the first hierarchical unit includes N, where N is a positive integer greater than 1. The classification module 320 includes a sampling submodule and a splicing submodule. The sampling submodule is used to downsample or upsample the calculation results of at least N-1 convolutional block attention modules to obtain respective sampling results, wherein all sampling results have the same size, and in the case of unsampled calculation results, all sampling results have the same size as the unsampled calculation results. The splicing submodule is used to splice all sampling results, and in the case of unsampled calculation results, splice all sampling results with the unsampled calculation results to obtain the second feature.

[0090] Exemplarily, the plurality of hierarchical units are all the first hierarchical units.

[0091] Exemplarily, the classification module 320 includes a first acquisition submodule, a second acquisition submodule and a third acquisition submodule. The first acquisition submodule is used to perform a convolution operation on the second feature to obtain a third feature. The second acquisition submodule is used to perform a global average pooling operation on the third feature to obtain a fourth feature. The third acquisition submodule is used to calculate the confidence that the defect of the target object belongs to different categories based on the fourth feature to obtain the classification result.

[0092] Exemplarily, the defect category detection device further includes a training module. The training module is used to train the classification model using labeled sample images based on a cross entropy loss function and a center loss function as loss functions.

[0093] Illustratively, according to yet another aspect of the present invention, an electronic device is also provided. Figure 4 A schematic block diagram of an electronic device 400 according to an embodiment of the present invention is shown. The electronic device 400 includes a processor 410 and a memory 420. The memory 420 stores computer program instructions, which are used by the processor 410 to execute the defect category detection method described above when the computer program instructions are executed.

[0094] Exemplarily, according to another aspect of the present invention, a storage medium is provided, on which program instructions are stored, and the program instructions are used to execute the above-mentioned defect category detection method when running. The storage medium may include, for example, an erasable programmable read-only memory (EPROM), a portable read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The storage medium may be any combination of one or more computer-readable storage media.

[0095] Illustratively, according to yet another aspect of the present invention, there is further provided a computer program product, comprising computer program instructions, wherein the computer program instructions are used to execute the defect category detection method as described above when being run.

[0096] A person skilled in the art can understand the specific implementation schemes and beneficial effects of the above-mentioned defect category detection devices, electronic devices, storage media and computer program products by reading the above-mentioned descriptions of the defect category detection methods. For the sake of brevity, they will not be elaborated here.

[0097] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application to this. Those of ordinary skill in the art may make various changes and modifications therein without departing from the scope and spirit of the present application. All these changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0098] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0100] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0101] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present application should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with features less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0102] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0103] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0104] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules in the defect category detection device according to the embodiment of the present application. The present application can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0105] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0106] The above is only a specific implementation or description of a specific implementation of the present application, and the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A defect category detection method, characterized in that: include: Acquire an image of a target object to be detected; as well as The image to be detected is input into a trained classification model to obtain a classification result of the defects of the target object in the image to be detected, wherein the classification model includes a feature extraction module, a convolution block attention module and a post-processing module connected in sequence; the feature extraction module is used to extract the first feature of the image to be detected; the convolution block attention module is used to perform channel attention calculation and / or spatial attention calculation on the received first feature, and determine the second feature according to the calculation result; the post-processing module is used to classify the second feature to obtain the classification result.

2. The defect type detection method according to claim 1, characterized in that: The feature extraction module is implemented using a hierarchical visual self-attention model based on a moving window, which includes a plurality of hierarchical units, the plurality of hierarchical units are connected in series, and the plurality of hierarchical units include at least one first-level unit, and each first-level unit is respectively connected to a convolutional block attention module.

3. The defect type detection method according to claim 2, characterized in that: The first hierarchical unit includes a last hierarchical unit among the plurality of hierarchical units.

4. The defect type detection method according to claim 2, characterized in that: The first level units include N units, where N is a positive integer greater than 1; The convolution block attention module determines the second feature according to the calculation result, including performing the following operations: Downsample or upsample the respective calculation results of at least N-1 convolutional block attention modules to obtain respective sampled results, wherein all sampled results have the same size, and in the case where there are unsampled calculation results, all sampled results have the same size as the unsampled calculation results; All sampling results are concatenated, and when there are unsampled calculation results, all sampling results are concatenated with the unsampled calculation results to obtain the second feature.

5. The defect type detection method according to claim 4, characterized in that: The plurality of hierarchical units are all the first hierarchical units.

6. The defect type detection method according to any one of claims 1 to 5, characterized in that: The post-processing module classifies the second feature to obtain the classification result, including performing the following operations: Performing a convolution operation on the second feature to obtain a third feature; Performing global average pooling on the third feature to obtain a fourth feature; The confidence that the defect of the target object belongs to different categories is calculated according to the fourth feature to obtain the classification result.

7. The defect type detection method according to any one of claims 1 to 5, characterized in that: The detection method further comprises: Based on the cross entropy loss function and the center loss function being used together as the loss function, the classification model is trained using the labeled sample images.

8. A defect type detection device, characterized in that: include: A first acquisition module is used to acquire an image to be detected of a target object; as well as A classification module is used to input the image to be detected into a trained classification model to obtain a classification result of the defects of the target object in the image to be detected, wherein the classification model includes a feature extraction module, a convolution block attention module and a post-processing module connected in sequence; the feature extraction module is used to extract the first feature of the image to be detected; the convolution block attention module is used to perform channel attention calculation and / or spatial attention calculation on the received first feature, and determine the second feature according to the calculation result; the post-processing module is used to classify the second feature to obtain the classification result.

9. An electronic device, comprising: A processor and a memory, characterized in that The memory stores computer program instructions, which are used by the processor to execute the defect category detection method according to any one of claims 1 to 7 when the processor executes the computer program instructions.

10. A storage medium having program instructions stored thereon, characterized in that: The program instructions are used to execute the defect category detection method according to any one of claims 1 to 7 when running.

11. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the defect category detection method according to any one of claims 1 to 7 when being executed.

Images