An image defect detection method, device and storage medium

Abstract

The application provides an image defect detection method, device and storage medium, belonging to the technical field of data processing, aiming to improve the reliability of defect detection, the method comprising: acquiring an image to be detected; performing multi-level feature extraction on the image to be detected based on a feature extraction module, obtaining feature maps output by each feature layer in the feature extraction module, and determining a plurality of initial feature maps from the feature maps output by each feature layer; performing feature aggregation on the plurality of initial feature maps to obtain a target feature map; the feature aggregation comprises: after adjusting the resolution and dimension of the plurality of initial feature maps, performing linear transformation to output the target feature map; the initial feature map comprises: the feature map with the highest contribution degree for distinguishing whether the image feature is a good product feature or a defect feature among the feature maps output by each feature layer; performing defect detection on the target feature map based on a detection module to obtain a defect detection result of the image to be detected.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, specifically relating to an image defect detection method, device, and storage medium. Background Technology

[0002] Defect detection is a crucial step in ensuring product quality. Manual inspection requires a significant workforce, while automated defect detection using deep learning algorithms based on acquired images greatly reduces this burden and lowers labor costs for businesses. However, existing defect detection methods have low accuracy in identifying defects, making it difficult to meet production requirements. Summary of the Invention

[0003] This application provides an image defect detection method, device, and storage medium to address the problem of insufficient reliability in existing image defect detection methods.

[0004] A first aspect of this application provides an image defect detection method, the method comprising:

[0005] Acquire the image to be detected;

[0006] The feature extraction module performs multi-level feature extraction on the image to be detected, obtaining feature maps output by each feature layer in the feature extraction module. Multiple initial feature maps are then determined from the feature maps output by each feature layer. Feature aggregation is performed on these multiple initial feature maps to obtain a target feature map. The feature aggregation includes adjusting the resolution and dimension of the multiple initial feature maps, followed by a linear transformation to output the target feature map. The initial feature maps include the feature maps with the highest contribution to distinguishing between good and defective image features from the feature maps output by each feature layer.

[0007] Based on the detection module, defect detection is performed on the target feature map to obtain the defect detection result of the image to be detected.

[0008] A second aspect of this application also provides an image defect detection apparatus, the apparatus being used to perform the image defect detection method described in the first aspect, the apparatus comprising:

[0009] The input module is used to acquire the image to be detected;

[0010] The feature extraction module is used to perform multi-level feature extraction on the image to be detected, obtain feature maps output by each feature layer in the feature extraction module, determine multiple initial feature maps from the feature maps output by each feature layer, and perform feature aggregation on the multiple initial feature maps to obtain a target feature map; the feature aggregation includes: adjusting the resolution and dimension of the multiple initial feature maps, and then performing a linear transformation to output the target feature map; the initial feature maps include: the feature map with the highest contribution to distinguishing image features as good or defective features from the feature maps output by each feature layer.

[0011] The detection module is used to perform defect detection on the target feature map to obtain the defect detection result of the image to be detected.

[0012] A third aspect of this application also provides an electronic device, including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the image defect detection method as described in the first aspect.

[0013] A fourth aspect of this application also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the image defect detection method as described in the first aspect.

[0014] The beneficial effects of this application are as follows: The image defect detection method proposed in this application performs multi-level image feature extraction through a feature extraction module, and then autonomously selects features that are conducive to distinguishing between good and defective images (i.e., determining multiple initial feature maps from the feature maps output by each feature layer), and then aggregates the selected appropriate initial feature maps, and uses the detection module to perform detection scoring to obtain the defect detection result of the input image to be detected, thereby improving the reliability of the image defect detection result.

[0015] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the scale in the drawings is for illustration only and does not represent the actual scale.

[0017] Figure 1 This is a flowchart of the steps of an image defect detection method according to an embodiment of this application;

[0018] Figure 2 This is a schematic diagram of the structure of multiple feature layers in a feature extraction module according to an embodiment of this application;

[0019] Figure 3 This is a schematic diagram of a defect detection process in an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of the training process of an image defect detection model in an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of an image defect detection process in an embodiment of this application;

[0022] Figure 6 This is a schematic diagram of a target feature layer selection process in an embodiment of this application;

[0023] Figure 7 This is a schematic diagram illustrating the effect of image restoration in one embodiment of this application;

[0024] Figure 8 This is a schematic diagram of the network structure of a detection module in one embodiment of this application;

[0025] Figure 9 This is a schematic diagram of the structure of an image defect detection device according to an embodiment of this application;

[0026] Figure 10 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0027] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, the first object can be one or at least two. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0029] Defect detection is a crucial step in ensuring product quality. Manual inspection requires a significant workforce, while automated defect detection using deep learning algorithms based on acquired images greatly reduces this burden and lowers labor costs for businesses. However, existing defect detection methods suffer from low accuracy, failing to meet production requirements. Specifically, current anomaly detection algorithms (such as RealNet and EfficientAD) typically only apply to one type of scenario. For multiple scenarios, multiple defect detection models need to be trained specifically. In actual industrial production, defects vary greatly in size, color, shape, and background, and new defect types are easily introduced during production, making it difficult to build a classifier applicable to new scenarios and capable of accurately identifying all types of defects.

[0030] In view of the above problems, this application provides an image defect detection method, device and storage medium. The method performs multi-level image feature extraction through a feature extraction module, and then autonomously selects features that are conducive to distinguishing between good and defective images (i.e., determine multiple initial feature maps from the feature maps output by each feature layer). The selected appropriate initial feature maps are then aggregated, and the detection module is used to perform detection scoring to obtain the defect detection result of the input image to be detected, so as to improve the reliability of the image defect detection result.

[0031] The first aspect of this application proposes an image defect detection method. The image defect detection method proposed in the embodiments of this application will be specifically described below with reference to sections 1.1-1.4.

[0032] 1.1 A general description of the overall scheme of image defect detection method.

[0033] This image defect detection method is applied to an image defect detection model, which includes at least a feature extraction module and a detection module. The feature extraction module includes multiple feature layers and an Adaptive Feature Aggregation Module (AFAM). (Refer to...) Figure 1 , Figure 1 A flowchart illustrating the steps of an image defect detection method is shown, as follows: Figure 1 As shown, the method includes:

[0034] Step S101: Obtain the image to be detected.

[0035] Specifically, the image to be detected refers to an image captured of a specific object (the image may include the image background and the target object). The method proposed in this application can be applied to the scenario of industrial defect detection. The image to be detected is an image captured of an industrial product against an industrial background. Through the above steps S101-S103, the method detects whether there are defects on the surface of the object in the image (i.e., defects, flaws, etc. on the surface of the industrial product). If there are defects on the surface of the object, the corresponding defect detection result indicates that the image to be detected is a defective image (i.e., the object is a defective object); if there are no defects on the surface of the object, the corresponding defect detection result indicates that the image to be detected is not a defective image (i.e., the object is a good object).

[0036] Step S102: Based on the feature extraction module, multi-level feature extraction is performed on the image to be detected to obtain feature maps output by each feature layer in the feature extraction module. Multiple initial feature maps are determined from the feature maps output by each feature layer. Feature aggregation is performed on the multiple initial feature maps to obtain a target feature map. The feature aggregation includes: adjusting the resolution and dimension of the multiple initial feature maps, and then performing a linear transformation to output the target feature map. The initial feature maps include: the feature map with the highest contribution to distinguishing image features as good or defective features from the feature maps output by each feature layer.

[0037] In this embodiment, the feature extraction module can be pre-trained weights, i.e., a classification network trained on large-scale natural scene images, such as the Wideresnet50 model trained on the ImageNet dataset. Pre-trained weights refer to model weights pre-trained on a large-scale public dataset like ImageNet (natural scene images), already possessing a certain feature extraction capability. For tasks with limited real-world data (making it difficult to retrain a network from scratch, prone to overfitting), an initial model (pre-trained weights) is typically trained on a large-scale natural scene dataset to give it some feature extraction capability. Then, the model is fine-tuned using a small-scale real-world scene dataset to adapt it to the task at hand. Pre-trained weight models, trained on natural scene datasets, are not entirely suitable for feature extraction from industrial scene images.

[0038] Specifically, refer to Figure 2 , Figure 2 This diagram illustrates the structure of multiple feature layers in a feature extraction module, such as... Figure 2 As shown, in the architecture of this feature extraction module, the input image first passes through a 7*7 convolutional layer, a BN+ReLU normalization and activation function layer, and a 3*3 Maxpool2d pooling layer, and then through four cascaded feature layers (i.e., ... Figure 2 The algorithm performs feature extraction step-by-step on modules 1-4 to obtain the feature extraction results (the feature maps output by each feature layer). Specifically, the first feature layer (module 1) consists of a bottleneck layer, a 1x1 convolutional layer, a BN normalization layer, and two bottleneck layers (bottleneck*2) in sequence; the second feature layer (module 2) consists of a bottleneck layer, a 1x1 convolutional layer, a BN normalization layer, and three bottleneck layers (bottleneck*3) in sequence; the third feature layer (module 3) consists of a bottleneck layer, a 1x1 convolutional layer, a BN normalization layer, and five bottleneck layers (bottleneck*5) in sequence; and the fourth feature layer (module 4) consists of a bottleneck layer, a 1x1 convolutional layer, a BN normalization layer, and two bottleneck layers (bottleneck*2) in sequence. The bottleneck layer consists of the following layers connected in series: a 1*1 convolutional layer with BN normalization, a 3*3 convolutional layer with BN normalization, and a 1*1 convolutional layer with BN normalization and ReLU activation function.

[0039] Based on the adaptive feature aggregation module, multiple initial feature maps are determined from the feature maps output by each feature layer. Feature aggregation is then performed on these initial feature maps to obtain the target feature map. The initial feature map refers to the feature map with the highest contribution in distinguishing between good and defective image features from the feature maps output by each feature layer. The process of feature aggregation on multiple initial feature maps may include adjusting the resolution and dimensions of the multiple initial feature maps, followed by a linear transformation to output the target feature map.

[0040] Because industrial images differ from images in the ImageNet dataset (a natural scene dataset), directly using the features output from pre-trained weights results in domain bias and is not entirely suitable for feature extraction from industrial scene images. Therefore, this embodiment adds an adaptive feature aggregation module after the pre-trained weights to generate features tailored to industrial scenes. Specifically, by training the model, the adaptive feature aggregation module learns to automatically select feature maps (initial feature maps) that are more conducive to distinguishing between good and defective features, thus indicating good and defective features.

[0041] Step S103: Based on the detection module, perform defect detection on the target feature map to obtain the defect detection result of the image to be detected. The defect detection result indicates whether a defect exists in the image to be detected. If no defect exists, the image is a good image; if a defect exists, the image is a defective image. Furthermore, the defect detection result may also include the specific location of the defect in the image.

[0042] Reference Figure 3 , Figure 3 A schematic diagram of a defect detection process is shown, such as... Figure 3 As shown, specifically, in step S102, the feature extraction module includes a multi-layered cascaded feature layer and an adaptive feature aggregation module. Feature extraction is performed level by level through each feature layer (multi-level feature extraction is performed on the image to be detected), and each feature layer outputs the feature map extracted at that layer. Then, the adaptive feature aggregation module selects initial feature maps from the various feature maps that are more conducive to distinguishing between good and defective features. These selected initial feature maps are then aggregated to obtain the target feature map. Finally, in step S103, defect detection and scoring are performed on the target feature map to obtain the defect detection result, thereby improving the reliability of defect detection.

[0043] 1.2 Specific scheme for selecting initial feature maps in the adaptive feature aggregation module.

[0044] This embodiment utilizes an adaptive feature aggregation module to determine multiple initial feature maps from the feature maps output by each feature layer of the feature extraction module. These initial feature maps are those that best distinguish between good and defective features; that is, the feature maps with the highest contribution to differentiating image features as good or defective features. In this embodiment, during the training of the image defect detection model, the contribution of the feature maps output by each feature layer can be calculated, and the multiple feature layers with the highest contribution can be determined as target feature layers. Therefore, in practical applications, the feature maps output by the target feature layers are used as the initial feature maps.

[0045] 1.2.1 First, a brief description of the training process of the image defect detection model (the feature extraction module and the detection module) will be given.

[0046] In one possible implementation, the feature extraction module, the adaptive feature aggregation module, and the detection module are trained through the following steps:

[0047] Step S201: Create a sample image pair consisting of a good product image and a defective image with the same background. In the sample image pair, the good product image refers to an image of a good object without defects, and the defective image refers to an image of a defective object with surface defects.

[0048] Image pairs with similar backgrounds refer to two images in a pair that share similar background content or structure. Specifically, this means that the texture, color, lighting, and other features of the background portion (i.e., non-primary target or defect area) of the two images are very similar or identical. This background similarity helps the algorithm better focus on the main target or defect area in the image, rather than being distracted by background variations. In the sample image pair, the good product image and the defective image are very similar in the background portion; the only difference is that the defective image contains some kind of defect (such as scratches, stains, etc.). By comparing these two images, the algorithm can more easily identify the defective area because the background similarity reduces the interference of the background on the detection results.

[0049] Reference Figure 4 , Figure 4 A schematic diagram illustrating the training process of an image defect detection model is shown, such as... Figure 4 As shown, sample image pairs can be created from three directions. One is based on defective images, using image inpainting algorithms to repair defects in defective images to obtain "good" images and generate sample image pairs. Another is based on good images, using image segmentation models to segment defects from other real defective images and paste them into the good image to obtain "defective" images and generate sample image pairs. The third is to extract image features from the good image, add Gaussian noise to the image features, obtain the corresponding "defective" images, and generate sample image pairs.

[0050] Step S202: Input the good product image and the defective image in the sample image pair into the feature extraction module for multi-level feature extraction to obtain the sample feature map output by each feature layer in the feature extraction module; the sample feature map includes: sample good product feature map and sample defective feature map.

[0051] Step S203: Based on the adaptive feature aggregation module of the feature extraction module, multiple target feature layers are determined from each feature layer, and the sample feature maps output by the target feature layers are determined as the initial sample feature maps. Feature aggregation is performed on the initial sample feature maps to obtain the target sample feature maps. The target feature layer refers to the feature layer among the various feature layers that contributes the most to distinguishing image features as good or defective features. Figure 4 As shown, after feature extraction of the sample image pairs, the adaptive feature aggregation module AFAM selects appropriate feature maps for feature aggregation to obtain sample target feature maps, including sample good product target feature maps and sample defect target feature maps.

[0052] Step S204: Input the sample target feature map into the detection module for defect detection to obtain the defect detection score of the sample image pair.

[0053] Step S205: Based on the defect detection score, calculate the loss function value, and update the parameters of the feature extraction module and the detection module according to the loss function value to obtain the trained feature extraction module and detection module. Specifically, each parameter update is considered as completing one round of model training. Then, repeat steps S202-S205 to complete multiple rounds of model training until the preset number of training iterations is reached or the loss function value converges, at which point training ends, and the trained image defect detection model is obtained.

[0054] In one possible implementation, the defect detection score includes: the defect detection score of each pixel in the good product image and the defect detection score of each pixel in the defective image;

[0055] During model training, when executing step S205, the loss function value is calculated based on the defect detection score, including:

[0056] The training objective is to set the defect detection score of pixels in normal regions of an image to be greater than a first preset threshold, and the defect detection score of pixels in defective regions to be less than a negative first preset threshold. The loss function value is calculated based on the defect detection scores using the following formula:

[0057] lp_loss=max(-lp_scores+th,0);

[0058] qx_loss=max(qx_scores·mask+th,0);

[0059] Loss=mean(lp_loss)+mean(qx_loss);

[0060] Wherein, lp_scores is the defect detection score of each pixel in the good product image, th is the first preset threshold; qx_scores is the defect detection score of each pixel in the defective image, mask is the defect score of each pixel in the mask image of the defective image, lp_loss is the loss of the good product image, qx_loss is the loss of the defective image, and Loss is the loss function value.

[0061] Specifically, this embodiment designs a corresponding class segmentation loss function. The training objective is to ensure that the defect detection score of pixels in the normal region without defects in the image is greater than 0, the defect detection score of pixels in the defective region is less than 0, and the calculated loss function value (loss) approaches 0. To prevent overfitting, truncation terms th and -th (i.e., the first preset threshold) are set. This means that through training, the defect detection score of pixels in the normal region of the feature map is greater than or equal to the first preset threshold th, and the defect detection score of pixels in the defective region is less than the negative first preset threshold -th. The first preset threshold th defaults to 0.5 and can be set manually. The defect detection scores of pixels at various locations in the good product image are denoted as lp_scores (the entire feature map is a normal region), and the defect detection scores of pixels at various locations in the defective image are denoted as qx_scores (the defective image contains both normal and defective regions). The mask image of the defective image marks the location of defects in the defective image (-1 in the mask represents a normal region, and 1 represents a defective region).

[0062] Reference Figure 5 , Figure 5 A schematic diagram of an image defect detection process is shown, such as... Figure 5 As shown, when training the model, this embodiment first obtains 'good product-defect' image pairs with the same background through the data acquisition module, and obtains 'good product-defect' image feature pairs through feature extraction. The detection module performs score prediction on each pixel in the feature pair to obtain the detection result that the object in the corresponding image belongs to good product or defect (where the dashed arrow part represents the processing flow of defective image). Through the contrastive learning mechanism, the model can better understand and distinguish between good product and defect features, and improve the identification ability.

[0063] 1.2.2 Methods for determining the target feature layer.

[0064] In one possible implementation, step S203, determining multiple target feature layers from each feature layer, includes:

[0065] Step S203-1: Calculate the difference between the good sample feature map and the defective sample feature map output by the feature layer of each layer.

[0066] Reference Figure 6 , Figure 6 A schematic diagram of the target feature layer selection process is shown, such as... Figure 6 As shown, the good and defective images in the sample image pair are respectively input into the feature extraction module for multi-level processing (e.g., ... Figure 6 The feature extraction shown in layers 1, 2, ..., 1i) involves the output of a sample good product feature map F (corresponding to the good product image) for each feature layer. lpi And the sample defect feature map F (corresponding to the defect image). qxi , where i represents the i-th feature layer. Calculate F lpi and F qxi The difference Δ between the two i This difference reflects the discrepancy between good and defective images in the feature space and is the core component of the adaptive feature aggregation module. By analyzing these differences and comparing them with the mask, the adaptive feature aggregation module can automatically select the feature layers and channels that are most effective in distinguishing between good and defective products (i.e., select multiple feature layers whose differences are closest to the mask), thereby improving the model's detection performance.

[0067] Step S203-2: Based on the difference between the sample good product feature map and the sample defect feature map, calculate the contribution of the feature layer in distinguishing image features as good product features or defect features. The contribution level can be calculated as the degree of closeness between the difference between the sample good product feature map and the sample defect feature map and the mask image in each feature layer. The closer the difference is to the mask, the higher the contribution level of the corresponding feature layer.

[0068] Step S203-4: The multiple feature layers with the highest contribution are determined as the target feature layers.

[0069] In the process of training the model using sample images, the contribution level of the feature maps output by each feature layer is calculated (to distinguish whether image features are good or defective features). The feature layers with the highest contribution levels are then identified as target feature layers, and the feature maps output by these target feature layers are identified as initial feature maps. For example, if feature layers 2 and 4 are found to have the highest contribution levels, they are selected as initial feature layers. In practical applications, the feature maps output by feature layers 2 and 4 are selected as initial feature maps, aggregated, and then the target feature map is obtained and input into the detection module for defect detection and scoring.

[0070] In one possible implementation, step S203-1, calculating the difference between the good sample feature map and the defective sample feature map output by each feature layer, includes:

[0071] For the sample good feature map and sample defect feature map output by the feature layer of each layer, the number of channels is compressed to 1 by global average pooling to obtain the compressed good feature map and compressed defect feature map. Specifically, the sample good feature map F output by the feature layer of the i-th layer... lpi and sample defect feature map F qxi F is obtained by global average pooling along the channel dimension. lpi and F qxi The number of channels is compressed to 1, resulting in the compressed good product feature map F. lpi ′ and the defect feature map F after compression qxi ′.

[0072] The square of the difference between the compressed good product feature map and the compressed defect feature map is calculated and normalized to obtain a first calculation result; the first calculation result characterizes the magnitude of the difference between the sample good product feature map and the sample defect feature map. Specifically, F is calculated. lpi ′ and F qxi The square of the difference between ′, i.e., |F lpi ′-F qxi ′| 2 Normalizing the square of this difference yields the first calculated result Δ. i The first calculation result characterizes the difference between the good product feature map and the defective product feature map of the sample.

[0073] Step S203-2, based on the difference between the sample good product feature map and the sample defect feature map, calculate the contribution of the feature layer in distinguishing image features as good product features or defect features, including:

[0074] After adjusting the first calculation result to the same resolution as the mask image of the sample defect feature map using bilinear interpolation, the square of the difference between the first calculation result and the mask image of the sample defect feature map is calculated to obtain the second calculation result. Specifically, the first calculation result Δ i After adjusting the resolution of the mask image to match that of the sample defect feature map using bilinear interpolation, the two (Δ) are calculated. i The square of the difference between (and mask), i.e., |Δ-mask| 2 Specifically, the mask image of the sample defect feature map refers to the mask image of the sample image pair, used to represent the defective region of the defective image in the sample image pair relative to the good product image. The mask image and the first calculation result Δ i The smaller the difference, the smaller the difference between the good and defective features extracted by this feature layer (the first calculation result Δ). i This is closer to the difference (mask) between a real good product image and a defective image. Therefore, the second calculation result |Δ-mask| 2 The smaller the value, the higher the contribution, and it is used to assess the difference in characteristics Δ. i The consistency between the mask and the actual defect location is an important indicator and also an important basis for model optimization.

[0075] Using N pairs of sample images, the mean of the second calculation result of each feature layer is calculated, which serves as the third calculation result for that feature layer. This third calculation result characterizes the contribution of the feature map output by that feature layer to distinguishing between good and defective image features. Specifically, each pair of sample images can yield a corresponding second calculation result |Δ-mask|. 2 The process involves calculating N second calculation results for each pair of sample images using N (N can be any positive integer greater than 1), then averaging these results to obtain a third calculation result. This third calculation result represents the average performance of the feature layer across the entire dataset (i.e., N pairs of sample images). The third calculation result indicates the contribution of the feature map output by the corresponding feature layer; a smaller value indicates a higher contribution. In steps S203-4, the feature layers with the highest contribution are determined as the initial feature layers, meaning the feature layers with the smallest third calculation results are selected as the initial feature layers.

[0076] Unlike manually setting the number of feature extraction layers, this embodiment designs an adaptive feature aggregation module, which enables the network to learn and autonomously select feature layers (i.e., target feature layers) that are conducive to distinguishing between good products and defects, so as to obtain more accurate and reliable defect detection results.

[0077] 1.2.3 Select a more suitable target channel from each channel in each feature layer.

[0078] This embodiment proposes selecting several suitable channels from each channel of the feature layer. Considering that each feature layer includes multiple channels, feature extraction is performed on each channel to output the corresponding feature map. Following the scheme in 1.2 for selecting the target feature layer that is more conducive to distinguishing between good and defective products, this embodiment proposes that after determining the target feature layer, an adaptive feature aggregation module is used to determine multiple target channels from each channel of the target feature layer. The target channel refers to the channel among all channels that is more conducive to distinguishing between good and defective features, that is, the channel that contributes the most to distinguishing image features as good or defective features.

[0079] In one possible implementation, after determining the target feature layer, the method further includes:

[0080] From each channel of each target feature layer, multiple target channels are determined, wherein the target channel is the channel in which the output feature map contributes the most to distinguishing image features as good features or defective features;

[0081] The feature maps output by the target channel are aggregated to obtain the sample feature maps output by the target feature layer.

[0082] In one possible implementation, determining a plurality of target channels from each channel of each of the target feature layers includes:

[0083] Step S301: For the first good feature map and the first defect feature map output by each channel of the target feature layer, the number of channels is compressed to 1 by global average pooling to obtain the second good feature map and the second defect feature map.

[0084] Specifically, the first good product feature map f corresponding to the good product image output by the i-th channel. lpi The first defect feature map f corresponding to the defect image qxi f is obtained by global average pooling along the channel dimension. lpi and f qxi The number of channels is compressed to 1, resulting in the second good product feature map f. lpi ′ and second defect feature map f qxi ′.

[0085] Step S302: Calculate the square of the difference between the second good feature map and the second defect feature map, and normalize it to obtain a fourth calculation result; the fourth calculation result represents the magnitude of the difference between the first good feature map and the first defect feature map.

[0086] Specifically, calculate flpi ′ and f qxi The square of the difference between ′, i.e., |f lpi ′-f qxi ′| 2 Normalizing the square of this difference yields the fourth calculated result Δ. i The fourth calculation result Δ i ′ represents the difference between the first good feature map and the first defect feature map extracted from the i-th channel of the feature layer.

[0087] Step S303: After adjusting the fourth calculation result to the same resolution as the mask image of the first defect feature map using bilinear interpolation, the square of the difference between the fourth calculation result and the mask image of the first defect feature map is calculated to obtain the fifth calculation result.

[0088] Specifically, the fourth calculation result Δ i After adjusting the resolution of the mask image to match that of the first defect feature map using bilinear interpolation, the two (Δ) are calculated. i The square of the difference between ′ and mask, i.e., |Δ i ′-mask| 2 Specifically, the mask image of the first defect feature map refers to the mask image of the sample image pair, used to represent the defective region of the defective image in the sample image pair relative to the good image. The mask image and the fourth calculation result Δ i The smaller the difference between ' and ', the smaller the difference between the good and defective features extracted by this channel (fourth calculation result Δ). i The result is closer to the difference between a real good image and a defective image (mask). Therefore, the fifth calculation result |Δ i ′-mask| 2 The smaller the value, the higher the contribution of that channel.

[0089] Step S304: Using N pairs of sample image pairs, calculate the mean of the fifth calculation result for each channel, and use it as the sixth calculation result for that channel. The sixth calculation result is used to characterize the contribution of the feature map output by that channel to distinguish whether the image features are good or defective features.

[0090] Specifically, for each pair of sample images, a corresponding fifth calculation result |Δ can be calculated. i ′-mask| 2Therefore, N fifth calculation results are obtained by using N (N can be any positive integer greater than 1) for each pair of sample images. The average value is then taken to obtain the sixth calculation result, which represents the average effect of that channel in the entire dataset (i.e., N pairs of sample images). The sixth calculation result represents the contribution of the feature map output by the corresponding channel. The smaller the value of the sixth calculation result, the higher the contribution.

[0091] Step S305: Determine the multiple channels with the smallest sixth calculation result as the target channels. This means determining the multiple channels with the highest contribution as target channels, i.e., determining the multiple channels with the smallest sixth calculation result as target channels.

[0092] This embodiment proposes that during the training of the image defect detection model, the contribution level of the feature map output by each channel can be calculated, and the channels with the highest contribution level can be determined as target channels. In practical applications, the feature map output by the target channel of the target feature layer can be determined as the initial feature map, thereby further improving the reliability of the final defect detection result.

[0093] 1.3 Creation of sample image pairs.

[0094] This embodiment proposes three methods for creating sample image pairs, which are described in sections 1.3.1, 1.3.2, and 1.3.3 respectively. In practical applications, one or more of the above methods can be used simultaneously to generate sample image pairs, thereby further enriching the training dataset. The sample image pairs can then be used to train the image defect detection model (feature extraction module, adaptive feature aggregation module, and detection module) to improve the model's defect detection capability.

[0095] 1.3.1 A method for generating defect features based on good product features.

[0096] In one possible implementation, the sample target feature map includes a sample good product target feature map and a sample defect target feature map; the method further includes: obtaining the sample target feature map according to the following steps;

[0097] Obtain a first sample image, which is a first good product image;

[0098] The first sample image is input into the feature extraction module for multi-level feature extraction to obtain the first sample feature map output by each feature layer in the feature extraction module;

[0099] The adaptive feature aggregation module based on the feature extraction module determines the initial feature map of the first sample from the first sample feature map output by each feature layer, and performs feature aggregation on the initial feature map of the first sample to obtain the target feature map of the first sample good product.

[0100] Noise is added to the first sample good product target feature map to obtain the first sample defect target feature map. The first sample good product target feature map and the first sample defect target feature map are combined as the sample target feature map.

[0101] In this embodiment, based on the acquired first good product image, after obtaining the good product image features (i.e., the first sample good product target features) through feature extraction, Gaussian noise is superimposed on the feature space. Specifically, firstly, noise addition positions are randomly generated (e.g., adding noise to a feature map of size h*w, randomly selecting several pixels from h*w positions as noise addition positions). Noise is added at these positions by adding the noise to the good product features, resulting in defect image features (i.e., the first sample defect target features). The feature pair composed of the good product image features and the defect image features is then used as training data. Specifically, firstly, a real first good product image is acquired. Then, feature extraction can be performed on the first good product image using a feature extraction module (or other feature extraction methods). The extracted first sample feature maps are aggregated to obtain the first sample good product target feature map. Gaussian noise is superimposed on the feature space to obtain the corresponding first sample defect target feature map, thus forming a feature map pair (first sample good product target feature map - first sample defect target feature map). This embodiment generates defect image features by adding noise to the good product image features, which can achieve the acquisition of richer training data.

[0102] In one possible implementation, the sample image pair includes: a first sample image pair;

[0103] The creation of sample image pairs consisting of good and defective images with the same background includes:

[0104] Obtain the first good product image;

[0105] Feature extraction is performed on the first good product image to obtain the features of the first good product image;

[0106] Gaussian noise is superimposed on the features of the first good product image to obtain the features of the first defective image, and a first defective image is generated based on the features of the first defective image.

[0107] The combination of the first good product image and the first defective image is used as the first sample image pair.

[0108] In this embodiment, based on the acquired good product image, after obtaining the good product image features through feature extraction, defect image features are obtained by superimposing Gaussian noise in the feature space. The feature pair composed of the good product image features and the defect image features is then used as training data. Specifically, a real first good product image is first acquired. Then, features are extracted from the first good product image using a feature extraction module (or other feature extraction methods) to obtain the first good product image features. Gaussian noise is then superimposed in the feature space to obtain the corresponding first defect image features. Based on the first defect image features, a first defect image with the same background as the first good product image is generated (the image generation method is not limited in this embodiment), thus forming a sample image pair (first good product image - first defect image). This embodiment generates defect image features by adding noise to the good product image features, which can achieve the generation of defect images with a richer variety.

[0109] 1.3.2 A method for generating defective images based on good product images.

[0110] In one possible implementation, the sample image pair includes: a second sample image pair;

[0111] The creation of sample image pairs consisting of good and defective images with the same background includes:

[0112] Using an image segmentation model, defect targets in real defect images are segmented to obtain defect target sub-images;

[0113] Obtain the image of the second good product;

[0114] After preprocessing the defective target sub-image, the second good product image is pasted onto it to obtain a second defective image with the same background as the second good product image. The preprocessing is to change at least one of the position, size and rotation angle of the defective target sub-image.

[0115] The combination of the second good product image and the second defective image is used as the second sample image pair.

[0116] In this embodiment, based on the acquired good product image, defects from other real defect images are copied and pasted into the good product image to generate a defect image with the same background. Specifically, a real second good product image is acquired, and then an image segmentation model (e.g., SAM segmentation model) is used to segment the defect target (e.g., surface blemishes) from the existing real defect image (e.g., an image taken by a camera targeting the defective surface of a defective product), obtaining a defect target sub-image. This defect target sub-image is then randomly adjusted in position, size, and rotation angle and pasted into the second good product image, thereby superimposing the defect area onto the good product image to generate a defect image. This results in a second defect image with the same background as the second good product image, forming a second sample image pair (second good product image - second defect image).

[0117] 1.3.3 A method for generating good product images based on defect images.

[0118] In one possible implementation, the sample image pair includes: a third sample image pair;

[0119] The creation of sample image pairs consisting of good and defective images with the same background includes:

[0120] Acquire a third defect image and a mask image that marks the region where the defect target is located in the third defect image;

[0121] Using an image segmentation model, the third defect image is segmented based on the mask image to remove the defect target in the third defect image and obtain a defect background image;

[0122] An image restoration algorithm is used to restore the area where the defective target is located in the defective background image to obtain a third good product image with the same background as the third defective image.

[0123] The combination of the third defect image and the third good product image is used as the third sample image pair.

[0124] In this embodiment, the acquired defect image is used to perform image inpainting on the region where the defective target is located, thereby eliminating the defective target and generating a good product image. Specifically, during the image inpainting process, the background of the good product image is used to repair the background of the defective image, generating an image pair with the same background but with the defective region repaired. Specifically, a real third defect image is acquired, and the location of the defective target in the third defect image is marked, resulting in a mask image of the third defective target (a black-and-white binary image with the same resolution as the third defect image, where black represents normal pixels and white represents the defect location). An image segmentation model (SAM segmentation model) is used to segment the third defect image, separating the region where the defective target is located while preserving the background, resulting in a defective background image. Then, an image inpainting algorithm is used to repair the missing parts in the defective background image, generating a third good product image with the same background as the third defective image. Considering that industrial images often possess regular texture information, this embodiment can use a texture synthesis-based image inpainting algorithm to repair defective areas. Specifically, the image inpainting algorithm can be the Criminisi image inpainting algorithm, which uses a mask combined with the real defective image to repair the defective area, resulting in a good product image. Optionally, provided that computing resources permit, stable diffusion inpainting technology can also be used as an image inpainting algorithm to repair defective areas.

[0125] For example, refer to Figure 7 , Figure 7 This illustrates the effect of an image restoration, such as... Figure 7 As shown, column (a) represents the actual defective image (i.e., the third defective image). First, [the image is...] Figure 7 The real image containing defects shown in (a) is segmented using the SAM segmentation model to determine the location of the defects, resulting in a mask image (b) showing the defect locations. The real image containing defects and its corresponding defect locations are then compared using the SAM segmentation model. Figure 7 Images (a) and (b) are fed into an image inpainting algorithm to repair the defective areas. After repair, the defect-free good image (i.e., the third good image) shown in (c) is obtained.

[0126] This embodiment uses one or more of the methods described in 1.3.1-1.3.3 above to create sample image pairs, thereby employing a multi-source data collaboration method (the above methods can obtain first good product images, second good product images, or third defect surprises from different sources), which greatly enriches the diversity of the training set, enabling the model to learn more types of defect patterns and improve the network's generalization ability when facing new scenarios and unknown defects.

[0127] 1.4 The specific processing procedure of the model.

[0128] 1.4.1 Scheme for feature aggregation of multiple target feature maps.

[0129] In step S102, multi-level feature extraction is performed on the image to be detected based on the feature extraction module. After obtaining the feature maps output by each feature layer in the feature extraction module, multiple initial feature maps are determined from the feature maps output by each feature layer. Feature aggregation is performed on the multiple initial feature maps to obtain the target feature map.

[0130] Specifically, step S102 involves performing feature aggregation on the multiple initial feature maps to obtain a target feature map, including:

[0131] The resolution of the initial feature map is unified by bilinear interpolation;

[0132] The initial feature map is adjusted to the first dimension by adaptive pooling and then stacked to obtain the first feature map;

[0133] The second feature map is obtained by adjusting the dimension of the first feature map to the first dimension through adaptive pooling;

[0134] The second feature map is linearly transformed to obtain the target feature map.

[0135] Specifically, an adaptive feature aggregation module extracts feature maps (i.e., initial feature maps) from a subset of feature layers (pre-trained weights WideResNet50). Using the maximum resolution as a baseline, bilinear interpolation is used to ensure all extracted initial feature maps have the same resolution. Assuming there are n final selected initial feature maps, adaptive pooling adjusts the dimensions of all n (n is usually 2) initial feature maps to the first dimension. The specific size of this first dimension can be set according to the actual application; an example first dimension could be h×w×1536, where 1536 represents the number of channels (i.e., the depth of the feature map). The specific value can be flexibly adjusted according to model design and task requirements. After stacking, a first feature map with dimensions h×w×n×1536 is obtained. Further adaptive pooling adjusts the dimension of the first feature map to h×w×1536 to obtain a second feature map. This second feature map is then transformed using a linear transformation layer to obtain the final feature map (dimension h×w×1536), i.e., the target feature map.

[0136] Correspondingly, during the training of the model, when executing step S203, based on the adaptive feature aggregation module, after determining the initial feature map of the sample from the output sample feature map of each feature layer, the initial feature map of the sample is aggregated to obtain the target feature map of the sample. The processing procedure is the same as the above process.

[0137] In one possible implementation, feature aggregation is performed on the initial feature map of the sample to obtain the target feature map of the sample, including:

[0138] The resolution of the initial feature map of the sample is unified by bilinear interpolation;

[0139] The dimensions of the initial feature maps of the samples are adjusted to the first dimension by adaptive pooling, and then stacked to obtain the first intermediate feature map;

[0140] The second intermediate feature map is obtained by adjusting the dimension of the first intermediate feature map to the first dimension through adaptive pooling.

[0141] A linear transformation is performed on the second intermediate feature map to obtain the sample target feature map.

[0142] 1.4.2 Processing scheme of the detection module.

[0143] In step S103, the detection module performs defect detection on the input target feature map to obtain the defect detection result of the image to be detected. In one possible implementation, performing defect detection on the target feature map to obtain the defect detection result of the image to be detected includes:

[0144] Step S1041: Perform a linear transformation on the target feature map to change the number of channels of the target feature map to 1024, thereby obtaining the third feature map.

[0145] Step S1042: Normalize the third feature map to obtain the normalization result.

[0146] Step S1043: Perform a nonlinear transformation on the normalization result using a nonlinear activation function to obtain the transformation result.

[0147] Step S1044: Perform a linear transformation on the transformation result to obtain the defect detection score of each pixel in the image to be detected.

[0148] Step S1045: Determine the defect detection result of the image to be detected based on the defect detection score of each pixel.

[0149] Specifically, the detection module can be a discriminator used to predict pixel-level scores for the target feature maps extracted from the image to be detected. This detection module employs a linear+BN+LeakyReLU+linear network structure. The extracted target feature maps all have a feature dimension of h×w×1536. These feature maps are sequentially processed through a linear layer (corresponding to step S1041, where the number of feature map channels changes from 1536 to 1024, resulting in an output feature map dimension of 2L×1024), a BN normalization layer (corresponding to step S1042, resulting in an output feature map dimension of 2L×1024), a LeakyReLU activation layer (corresponding to step S1043, resulting in an output feature map dimension of 2L×1024), and a linear layer (corresponding to step S1044, where the number of feature map channels is reduced to 1, resulting in an output feature map dimension of 2L×1) to obtain the predicted score map, which represents the defect detection score for each pixel in the image to be detected.

[0150] Correspondingly, during the training process, step S204 is executed, where the sample target feature map is input into the detection module for defect detection. The process of obtaining the defect detection score for the sample image pair is similar to the above-described process. Specifically, the sample target features include: a sample good product target feature map corresponding to a good product image, and a sample defect target feature map corresponding to a defective image. The feature dimensions of both the sample good product target feature map and the sample defect target feature map are h×w×1536. (Refer to...) Figure 8 , Figure 8 A schematic diagram of the processing flow of a detection module is shown, such as... Figure 8 As shown, the sample target feature map is transformed into an L*1536 dimension feature map by Reshape and then concatenated to obtain a 2L*1536 dimension feature map (L=h*w). Then, it is sequentially passed through a linear (corresponding to step S1041, the number of feature map channels is changed from 1536 to 1024, and the output feature map dimension is 2L×1024), a BN normalization layer (corresponding to step S1042, the output feature map dimension is 2L×1024), a LeakyReLU activation layer (corresponding to step S1043, the output feature map dimension is 2L×1024), and a linear (corresponding to step S1044, the number of feature map channels is changed to 1, and the output feature map dimension is 2L×1) to obtain the prediction score map, that is, the defect detection score of the good product image and the defect detection score of the defective image.

[0151] In one possible implementation, determining the defect detection result of the image to be detected based on the defect detection scores of each pixel includes:

[0152] The total defect score of the image to be detected is determined based on the defect detection scores of each pixel.

[0153] The defect detection result of the image to be detected is determined based on whether the total defect score exceeds the second preset threshold.

[0154] Specifically, a negative defect detection score indicates that the pixel belongs to the region containing the target defect in the image, while a positive defect detection score indicates that the pixel belongs to the normal region in the image. The maximum negative value among all pixel defect detection scores is taken as the total defect score of the current image to be detected (i.e., image defect score = max(-discriminator(feature))). When the total defect score is less than or equal to the second preset threshold, the current image to be detected is considered a good image; otherwise, it is a defective image (containing defects).

[0155] Furthermore, to better balance the precision and recall of defect detection, a local test set (with an equal number of good and defective images) is constructed by selecting a subset of sample images. The defect detection score of each image in the test set is obtained sequentially using the same method. The scores of all images are sorted in ascending order and used as thresholds to plot the PR (Precision-Recall) curve. The threshold at which Precision + Recall is maximized is taken as the second preset threshold for distinguishing between good and defective images.

[0156] A second aspect of this application also provides an image defect detection apparatus, the apparatus being used to perform the image defect detection method described in the first aspect, with reference to... Figure 9 , Figure 9 A schematic diagram of an image defect detection device is shown, such as... Figure 9 As shown, the device includes:

[0157] The input module is used to acquire the image to be detected;

[0158] The feature extraction module is used to perform multi-level feature extraction on the image to be detected, obtain the feature maps output by each feature layer in the feature extraction module, determine multiple initial feature maps from the feature maps output by each feature layer, and perform feature aggregation on the multiple initial feature maps to obtain the target feature map;

[0159] The detection module is used to perform defect detection on the target features and obtain the defect detection result of the image to be detected.

[0160] In one possible implementation, the feature extraction module and the detection module are trained through the following steps:

[0161] Create sample image pairs consisting of good product images and defective product images with the same background;

[0162] The good product image and the defective image in the sample image pair are respectively input into the feature extraction module for multi-level feature extraction to obtain the sample feature map output by each feature layer in the feature extraction module; the sample feature map includes: sample good product feature map and sample defective feature map;

[0163] Based on the adaptive feature aggregation module of the feature extraction module, multiple target feature layers are determined from each feature layer, and the sample feature map output by the target feature layer is determined as the initial sample feature map. Feature aggregation is performed on the initial sample feature map to obtain the target sample feature map. The target feature layer refers to the feature layer with the highest contribution in distinguishing image features as good or defective features among all feature layers.

[0164] The sample target feature map is input into the detection module for defect detection to obtain the defect detection score of the sample image pair;

[0165] Based on the defect detection score, a loss function value is calculated, and the parameters of the feature extraction module and the detection module are updated according to the loss function value to obtain the trained feature extraction module and detection module.

[0166] In one possible implementation, the sample target feature map includes a sample good product target feature map and a sample defect target feature map; the device further includes: a feature acquisition module, used to acquire the sample target feature map according to the following steps;

[0167] Obtain a first sample image, which is a first good product image;

[0168] The first sample image is input into the feature extraction module for multi-level feature extraction to obtain the first sample feature map output by each feature layer in the feature extraction module;

[0169] The adaptive feature aggregation module based on the feature extraction module determines the initial feature map of the first sample from the first sample feature map output by each feature layer, and performs feature aggregation on the initial feature map of the first sample to obtain the target feature map of the first sample good product.

[0170] Noise is added to the first sample good product target feature map to obtain the first sample defect target feature map. The first sample good product target feature map and the first sample defect target feature map are combined as the sample target feature map.

[0171] In one possible implementation, the sample image pair includes: a second sample image pair;

[0172] The creation of sample image pairs consisting of good and defective images with the same background includes:

[0173] Using an image segmentation model, defect targets in real defect images are segmented to obtain defect target sub-images;

[0174] Obtain the image of the second good product;

[0175] After preprocessing the defective target sub-image, the second good product image is pasted onto it to obtain a second defective image with the same background as the second good product image. The preprocessing is to change at least one of the position, size and rotation angle of the defective target sub-image.

[0176] The combination of the second good product image and the second defective image is used as the second sample image pair.

[0177] In one possible implementation, the sample image pair includes: a third sample image pair;

[0178] The creation of sample image pairs consisting of good and defective images with the same background includes:

[0179] Acquire a third defect image and a mask image that marks the region where the defect target is located in the third defect image;

[0180] Using an image segmentation model, the third defect image is segmented based on the mask image to remove the defect target in the third defect image and obtain a defect background image;

[0181] An image restoration algorithm is used to restore the area where the defective target is located in the defective background image to obtain a third good product image with the same background as the third defective image.

[0182] The combination of the third defect image and the third good product image is used as the third sample image pair.

[0183] In one possible implementation, multiple target feature layers are determined from the various feature layers, including:

[0184] Calculate the difference between the good sample feature map and the defect sample feature map output by the feature layer of each layer;

[0185] Based on the difference between the sample good product feature map and the sample defect feature map, the contribution of this feature layer to distinguish image features as good product features or defect features is calculated.

[0186] The multiple feature layers with the highest contribution are identified as the target feature layers.

[0187] In one possible implementation, calculating the difference between the good sample feature map and the defective sample feature map output by each feature layer includes:

[0188] For the sample good product feature map and the sample defect feature map output by the feature layer of each layer, the number of channels is compressed to 1 by global average pooling to obtain the compressed good product feature map and the compressed defect feature map.

[0189] The square of the difference between the compressed good product feature map and the compressed defect feature map is calculated and normalized to obtain a first calculation result; the first calculation result represents the magnitude of the difference between the sample good product feature map and the sample defect feature map.

[0190] Based on the difference between the sample good product feature map and the sample defect feature map, the contribution of this feature layer to distinguishing image features as good product features or defect features is calculated, including:

[0191] After adjusting the first calculation result to the same resolution as the mask image of the sample defect feature map using bilinear interpolation, the square of the difference between the first calculation result and the mask image of the sample defect feature map is calculated to obtain the second calculation result.

[0192] Using N pairs of sample images, the mean of the second calculation result of the feature layer of each layer is calculated as the third calculation result of the feature layer. The third calculation result is used to characterize the degree of contribution of the feature map output by the feature layer to distinguish whether the image features are good or defective.

[0193] In one possible implementation, the apparatus further includes: a target channel determination module, configured to determine a plurality of target channels from each channel of each target feature layer after determining the target feature layer, wherein the target channel refers to the channel in which the output feature map contributes the most to distinguishing image features as good features or defective features;

[0194] The feature maps output by the target channel are aggregated to obtain the sample feature maps output by the target feature layer.

[0195] In one possible implementation, determining a plurality of target channels from each channel of each of the target feature layers includes:

[0196] For each channel of the target feature layer, the output of the first good feature map and the first defect feature map is compressed to 1 by global average pooling to obtain the second good feature map and the second defect feature map.

[0197] The square of the difference between the second good feature map and the second defect feature map is calculated and normalized to obtain a fourth calculation result; the fourth calculation result represents the magnitude of the difference between the first good feature map and the first defect feature map.

[0198] After adjusting the fourth calculation result to the same resolution as the mask image of the first defect feature map using bilinear interpolation, the square of the difference between the fourth calculation result and the mask image of the first defect feature map is calculated to obtain the fifth calculation result.

[0199] Using N pairs of sample images, the mean of the fifth calculation result for each channel is calculated as the sixth calculation result for that channel. The sixth calculation result is used to characterize the contribution of the feature map output by that channel to distinguish between good and defective image features.

[0200] The channels with the smallest sixth calculation result are determined as the target channels.

[0201] In one possible implementation, feature aggregation is performed on the plurality of initial feature maps to obtain a target feature map, including:

[0202] The resolution of the initial feature map is unified by bilinear interpolation;

[0203] The dimensions of the initial feature map are adjusted to h×w×1536 using adaptive pooling, and then stacked to obtain the first feature map;

[0204] The second feature map is obtained by adjusting the dimension of the first feature map to h×w×1536 through adaptive pooling.

[0205] The second feature map is linearly transformed to obtain the target feature map.

[0206] In one possible implementation, the defect detection score includes: the defect detection score of each pixel in the good product image and the defect detection score of each pixel in the defective image;

[0207] Based on the defect detection score, the loss function value is calculated, including:

[0208] The training objective is to set the defect detection score of pixels in normal regions of an image to be greater than a first preset threshold, and the defect detection score of pixels in defective regions to be less than a negative first preset threshold. The loss function value is calculated based on the defect detection scores using the following formula:

[0209] lp_loss=max(-lp_scores+th,0);

[0210] qx_loss=max(qx_scores·mask+th,0);

[0211] Loss=mean(lp_loss)+mean(qx_loss);

[0212] Wherein, lp_scores is the defect detection score of each pixel in the good product image, th is the first preset threshold; qx_scores is the defect detection score of each pixel in the defective image, mask is the defect score of each pixel in the mask image of the defective image, lp_loss is the loss of the good product image, qx_loss is the loss of the defective image, and Loss is the loss function value.

[0213] In one possible implementation, defect detection is performed on the target feature map to obtain the defect detection result of the image to be detected, including:

[0214] A linear transformation is performed on the target feature map to change the number of channels to 1024, resulting in a third feature map;

[0215] The third feature map is normalized to obtain the normalization result;

[0216] The normalization result is subjected to a nonlinear transformation using a nonlinear activation function to obtain the transformed result;

[0217] A linear transformation is performed on the transformation result to obtain the defect detection score of each pixel in the image to be detected.

[0218] Based on the defect detection scores of each pixel, the defect detection result of the image to be detected is determined.

[0219] In one possible implementation, determining the defect detection result of the image to be detected based on the defect detection scores of each pixel includes:

[0220] The total defect score of the image to be detected is determined based on the defect detection scores of each pixel.

[0221] The defect detection result of the image to be detected is determined based on whether the total defect score exceeds the second preset threshold.

[0222] This application also provides an electronic device, see embodiments thereof. Figure 10 , Figure 10 This is a schematic diagram of the electronic device proposed in an embodiment of this application. Figure 10As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus for communication. The memory 110 stores a computer program that can run on the processor 120 to implement the steps of the image defect detection method disclosed in the first aspect of the present application.

[0223] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the image defect detection method as disclosed in the first aspect of this application.

[0224] This application also provides a computer program product that, when executed on an electronic device, causes a processor to implement the steps of the image defect detection method as disclosed in the first aspect of this application. The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably.

[0225] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0226] The above provides a detailed description of the image defect detection method, device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0227] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0228] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0229] The terms "an embodiment," "embodiment," or "one or more embodiments" as used herein mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this application. Furthermore, please note that the examples of the phrase "in one embodiment" do not necessarily all refer to the same embodiment.

[0230] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0231] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0232] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image defect detection method, characterized in that, The method includes: Acquire the image to be detected; The feature extraction module performs multi-level feature extraction on the image to be detected, obtaining feature maps output by each feature layer in the feature extraction module. Multiple initial feature maps are then determined from the feature maps output by each feature layer. Feature aggregation is performed on these multiple initial feature maps to obtain a target feature map. The feature aggregation includes adjusting the resolution and dimension of the multiple initial feature maps, followed by a linear transformation to output the target feature map. The initial feature maps include the feature maps with the highest contribution to distinguishing between good and defective image features from the feature maps output by each feature layer. Based on the detection module, defect detection is performed on the target feature map to obtain the defect detection result of the image to be detected.

2. The image defect detection method according to claim 1, characterized in that, The feature extraction module and the detection module are trained through the following steps: Create sample image pairs consisting of good product images and defective product images with the same background; The good product image and the defective image in the sample image pair are respectively input into the feature extraction module for multi-level feature extraction to obtain the sample feature map output by each feature layer in the feature extraction module. The sample feature map includes: sample good product feature map and sample defective feature map. Based on the adaptive feature aggregation module of the feature extraction module, multiple target feature layers are determined from each feature layer, and the sample feature map output by the target feature layer is determined as the initial sample feature map. Feature aggregation is performed on the initial sample feature map to obtain the target sample feature map. The sample target feature map is input into the detection module for defect detection to obtain a defect detection score. Based on the defect detection score, a loss function value is calculated, and the parameters of the feature extraction module and the detection module are updated according to the loss function value to obtain the trained feature extraction module and detection module.

3. The image defect detection method according to claim 2, characterized in that, The sample target feature map includes a sample good product target feature map and a sample defective target feature map; the method further includes: obtaining the sample target feature map according to the following steps: Obtain a first sample image, which is a first good product image; The first sample image is input into the feature extraction module for multi-level feature extraction to obtain the first sample feature map output by each feature layer in the feature extraction module; The adaptive feature aggregation module based on the feature extraction module determines the initial feature map of the first sample from the first sample feature map output by each feature layer, and performs feature aggregation on the initial feature map of the first sample to obtain the target feature map of the first sample good product. Noise is added to the first sample good product target feature map to obtain the first sample defect target feature map. The first sample good product target feature map and the first sample defect target feature map are combined as the sample target feature map.

4. The image defect detection method according to claim 2, characterized in that, The sample image pair includes: a second sample image pair; The creation of sample image pairs consisting of good and defective images with the same background includes: Using an image segmentation model, defect targets in real defect images are segmented to obtain defect target sub-images; Obtain the image of the second good product; After preprocessing the defective target sub-image, the second good product image is pasted onto it to obtain a second defective image with the same background as the second good product image. The preprocessing is to change at least one of the position, size and rotation angle of the defective target sub-image. The combination of the second good product image and the second defective image is used as the second sample image pair.

5. The image defect detection method according to claim 2, characterized in that, The sample image pair includes: a third sample image pair; The creation of sample image pairs consisting of good and defective images with the same background includes: Acquire a third defect image and a mask image that marks the region where the defect target is located in the third defect image; Using an image segmentation model, the third defect image is segmented based on the mask image to remove the defect target in the third defect image and obtain a defect background image; An image restoration algorithm is used to restore the area where the defective target is located in the defective background image to obtain a third good product image with the same background as the third defective image. The combination of the third defect image and the third good product image is used as the third sample image pair.

6. The image defect detection method according to claim 2, characterized in that, Multiple target feature layers are determined from each feature layer, including: Calculate the difference between the good sample feature map and the defect sample feature map output by the feature layer of each layer; Based on the difference between the sample good product feature map and the sample defect feature map, the contribution of this feature layer to distinguish image features as good product features or defect features is calculated. The multiple feature layers with the highest contribution are identified as the target feature layers.

7. The image defect detection method according to claim 6, characterized in that, Calculate the difference between the good sample feature map and the defective sample feature map output by each feature layer, including: For the sample good product feature map and the sample defect feature map output by the feature layer of each layer, the number of channels is compressed to 1 by global average pooling to obtain the compressed good product feature map and the compressed defect feature map. The square of the difference between the compressed good product feature map and the compressed defect feature map is calculated and normalized to obtain a first calculation result; the first calculation result represents the magnitude of the difference between the sample good product feature map and the sample defect feature map. Based on the difference between the sample good product feature map and the sample defect feature map, the contribution of this feature layer to distinguishing image features as good product features or defect features is calculated, including: After adjusting the first calculation result to the same resolution as the mask image of the sample defect feature map using bilinear interpolation, the square of the difference between the first calculation result and the mask image of the sample defect feature map is calculated to obtain the second calculation result. Using N pairs of sample images, the mean of the second calculation result of the feature layer of each layer is calculated as the third calculation result of the feature layer. The third calculation result is used to characterize the degree of contribution of the feature map output by the feature layer to distinguish whether the image features are good or defective features.

8. The image defect detection method according to claim 6, characterized in that, After determining the target feature layer, the method further includes: From each channel of each target feature layer, multiple target channels are determined, wherein the target channel is the channel in which the output feature map contributes the most to distinguishing image features as good features or defective features; The feature maps output by the target channel are aggregated to obtain the sample feature maps output by the target feature layer.

9. The image defect detection method according to claim 8, characterized in that, From each channel of each of the target feature layers, multiple target channels are determined, including: For each channel of the target feature layer, the output of the first good feature map and the first defect feature map is compressed to 1 by global average pooling to obtain the second good feature map and the second defect feature map. The square of the difference between the second good feature map and the second defect feature map is calculated and normalized to obtain a fourth calculation result; the fourth calculation result represents the magnitude of the difference between the first good feature map and the first defect feature map. After adjusting the fourth calculation result to the same resolution as the mask image of the first defect feature map using bilinear interpolation, the square of the difference between the fourth calculation result and the mask image of the first defect feature map is calculated to obtain the fifth calculation result. Using N pairs of sample images, the mean of the fifth calculation result for each channel is calculated as the sixth calculation result for that channel. The sixth calculation result is used to characterize the contribution of the feature map output by that channel to distinguish between good and defective image features. The channels with the smallest sixth calculation result are determined as the target channels.

10. The image defect detection method according to claim 2, characterized in that, The defect detection score includes: the defect detection score of each pixel in the good product image and the defect detection score of each pixel in the defective image; Based on the defect detection score, the loss function value is calculated, including: The training objective is to set the defect detection score of pixels in normal regions of an image to be greater than a first preset threshold, and the defect detection score of pixels in defective regions to be less than a negative first preset threshold. The loss function value is calculated based on the defect detection scores using the following formula: lp_loss=max(-lp_scores+th,0); qx_loss=max(qx_scores·mask+th,0); Loss=mean(lp_loss)+mean(qx_loss); Wherein, lp_scores is the defect detection score of each pixel in the good product image, th is the first preset threshold; qx_scores is the defect detection score of each pixel in the defective image, mask is the defect score of each pixel in the mask image of the defective image, lp_loss is the loss of the good product image, qx_loss is the loss of the defective image, and Loss is the loss function value.

11. The image defect detection method according to claim 2, characterized in that, The initial feature map of the sample is subjected to feature aggregation to obtain the target feature map of the sample, including: The resolution of the initial feature map of the sample is unified by bilinear interpolation; The dimensions of the initial feature maps of the samples are adjusted to the first dimension by adaptive pooling, and then stacked to obtain the first intermediate feature map; The second intermediate feature map is obtained by adjusting the dimension of the first intermediate feature map to the first dimension through adaptive pooling. A linear transformation is performed on the second intermediate feature map to obtain the sample target feature map.

12. The image defect detection method according to claim 1, characterized in that, The target feature map is obtained by feature aggregation of the multiple initial feature maps, including: The resolution of the initial feature map is unified by bilinear interpolation; The initial feature map is adjusted to the first dimension by adaptive pooling and then stacked to obtain the first feature map; The second feature map is obtained by adjusting the dimension of the first feature map to the first dimension through adaptive pooling; The second feature map is linearly transformed to obtain the target feature map.

13. The image defect detection method according to claim 1, characterized in that, Defect detection is performed on the target feature map to obtain the defect detection result of the image to be detected, including: A linear transformation is performed on the target feature map to change the number of channels to 1024, resulting in a third feature map; The third feature map is normalized to obtain the normalization result; The normalization result is subjected to a nonlinear transformation using a nonlinear activation function to obtain the transformed result; A linear transformation is performed on the transformation result to obtain the defect detection score of each pixel in the image to be detected. Based on the defect detection scores of each pixel, the defect detection result of the image to be detected is determined.

14. The image defect detection method according to claim 13, characterized in that, Determining the defect detection result of the image to be detected based on the defect detection score of each pixel includes: The total defect score of the image to be detected is determined based on the defect detection scores of each pixel. The defect detection result of the image to be detected is determined based on whether the total defect score exceeds the second preset threshold.

15. An electronic device, characterized in that, It includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the image defect detection method as described in any one of claims 1-14.

16. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the image defect detection method as described in any one of claims 1-14.

Images