Defect detection method and device, electronic equipment and storage medium

Abstract

The invention provides a defect detection method and device, electronic equipment and a storage medium. The defect detection method comprises the following steps: acquiring a target image comprising a target object and a reference image comprising a reference object; and inputting the target image and the reference image into a trained defect detection model to obtain a defect detection result of the target object. According to the invention, the trained spatial transformation module can be used for correcting the target image to obtain the target correction image which is spatially aligned with the reference image, so that the influence of errors caused by shooting angles, target object postures, target object position offset and the like on the precision of a defect detection result is effectively reduced. According to the scheme, the image alignment task can be closely adapted to the defect detection task, so that the accuracy of the defect detection result obtained for the target image can be improved.

Description

Technical Field

[0001] This invention relates to the field of defect detection, and more specifically to a defect detection method, a defect detection device, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] In many fields such as precision electronics and automotive parts manufacturing, defect detection is one of the core steps to ensure product quality and reduce production losses. For example, in semiconductor chip manufacturing, defects such as tiny scratches and dents on the chip surface can directly affect the chip's electrical performance.

[0003] Current mainstream methods typically utilize a small number of real-world samples to build defect detection models. However, in real-world industrial scenarios, it is difficult to accurately align the images to be detected, leading to low accuracy in defect detection results. Therefore, how to accurately detect defects in images is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The present invention was proposed in view of the above-mentioned problems. The present invention provides a defect detection method, a defect detection device, an electronic device, a storage medium, and a computer program product.

[0005] According to one aspect of the present invention, a defect detection method is provided. The detection method includes: acquiring a target image including a target object and a reference image including a reference object, wherein the reference object and the target object are of the same object type; inputting the target image and the reference image into a trained defect detection model to obtain a defect detection result for the target object; wherein the trained defect detection model includes a trained spatial transformation module and a trained feature extraction module, the trained spatial transformation module is used to correct the target image to obtain a corrected target image spatially aligned with the reference image, and the trained feature extraction module is used to perform feature extraction processing on the corrected target image; the trained defect detection model is based on multiple sample images including sample objects, a template image, differential images corresponding to each of the multiple sample images, and multiple sample images. The corresponding defect annotation results are trained. The sample objects and the target objects have the same object type. Multiple sample images are obtained through image alignment processing. The template image is obtained based on multiple sample images through image fusion processing. For each difference image in the difference image corresponding to each of the multiple sample images, the difference image is determined based on the sample image corresponding to the difference image and the template image corresponding to the difference image. For each sample image in the multiple sample images, the fusion feature output by the untrained feature extraction module is determined based on the template image and the difference image corresponding to the sample image. The defect detection result is used to indicate the defect type of the target object or whether the target object has a defect. The defect annotation result corresponding to the sample image is used to indicate the defect type of the sample object in the sample image or whether the sample object has a defect.

[0006] For example, the multiple sample images include at least one set of sample images. For each set of sample images in the at least one set of sample images, each sample image in the set of sample images is obtained through image alignment processing. The trained defect detection model is obtained through the following steps:

[0007] For each set of sample images in at least one set of sample images,

[0008] Image fusion processing is performed on all sample images in the set of sample images to obtain the template image corresponding to the set of sample images. Based on the image size of the set of sample images, the first grid corresponding to the set of sample images is determined. The template image corresponding to the set of sample images is spatially aligned with each sample image in the set of sample images. The first grid corresponding to the set of sample images is used to represent the normalized result of the position of each pixel in each sample image in the set of sample images in its respective sample image.

[0009] For each sample image in this set of sample images

[0010] The sample image and the first grid corresponding to the set of sample images are input into the untrained spatial transformation module to obtain the first image and the second grid.

[0011] The difference between the first image and the template image corresponding to the sample images in the group is calculated to obtain the difference image corresponding to the sample image;

[0012] The difference image corresponding to the sample image, the template image corresponding to the group of sample images, and the second grid are input into the untrained feature extraction module to obtain the fused features corresponding to the sample image;

[0013] Based on the fusion features corresponding to the sample image, the overall loss value is determined;

[0014] Based on the overall loss value, adjust the module parameters of the untrained spatial transformation module and the untrained feature extraction module.

[0015] For example, the overall loss value includes a first loss value. The overall loss value is determined based on the fusion features corresponding to the sample image, including:

[0016] The fusion features corresponding to the sample image are input into the decoding module to obtain the decoded image;

[0017] Based on the processing parameters of the untrained spatial transformation module, the decoded image is subjected to inverse transformation to obtain the reconstructed image;

[0018] The difference between the reconstructed image and the sample image is determined and used as the first loss value.

[0019] For example, the overall loss value also includes a second loss value. The overall loss value is determined based on the fusion features corresponding to the sample image, and further includes:

[0020] The difference between the second grid and the first grid is determined as the second loss value.

[0021] For example, the multiple sample images include multiple local images, where each local image is a sample image that includes a portion of the sample object.

[0022] For example, the multiple sample images include at least one noisy image, which is obtained by image noise addition processing based on the original image including the sample object.

[0023] For example, the target image and reference image are input into a trained defect detection model to obtain defect detection results for the target object, including:

[0024] Based on the image size of the target image, the third grid corresponding to the target image is determined, where the third grid corresponding to the target image is used to represent the normalized result of the position of each pixel in the target image.

[0025] The target image and its corresponding third grid are input into the trained spatial transformation module to obtain the second image and the fourth grid.

[0026] The difference between the second image and the reference image is calculated to obtain the difference image corresponding to the target image;

[0027] Input the difference image, reference image, and fourth grid corresponding to the target image into the trained feature extraction module to obtain the fused features corresponding to the target image;

[0028] Based on the fusion features corresponding to the target image, the defect detection result of the target object is determined.

[0029] According to another aspect of the present invention, a defect detection device is also provided, the defect detection device comprising: an image acquisition module and a defect detection result determination module. The image acquisition module is used to acquire a target image including the target object and a reference image including a reference object, wherein the reference object and the target object have the same object type. The defect detection result determination module is used to input the target image and the reference image into a trained defect detection model to obtain the defect detection result of the target object. The trained defect detection model includes a trained spatial transformation module and a trained feature extraction module. The trained spatial transformation module is used to correct the target image to obtain a corrected target image spatially aligned with the reference image. The trained feature extraction module is used to perform feature extraction processing on the corrected target image. The trained defect detection model is trained based on multiple sample images including sample objects, a template image, and differential images corresponding to each of the multiple sample images. The sample objects and the target object have the same object type. The multiple sample images are obtained through image alignment processing. The template image is obtained based on multiple sample images through image fusion processing. For each differential image corresponding to each of the multiple sample images, the differential image is determined based on the sample image corresponding to the differential image and the template image corresponding to the differential image.

[0030] According to another aspect of the present invention, an electronic device is also provided, the electronic device including a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program to implement the above-described defect detection method.

[0031] According to another aspect of the present invention, a storage medium is also provided, which stores computer program instructions that, when executed, are used to perform the above-described defect detection method.

[0032] According to another aspect of the present invention, a computer program product is also provided, the computer program product comprising computer program instructions which, when executed by a processor, are used to perform the defect detection method described above.

[0033] According to the above-described scheme of the present invention, a target image including the target object and a reference image including the reference object can be obtained. Then, the target image and the reference image are input into a trained defect detection model to obtain the defect detection result of the target object. The above scheme can use a trained spatial transformation module to correct the target image, obtaining a corrected target image spatially aligned with the reference image, effectively reducing the impact of errors caused by shooting angle, target object pose, target object position offset, etc., on the accuracy of the defect detection result. Furthermore, since the trained defect detection model in the above scheme includes a trained spatial transformation module and a trained feature extraction module, compared to the method of first aligning the image and then inputting it into the defect detection model, the above scheme of the present invention is advantageous in closely adapting the image alignment task to the defect detection task, thereby improving the accuracy of the defect detection result obtained for the target image. The above scheme also has higher robustness. In cases where the target object and the reference object have inherent morphological differences or texture changes unrelated to the defect, a strict image alignment algorithm may introduce a large amount of noise after correction. Since the image alignment task in the above-mentioned solution of the present invention is closely adapted to the defect detection task, the inherent morphological differences or texture changes that are unrelated to defects do not need to be over-corrected, and the impact on the accuracy of subsequent defect detection results is small. Thus, the ultimate goal of the overall processing flow is to obtain a more accurate defect detection result, rather than to obtain the most accurate image alignment result, which also meets the actual needs of industrial scenarios. Attached Figure Description

[0034] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0035] Figure 1 A schematic flowchart of a defect detection method according to an embodiment of the present invention is shown;

[0036] Figure 2 A schematic diagram illustrating the process of determining a first loss value according to a specific embodiment of the present invention is shown;

[0037] Figure 3A schematic block diagram of a defect detection apparatus according to an embodiment of the present invention is shown;

[0038] Figure 4 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0040] To at least partially solve the above problems, embodiments of the present invention provide a defect detection method. Figure 1 A schematic flowchart of a defect detection method according to an embodiment of the present invention is shown. Figure 1 As shown, the method may include steps S110 and S120.

[0041] In step S110, a target image including the target object and a reference image including the reference object are obtained.

[0042] The reference object and the target object are of the same object type. It is understood that the reference object and the target object can be industrial products of the same type and model, such as wafers, dies, or other electronic components. In some embodiments, the target object can be the product to be inspected, and the reference object can be an existing qualified product (without defects). In still other embodiments, the target object can be the product to be inspected, and the reference object can be a simulated product obtained based on multiple existing qualified products. Specifically, the reference image including the reference object is a fused image obtained based on multiple product images including qualified products, and the reference object in this fused image can be a simulated object.

[0043] In some embodiments, the target image may include the complete target object. In other embodiments, the target image includes only a portion of the target object, i.e., the target object contains a region to be detected, and the target image is an image that includes the region to be detected of the target object.

[0044] In some embodiments, the target image may be the original image obtained for the target object. In other embodiments, the target image may be the original image after processing such as image enhancement or image cropping.

[0045] In step S120, the target image and the reference image are input into the trained defect detection model to obtain the defect detection result of the target object.

[0046] A trained defect detection model includes a trained spatial transformation module and a trained feature extraction module. It can be understood that a trained defect detection model may also include trained intermediate modules (such as intermediate network layers in a deep learning network) and a trained detection head.

[0047] The trained spatial transformation module is used to correct the target image to obtain a corrected target image spatially aligned with the reference image. The trained feature extraction module is used to extract features from the corrected target image. The trained intermediate module performs pooling, convolution, and other processing on the features output by the trained feature extraction module. The trained detection head performs activation processing on the features output by the trained intermediate module to determine the defect detection result corresponding to the target image.

[0048] The trained defect detection model is trained based on multiple sample images, template images, differential images corresponding to each of the sample images, and defect annotation results corresponding to each of the sample images.

[0049] The sample object and the target object are of the same type. This allows the trained defect detection model to learn visual features closely related to the target object. Compared to using a general object detection model to output the region where the target object is located and then performing defect detection on that region, the above-described scheme of this invention reduces the data domain difference between the sample image and the actual scene, thereby improving the accuracy of the defect detection results. Specifically, for example, the sample object and the target object can be industrial products of the same type and model. In one example, the sample object can be an existing qualified product or a simulated qualified product obtained based on an existing qualified product; please refer to the relevant content of the reference object above for details. In another example, the sample object can also be an existing defective product (hereinafter referred to as a defective product) or a simulated product obtained based on an existing defective product.

[0050] It is understood that the sample image includes the aforementioned sample objects. If the sample objects included in the sample image are qualified products, the defect annotation result corresponding to the sample image can be used to indicate that the sample objects in the sample image do not have defects. If the sample objects included in the sample image are defective products, the defect annotation result corresponding to the sample image can be used to indicate the specific defect type of the sample objects in the sample image or that the sample objects have defects.

[0051] Multiple sample images are obtained through image alignment processing. In some embodiments, the multiple sample objects are arranged in the same direction. Using the same image acquisition method, multiple original images including the sample objects in the same direction can be obtained based on these multiple original images including the sample objects. By performing image alignment processing on these multiple original images including the sample objects, multiple almost overlapping sample images can be obtained. In some alternative embodiments, the sample objects are located at different positions in the multiple original images including the sample objects. Image alignment processing can make the positions of the sample objects in the multiple original images including the sample objects the same in the aligned reference coordinate system. In some embodiments, image enhancement processing and image alignment processing can be performed on the multiple original images including the sample objects to obtain multiple sample images.

[0052] The template image is obtained through image fusion processing based on multiple sample images. This template image can be used to represent the benchmark for correcting the multiple sample images. The template image obtained through image fusion processing of multiple sample images can have the same size as the multiple sample images. Image fusion methods such as logistic filtering and weighted averaging can be used to determine the template image, which can make the image texture less complex than the sample images, thus improving the accuracy of the defect detection results output by the defect detection model trained based on it.

[0053] For each difference image among multiple sample images, the difference image is determined based on the sample image corresponding to that difference image and the template image corresponding to that difference image. The difference image can be obtained by subtracting the template image corresponding to the sample image (after processing by an untrained spatial transformation module) from the template image. In one example, the sample image can be directly input into the untrained spatial transformation module. In another example, the sample image can be processed using image correlation algorithms (e.g., algorithms for noise reduction, sharpness adjustment, and size adjustment) before being input into the untrained spatial transformation module. It is understandable that the difference image can also be obtained by processing the subtracted image using image correlation algorithms.

[0054] According to the above-described scheme of the present invention, a target image including the target object and a reference image including the reference object can be obtained. Then, the target image and the reference image are input into a trained defect detection model to obtain the defect detection result of the target object. The above scheme can use a trained spatial transformation module to correct the target image, obtaining a corrected target image spatially aligned with the reference image, effectively reducing the impact of errors caused by shooting angle, target object pose, target object position offset, etc., on the accuracy of the defect detection result. Furthermore, since the trained defect detection model in the above scheme includes a trained spatial transformation module and a trained feature extraction module, compared to the method of first aligning the image and then inputting it into the defect detection model, the above scheme of the present invention is advantageous in closely adapting the image alignment task to the defect detection task, thereby improving the accuracy of the defect detection result obtained for the target image. The above scheme also has higher robustness. In cases where the target object and the reference object have inherent morphological differences or texture changes unrelated to the defect, a strict image alignment algorithm may introduce a large amount of noise after correction. Since the image alignment task in the above-mentioned solution of the present invention is closely adapted to the defect detection task, the inherent morphological differences or texture changes that are unrelated to defects do not need to be over-corrected, and the impact on the accuracy of subsequent defect detection results is small. Thus, the ultimate goal of the overall processing flow is to obtain a more accurate defect detection result, rather than to obtain the most accurate image alignment result, which also meets the actual needs of industrial scenarios.

[0055] For example, the multiple sample images include at least one set of sample images, and for each set of sample images in the at least one set of sample images, each sample image in the set of sample images is obtained through image alignment processing. The trained defect detection model is obtained through steps S210 to S260.

[0056] In step S210, for each set of sample images in at least one set of sample images, image fusion processing is performed on all sample images in the set of sample images to obtain the template image corresponding to the set of sample images, and the first grid corresponding to the set of sample images is determined based on the image size of the set of sample images.

[0057] The template image corresponding to the set of sample images is spatially aligned with each sample image in the set of sample images.

[0058] It can be understood that the sample object corresponding to each sample image in a set of sample images is a sample object under the same shooting environment or the same sample object. For example, the sample object under the same shooting environment can be an industrial product of the same type and model (furthermore, it can be the same batch) with the same defects under the same lighting environment.

[0059] The process of determining the template image corresponding to a set of sample images can be referred to above, and will not be repeated here in the embodiments of the present invention. It can be understood that the template images corresponding to different sets of sample images may be different.

[0060] The first grid corresponding to a set of sample images is used to represent the normalized result of the position of each pixel in each sample image within the set of sample images. The coordinates of the pixel in the sample image can be normalized to obtain the coordinates of the grid point corresponding to the pixel in the first grid. For example, if each row of the sample image includes 5 pixels and each column includes 5 pixels, then each row of the first grid may include 5 grid points and each column may include 5 grid points. The coordinates of the grid point corresponding to the pixel in the first row and first column of the first grid are (0, 0), the coordinates of the grid point corresponding to the pixel in the first row and second column of the first grid are (0.25, 0), and the coordinates of the grid point corresponding to the pixel in the first row and fifth column of the first grid are (1, 0).

[0061] The first grid is an orthogonal grid. In some embodiments, for a set of sample images, the resolution of the template image may be the same as that of the sample images. The first grid corresponding to the set of sample images can be determined based on the image size of the set of sample images. The horizontal grid size of the first grid may be positively correlated with the horizontal image size of the sample images. The vertical grid size of the first grid may be positively correlated with the vertical image size of the sample images. In some alternative embodiments, for a set of sample images, the resolution of the template image may be different from that of the sample images. The resolution of the template image may be adjusted to be consistent with that of the sample images, and then the first grid corresponding to the set of sample images can be determined based on the image size of the set of sample images.

[0062] In step S220, for each set of sample images in at least one set of sample images, for each sample image in the set of sample images, the sample image and the first grid corresponding to the set of sample images are input to the untrained spatial transformation module to obtain the first image and the second grid.

[0063] In some embodiments, an untrained spatial transformation module transforms the first mesh into a second mesh using an affine transformation. The affine transformation includes translation, rotation, shearing, and scaling operations. Specifically, the affine transformation includes six associated parameters. In other embodiments, the untrained spatial transformation module may also perform a non-affine transformation on the first mesh to obtain other transformation parameters. It is understood that the spatial transformation module may also include trainable module parameters used to predict the aforementioned transformation parameters. For each sample image, the first image and the second mesh use the same transformation parameters.

[0064] In step S230, for each set of sample images in at least one set of sample images, for each sample image in the set of sample images, the difference between the first image and the template image corresponding to the set of sample images is calculated to obtain the difference image corresponding to the sample image.

[0065] For each pixel in the first image, the pixel value can be subtracted from the pixel value of the corresponding pixel in the template image (e.g., the pixel with the same coordinates as the pixel in the first image) to obtain the pixel value of the corresponding pixel in the difference image (e.g., the pixel with the same coordinates as the pixel in the first image). In some embodiments, for each pixel in the first image, if the difference between the pixel value of the pixel in the first image and the pixel value of the corresponding pixel in the template image is negative, the absolute value of the negative value can be determined as the pixel value of the corresponding pixel in the difference image. In some alternative embodiments, for each pixel in the first image, if the difference between the pixel value of the pixel in the first image and the pixel value of the corresponding pixel in the template image is negative, the pixel value of the corresponding pixel in the difference image can be determined as zero.

[0066] In step S240, for each set of sample images in at least one set of sample images, for each sample image in the set of sample images, the difference image corresponding to the sample image, the template image corresponding to the set of sample images, and the second grid are input into the untrained feature extraction module to obtain the fused feature corresponding to the sample image.

[0067] In some embodiments, the untrained feature extraction module first extracts features from the difference image, template image, and second grid respectively through an encoder, and then fuses the three features to obtain fused features. Specifically, the three features can be weighted and fused to obtain fused features. For example, an attention mechanism can be used to dynamically adjust the fusion weights of the features. The aforementioned second grid can be used to represent the corrected position of each pixel in the sample image, thereby allowing the trained defect detection model to focus more on the aforementioned positional changes, which is beneficial to improving the correction accuracy of the target image and thus improving the accuracy of the defect detection results. The aforementioned feature extraction module can be an encoder, which may not perform downsampling or perform only a small amount of downsampling, thereby effectively using the high-resolution visual information in the training images for training.

[0068] In step S250, for each set of sample images in at least one set of sample images, for each sample image in that set of sample images, the overall loss value is determined based on the fusion features corresponding to that sample image.

[0069] The overall loss value is used to adjust the parameters of the trainable modules in the trainable modules.

[0070] The overall loss value can be determined based on the fused features output by the untrained feature extraction module and the sample image. It can be understood that the higher the correspondence between the fused features and the sample image, the smaller the overall loss value. Processing the aforementioned fused features through an untrained intermediate module and an untrained detection head yields the defect prediction result corresponding to the sample image. The overall loss value may also include the degree of difference between the defect prediction result and the defect annotation result.

[0071] In step S260, for each set of sample images in at least one set of sample images, for each sample image in that set of sample images, the module parameters of the untrained spatial transformation module and the untrained feature extraction module are adjusted based on the overall loss value.

[0072] The module parameters in an incompletely trained feature extraction module can be adjusted based on the overall loss value. Specifically, for an incompletely trained feature extraction module, trainable module parameters can include weights, biases, etc. Similarly, the module parameters in an incompletely trained spatial transformation module can be adjusted based on the overall loss value.

[0073] In some embodiments, the trainable modules include an untrained feature extraction module and an untrained spatial transformation module. In other embodiments, the untrained defect detection model also includes other trainable modules (such as the untrained intermediate modules and the untrained detection head mentioned above), so the overall loss value can also be used to adjust the parameters of other trainable modules. Specifically, the untrained defect detection model may include trainable intermediate modules, trainable detection heads, etc.

[0074] It is understandable that training the untrained spatial transformation module together with the untrained feature extraction module can make the image alignment task closely adapt to the defect detection task. Therefore, for inherent morphological differences or texture changes that are unrelated to defects, over-correction is not necessary, and the impact on the accuracy of subsequent defect detection results can be small. Thus, the ultimate goal of the overall processing flow is to obtain more accurate defect detection results, rather than to obtain the most accurate image alignment results, which also meets the actual needs of industrial scenarios.

[0075] According to the above-described scheme of the present invention, during the training process of the defect detection model, for each set of sample images in at least one set of sample images, all sample images in that set of sample images can be subjected to image fusion processing to obtain a template image corresponding to that set of sample images, and a first grid corresponding to that set of sample images is determined based on the image size of that set of sample images. Then, for each sample image in that set of sample images, the sample image and the first grid corresponding to that set of sample images are first input to the untrained spatial transformation module to obtain a first image and a second grid. Next, the difference between the first image and the template image corresponding to that set of sample images is calculated to obtain a difference image corresponding to that sample image. Then, the difference image corresponding to that sample image, the template image corresponding to that set of sample images, and the second grid are input to the untrained feature extraction module to obtain the fusion feature corresponding to that sample image. Based on the fusion feature corresponding to that sample image, the overall loss value is determined. Finally, based on the overall loss value, the module parameters of the untrained spatial transformation module and the untrained feature extraction module are adjusted. The above scheme obtains the fusion features corresponding to the sample image by using the untrained spatial transformation module and the untrained feature extraction module, and then adjusts the module parameters of the untrained spatial transformation module and the untrained feature extraction module based on the fusion features. This helps the defect detection model learn the positional changes of the sample image before and after correction and the visual features of the sample object, thereby improving the accuracy of the defect detection results output by the defect detection model trained in this way.

[0076] For example, the overall loss value includes a first loss value. For each set of sample images in at least one set of sample images, for each sample image in that set of sample images, the determination of the overall loss value in step S250 based on the fusion features corresponding to that sample image includes steps S251a to S253a.

[0077] In step S251a, the fusion features corresponding to the sample image are input into the decoding module to obtain the decoded image.

[0078] The fused features output by the untrained feature extraction module can be decoded based on the decoding module corresponding to the encoder in the untrained feature extraction module to obtain a decoded image.

[0079] In step S252a, based on the processing parameters of the untrained spatial transformation module, the decoded image is subjected to inverse transformation to obtain the reconstructed image.

[0080] In some embodiments, the untrained spatial transformation module is used to generate a first image from a sample image through an affine transformation. The parameters of the inverse affine transformation can be determined based on the six parameters of the affine transformation in the untrained spatial transformation module, and then the decoded image can be subjected to inverse affine processing to obtain a reconstructed image.

[0081] In step S253a, the difference between the reconstructed image and the sample image is determined as the first loss value.

[0082] It is understandable that, ideally, the reconstructed image obtained based on the fusion features through steps S251a and S252a should be consistent with the sample image. Therefore, the difference between the reconstructed image and the sample image can be determined as the first loss value. After batch training with multiple sample images, the first loss value gradually decreases. When the overall loss value, including the first loss value, is less than the loss value threshold, the adjustment of the module parameters of the defect detection model can be stopped to obtain the trained defect detection model.

[0083] Figure 2 A schematic diagram illustrating the process of determining a first loss value according to a specific embodiment of the present invention is shown. First, a sample image is input into a spatial transformation module. Then, a difference image is determined based on the output of the spatial transformation module and a template image. Next, the difference image and a second grid are input into a feature extraction module to obtain fused features. Then, a decoded image corresponding to the fused features is obtained through a decoding module. Finally, the reconstructed image obtained by inverse transformation processing of the decoded image (corresponding to the spatial transformation module) is compared with the sample image to determine the first loss value.

[0084] According to the above-described scheme of the present invention, the fusion features corresponding to the sample image can be input into the decoding module to obtain a decoded image. Then, based on the processing parameters of the untrained spatial transformation module, the decoded image is subjected to inverse transformation processing to obtain a reconstructed image. Finally, the difference between the reconstructed image and the sample image is determined as the first loss value. The above scheme determines the overall loss value through the fusion features corresponding to the sample image, which can improve the correction accuracy of the trained defect detection model, and thus improve the accuracy of the defect detection results.

[0085] For example, the overall loss value also includes a second loss value. Step S250, for each set of sample images in at least one set of sample images, for each sample image in the set of sample images, determines the overall loss value based on the fusion features corresponding to the sample image, including step S251b.

[0086] In step S251b, the difference between the second grid and the first grid is determined as the second loss value.

[0087] The untrained spatial transformation module transforms the input first grid into a second grid. To prevent over-transformation of the sample image, the difference between the first and second grids can be used to determine whether there are simpler transformation parameters.

[0088] According to the above-described scheme of the present invention, the difference between the second grid and the first grid can be determined as a second loss value. The above scheme, by training an incompletely trained spatial transformation module using the second loss value, can reduce the possibility of over-transformation caused by a fully trained spatial transformation module.

[0089] For example, the multiple sample images include multiple local images, where each local image is a sample image that includes a portion of the sample object.

[0090] Multiple sample images include at least one set of sample images. Each sample image in this set of sample images is a local image.

[0091] It is understood that the sample object may include a region to be detected and a non-detection region. The sample image may be an image including the region to be detected. In some embodiments, the non-detection region includes regions that do not affect product quality and therefore do not require image detection. In other embodiments, the non-detection region includes regions that require further detection without utilizing the trained defect detection model. Specifically, the detection may be a performance detection, i.e., the non-detection region represents regions that are not detected in the image detection step of the trained defect detection model but are subject to performance detection in a subsequent step. In still other embodiments, the non-detection region includes regions that do not affect product quality and therefore do not require image detection, as well as regions that require further detection without utilizing the trained defect detection model.

[0092] According to the above-described scheme of the present invention, an incompletely trained defect detection model can be trained using sample images including local portions of the sample object. This scheme improves the robustness of the defect detection model trained accordingly for defect detection of target objects at different scales, thus helping to maintain the accuracy of defect detection results for target objects at different scales.

[0093] For example, the multiple sample images include at least one noisy image, which is obtained by image noise addition processing based on the original image including the sample object.

[0094] It is understandable that noisy images are used to improve the stability of defect detection models that have not been fully trained.

[0095] The multiple sample images include at least one set of sample images. In some embodiments, for each set of sample images, the set of sample images includes at least one noisy image. In still other embodiments, only a portion of the set of sample images includes the noisy image.

[0096] A set of sample images including noisy images may have a different total number of noisy images and a different degree of noise application. In some embodiments, a set of sample images includes only one noisy image. In still other embodiments, a set of sample images includes multiple noisy images.

[0097] In some embodiments, the plurality of sample images includes a plurality of sets of sample images, wherein at least one set of sample images contains noisy images.

[0098] According to the above-described scheme of the present invention, an incompletely trained defect detection model can be trained based on sample images including noisy images. By training based on noisy images, the above scheme can maintain the accuracy of the defect detection results output by the already trained defect detection model. This is especially useful in real-world industrial inspection scenarios where the acquired target images often contain a certain amount of noise.

[0099] For example, step S120 involves inputting the target image and the reference image into the trained defect detection model to obtain the defect detection result of the target object, including steps S121 to S125.

[0100] In step S121, the third grid corresponding to the target image is determined based on the image size of the target image.

[0101] The third grid corresponding to the target image is used to represent the normalized result of the position of each pixel in the target image. Specifically, the third grid corresponding to the target image can be determined based on the image size of the target image, referring to the relevant content of step S210 above. The embodiments of the present invention will not be elaborated here.

[0102] The third grid is an orthogonal grid. In some embodiments, the resolution of the reference image is the same as that of the target image, and the third grid corresponding to the target image is determined based on the image size of the target image. In some alternative embodiments, the resolution of the reference image is different from that of the target image. The resolution of the reference image can be adjusted to be the same as that of the target image, and then the third grid corresponding to the target image is determined based on the image size of the target image.

[0103] In step S122, the target image and the corresponding third grid are input to the trained spatial transformation module to obtain the second image and the fourth grid.

[0104] The generation process of the second image and the fourth grid can be referred to the relevant content of step S220 above, and will not be repeated here in the embodiments of the present invention.

[0105] In step S123, the difference between the second image and the reference image is calculated to obtain the difference image corresponding to the target image.

[0106] Based on the second image and the reference image, and referring to the relevant content of step S230 above, the difference image corresponding to the target image can be determined. The embodiments of the present invention will not be described in detail here.

[0107] In step S124, the difference image, reference image, and fourth grid corresponding to the target image are input into the trained feature extraction module to obtain the fused features corresponding to the target image.

[0108] In some embodiments, the trained feature extraction module first extracts features from the difference image, the reference image, and the fourth grid through the encoder, and then fuses the three features to obtain the fused features. For details, please refer to the relevant content of step S240. The embodiments of the present invention will not be described in detail here.

[0109] In step S125, the defect detection result of the target object is determined based on the fusion features corresponding to the target image.

[0110] The aforementioned fused features can be input into the trained intermediate module and the trained detection head to obtain the defect detection results of the target object. The specific processing flow for the fused features can be adjusted based on the specific model structure of the defect detection model; this embodiment of the invention does not impose any limitations on this.

[0111] According to the above-described scheme of the present invention, a third grid corresponding to the target image can be determined based on the image size of the target image. Then, the target image and its corresponding third grid are input into a trained spatial transformation module to obtain a second image and a fourth grid. The difference between the second image and the reference image is then calculated to obtain a difference image corresponding to the target image. Next, the difference image, the reference image, and the fourth grid are input into a trained feature extraction module to obtain fusion features corresponding to the target image. Finally, based on the fusion features corresponding to the target image, the defect detection result of the target object is determined. The trained defect detection model in the above scheme can effectively utilize the positional information of the grid points in the fourth grid and the visual features in the reference image, thereby improving the accuracy of the final defect detection result.

[0112] This invention also provides a defect detection device. Figure 3 A schematic block diagram of a defect detection apparatus 300 according to an embodiment of the present invention is shown. (In conjunction with...) Figure 3 As shown, the defect detection device 300 may include an image acquisition module 310 and a defect detection result determination module 320.

[0113] The image acquisition module 310 is used to acquire a target image including a target object and a reference image including a reference object, wherein the reference object and the target object have the same object type.

[0114] The defect detection result determination module 320 is used to input the target image and the reference image into the trained defect detection model to obtain the defect detection result of the target object. The trained defect detection model includes a trained spatial transformation module and a trained feature extraction module. The trained spatial transformation module is used to correct the target image to obtain a corrected target image spatially aligned with the reference image. The trained feature extraction module is used to perform feature extraction processing on the corrected target image. The trained defect detection model is trained based on multiple sample images including the sample object, template images, difference images corresponding to each of the multiple sample images, and defect annotation results corresponding to each of the multiple sample images. The target object has the same object type. Multiple sample images are obtained through image alignment. The template image is obtained through image fusion based on multiple sample images. For each difference image in the difference image corresponding to each of the multiple sample images, the difference image is determined based on the sample image corresponding to the difference image and the template image corresponding to the difference image. For each sample image in the multiple sample images, the fusion feature output by the untrained feature extraction module is determined based on the template image and the difference image corresponding to the sample image. The defect detection result is used to indicate the defect type of the target object or whether the target object has a defect. The defect annotation result corresponding to the sample image is used to indicate the defect type of the sample object in the sample image or whether the sample object has a defect.

[0115] For example, the plurality of sample images includes at least one set of sample images, and for each set of sample images in the at least one set of sample images, each sample image in the set of sample images is obtained through image alignment processing. The detection device 300 also includes a first network determination module, a second network determination module, a first difference image determination module, a first fusion feature determination module, an overall loss value determination module, and a parameter adjustment module.

[0116] The first network determination module is used to perform image fusion processing on all sample images in each of at least one set of sample images to obtain a template image corresponding to the set of sample images, and determine a first grid corresponding to the set of sample images based on the image size of the set of sample images. The template image corresponding to the set of sample images is spatially aligned with each sample image in the set of sample images, and the first grid corresponding to the set of sample images is used to represent the normalized result of the position of each pixel in each sample image in the set of sample images in its respective sample image.

[0117] The second network determination module is used to input the sample image and the corresponding first grid to the untrained spatial transformation module for each sample image in at least one set of sample images, so as to obtain the first image and the second grid.

[0118] The first difference image determination module is used to, for each of the at least one set of sample images, subtract the first image from the template image corresponding to the first image and the template image corresponding to the first image to obtain the difference image corresponding to the first image.

[0119] The first fusion feature determination module is used to, for each sample image in at least one set of sample images, input the difference image corresponding to the sample image, the template image corresponding to the sample image, and the second grid into the untrained feature extraction module to obtain the fusion feature corresponding to the sample image.

[0120] The overall loss value determination module is used to determine the overall loss value for each sample image in at least one set of sample images, based on the fusion features corresponding to that sample image.

[0121] The parameter adjustment module is used to adjust the module parameters of the untrained spatial transformation module and the untrained feature extraction module for each sample image in at least one set of sample images, based on the overall loss value.

[0122] For example, the overall loss value includes a first loss value, and the overall loss value determination module includes a decoded image determination module, a reconstructed image determination module, and a first loss value determination module.

[0123] The decoded image determination module inputs the fused features corresponding to the sample image into the decoding module to obtain the decoded image. The reconstructed image determination module performs an inverse transform on the decoded image based on the processing parameters of the untrained spatial transform module to obtain the reconstructed image. The first loss value determination module determines the difference between the reconstructed image and the sample image as the first loss value.

[0124] For example, the overall loss value also includes a second loss value, and the overall loss value determination module also includes a second loss value determination module.

[0125] The second loss value determination module is used to determine the difference between the second grid and the first grid, as the second loss value.

[0126] For example, the multiple sample images include multiple local images, where each local image is a sample image that includes a portion of the sample object.

[0127] For example, the multiple sample images include at least one noisy image, which is obtained by image noise addition processing based on the original image including the sample object.

[0128] For example, the defect detection result determination module includes a third network determination module, a fourth network determination module, a second differential image determination module, a second fusion feature determination module, and a defect detection result determination submodule.

[0129] The third network determination module determines the third grid corresponding to the target image based on its image size. This third grid represents the normalized position of each pixel in the target image. The fourth network determination module inputs the target image and its corresponding third grid into a trained spatial transformation module to obtain the second image and the fourth grid. The second difference image determination module subtracts the second image from the reference image to obtain the difference image corresponding to the target image. The second fusion feature determination module inputs the difference image, the reference image, and the fourth grid into a trained feature extraction module to obtain the fusion features corresponding to the target image. The defect detection result determination submodule determines the defect detection result of the target object based on the fusion features corresponding to the target image.

[0130] According to another aspect of the present invention, an electronic device is also provided. Figure 4 A schematic block diagram of an electronic device 400 according to an embodiment of the present invention is shown. Figure 4 As shown, the electronic device 400 includes a processor 410 and a memory 420. The memory 420 stores a computer program, and the computer program instructions are executed by the processor 410 to perform the aforementioned defect detection method.

[0131] Furthermore, according to another aspect of the present invention, a storage medium is provided, on which program instructions are stored. When the program instructions are executed by a computer or processor, the computer or processor performs corresponding steps of the defect detection method described in the embodiments of the present invention, and is used to implement corresponding modules in the defect detection apparatus or electronic device described in the embodiments of the present invention. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media. According to yet another aspect of the present invention, a computer program product is also provided, including computer program instructions. When the computer program instructions are executed by a computer or processor, the computer or processor performs corresponding steps of the defect detection method described above.

[0132] Those skilled in the art can understand the specific implementation schemes of the above-mentioned electronic devices and storage media by reading the relevant descriptions of the defect detection methods, and for the sake of brevity, they will not be described in detail here.

[0133] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention thereto. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0134] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0135] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0136] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0137] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention. However, this approach should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0138] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0139] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0140] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the defect detection apparatus according to embodiments of the present invention. The present invention can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0141] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0142] The above description is merely a specific embodiment of the present invention or an explanation of that embodiment. The scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A defect detection method, characterized in that, The method includes: Acquire a target image including a target object and a reference image including a reference object, wherein the reference object and the target object are of the same object type; The target image and reference image are input into a trained defect detection model to obtain the defect detection result of the target object. The trained defect detection model includes a trained spatial transformation module and a trained feature extraction module. The trained spatial transformation module corrects the target image to obtain a corrected target image spatially aligned with the reference image. The trained feature extraction module performs feature extraction processing on the corrected target image. The trained defect detection model is trained based on multiple sample images including a sample object, a template image, difference images corresponding to each of the multiple sample images, and defect annotation results corresponding to each of the multiple sample images. The sample object and the target object... The objects are of the same type. The multiple sample images are obtained through image alignment processing. The template image is obtained based on the multiple sample images through image fusion processing. For each difference image in the difference image corresponding to each of the multiple sample images, the difference image is determined based on the sample image corresponding to the difference image and the template image corresponding to the difference image. For each sample image in the multiple sample images, the fusion feature output by the feature extraction module that has not been trained is determined based on the template image and the difference image corresponding to the sample image. The defect detection result is used to indicate the defect type of the target object or whether the target object has a defect. The defect annotation result corresponding to the sample image is used to indicate the defect type of the sample object in the sample image or whether the sample object has a defect.

2. The method as described in claim 1, characterized in that, The plurality of sample images includes at least one set of sample images. For each set of sample images in the at least one set of sample images, each sample image in the set of sample images is obtained through image alignment processing. The trained defect detection model is obtained through the following steps: For each of the at least one set of sample images, Image fusion processing is performed on all sample images in the set of sample images to obtain the template image corresponding to the set of sample images. Based on the image size of the set of sample images, the first grid corresponding to the set of sample images is determined. The template image corresponding to the set of sample images is spatially aligned with each sample image in the set of sample images. The first grid corresponding to the set of sample images is used to represent the normalized result of the position of each pixel in each sample image in the set of sample images in its respective sample image. For each sample image in this set of sample images The sample image and the first grid corresponding to the set of sample images are input into the untrained spatial transformation module to obtain the first image and the second grid. The difference between the first image and the template image corresponding to the group of sample images is calculated to obtain the difference image corresponding to the sample image; The difference image corresponding to the sample image, the template image corresponding to the group of sample images, and the second grid are input into the untrained feature extraction module to obtain the fusion feature corresponding to the sample image; Based on the fusion features corresponding to the sample image, the overall loss value is determined; Based on the overall loss value, adjust the module parameters of the untrained spatial transformation module and the untrained feature extraction module.

3. The method as described in claim 2, characterized in that, The overall loss value includes a first loss value, and determining the overall loss value based on the fusion features corresponding to the sample image includes: The fusion features corresponding to the sample image are input into the decoding module to obtain the decoded image; Based on the processing parameters of the untrained spatial transformation module, the decoded image is subjected to inverse transformation to obtain the reconstructed image; The difference between the reconstructed image and the sample image is determined and used as the first loss value.

4. The method as described in claim 3, characterized in that, The overall loss value also includes a second loss value, and the process of determining the overall loss value based on the fusion features corresponding to the sample image further includes: The difference between the second grid and the first grid is determined as the second loss value.

5. The method as described in claim 1, characterized in that, The plurality of sample images includes a plurality of local images, wherein each local image is a sample image that includes a portion of the sample object.

6. The method as described in claim 1, characterized in that, The plurality of sample images includes at least one noisy image, which is obtained by image noise addition processing based on the original image including the sample object.

7. The method as described in claim 1, characterized in that, The step of inputting the target image and reference image into the trained defect detection model to obtain the defect detection result of the target object includes: Based on the image size of the target image, a third grid corresponding to the target image is determined, wherein the third grid corresponding to the target image is used to represent the normalized result of the position of each pixel in the target image in the target image; The target image and the corresponding third grid are input into the trained spatial transformation module to obtain the second image and the fourth grid. The difference between the second image and the reference image is calculated to obtain the difference image corresponding to the target image; The difference image corresponding to the target image, the reference image, and the fourth grid are input into the trained feature extraction module to obtain the fused features corresponding to the target image; Based on the fusion features corresponding to the target image, the defect detection result of the target object is determined.

8. A defect detection device, characterized in that, The device includes: An image acquisition module is used to acquire a target image including a target object and a reference image including a reference object, wherein the reference object and the target object are of the same object type; A defect detection result determination module is used to input a target image and a reference image into a trained defect detection model to obtain the defect detection result of the target object. The trained defect detection model includes a trained spatial transformation module and a trained feature extraction module. The trained spatial transformation module corrects the target image to obtain a corrected target image spatially aligned with the reference image. The trained feature extraction module performs feature extraction processing on the corrected target image. The trained defect detection model is trained based on multiple sample images including a sample object, a template image, difference images corresponding to each of the multiple sample images, and defect annotation results corresponding to each of the multiple sample images. The image is of the same type as the target object. The multiple sample images are obtained through image alignment processing. The template image is obtained through image fusion processing based on the multiple sample images. For each difference image in the difference image corresponding to each of the multiple sample images, the difference image is determined based on the sample image corresponding to the difference image and the template image corresponding to the difference image. For each sample image in the multiple sample images, the fusion feature output by the untrained feature extraction module is determined based on the template image and the difference image corresponding to the sample image. The defect detection result is used to indicate the defect type of the target object or whether the target object has a defect. The defect annotation result corresponding to the sample image is used to indicate the defect type of the sample object in the sample image or whether the sample object has a defect.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program to implement the defect detection method as described in any one of claims 1-7.

10. A storage medium storing computer program instructions, characterized in that, The computer program instructions are used to execute the defect detection method as described in any one of claims 1-7 when the program is run.

11. A computer program product comprising computer program instructions, characterized in that, The computer program instructions, when executed by a processor, are used to perform the defect detection method as described in any one of claims 1-7.

Images