Defect detection method and device, electronic equipment and storage medium

Abstract

The invention provides a defect detection method and device, electronic equipment and a storage medium. The detection method comprises the following steps: acquiring a target image comprising a target crystal grain; and through the trained defect detection model, determining a target defect type of the target crystal grain and a target defect region corresponding to the defective part in the target crystal grain in the target image. According to the defect detection model, a segmented training strategy is adopted, so that the accuracy of defect type classification of the model is ensured. And then combined training is carried out for the training images of the crystal grain scene, and the plurality of first projection modules are trained through the training images including the trained crystal grains, so that the adaptability of the model to crystal grain related image processing is improved on the premise of ensuring that the model can accurately classify the defect types and position the defect areas. The trained defect detection model can reduce the misjudgment rate of the model for normal textures and defect textures to a certain extent.

Description

Defect detection methods and devices, electronic equipment, and storage media Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a defect detection method, a defect detection device, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] In the field of computer vision, the development of visual coding models has greatly promoted the performance improvement of various visual tasks. General visual coding models, represented by Contrastive Language-Image Pre-training (CLIP), have demonstrated excellent transfer learning capabilities and generalization performance thanks to their cross-modal pre-training paradigm. These models, trained on large-scale natural scene datasets, can learn universal visual feature representations. They can be well adapted to various general visual tasks such as image classification and object detection without requiring extensive fine-tuning with labeled data for specific tasks, providing an efficient solution for the industrial application of computer vision technology.

[0003] However, in actual industrial production, the application of general-purpose visual coding models such as CLIP faces significant bottlenecks. The core issue is that the training data for these models primarily comes from natural scenes, and the visual features they learn focus more on the global contours and general semantic information of objects in those scenes. However, for vertical domains, especially defect segmentation and classification, general-purpose visual coding models like CLIP suffer from the loss of both global and local information. Therefore, how to efficiently and accurately detect defects on wafers is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The present invention was proposed in view of the above-mentioned problems. The present invention provides a defect detection method, a defect detection device, an electronic device, a storage medium, and a computer program product.

[0005] According to one aspect of the present invention, a defect detection method is provided. The detection method includes: acquiring a target image including a target grain; determining, through a trained defect detection model, the target defect type of the target grain and the target defect region corresponding to the defective portion of the target grain in the target image; wherein the trained defect detection model is trained based on multiple training images including training grains, the training defect region corresponding to the defective portion of the training grain in each training image, and the training defect type corresponding to each training image; the trained defect detection model includes multiple sequentially connected type detection pre-training modules, multiple trained first projection modules, and multiple sequentially connected... The type detection pre-training module is used to determine the target defect type, which represents the defect type present in the target grain. Multiple trained first projection modules are used to determine the target defect region. For each of the multiple sequentially connected type detection pre-training modules, the output of the type detection pre-training module is connected to the input of a trained first projection module. The input of the type detection pre-training module whose input is connected to the output of the type detection pre-training module is connected to the output of the trained first projection module. Furthermore, the trained first projection module is different from the trained first projection modules whose input is connected to the output of other type detection pre-training modules.

[0006] For example, the trained defect detection model also includes a pre-trained defect type classification module. The input of the pre-trained defect type classification module is connected to the output of the trained second projection module. The input of the trained second projection module is connected to the output of the last type detection pre-trained module among a plurality of sequentially connected type detection pre-trained modules, and is also connected to the output of the trained first projection module, which is connected to both the input and the output of the last type detection pre-trained module.

[0007] For example, the trained defect detection model also includes a trained feature extraction module. The trained feature extraction module is used to output the overall target features and the local target features to the first type detection pre-training module among a plurality of sequentially connected type detection pre-training modules. The overall target features are used to represent the overall visual features of the target image, and the local target features are used to represent the local visual features of the target image.

[0008] For example, the trained defect detection model also includes a pre-trained feature extraction module. The output of the pre-trained feature extraction module is connected to the trained third projection module. The trained third projection module is used to output the overall target features and local target features to the first type detection pre-training module among a plurality of sequentially connected type detection pre-training modules. The overall target features are used to represent the overall visual features of the target image, and the local target features are used to represent the local visual features of the target image.

[0009] For example, the trained defect detection model also includes a trained region segmentation and classification module, which is connected to multiple trained first projection modules. The trained region segmentation and classification module is used to determine the target defect region based on the average features of the output features of the multiple trained first projection modules.

[0010] For example, the detection method further includes:

[0011] Acquire the original image including the target grain;

[0012] Based on a reference image including a reference grain, the original image is subjected to image correction processing to obtain the target image, wherein the reference grain is a grain without defects.

[0013] According to another aspect of the present invention, a defect detection device is also provided. The detection device includes: a target image acquisition module, which can be used to acquire a target image including a target grain; and a defect determination module, which can be used to determine the target defect type of the target grain and the target defect region corresponding to the defective portion in the target grain in the target image through a trained defect detection model; wherein the trained defect detection model is trained based on multiple training images including training grains, the training defect region corresponding to the defective portion in the training grain in each training image, and the training defect type corresponding to each training image. The trained defect detection model includes multiple sequentially connected type detection pre-training modules and multiple trained first... The projection module comprises multiple sequentially connected type detection pre-training modules used to determine the target defect type, which represents the defect type present in the target grain. Multiple trained first projection modules are used to determine the target defect region. For each of the multiple sequentially connected type detection pre-training modules, the output of the type detection pre-training module is connected to the input of a trained first projection module. The input of a type detection pre-training module whose input is connected to the output of the type detection pre-training module is connected to the output of the trained first projection module. Furthermore, the trained first projection module is different from the trained first projection modules whose inputs are connected to the outputs of other type detection pre-training modules.

[0014] According to another aspect of the present invention, an electronic device is also provided. The electronic device includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, are used to perform the aforementioned defect detection method.

[0015] According to another aspect of the present invention, a storage medium is also provided. Program instructions are stored on this storage medium, which, when executed, are used to perform the aforementioned defect detection method.

[0016] According to another aspect of the present invention, a computer program product is also provided. This computer program product includes computer program instructions that, when executed by a processor, are used to perform the aforementioned defect detection method.

[0017] According to the above-described scheme of the present invention, a target image including the target grain can be obtained. Then, the target defect type of the target grain and the target defect region corresponding to the defective portion in the target grain in the target image can be determined through a trained defect detection model. On one hand, by inputting the target image into the trained defect detection model, the above scheme can simultaneously determine the type of defect in the target grain and locate its region, which is beneficial to improving the efficiency of defect detection. On the other hand, the above defect detection model adopts a segmented training strategy, that is, it pre-trains multiple type detection pre-training modules using a large number of images, which helps to ensure the accuracy of the model in classifying defect types. Then, joint training is carried out on the training images of the grain scene, training multiple first projection modules using training images including the training grain, ensuring that the model can accurately classify defect types and locate defect regions while improving its adaptability to grain-related image processing. Furthermore, the trained defect detection model in the above scheme can, to a certain extent, reduce the model's misclassification rate for normal textures and defect textures, which is beneficial to improving the reliability of the target defect type and the target defect region. Attached Figure Description

[0018] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0019] Figure 1 shows a schematic flowchart of a defect detection method according to an embodiment of the present invention;

[0020] Figure 2 shows a schematic diagram of a defect detection model according to an embodiment of the present invention;

[0021] Figure 3 shows a schematic block diagram of a defect detection device according to an embodiment of the present invention;

[0022] Figure 4 shows a schematic block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0024] To at least partially address the aforementioned problems, embodiments of the present invention provide a defect detection method. Figure 1 shows a schematic flowchart of a defect detection method according to an embodiment of the present invention. As shown in Figure 1, the method may include steps S110 and S120.

[0025] In step S110, a target image including the target grain is obtained.

[0026] The target grain can be the grain that needs to be defect-detected.

[0027] The target image of the target chip can be a static image or any video frame from a dynamic video. The target image of the target chip can be the original image captured by an image acquisition device (e.g., the original image captured by the image sensor in a camera), or it can be an image obtained after preprocessing the original image (e.g., digitization, normalization, smoothing, etc.).

[0028] In step S120, the target defect type of the target grain and the target defect region in the target image corresponding to the defective part in the target grain are determined by the trained defect detection model.

[0029] The trained defect detection model is trained based on multiple training images including training dies, the training defect region corresponding to the defective portion of the training die in each training image, and the training defect type corresponding to each training image. For example, the defect detection model can be a visual encoding model based on a transformer, a model combining a convolutional neural network (CNN) and visual encoding, etc. The training dies can include dies with defects (e.g., foreign objects, scratches, etc.) and dies without any defects. The process of acquiring multiple training images including training dies can refer to the target image acquisition method in step S110 above, and will not be elaborated here in this embodiment of the invention. The training defect region can be manually labeled in the training image according to the actual situation of the training die. When there is a training defect region in the training image, the training defect type can be used to represent the defect type corresponding to each training defect region (e.g., "scratch", "breakage", "broken line", "foreign object", etc.). When there is no training defect region in the training image, the training defect type can be used to represent the absence of defect regions (e.g., "normal"). The training defect type can also be manually labeled according to the actual performance of the training image. The aforementioned training defect regions are used to determine the difference (with weights) between the training defect regions and the predicted defect regions output by the defect detection model based on the training images, serving as the first loss value. The aforementioned training defect types are used to determine the difference (with weights) between the training defect types and the predicted defect types output by the defect detection model based on the training images, serving as the second loss value. It can be understood that, considering the correlation between the training defect regions and training defect types, the first loss value can also be determined based on the second loss value. The overall loss value of the defect detection model includes at least the aforementioned first and second loss values.

[0030] The trained defect detection model includes multiple sequentially connected type detection pre-training modules and multiple trained first projection modules. The multiple sequentially connected type detection pre-training modules are used to determine the target defect type, which represents the defect type present in the target grain. The multiple trained first projection modules are used to determine the target defect region. For example, Figure 2 shows a schematic diagram of a defect detection model according to an embodiment of the present invention. As shown in Figure 2, the trained defect detection model may include type detection pre-training modules L1 to L2. n First projection module T1 to first projection module T n Type detection pre-training module L1 to type detection pre-training module L n It can be used to determine the type of target defect, from the first projection module T1 to the first projection module T nThis can be used to determine the target defect region. It is worth noting that the aforementioned type detection pre-training module can be trained before the first projection module. Specifically, for example, a defect detection model that does not include the aforementioned multiple first projection modules can be pre-trained using a large number of images. That is, only the aforementioned type detection pre-training modules are trained at this stage, enabling the type detection pre-training module to detect whether a defect exists in the input image and, if so, the type of each defect. After training the aforementioned type detection pre-training modules, they can be frozen, and the aforementioned multiple first projection modules can be connected to the defect detection model. By training the multiple first projection modules using multiple training images including training grains, the trained defect detection model can accurately determine whether a defect exists in the target image and, if so, the type of each defect (i.e., the aforementioned target defect type), while also accurately dividing the defect region (i.e., the aforementioned target defect region) in the target image. It is understood that the images used to train the multiple type detection pre-training modules can include images related to the grains or images unrelated to the grains.

[0031] For each of a series of sequentially connected type detection pre-training modules, its output is connected to the input of a trained first projection module. The input of a type detection pre-training module connected to its output is then connected to the output of that trained first projection module, and this trained first projection module is different from those connected to the inputs of other type detection pre-training modules. For example, as shown in Figure 2, the output of type detection pre-training module L1 is connected to the input of the next type detection pre-training module. n The input terminal can be connected to the output terminal of the previous type detection pre-training module, that is, the type detection pre-training module L1 to the type detection pre-training module L... n This consists of multiple type detection pre-training modules connected in sequence. Taking type detection pre-training module L1 as an example, the output of type detection pre-training module L1 is connected to the input of the first projection module T1. The input of a type detection pre-training module connected to the output of type detection pre-training module L1 (i.e., the next type detection pre-training module after type detection pre-training module L1) can be connected to the output of the first projection module T1. In Figure 2, the type detection pre-training module connected to the output of the first projection module T1 is different from the type detection pre-training modules connected to the outputs of other first projection modules, which is equivalent to different type detection pre-training modules being connected to different outputs of different first projection modules.

[0032] For example, each type detection pre-training module can generate and output a general feature vector based on the input data. This general feature vector may include a global feature vector (which can be used to represent the semantic information of the entire input image) and local feature vectors. Each local feature vector may include m local feature sub-vectors, each of which can be used to represent the semantic information of an image patch corresponding to the input image. Here, m is a positive integer, and its value can be determined by the image size of the input defect detection model and the image patch size set by the defect detection model. It is understood that the general feature vectors output by different type detection pre-training modules correspond to different levels of semantic information. The level of semantic information gradually deepens from the general feature vector output by the first type detection pre-training module to the general feature vector output by the last type detection pre-training module. For example, the level of semantic information corresponding to the general feature vector output by the first type detection pre-training module may be low-level visual features (e.g., edges, colors, textures, etc.), while the level of semantic information corresponding to the general feature vector output by the last type detection pre-training module may be high-level visual features (e.g., defect contours, etc.).

[0033] Each type detection pre-training module (as shown in Figure 2, type detection pre-training module L1 to type detection pre-training module L) n Each of the input and output general feature vectors can output them to the first projection module connected to the input. The first projection module generates a grain feature vector, including global and local feature vectors, based on the input general feature vector and the feature mapping relationship corresponding to the first projection module. The global feature vector in this grain feature vector is obtained by the first projection module mapping the global feature vector in the input general feature vector to a more suitable global feature vector for grain image detection, based on its corresponding feature mapping relationship. The local feature vector in this grain feature vector is obtained by the first projection module mapping the local feature vector in the input general feature vector to a more suitable local feature vector for grain image detection, based on its corresponding feature mapping relationship. It is understood that the feature mapping relationship corresponding to each first projection module is obtained through training, and the feature mapping relationships corresponding to different first projection modules are different.

[0034] In one example, for each first projection module whose input is connected to the type detection pre-training module, the output of the first projection module can be connected to the next module (e.g., the next type detection pre-training module, or another module that needs to be connected to the output of the first projection module) through a residual summing module. Similarly, the output of the type detection pre-training module connected to the input of the first projection module can also be connected to the next module through the residual summing module. Taking the first projection module T1 in Figure 2 as an example, its output can be connected to the input of the next type detection pre-training module after type detection pre-training module L1 through the residual summing module. The output of type detection pre-training module L1 can also be connected to the input of the residual summing module. Type detection pre-training module L1 can output the generated general feature vector to the residual summing module, and the first projection module T1 can output the generated grain feature vector to the residual summing module. The residual summing module can perform residual summation on the received general feature vector and grain feature vector, and output the result as a comprehensive feature vector to the next type detection pre-training module after type detection pre-training module L1. The comprehensive feature vector may include global feature vectors and local feature vectors. The global feature vector in the comprehensive feature vector can be calculated using the residual summation formula, the global feature vector in the general feature vector, and the global feature vector in the grain feature vector. The local feature vector in the comprehensive feature vector can also be calculated using the residual summation formula, the local feature vector in the general feature vector, and the local feature vector in the grain feature vector. The aforementioned general feature vector, grain feature vector, and global feature vector in the comprehensive feature vector can be used to determine the target defect type. The aforementioned general feature vector, grain feature vector, and local feature vector in the comprehensive feature vector can be used to determine the target defect region. It is understood that the output terminals of different first projection modules are connected to different residual summation modules.

[0035] In another example, for each first projection module whose input is connected to the type detection pre-training module, the output of the first projection module can be connected to the next module (e.g., the next type detection pre-training module, or another module that needs to be connected to the output of the first projection module) through a feature fusion module. The output of the type detection pre-training module connected to the input of the first projection module can also be connected to the next module through the feature fusion module. Taking the first projection module T1 in Figure 2 as an example, its output can be connected to the input of the next type detection pre-training module after type detection pre-training module L1 through the feature fusion module. The output of type detection pre-training module L1 can also be connected to the input of the feature fusion module. Type detection pre-training module L1 can output the generated general feature vector to the feature fusion module, and the first projection module T1 can output the generated grain feature vector to the feature fusion module. The feature fusion module can perform feature fusion on the received general feature vector and grain feature vector (e.g., through channel splicing, attention fusion, etc.), and output the fusion result as a comprehensive feature vector to the next type detection pre-training module after type detection pre-training module L1. The comprehensive feature vector may include global feature vectors and local feature vectors. The global feature vector in the comprehensive feature vector can be calculated based on the feature fusion formula, the global feature vector in the general feature vector, and the global feature vector in the grain feature vector. The local feature vectors in the comprehensive feature vector can also be calculated based on the feature fusion formula, the local feature vectors in the general feature vector, and the local feature vectors in the grain feature vector. The aforementioned general feature vector, grain feature vector, and global feature vector in the comprehensive feature vector can be used to determine the target defect type. The aforementioned general feature vector, grain feature vector, and local feature vector in the comprehensive feature vector can be used to determine the target defect region. It is understood that the output terminals of different first projection modules are connected to different feature fusion modules.

[0036] This embodiment of the invention provides an example for reference. The target image is input into a trained defect detection model. If the defect types present in the target image are "foreign object" and "scratch," the trained defect detection model can output the target defect types representing "foreign object" and "scratch," and output the target defect regions corresponding to the "foreign object" and "scratch" target defect types, respectively. If there are no defects in the target image, the trained defect detection model can output the target defect type representing "normal," based on the input target image.

[0037] According to the above-described scheme of the present invention, a target image including the target grain can be acquired. Then, the target defect type of the target grain and the target defect region corresponding to the defective portion in the target grain in the target image can be determined through a trained defect detection model. On one hand, by inputting the target image into the trained defect detection model, the above scheme can simultaneously determine the type of defect in the target grain and locate its region, which is beneficial to improving the efficiency of defect detection. On the other hand, the above defect detection model adopts a segmented training strategy, that is, it pre-trains multiple type detection pre-training modules using a large number of images, which helps to ensure the accuracy of the model in classifying defect types. Then, multiple first projection modules are trained using training images including the training grain, ensuring that the model can accurately classify defect types and locate defect regions while improving its adaptability to grain-related image processing. Furthermore, the trained defect detection model in the above scheme can, to a certain extent, reduce the model's misclassification rate for normal textures and defect textures, which is beneficial to improving the reliability of the target defect type and the target defect region.

[0038] For example, the trained defect detection model also includes a pre-trained defect type classification module.

[0039] The pre-trained defect type classification module described above can be trained together with the type detection pre-training module. This pre-trained defect type classification module can be used to classify the defect types in the input image based on the output data of the last type detection pre-training module during the training of the type detection pre-training module. It should be understood that this classification applies to defects in various types of input images, not specifically to defects in images related to grains.

[0040] The input of the pre-trained defect type classification module is connected to the output of the trained second projection module. The input of the trained second projection module is connected to the output of the last type detection pre-training module in a series of sequentially connected type detection pre-training modules, and also connected to the output of the trained first projection module, which is connected to both the input and the output of the last type detection pre-training module. For example, as shown in Figure 2, the type detection pre-training module L... n The output of the last type detection pre-training module (i.e., the last type detection pre-training module mentioned above) can be connected to the input of the second projection module, and the first projection module T nThe output of the first projection module (which is connected to the input of the last type detection pre-training module and the output of the first projection module after training) is connected to the input of the second projection module, and the output of the second projection module is connected to the input of the pre-trained defect type classification module. Specifically, for example, referring to the content corresponding to the residual addition module and feature fusion module in step S120 above, it can be seen that the type detection pre-training module L... n The output terminal can be connected to the first projection module T n The input is connected, and the type detection pre-training module L n The output terminal and the first projection module T n The output can be connected to the input of the second projection module via a residual summing module or a feature fusion module. Here, we take the type detection pre-training module L as an example. n The output terminal and the first projection module T n Taking the output of the L type detection pre-training module as an example, which is connected to the input of the second projection module through a residual summing module. n Based on the input comprehensive feature vector, a general feature vector can be generated and output to the first projection module T. n And in the residual addition module. First projection module T n The grain feature vector generated and output based on the general feature vector is fed into the residual addition module. The residual addition module generates and outputs a comprehensive feature vector to the second projection module based on the received general feature vector and grain feature vector. The second projection module generates and outputs a final feature vector based on the input comprehensive feature vector and its corresponding feature mapping relationship (obtained through training). The final feature vector may include global and local feature vectors. The global feature vector in the final feature vector can be obtained by the second projection module mapping the global feature vector in the input comprehensive feature vector to a global feature vector more suitable for grain image detection. The local feature vector in the final feature vector can be obtained by the second projection module mapping the local feature vector in the input comprehensive feature vector to a local feature vector more suitable for grain image detection. The second projection module generates and outputs the final feature vector to the defect type classification module. The defect type classification module outputs the target defect type of the target grain based on the global feature vector in the input final feature vector. Specifically, for example, the defect type classification module can make classification decisions based on the global feature vector in the final input feature vector, and output the probability between the input image and each defect type (which can be determined based on the training defect types). Then, the defect type corresponding to the probability value greater than a preset probability value is taken as the defect type corresponding to the input image. When the input image is a target image, the defect type corresponding to the probability value greater than the preset probability value is the target defect type. The aforementioned preset probability value can be determined manually.

[0041] According to the above-described scheme of the present invention, the trained defect detection model may further include a pre-trained defect type classification module. The input of the pre-trained defect type classification module is connected to the output of the trained second projection module. The input of the trained second projection module is connected to the output of the last type detection pre-trained module among a plurality of sequentially connected type detection pre-trained modules, and is also connected to the output of the trained first projection module, which is connected to both the input and the output of the last type detection pre-trained module. The above scheme, through the pre-trained defect type classification module and the second projection module, helps to improve the accuracy of the defect detection model in classifying defect types in target images including target grains.

[0042] For example, the trained defect detection model also includes a trained feature extraction module.

[0043] It is understandable that developers may omit the feature extraction module depending on the actual situation. The trained feature extraction module outputs the overall and local features of the target to the first type detection pre-training module among multiple sequentially connected type detection pre-training modules. For example, the feature extraction module can be trained together with the type detection pre-training module during its training, and then trained together with the multiple first projection modules after the type detection pre-training module has completed training and been frozen. Specifically, referring to Figure 2, the input of the trained feature extraction module can be used to receive the target image. The output of the feature extraction module can be connected to the input of the type detection pre-training module L1 (i.e., the first type detection pre-training module among the multiple sequentially connected type detection pre-training modules). The feature extraction module can be used to extract features from the target image, generating and outputting the overall and local features of the target to the type detection pre-training module L1. The type detection pre-training module L1 can generate a global feature vector in the general feature vector based on the overall target features output by the feature extraction module, and generate a local feature vector in the general feature vector based on the local target features output by the feature extraction module. The aforementioned overall target features can be used to represent the overall visual features of the target image (e.g., the overall shape, color distribution, and texture of the target image). Local target features are used to represent the local visual features of the target image (e.g., edges, corners, and texture details of specific regions within the target image).

[0044] According to the above-described scheme of the present invention, the trained defect detection model may further include a trained feature extraction module. The trained feature extraction module outputs the overall target features and local target features to the first type detection pre-training module among a plurality of sequentially connected type detection pre-training modules. The overall target features represent the overall visual features of the target image, and the local target features represent the local visual features of the target image. In the above scheme, the feature extraction module can be trained together with multiple first projection modules, ensuring that the feature extraction module in the trained defect detection model is more suitable for feature extraction of grain-related images, thereby improving the accuracy of the target defect type and target defect region output by the defect detection model.

[0045] For example, the trained defect detection model also includes a pre-trained feature extraction module.

[0046] The pre-trained feature extraction module described above can be trained together with the type detection pre-trained module described above. When training the first projection module, both the feature extraction module and the type detection pre-trained module can be frozen.

[0047] The output of the pre-trained feature extraction module is connected to the trained third projection module. The trained third projection module outputs the overall target features and local target features to the first type detection pre-training module in a series of sequentially connected type detection pre-training modules. The overall target features represent the overall visual features of the target image, and the local target features represent the local visual features of the target image. For example, the feature extraction module can generate and output feature extraction results to the third projection module based on the target image. The third projection module can obtain the overall target features and local target features by mapping the received feature extraction results to features more suitable for grain image detection, based on the received feature extraction results and their corresponding feature mapping relationship (obtained through training). The third projection module can then output the generated overall target features and local target features to the first type detection pre-training module (e.g., type detection pre-training module L1 in Figure 2). Compared to the overall target features and local target features generated by the feature extraction module, the overall target features and local target features generated by the third projection module focus more on grain-related features in the input image.

[0048] According to the above-described scheme of the present invention, the trained defect detection model may further include a pre-trained feature extraction module. The output of the pre-trained feature extraction module is connected to the trained third projection module. The trained third projection module is used to output the overall target features and local target features to the first type detection pre-training module among a plurality of sequentially connected type detection pre-training modules. The overall target features represent the overall visual features of the target image, and the local target features represent the local visual features of the target image. The above scheme can pre-train the feature extraction module with a large number of general images, which can ensure that the feature extraction module can be fully trained and learn a stable, general, and robust visual feature representation. This helps to avoid overfitting and parameter instability problems that may occur when training directly on small samples or specific domain data. In addition, under the premise of pre-training the feature extraction module, adding a third projection module between the feature extraction module and the first type detection pre-training module ensures that the overall target features and local target features input to the first type detection pre-training module can more accurately adapt to the grain defect detection task, thereby improving the accuracy of the target defect type and target defect region output by the defect detection model.

[0049] For example, the trained defect detection model also includes a trained region segmentation and classification module, which is connected to multiple trained first projection modules. The trained region segmentation and classification module is used to determine the target defect region based on the average features of the output features of the multiple trained first projection modules.

[0050] As shown in Figure 2, the output of each first projection module is also connected to the region segmentation and classification module. Each first projection module can output its generated grain feature vector to the region segmentation and classification module. The region segmentation and classification module can calculate the above-mentioned average feature based on multiple local feature vectors among the received multiple grain feature vectors. Furthermore, the region segmentation and classification module can calculate the above-mentioned average feature based on its corresponding weights and bias values ​​to determine the target defect region in the target image. Specifically, for example, each grain feature vector output by the first projection module to the region segmentation and classification module includes m local feature sub-vectors. Each local feature sub-vector corresponds to an image block. For each image block with the same pixel coordinates, the average value of the local feature sub-vectors corresponding to the image block in each grain feature vector is calculated, and this average value is used as the average feature corresponding to the image block. The region segmentation and classification module can calculate the average feature of each image block based on its corresponding weights and bias values ​​to obtain the probability of the defect type corresponding to the image block. Image blocks with probability values ​​higher than a preset probability value are determined as target defect regions. It can be understood that the above-mentioned weights and bias values ​​can be obtained through supervised learning (using labeled training data).

[0051] According to the above-described scheme of the embodiments of the present invention, the trained defect detection model may further include a trained region segmentation and classification module. The trained region segmentation and classification module is connected to multiple trained first projection modules. The trained region segmentation and classification module is used to determine the target defect region based on the average features of the output features of the multiple trained first projection modules. The above scheme can effectively fuse multi-scale visual information, which is beneficial to improving the accuracy and segmentation continuity of target defect region localization, and can reduce false detection and false negative rates. For target images with minute defects and complex texture backgrounds, the above scheme can improve the stability of target defect region recognition to a certain extent.

[0052] For example, the defect detection method further includes steps S210 and S220.

[0053] In step S210, an original image including the target grain is acquired.

[0054] The method for obtaining the original image described above can be found in the method for obtaining the target image in step S110 above, and will not be repeated here in this embodiment of the invention.

[0055] In step S220, the original image is subjected to image correction processing based on the reference image including the reference grains to obtain the target image.

[0056] The reference grain is a grain without defects. For example, a reference grain without any defects can be equivalent to a target grain with a target defect type of "normal" in step S120 above.

[0057] The location of the region corresponding to the reference grain in the reference image can be close to the location of the region corresponding to the training grain in the training image. For example, a training grain without defects can be directly used as the reference grain, and the training image of this defect-free training grain can be used as the reference image.

[0058] This invention provides an example for reference. The original image can be processed using image processing methods (e.g., reduction, enlargement, rotation, affine transformation, perspective transformation, etc.) to ensure that the overlap between a specified region in the reference image and its corresponding region in the processed original image (i.e., the target image) is greater than a preset threshold. Specifically, the specified region in the reference image (which can be determined manually based on actual conditions) can be the region corresponding to the pads on the reference die in the reference image. Hereinafter, the specified region in the reference image is referred to as the reference pad region, and the region corresponding to this specified region in the original image is referred to as the original pad region. The overlap between the reference pad region and the original pad region can be determined based on the pixel coordinates corresponding to the reference pad region and the original pad region. When the overlap is greater than the preset threshold, the original image can be used as the target image. When the overlap is less than the preset threshold, the original image can be processed to ensure that the overlap between the original pad region and the reference pad region in the processed original image is greater than the preset threshold; in this case, the processed original image can be used as the target image.

[0059] According to the above-described scheme of the present invention, an original image including the target grain can be acquired. Then, based on a reference image including a reference grain, image correction processing is performed on the original image to obtain the target image. This scheme helps eliminate image distortion caused by factors such as shooting angle and positional offset, ensuring accurate alignment of grain feature positions in the target image. It provides a high-precision, highly consistent input image for subsequent defect detection, thereby improving the accuracy of grain defect detection and system reliability.

[0060] This invention also provides a defect detection device. Figure 3 shows a schematic block diagram of a defect detection device 300 according to an embodiment of the present invention. As shown in Figure 3, the detection device 300 may include a target image acquisition module 310 and a defect determination module 320.

[0061] The target image acquisition module 310 can be used to acquire a target image including the target grains.

[0062] The defect determination module 320 can be used to determine the target defect type of the target grain and the target defect region in the target image corresponding to the defective part in the target grain through the trained defect detection model. The trained defect detection model is trained based on multiple training images including training dies, the training defect region corresponding to the defective portion of the training die in each training image, and the training defect type corresponding to each training image. The trained defect detection model includes multiple sequentially connected type detection pre-training modules and multiple trained first projection modules. The multiple sequentially connected type detection pre-training modules are used to determine the target defect type, which represents the defect type of the defect present in the target die. The multiple trained first projection modules are used to determine the target defect region. For each type detection pre-training module, the output of the type detection pre-training module is connected to the input of a trained first projection module. The input of the type detection pre-training module whose input is connected to the output of the type detection pre-training module is connected to the output of the trained first projection module. Furthermore, the trained first projection module is different from the trained first projection modules whose input is connected to the output of other type detection pre-training modules.

[0063] For example, the trained defect detection model also includes a pre-trained defect type classification module. The input of the pre-trained defect type classification module is connected to the output of the trained second projection module. The input of the trained second projection module is connected to the output of the last type detection pre-trained module among a plurality of sequentially connected type detection pre-trained modules, and is also connected to the output of the trained first projection module, which is connected to both the input and the output of the last type detection pre-trained module.

[0064] For example, the trained defect detection model also includes a trained feature extraction module. The trained feature extraction module is used to output the overall target features and the local target features to the first type detection pre-training module among a plurality of sequentially connected type detection pre-training modules. The overall target features are used to represent the overall visual features of the target image, and the local target features are used to represent the local visual features of the target image.

[0065] For example, the trained defect detection model also includes a pre-trained feature extraction module. The output of the pre-trained feature extraction module is connected to the trained third projection module. The trained third projection module is used to output the overall target features and local target features to the first type detection pre-training module among a plurality of sequentially connected type detection pre-training modules. The overall target features are used to represent the overall visual features of the target image, and the local target features are used to represent the local visual features of the target image.

[0066] For example, the trained defect detection model also includes a trained region segmentation and classification module, which is connected to multiple trained first projection modules. The trained region segmentation and classification module is used to determine the target defect region based on the average features of the output features of the multiple trained first projection modules.

[0067] For example, the detection device further includes: an original image acquisition module and a correction processing module.

[0068] The raw image acquisition module is used to acquire a raw image including the target grain. The correction processing module is used to perform image correction processing on the raw image based on a reference image including a reference grain to obtain the target image, wherein the reference grain is a grain without defects.

[0069] According to another aspect of the present invention, an electronic device is also provided. FIG4 shows a schematic block diagram of an electronic device 400 according to an embodiment of the present invention. As shown in FIG4, the electronic device 400 includes a processor 410 and a memory 420, wherein the memory 420 stores a computer program, and the computer program instructions are executed by the processor 410 to perform the above-described defect detection method.

[0070] Furthermore, according to another aspect of the present invention, a storage medium is also provided, on which program instructions are stored. When the program instructions are executed by a computer or processor, the computer or processor performs corresponding steps of the defect detection method described above in the embodiments of the present invention, and is used to implement corresponding modules in the defect detection apparatus or electronic device described above in the embodiments of the present invention. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media.

[0071] According to another aspect of the present invention, a computer program product is also provided, comprising computer program instructions, which, when executed by a computer or processor, cause the computer or processor to perform corresponding steps of the defect detection method described above.

[0072] Those skilled in the art can understand the specific implementation schemes of the above-mentioned electronic devices and storage media by reading the relevant descriptions of the defect detection methods. For the sake of brevity, they will not be described in detail here.

[0073] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0074] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0075] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0076] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0077] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention. However, this approach should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0078] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0079] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0080] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the defect detection apparatus according to embodiments of the present invention. The present invention can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0081] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0082] The above description is merely a specific embodiment of the present invention or an explanation of that embodiment. The scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting defects, characterized in that, The method includes: acquiring a target image including a target grain; determining the target defect type of the target grain and the target defect region corresponding to the defective portion of the target grain in the target image using a trained defect detection model; wherein, the trained defect detection model is trained based on multiple training images including training grains, the training defect region corresponding to the defective portion of the training grain in each training image, and the training defect type corresponding to each training image; the trained defect detection model includes multiple sequentially connected type detection pre-training modules and multiple trained first projection modules, the multiple sequentially connected type detection pre-training modules being used for... The target defect type is determined, which represents the type of defect present in the target grain. The plurality of trained first projection modules are used to determine the target defect region. For each of the plurality of sequentially connected type detection pre-training modules, the output of the type detection pre-training module is connected to the input of a trained first projection module. The input of the type detection pre-training module whose input is connected to the output of the type detection pre-training module is connected to the output of the trained first projection module. Furthermore, the trained first projection module is different from the trained first projection modules whose inputs are connected to the outputs of other type detection pre-training modules.

2. The method as described in claim 1, characterized in that, The trained defect detection model also includes a pre-trained defect type classification module. The input of the pre-trained defect type classification module is connected to the output of the trained second projection module. The input of the trained second projection module is connected to the output of the last type detection pre-training module among the multiple sequentially connected type detection pre-training modules, and is also connected to the output of the trained first projection module, which is connected to both the input and the output of the last type detection pre-training module.

3. The method as described in claim 2, characterized in that, The trained defect detection model also includes a trained feature extraction module. The trained feature extraction module is used to output the overall target features and the local target features to the first type detection pre-training module among the multiple sequentially connected type detection pre-training modules. The overall target features are used to represent the overall visual features of the target image, and the local target features are used to represent the local visual features of the target image.

4. The method as described in claim 2, characterized in that, The trained defect detection model also includes a pre-trained feature extraction module. The output of the pre-trained feature extraction module is connected to the trained third projection module. The trained third projection module is used to output the overall target features and local target features to the first type detection pre-training module among the multiple sequentially connected type detection pre-training modules. The overall target features are used to represent the overall visual features of the target image, and the local target features are used to represent the local visual features of the target image.

5. The method as described in claim 1, characterized in that, The trained defect detection model also includes a trained region segmentation and classification module, which is connected to multiple trained first projection modules. The trained region segmentation and classification module is used to determine the target defect region based on the average feature of the output features of the multiple trained first projection modules.

6. The method as described in claim 1, characterized in that, The method further includes: acquiring an original image including a target grain; and performing image correction processing on the original image based on a reference image including a reference grain to obtain the target image, wherein the reference grain is a grain without defects.

7. A defect detection device, characterized in that, The device includes: a target image acquisition module, used to acquire a target image including a target grain; and a defect determination module, used to determine, through a trained defect detection model, the target defect type of the target grain and the target defect region corresponding to the defective portion of the target grain in the target image; wherein, the trained defect detection model is trained based on multiple training images including training grains, the training defect region corresponding to the defective portion of the training grain in each training image, and the training defect type corresponding to each training image; the trained defect detection model includes multiple sequentially connected type detection pre-training modules and multiple trained first projection modules, the multiple sequentially connected... The type detection pre-training module is used to determine the target defect type, which represents the defect type present in the target grain. The plurality of trained first projection modules are used to determine the target defect region. For each of the plurality of sequentially connected type detection pre-training modules, the output of the type detection pre-training module is connected to the input of a trained first projection module. The input of the type detection pre-training module whose input is connected to the output of the type detection pre-training module is connected to the output of the trained first projection module. Furthermore, the trained first projection module is different from the trained first projection modules whose inputs are connected to the outputs of other type detection pre-training modules.

8. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program to implement the defect detection method as described in any one of claims 1-6.

9. A storage medium storing computer program instructions, characterized in that, The computer program instructions, when executed, are used to perform the defect detection method as described in any one of claims 1-6.

10. A computer program product comprising computer program instructions, characterized in that, The computer program instructions, when executed by a processor, are used to perform the defect detection method as described in any one of claims 1-6.

Images