A defect detection method, a model training method, and related devices

Abstract

This application provides a defect detection method, a model training method, and related apparatus. The method involves acquiring a test image and a reference image of the same type, where the reference image does not contain defects. A differential defect detection model is configured to perform difference comparison and defect analysis based on the test image and the reference image. The test image and the reference image are input into the differential defect detection model to obtain the detection result of the test image. The detection result is used to identify whether defects exist in the test image and to annotate the areas where defects are located. Thus, by using a trained differential defect detection model, and inputting the test image and the reference image, the model can deeply analyze the differences between the two images, accurately identify which differences are caused by defects, thereby achieving accurate identification of defects in the test image and improving defect detection accuracy.

Description

Technical Field

[0001] This application relates to the field of defect detection, and in particular to a defect detection method, a model training method, and related apparatus. Background Technology

[0002] In industrial production, product defect detection is a crucial step in ensuring product quality. As industrial products develop towards higher precision and larger formats, traditional defect detection methods are increasingly revealing numerous problems. Existing defect detection methods have low accuracy, failing to accurately identify defects in the images being tested, and are prone to false positives and false negatives. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a defect detection method, a model training method, and related apparatus, which can achieve accurate identification of defects in the image under test and improve the defect detection accuracy. The specific solution is as follows: On the one hand, this application provides a defect detection method, including: Acquire the image to be tested and a reference image of the same type as the image to be tested, wherein the reference image is free of defects; The image to be tested and the reference image are input into the differential defect detection model to obtain the detection result of the image to be tested. The detection result is used to identify whether there is a defect in the image to be tested and to annotate the area where the defect is located. The differential defect detection model is configured to perform difference comparison and defect analysis based on the image to be tested and the reference image.

[0004] In one possible implementation, when both the image to be tested and the reference image have a first image size, and the first image size is larger than the image segmentation size, the step of inputting the image to be tested and the reference image into the differential defect detection model to obtain the detection result of the image to be tested includes: The image to be tested and the reference image are segmented based on a preset image size to obtain multiple sub-images of the image to be tested and multiple reference sub-images of the reference image. The plurality of sub-images to be tested and the plurality of reference sub-images are input into the differential defect detection model, and the detection sub-results of each sub-image to be tested are output. The detection sub-results are used to identify whether there is a defect in the sub-image to be tested, and to annotate the area where the defect is located. Based on the positional distribution of each of the sub-images to be tested in the image to be tested, the defective images of multiple detection sub-results with defects are stitched together to obtain the detection result of the image to be tested.

[0005] In one possible implementation, the differential defect detection model includes a feature segmentation network and a feature alignment network. The step of inputting the image to be tested and the reference image into the differential defect detection model to obtain the detection result of the image to be tested includes: The feature alignment network is used to extract the test image features of the test image and the reference image features of the reference image. The feature alignment network determines the feature transformation parameters between the features of the image to be tested and the features of the reference image. Based on the feature transformation parameters, the image under test is geometrically corrected through the feature alignment network to obtain the corrected image under test; the texture misalignment in the corrected image under test is less than the misalignment threshold. The corrected image to be tested and the reference image are input into the feature alignment network to obtain the detection result of the image to be tested.

[0006] In one possible implementation, the type of the image to be tested is the type of the object to which the image to be tested is located, and the type of object includes flexible cable, fabric, metal or glass.

[0007] Furthermore, this application also provides a model training method, including: Obtain a positive sample set of template images, which includes multiple template images and the template images are free of defects; The initial model is pre-trained based on the positive sample set of the template images to obtain a pre-trained model. The template image positive sample set is input into the defect generation model to obtain the first image negative sample set. The first image negative sample set includes multiple first images, which are obtained by adding defects to the template image. The pre-trained model is fine-tuned and trained based on the first negative image sample set to obtain the differential defect detection model.

[0008] In one possible implementation, the step of fine-tuning the pre-trained model based on the first image negative sample set to obtain the differential defect detection model includes: An enhanced training sample set is constructed based on the first negative image sample set and the real negative image sample set, wherein the real negative image sample set includes multiple defective real images collected. The pre-trained model is fine-tuned using the enhanced training sample set to obtain the differential defect detection model.

[0009] In one possible implementation, the step of fine-tuning the pre-trained model using the enhanced training sample set to obtain the differential defect detection model includes: The pre-trained model is fine-tuned using the augmented training sample set and the labeled sample set to obtain the differential defect detection model. The labeled sample set includes multiple sample pairs with labels. Each sample pair includes a sample image from the augmented training sample set and a template image. The label is a defect image of a defect in the sample image.

[0010] In one possible implementation, the step of fine-tuning the pre-trained model using the enhanced training sample set and the labeled sample set to obtain the differential defect detection model includes: The pre-trained model is fine-tuned using the enhanced training sample set and the labeled sample set to obtain a first model; the first model includes a feature segmentation network. A feature alignment network is added to the first model to obtain the second model; The labeled sample set is input into the feature alignment network in the second model to obtain a corrected labeled sample set. The corrected labeled sample set includes multiple corrected sample pairs with the label. Each corrected sample pair includes a geometrically corrected sample image and the template image. Based on the calibration labeled sample set, the feature segmentation network in the second model is trained to obtain the differential defect detection model.

[0011] In one possible implementation, the method further includes: Obtain multiple raw template images; The window size is determined based on the texture density of the original template image; Based on the window size, the original template image is segmented to obtain multiple template images; The template image positive sample set is constructed based on multiple template images.

[0012] In one possible implementation, the window size and the texture density are negatively correlated.

[0013] In one possible implementation, inputting the labeled sample set into the feature alignment network in the second model to obtain the corrected labeled sample set includes: The feature alignment network is used to extract the sample image features of the sample image and the template image features of the template image. The feature alignment network determines the feature transformation parameters between the sample image features and the template image features. Based on the feature transformation parameters, the sample image is geometrically corrected using the feature alignment network to obtain the geometrically corrected sample image. Based on the geometrically corrected sample image and the template image, the corrected annotation sample set is constructed.

[0014] In one possible implementation, the defect generation model and the pre-trained model are the same model.

[0015] In another aspect, this application also provides a defect detection device, comprising: The first acquisition unit is used to acquire the image to be tested and a reference image of the same type as the image to be tested, wherein the reference image does not have defects; The detection unit is used to input the image to be tested and the reference image into the differential defect detection model to obtain the detection result of the image to be tested. The detection result is used to identify whether there is a defect in the image to be tested and to annotate the area where the defect is located. The differential defect detection model is configured to perform difference comparison and defect analysis based on the image to be tested and the reference image.

[0016] In another aspect, embodiments of this application provide a computer-readable storage medium for storing a computer program for performing the methods described above.

[0017] This application provides a defect detection method, a model training method, and related apparatus. The method involves acquiring a test image and a reference image of the same type, where the reference image does not contain defects. A differential defect detection model is configured to perform difference comparison and defect analysis based on the test image and the reference image. The test image and the reference image are input into the differential defect detection model to obtain the detection result of the test image. The detection result is used to identify whether defects exist in the test image and to annotate the areas where defects are located. Thus, by using a trained differential defect detection model, inputting the test image and the reference image, the model can deeply analyze the differences between the two images, accurately identify which differences are caused by defects, thereby achieving accurate identification of defects in the test image and improving defect detection accuracy. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a defect detection method provided in an embodiment of this application is shown; Figure 2 The diagram illustrates a defect detection method provided in an embodiment of this application. Figure 3 A schematic flowchart of a model training method provided in an embodiment of this application is shown; Figure 4 This illustration shows a schematic diagram related to geometric correction of an image under test according to an embodiment of this application; Figure 5 A schematic diagram of a geometric correction provided in an embodiment of this application is shown; Figure 6 This paper presents a schematic diagram of the overall process of a model training method provided in an embodiment of this application. Figure 7 A schematic diagram of a defect detection device provided in an embodiment of this application is shown; Figure 8 A schematic diagram of a model training apparatus provided in an embodiment of this application is shown; Figure 9 This is a structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0020] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0022] The defect detection method or model training method provided in this application can be implemented using computer equipment, which can be a terminal device or a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Terminal devices include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, and vehicle terminals. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations in this regard.

[0023] For ease of understanding, the following detailed description, in conjunction with the accompanying drawings, provides a defect detection method, a model training method, and related apparatus provided in the embodiments of this application.

[0024] refer to Figure 1The diagram shown is a flowchart of a defect detection method provided in an embodiment of this application. The method may include the following steps.

[0025] S101, acquire the image to be tested and a reference image of the same type as the image to be tested, wherein the reference image does not have defects.

[0026] The image to be tested refers to the image that needs to be inspected for defects. The image to be tested is acquired by an image acquisition device, such as a photodetector. Specifically, the image to be tested is an image of the object to be tested, but this application does not make specific limitations on the image to be tested or the object to be tested.

[0027] A reference image is an image used as a reference for the image under test, and it is free of defects. Since different types of images typically have different textures, the reference image and the image under test belong to the same type, so that the reference image can provide the image characteristics of a defect-free image of that type for the image under test. As an example, both the reference image and the image under test may be images captured for a flexible flat cable, or both may be images captured for glass.

[0028] In one possible implementation, the type of the image to be tested is the type of the object to which the image to be tested is located, and the object type includes flexible cable, fabric, metal or glass.

[0029] In other words, the image to be tested can be an image taken of a flexible flat cable, fabric, metal, or glass. The image may or may not contain defects, which needs to be determined subsequently based on a differential defect detection model. For example, the image to be tested may contain defects such as missing wires in the flexible flat cable, broken yarns in the fabric, scratches on the metal, or bubbles in the glass. In this case, the reference image could be, for example, a flexible flat cable, fabric, metal, or glass without defects.

[0030] In other words, this application can detect images of various types of items, such as fabrics, packaging patterns, etc., and metals, flexible cables, and glass, such as film structures in semiconductor devices, thereby broadening the applicable scenarios of defect detection methods and meeting the detection needs of various items.

[0031] S102, the image to be tested and the reference image are input into the differential defect detection model to obtain the detection result of the image to be tested. The detection result is used to identify whether there is a defect in the image to be tested, and to annotate the area where the defect is located. The differential defect detection model is configured to perform difference comparison and defect analysis based on the image to be tested and the reference image.

[0032] The differential defect detection model is a pre-trained neural network model. This model primarily compares the differences between a pair of images (the image to be tested and a reference image) and deeply analyzes whether the differences are caused by defects or by different textures in different images of the same type, thus achieving accurate defect detection. In other words, the differential defect detection model can be used in image segmentation tasks to identify defective regions using pixel-level annotations.

[0033] During the model application phase, both the test image and the reference image of the same type can be input into the differential defect detection model to obtain the detection result for the test image. The detection result can show whether there is a defect in the test image, and if so, the defective area in the test image is marked. As an example, the detection result can be a binary map of the defect.

[0034] refer to Figure 2 The diagram shown is a schematic representation of a defect detection method provided in an embodiment of this application. Figure 2 (a) in the image is the image to be tested, which contains a texture pattern. Figure 2 (b) in the image is a reference image without defects. Figure 2 (c) in the image represents the detection result, specifically a black and white binary image, where the white portion represents the defect.

[0035] In this way, by inputting the image to be tested and the reference image as a reference into the trained differential defect detection model, the model can deeply analyze the differences between the two images and accurately identify which differences are caused by defects, thereby achieving accurate identification of defects in the image to be tested and improving the defect detection accuracy.

[0036] Because the image size of the acquired image to be tested may be too large, and the amount of data in the image is huge, the processing speed of the model is slow, and the problem of missing edge differences is easy to occur. In order to improve the processing speed of the model, in one possible implementation, when the image size of both the image to be tested and the reference image is a first image size, and the first image size is larger than the image segmentation size, in step S102, the image to be tested and the reference image are input into the differential defect detection model to obtain the detection result of the image to be tested, which may include steps S1021-S1023.

[0037] S1021, the image to be tested and the reference image are segmented according to the preset image size to obtain multiple sub-images of the image to be tested and multiple reference sub-images of the reference image.

[0038] When the image to be tested and the reference image have the same size, and both are at the first image size, if the first image size is larger than the image segmentation size, it indicates that the first image size is too large, and the image needs to be segmented before detection. The image segmentation size is the minimum size requirement for image segmentation and can be set based on actual needs.

[0039] The preset image size is the image size set beforehand for image segmentation, for example, 512x512. The test image can be segmented according to the preset image size to obtain multiple test sub-images, and a reference image can be segmented according to the preset image size to obtain multiple reference sub-images. Here, the test sub-image refers to a portion of the test image, and the reference sub-image refers to a portion of the reference image. Both the test sub-image and the reference sub-images have the preset image size, and the number of images is equal.

[0040] S1022, input multiple sub-images to be tested and multiple reference sub-images into the differential defect detection model, and output the detection sub-results of each sub-image to be tested. The detection sub-results are used to identify whether there are defects in the sub-images to be tested, and to annotate the areas where defects are located.

[0041] Specifically, all segmented sub-images to be tested and all reference sub-images can be input into the differential defect detection model to obtain the detection sub-results corresponding to each sub-image to be tested. As an example, the sub-images to be tested and the reference sub-images can be input as image pairs, and within an image pair, the position of the sub-image to be tested in the image to be tested is the same as the position of the reference sub-image in the image to be tested; for example, both are images located in the upper left corner. Of course, their positions in the images can also be different, and this application does not limit this.

[0042] A detection sub-result refers to the result obtained by detecting a single sub-image to be tested. The aforementioned detection results include multiple detection sub-results. The detection sub-results can reflect whether defects exist in the sub-image to be tested, and if so, mark the area where the defects are located. Furthermore, to avoid defects being truncated or undetectable due to image segmentation, there can be a 10% - 20% overlap between the sub-images to be tested.

[0043] S1023, based on the positional distribution of each sub-image to be tested in the image to be tested, the multiple detection sub-results with defects are stitched together to obtain the detection result of the image to be tested.

[0044] When obtaining the detection sub-results of each sub-image to be tested, the multiple detection sub-results with defects can be stitched together according to their position distribution within the image to be tested. For example, the binary image of the defect can be stitched together according to the image to be tested, thereby obtaining the entire detection result corresponding to the image to be tested, ensuring the integrity of the defective region in the image to be tested. As an example, a weighted stitching algorithm can be used to implement image stitching.

[0045] In summary, by first segmenting the image to be tested and the reference image into smaller sub-images, and then inputting them into the differential defect detection model for defect detection, the model's processing speed for small-sized images can be improved as much as possible while ensuring detection accuracy, thereby improving defect detection efficiency.

[0046] During image capture, the image under test is prone to texture misalignment due to camera shake. Traditional defect detection methods cannot effectively eliminate this shake effect, resulting in a high false detection rate and severely impacting detection accuracy. To make defect detection more adaptable to this shake-prone scenario, one possible implementation includes a differential defect detection model comprising a feature segmentation network and a feature alignment network. In step S102, the image under test and a reference image are input into the differential defect detection model to obtain the detection result of the image under test, which may include steps S1024-S1027.

[0047] S1024, extract the test image features of the test image and the reference image features of the reference image through the feature alignment network.

[0048] The differential defect detection model can include a feature segmentation network and a feature alignment network. The feature segmentation network is mainly used for defect detection, while the feature alignment network is mainly used to eliminate texture misalignment in the test image before defect detection, so that the misaligned texture in the test image is aligned with the non-misaligned texture in the reference image.

[0049] Specifically, the feature alignment network first extracts features from the test image to obtain test image features, and then extracts features from the reference image to obtain reference image features. The test image features are used to represent the image characteristics of the test image, and the reference image features are used to represent the image characteristics of the reference image. As an example, image features can be understood as key point features, such as texture intersections, edge corners, etc.

[0050] S1025, the feature transformation parameters between the features of the image to be tested and the features of the reference image are determined by the feature alignment network.

[0051] The feature alignment network can also determine the feature transformation parameters between the features of the test image and the reference image. These feature transformation parameters are used to perform alignment transformations between the test image with texture misalignment and the reference image without texture misalignment, thereby mitigating or even completely eliminating texture misalignment in the test image. For example, feature transformation parameters can include translation parameters, rotation parameters, scaling parameters, etc., and specifically, can be affine transformation matrices.

[0052] S1026, Based on the feature transformation parameters, the feature alignment network performs geometric correction on the image under test to obtain the corrected image under test; the texture misalignment in the corrected image under test is less than the misalignment threshold.

[0053] Feature alignment networks can perform geometric correction on the test image based on feature transformation parameters. In other words, they align the image with a reference image free of texture misalignment, mitigating or even completely eliminating texture misalignment caused by jitter, resulting in a corrected test image. The corrected test image has minimal texture misalignment, meeting the elimination requirement. The misalignment threshold refers to the maximum permissible degree of texture misalignment in the test image, for example, one pixel.

[0054] S1027, The corrected image to be tested and the reference image are input into the feature alignment network to obtain the detection result of the image to be tested.

[0055] Since there is almost no texture misalignment in the corrected test image, it is possible to avoid mistaking texture misalignment for defects. The corrected test image and reference image are input into the feature alignment network to obtain the detection result of the test image.

[0056] In summary, the feature alignment network can basically eliminate texture misalignment in the image under test, making the model still applicable to shaky shooting scenarios and further improving the accuracy of defect detection.

[0057] Next, we will mainly explain the model training process of the differential defect detection model.

[0058] Traditional defect detection algorithms rely heavily on a large number of labeled defect samples for model training. However, in real-world industrial scenarios, defect samples are often scarce, and the labeling process is time-consuming and labor-intensive, resulting in high model training costs and long cycles, making it difficult to meet the demands of efficient production. Taking the inspection of flexible flat cable products as an example, their surface texture is complex, and traditional algorithms need to collect hundreds or even thousands of labeled defect samples to ensure basic detection accuracy, resulting in an enormous workload for labeling.

[0059] Meanwhile, for the detection of large-format textured products (such as large-format fabrics, glass, and flexible cables), traditional algorithms often use whole-image input processing. Due to the massive amount of data in large images, the processing speed is slow, and edge difference detection is prone to being missed. Furthermore, during the shooting process, the image under test is prone to texture misalignment due to device shake. Traditional differential algorithms cannot effectively eliminate this shake effect, resulting in a high false difference false detection rate, seriously affecting detection accuracy. Therefore, this application provides a differential defect detection model that can reduce dependence on labeled samples, improve the efficiency of large-format image processing, eliminate the effect of shake, and ensure detection accuracy.

[0060] refer to Figure 3 The diagram shown is a flowchart of a model training method provided in an embodiment of this application, which includes steps S301-S305.

[0061] S301, Obtain a positive sample set of template images. The positive sample set of template images includes multiple template images, and the template images are not defective.

[0062] The positive sample set of template images contains a large number of template images with defects. For example, the template images can be defect-free images of products such as flat cables, fabrics, metals, or glass. The positive sample set of template images can include various types of template images.

[0063] To improve the processing speed of the model for larger images, in one possible implementation, the method may further include: acquiring multiple original template images; determining the window size based on the texture density of the original template images; performing image segmentation on the original template images based on the window size to obtain multiple template images; and constructing a positive sample set of template images based on the multiple template images.

[0064] The original template image refers to the original template image directly acquired by the image acquisition device. Its image size is usually large. In order to improve the processing speed of the model, it can be segmented and the template image with smaller image size can be used for model training. This allows the smaller test sub-image and reference sub-image to be used directly for defect detection during the model application stage.

[0065] Specifically, an adaptive sliding window segmentation strategy can be used to process the original template image. First, the texture density of the original template image is determined; a higher texture density indicates a larger amount of data in the image. Based on the texture density, the corresponding window size is determined, and the window is slid across the original template image according to this size to perform image segmentation, resulting in multiple smaller template images, thus forming a positive sample set of template images. Furthermore, 10%-20% overlap can be retained at the edges of the sliding window to prevent differences from being truncated during segmentation.

[0066] In summary, by first segmenting the larger original template image and then using the smaller template image for subsequent model training, the large image processing performance of the model is optimized. The large image processing speed is 3 times faster than the whole image input, and the overlapping area design avoids missed detection of edge differences, making it suitable for large-format industrial product inspection.

[0067] To achieve adaptive segmentation of large-sized original template images, one possible implementation is that the window size and texture density are negatively correlated.

[0068] Specifically, when determining the texture density of the original template image, the texture density can be defined as the texture density of a portion of the original template image. If the texture density is large, it indicates that the texture in that region is relatively dense, making it a texture-dense region (e.g., a dense area of ​​flat cable lines). Since the data volume in this region is large, the window size can be reduced, for example, to 512×512. This ensures that a smaller template image is obtained from the dense texture region, avoiding the dense texture reducing the model's data processing speed.

[0069] If the texture density is low, it means that the texture in this area is relatively sparse, which is a sparse texture region. In this case, the window size can be increased, for example to 1024×1024, so as to minimize the number of template images obtained from segmentation and improve the data processing speed of the model.

[0070] S302, the initial model is pre-trained based on the positive sample set of template images to obtain the pre-trained model.

[0071] Specifically, a set of positive samples of template images can be input into the initial model for pre-training to obtain a pre-trained model. The initial model can be, for example, a deep learning model capable of feature extraction, such as a Residual Network (ResNet), the YOLO (You Only Look Once) convolutional neural network backbone, a recurrent neural network (RNN), or a graph neural network (GNN), etc., without specific limitations. YOLO is a deep learning model used for real-time object detection, and the backbone is a fundamental part of the convolutional neural network responsible for extracting features from input data (such as images). This application pre-trains the initial model, enabling it to learn the inherent feature distributions of different texture products, including the periodicity, directionality, and grayscale variation patterns of the texture, thus completing the foundational construction of differential feature transfer. In short, the pre-trained model has the ability to extract texture features from images and has learned the texture characteristics of various types of images.

[0072] S303, input the template image positive sample set into the defect generation model to obtain the first image negative sample set. The first image negative sample set includes multiple first images, which are obtained by adding defects to the template image.

[0073] A defect generation model is a model used to generate images with defects. A positive sample set of template images is input into the defect generation model, allowing it to generate a first image with defects based on the template images, thus constructing a negative sample set for the first image. In this way, using template images to generate the first image with defects makes the defects in the first image more consistent with the texture and background characteristics of the template image. Virtual defect types can cover fabric yarn breaks, metal scratches, glass bubbles, and broken flexible cable traces, etc., and the generated defects blend naturally with the background texture in terms of shape, size, and grayscale, improving the realism of the first image.

[0074] In one possible implementation, the defect generation model and the pre-trained model are the same model. That is, when the pre-trained model generates the first image with defects, it can utilize the texture characteristics that have been learned, so that the generated defects are more in line with the texture background characteristics, thereby further improving the realism of the defects.

[0075] Of course, in other implementation methods, existing software scripts can be used to obtain a defective first image by executing function operations.

[0076] S304. Based on the negative sample set of the first image, the pre-trained model is fine-tuned to obtain the differential defect detection model.

[0077] Specifically, the pre-trained model can be fine-tuned using the negative sample set of the first image, resulting in a differential defect detection model. During training, the model can mine potential defect patterns from the sample set, thereby achieving defect detection.

[0078] In one possible implementation, S304, the pre-trained model is fine-tuned and trained based on the first image negative sample set to obtain a differential defect detection model, which may specifically include S3041-S3042.

[0079] S3041, construct an enhanced training sample set based on the first image negative sample set and the real image negative sample set. The real image negative sample set includes multiple defective real images collected.

[0080] Specifically, this application also introduces an adversarial training mechanism, which mixes the generated virtual defect samples (i.e., the first image negative sample set) with a small number of real defect samples (i.e., the real image negative sample set) to construct an enhanced training sample set.

[0081] S3042, by enhancing the training sample set, the pre-trained model is fine-tuned to obtain the differential defect detection model.

[0082] In this way, the enhanced training sample set includes both negative samples with real defects and falsely generated defective negative samples. Fine-tuning the training based on these samples can further improve the training accuracy.

[0083] In one possible implementation, S3042, the pre-trained model is fine-tuned by enhancing the training sample set to obtain the differential defect detection model. Specifically, the pre-trained model is fine-tuned by enhancing the training sample set and the labeled sample set to obtain the differential defect detection model. The labeled sample set includes multiple sample pairs with labels. Each sample pair includes a sample image from the enhanced training sample set and a template image. The label is a defect image with a defect in the sample image.

[0084] Specifically, the labeled sample set is a set of samples that have already been labeled. All sample pairs in this sample set have labels. A sample pair includes a sample image from the augmented training sample set and a template image without defects. The label of the sample pair is, for example, a binary image of the defects in the sample image.

[0085] The enhanced training sample set and the labeled sample set can be input into the pre-trained model for model fine-tuning to obtain the differential defect detection model. This enables the differential defect detection model to quickly grasp the discrimination boundary between "normal texture and abnormal difference" and complete the optimization of the model's core parameters.

[0086] In summary, by first pre-training with defect-free template images and then fine-tuning the model using first defective images generated from the template images, the model's dependence on labeled samples is reduced. Through the combination of positive sample pre-training and fine-tuning with a small number of labeled samples, the number of labeled samples is reduced to 1 / 10 of traditional methods, while still maintaining a difference detection accuracy of over 95%, significantly reducing labeling costs and training time. Furthermore, the model's ability to identify minute defects is improved; the generated virtual defect samples are highly similar to real defect samples. Combined with adversarial training, the recall rate for identifying minute defects with an area <0.5mm² is increased by 15%, meeting the requirements for high-precision detection.

[0087] To optimize the model's adaptability to jittery scenarios, in one possible implementation, S305, the pre-trained model is fine-tuned by enhancing the training sample set and the labeled sample set to obtain a differential defect detection model, which may include S3051-S3054.

[0088] S3051, by enhancing the training sample set and the labeled sample set, the pre-trained model is fine-tuned to obtain the first model; the first model includes a feature segmentation network.

[0089] In other words, the model can be fine-tuned a second time to adapt it to shaking scenarios. The model obtained after fine-tuning the pre-trained model by enhancing the training sample set and the labeled sample set is denoted as the first model. The first model includes a feature segmentation network, which is mainly used to achieve defect detection.

[0090] As an example, the first model can employ an improved U-Net architecture, a convolutional neural network architecture comprising an encoder and a decoder. The encoder incorporates depthwise separable convolutions and an attention mechanism, while the decoder fuses multi-scale features through skip connections, outputting a difference mask (i.e., an image of the defect region). The improved U-Net architecture has only one-third the number of parameters of traditional segmentation networks, achieves twice the inference speed, and keeps the pixel error at the edge of the difference region within one pixel, balancing real-time performance and accuracy.

[0091] S3052, a feature alignment network is added to the first model to obtain the second model.

[0092] The feature alignment network is embedded into the first model obtained after fine-tuning to obtain the second model.

[0093] S3053, the labeled sample set is input into the feature alignment network in the second model to obtain the corrected labeled sample set. The corrected labeled sample set includes multiple corrected sample pairs with labels. The corrected sample pairs include geometrically corrected sample images and template images.

[0094] The sample images in the labeled sample set are geometrically corrected by a feature alignment network to obtain the corrected sample images, and thus obtain the corrected labeled sample set.

[0095] S3054, based on the calibration labeled sample set, the feature segmentation network in the second model is trained to obtain the differential defect detection model.

[0096] Using the calibration labeled sample set, the feature segmentation network in the second model is trained, thus obtaining the differential defect detection model.

[0097] In summary, the feature alignment network in the model can eliminate texture misalignment caused by shooting shake through geometric correction, avoid misdetecting it as a defect, reduce the false detection rate to below 3%, and improve detection accuracy.

[0098] As an example, refer to Figure 4The diagram shown is a schematic representation of geometric correction of an image under test according to an embodiment of this application. Figure 4 In the image (a), the template image does not have texture misalignment. Figure 4 (b) in the image is a sample image with texture misalignment. Figure 4 (c) in the image is the corrected sample image after geometric correction, which significantly reduces texture misalignment.

[0099] To ensure the accuracy of geometric correction, in one possible implementation, S3053, the labeled sample set is input into the feature alignment network in the second model to obtain the corrected labeled sample set, which may include S30531-S30534.

[0100] S30531 extracts sample image features of sample images and template image features of template images through a feature alignment network.

[0101] The feature alignment network is used to extract sample image features from the sample image and template image features from the template image. Sample image features can reflect the key point features in the sample image, and template image features can reflect the key point features in the template image, such as texture intersections, edge corners, etc.

[0102] S30532 determines the feature transformation parameters between sample image features and template image features through a feature alignment network.

[0103] The feature transformation parameters between the sample image features and the template image features are calculated using a feature alignment network. The specific details of the feature transformation parameters will not be explained here; please refer to the relevant records in the model application stage mentioned above.

[0104] S30533, based on feature transformation parameters, performs geometric correction on the sample image through a feature alignment network to obtain a geometrically corrected sample image.

[0105] Based on the feature transformation parameters, a feature alignment network can be used to perform geometric correction on sample images with texture misalignment to obtain geometrically corrected sample images.

[0106] S30534, Based on the geometrically corrected sample images and template images, construct a correction annotation sample set.

[0107] As an example, refer to Figure 5 The diagram shown is a schematic of geometric correction provided in an embodiment of this application. The sample image and the template image are input into the feature alignment network to obtain feature transformation parameters (i.e., geometric parameters). The feature alignment network performs geometric correction based on the feature transformation parameters to obtain the corrected sample image and template image.

[0108] In summary, by calculating the feature transformation parameters between sample images with texture misalignment and template images without texture misalignment, and then performing misalignment elimination based on these parameters, the accuracy of misalignment elimination can be improved, and geometric correction can be made more precise.

[0109] Next, combined Figure 6 This application provides an overall description of a model training method provided in its embodiments. Figure 6 This is a schematic diagram illustrating the overall process of a model training method provided in this application embodiment. The acquired original template image is segmented based on the window size to obtain a smaller template image. The template image is input into an initial model for pre-training to obtain a pre-trained model. Furthermore, the template image is input into a defect generation model (e.g., the aforementioned pre-trained model) to obtain a first image with defects, thus constructing a first image negative sample set. The first image negative sample set and the real image negative sample set are mixed to construct an enhanced training sample set. The enhanced training sample set and the labeled sample set are input into the pre-trained model for the first fine-tuning training, thereby obtaining a first model, which includes a feature segmentation network. A feature alignment network is embedded in the first model to obtain a second model. To adapt to jitter scenarios, the sample image and template image are input into the second model, and the feature alignment network is used to obtain geometrically corrected sample images and template images. Then, the feature segmentation network is used to perform a second fine-tuning training using the geometrically corrected sample images and template images, resulting in a differential defect detection model.

[0110] In this way, during the model application stage, the image to be tested and the reference image can be acquired, transformed to a specific image size, and then input into the differential defect detection model, so that defects in the image to be tested can be detected, such as pixel-level labeled images that identify defective areas.

[0111] refer to Figure 7 As shown, Figure 7 This illustration shows a schematic diagram of a defect detection device provided in an embodiment of this application. The defect detection device includes: The first acquisition unit 701 is used to acquire the image to be tested and a reference image of the same type as the image to be tested, wherein the reference image does not have defects.

[0112] The detection unit 702 is used to input the image to be tested and the reference image into the differential defect detection model to obtain the detection result of the image to be tested. The detection result is used to identify whether there is a defect in the image to be tested and to annotate the area where the defect is located. The differential defect detection model is configured to perform difference comparison and defect analysis based on the image to be tested and the reference image.

[0113] In one possible implementation, when the image size of both the image to be tested and the reference image is a first image size, and the first image size is larger than the image segmentation size, the detection unit is configured to: segment the image to be tested and the reference image based on a preset image size to obtain multiple sub-images to be tested of the image to be tested and multiple reference sub-images of the reference image; input the multiple sub-images to be tested and the multiple reference sub-images into a differential defect detection model, and output the detection sub-results of each sub-image to be tested, wherein the detection sub-results are used to identify whether there is a defect in the sub-image to be tested and to perform image annotation on the area where the defect is located; and based on the positional distribution of each sub-image to be tested in the image to be tested, stitch together the multiple detection sub-results with defects to obtain the detection result of the image to be tested.

[0114] In one possible implementation, the differential defect detection model includes a feature segmentation network and a feature alignment network. The detection unit is used to: extract the test image features of the test image and the reference image features of the reference image through the feature alignment network; determine the feature transformation parameters between the test image features and the reference image features through the feature alignment network; perform geometric correction on the test image based on the feature transformation parameters through the feature alignment network to obtain the corrected test image; ensure that the texture misalignment in the corrected test image is less than the misalignment threshold; and input the corrected test image and the reference image into the feature alignment network to obtain the detection result of the test image.

[0115] In one possible implementation, the type of the image to be tested is the type of the object to which the image to be tested is located, and the object type includes flexible cable, fabric, metal or glass.

[0116] refer to Figure 8 As shown, Figure 8 This illustration shows a schematic diagram of a model training apparatus provided in an embodiment of this application. The model training apparatus includes: The second acquisition unit 801 is used to acquire a template image positive sample set, which includes multiple template images, and the template images are free of defects.

[0117] The first training unit 802 is used to pre-train the initial model based on the positive sample set of template images to obtain the pre-trained model.

[0118] The determining unit 803 is used to input the template image positive sample set into the defect generation model to obtain the first image negative sample set. The first image negative sample set includes multiple first images, which are obtained by adding defects to the template image.

[0119] The second training unit 804 is used to fine-tune the pre-trained model based on the first image negative sample set to obtain the differential defect detection model.

[0120] In one possible implementation, the second training unit 804 is used to: construct an enhanced training sample set based on the first image negative sample set and the real image negative sample set, wherein the real image negative sample set includes multiple real images with defects that have been collected; and fine-tune the pre-trained model using the enhanced training sample set to obtain a differential defect detection model.

[0121] In one possible implementation, the second training unit 804 is used to: fine-tune the pre-trained model by augmenting the training sample set and the labeled sample set to obtain a differential defect detection model. The labeled sample set includes multiple sample pairs with labels. Each sample pair includes a sample image from the augmented training sample set and a template image, labeled as a defect image with a defect in the sample image.

[0122] In one possible implementation, the second training unit is used to: fine-tune the pre-trained model by enhancing the training sample set and the labeled sample set to obtain a first model; the first model includes a feature segmentation network; add a feature alignment network to the first model to obtain a second model; input the labeled sample set into the feature alignment network in the second model to obtain a corrected labeled sample set, the corrected labeled sample set including multiple corrected sample pairs with labels, the corrected sample pairs including geometrically corrected sample images and template images; and train the model in the feature segmentation network of the second model based on the corrected labeled sample set to obtain a differential defect detection model.

[0123] In one possible implementation, the apparatus further includes a construction unit for: acquiring multiple original template images; determining a window size based on the texture density of the original template images; performing image segmentation on the original template images based on the window size to obtain multiple template images; and constructing a positive sample set of template images based on the multiple template images.

[0124] In one possible implementation, window size and texture density are negatively correlated.

[0125] In one possible implementation, the second training unit is used to: extract sample image features of the sample image and template image features of the template image through a feature alignment network; determine feature transformation parameters between the sample image features and the template image features through the feature alignment network; perform geometric correction on the sample image through the feature alignment network based on the feature transformation parameters to obtain a geometrically corrected sample image; and construct a correction annotation sample set based on the geometrically corrected sample image and the template image.

[0126] In one possible implementation, the defect generation model and the pre-trained model are the same model.

[0127] In another aspect, embodiments of this application provide a computer device, with reference to Figure 9The diagram shown is a structural diagram of a computer device provided in an embodiment of this application. The computer device includes a processor 310 and a memory 320. The memory 320 is used to store program code and transfer the program code to the processor 310; The processor 310 is used to execute the method provided in the above embodiments according to the instructions in the program code.

[0128] The computer device may include a terminal device or a server, and the aforementioned apparatus may be configured in the computer device.

[0129] In another aspect, embodiments of this application also provide a storage medium for storing a computer program for executing the methods provided in the above embodiments.

[0130] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by program instructions in hardware. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk, or optical disk, etc., and other media capable of storing program code.

[0131] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0132] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0133] The above description is merely a preferred embodiment of this application. Although this application has disclosed preferred embodiments above, it is not intended to limit this application. Any person skilled in the art can make many possible variations and modifications to the technical solutions of this application using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the technical solutions of this application. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of this application without departing from the content of the technical solutions of this application shall still fall within the protection scope of the technical solutions of this application.

Claims

1. A defect detection method, characterized in that, include: Acquire the image to be tested and a reference image of the same type as the image to be tested, wherein the reference image is free of defects; The image to be tested and the reference image are input into the differential defect detection model to obtain the detection result of the image to be tested. The detection result is used to identify whether there is a defect in the image to be tested and to annotate the area where the defect is located. The differential defect detection model is configured to perform difference comparison and defect analysis based on the image to be tested and the reference image.

2. The method according to claim 1, characterized in that, When both the image to be tested and the reference image have a first image size, and the first image size is larger than the image segmentation size, the step of inputting the image to be tested and the reference image into the differential defect detection model to obtain the detection result of the image to be tested includes: The image to be tested and the reference image are segmented based on a preset image size to obtain multiple sub-images of the image to be tested and multiple reference sub-images of the reference image. The plurality of sub-images to be tested and the plurality of reference sub-images are input into the differential defect detection model, and the detection sub-results of each sub-image to be tested are output. The detection sub-results are used to identify whether there is a defect in the sub-image to be tested, and to annotate the area where the defect is located. Based on the positional distribution of each of the sub-images to be tested in the image to be tested, the defective images of multiple detection sub-results with defects are stitched together to obtain the detection result of the image to be tested.

3. The method according to claim 1, characterized in that, The differential defect detection model includes a feature segmentation network and a feature alignment network. The step of inputting the image to be tested and the reference image into the differential defect detection model to obtain the detection result of the image to be tested includes: The feature alignment network is used to extract the test image features of the test image and the reference image features of the reference image. The feature alignment network determines the feature transformation parameters between the features of the image to be tested and the features of the reference image. Based on the feature transformation parameters, the image under test is geometrically corrected through the feature alignment network to obtain the corrected image under test; the texture misalignment in the corrected image under test is less than the misalignment threshold. The corrected image to be tested and the reference image are input into the feature alignment network to obtain the detection result of the image to be tested.

4. The method according to claim 1, characterized in that, The type of the image to be tested is the type of the object to be tested, which includes flexible flat cable, fabric, metal or glass.

5. A model training method, characterized in that, include: Obtain a positive sample set of template images, which includes multiple template images and the template images are free of defects; The initial model is pre-trained based on the positive sample set of the template images to obtain a pre-trained model. The template image positive sample set is input into the defect generation model to obtain the first image negative sample set. The first image negative sample set includes multiple first images, which are obtained by adding defects to the template image. The pre-trained model is fine-tuned and trained based on the first negative image sample set to obtain the differential defect detection model.

6. The method according to claim 5, characterized in that, The step of fine-tuning the pre-trained model based on the first image negative sample set to obtain the differential defect detection model includes: An enhanced training sample set is constructed using the first negative image sample set and the real negative image sample set, wherein the real negative image sample set includes multiple defective real images collected. The pre-trained model is fine-tuned using the enhanced training sample set to obtain the differential defect detection model.

7. The method according to claim 6, characterized in that, The step of fine-tuning the pre-trained model using the enhanced training sample set to obtain the differential defect detection model includes: The pre-trained model is fine-tuned using the enhanced training sample set and the preset labeled sample set to obtain the differential defect detection model. The labeled sample set includes multiple sample pairs with labels. Each sample pair includes a sample image from the enhanced training sample set and a template image. The label is a defect image of a defect in the sample image.

8. The method according to claim 7, characterized in that, The step of fine-tuning the pre-trained model using the enhanced training sample set and the labeled sample set to obtain the differential defect detection model includes: The pre-trained model is fine-tuned using the enhanced training sample set and the labeled sample set to obtain a first model; the first model includes a feature segmentation network. A feature alignment network is added to the first model to obtain the second model; The labeled sample set is input into the feature alignment network in the second model to obtain a corrected labeled sample set. The corrected labeled sample set includes multiple corrected sample pairs with the label. Each corrected sample pair includes a geometrically corrected sample image and the template image. Based on the calibration labeled sample set, the feature segmentation network in the second model is trained to obtain the differential defect detection model.

9. The method according to claim 5, characterized in that, The method further includes: Obtain multiple raw template images; The window size is determined based on the texture density of the original template image; Based on the window size, the original template image is segmented to obtain multiple template images; The template image positive sample set is constructed based on multiple template images.

10. The method according to claim 9, characterized in that, The window size and the texture density are negatively correlated.

11. The method according to claim 8, characterized in that, The step of inputting the labeled sample set into the feature alignment network in the second model to obtain the corrected labeled sample set includes: The feature alignment network is used to extract the sample image features of the sample image and the template image features of the template image. The feature alignment network determines the feature transformation parameters between the sample image features and the template image features. Based on the feature transformation parameters, the sample image is geometrically corrected using the feature alignment network to obtain the geometrically corrected sample image. Based on the geometrically corrected sample image and the template image, the corrected annotation sample set is constructed.

12. The method according to claim 5, characterized in that, The defect generation model and the pre-trained model are the same model.

13. A defect detection device, characterized in that, include: The first acquisition unit is used to acquire the image to be tested and a reference image of the same type as the image to be tested, wherein the reference image does not have defects; The detection unit is used to input the image to be tested and the reference image into the differential defect detection model to obtain the detection result of the image to be tested; the detection result is used to identify whether there is a defect in the image to be tested, and to annotate the area where the defect is located; the differential defect detection model is configured to perform difference comparison and defect analysis based on the image to be tested and the reference image.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method according to any one of claims 1-4 or 5-12.

Images