Defect detection method and defect detection device

Abstract

The invention discloses a defect detection method and a defect detection device, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a prompt image with a label and a to-be-detected target image; performing semantic segmentation on the prompt image and the target image to obtain a first defect area corresponding to the prompt image and a second defect area corresponding to the target image; extracting a first defect image corresponding to the first defect area from the prompt image, and extracting a second defect image corresponding to the second defect area from the target image; acquiring a first defect category corresponding to the first defect image and a second defect category corresponding to the second defect image; and obtaining a defect detection result of the target image based on the first defect category and the second defect category. According to the method, the defect detection precision can be effectively improved under the condition that the number of labeled training samples is small.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a defect detection method and a defect detection device. Background Art

[0002] Visual technology is widely used in industrial defect detection, usually relying on large-scale data sets to train neural networks or models to perform defect detection tasks in order to improve production efficiency and product quality.

[0003] However, in the actual production process, the number of defective products is often small, and in the early stages of product production, it is unrealistic to obtain a large number of labeled training samples, resulting in low defect detection accuracy in industrial scenarios. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a defect detection method and a defect detection device, which can effectively improve the defect detection accuracy in industrial scenarios.

[0005] In a first aspect, the present application provides a defect detection method, the method comprising:

[0006] Obtain a labeled prompt image and a target image to be detected;

[0007] Performing semantic segmentation on the prompt image and the target image to obtain a first defect area corresponding to the prompt image and a second defect area corresponding to the target image;

[0008] Extracting a first defect image corresponding to the first defect area from the prompt image, and extracting a second defect image corresponding to the second defect area from the target image;

[0009] Acquire a first defect category corresponding to the first defect image and a second defect category corresponding to the second defect image;

[0010] Based on the first defect category and the second defect category, a defect detection result of the target image is obtained.

[0011] According to the defect detection method of the present application, by performing semantic segmentation without category distinction, the defect areas of the target image and the prompt image are detected, the defect images are cut out according to the defect areas for defect classification, and the defect detection results of the target image are obtained according to the defect categories of the target image and the prompt image. In the case of a small number of labeled training samples, the defect detection accuracy in industrial scenarios can be effectively improved, which helps to improve the production efficiency and product quality of industrial production.

[0012] According to an embodiment of the present application, the semantic segmentation of the prompt image and the target image to obtain a first defect area corresponding to the prompt image and a second defect area corresponding to the target image includes:

[0013] Inputting the prompt image into a semantic segmentation model to obtain the first defect region output by the semantic segmentation model, and inputting the target image into the semantic segmentation model to obtain the second defect region output by the semantic segmentation model;

[0014] The semantic segmentation model is obtained by training based on the first image sample set and the sample defect area corresponding to the first image sample set.

[0015] According to an embodiment of the present application, the first image sample set includes a first image sample and a second image sample, and the second image sample is obtained by performing data enhancement on the first image sample.

[0016] According to one embodiment of the present application, the second image sample is obtained by the following steps:

[0017] Acquiring defect morphological features of the first image sample;

[0018] Based on the defect morphological features, at least one of color transformation, defect mapping, image scaling, image translation, illumination transformation, image blurring and image inversion is performed on the first image sample to obtain the second image sample.

[0019] According to an embodiment of the present application, the acquiring a first defect category corresponding to the first defect image and a second defect category corresponding to the second defect image includes:

[0020] Inputting the first defect image into a defect classification model to obtain the first defect category output by the defect classification model, and inputting the second defect image into the defect classification model to obtain the second defect category output by the defect classification model;

[0021] The defect classification model is obtained by training based on a defect image sample set and defect category labels corresponding to the defect image sample set.

[0022] According to an embodiment of the present application, the defect image sample set includes multiple defect image sample subsets, and the defect category labels corresponding to the respective defect image sample subsets are different.

[0023] According to an embodiment of the present application, the defect image sample set further includes a background image sample subset, and the defect category label corresponding to the background image sample subset is an unknown category.

[0024] According to an embodiment of the present application, obtaining a defect detection result of the target image based on the first defect category and the second defect category includes:

[0025] When the first defect category and the second defect category are the same, defect area information and defect category information of the target image are obtained.

[0026] According to an embodiment of the present application, obtaining a defect detection result of the target image based on the first defect category and the second defect category includes:

[0027] When the first defect category and the second defect category are different, it is determined that no defect is detected in the target image.

[0028] In a second aspect, the present application provides a defect detection device, the device comprising:

[0029] A first processing module is used to obtain a labeled prompt image and a target image to be detected;

[0030] A second processing module is used to perform semantic segmentation on the prompt image and the target image to obtain a first defect area corresponding to the prompt image and a second defect area corresponding to the target image;

[0031] A third processing module, configured to extract a first defect image corresponding to the first defect area from the prompt image, and to extract a second defect image corresponding to the second defect area from the target image;

[0032] A fourth processing module, configured to obtain a first defect category corresponding to the first defect image and a second defect category corresponding to the second defect image;

[0033] A fifth processing module is used to obtain a defect detection result of the target image based on the first defect category and the second defect category.

[0034] According to the defect detection device of the present application, by performing semantic segmentation without category distinction, the defect areas of the target image and the prompt image are detected, the defect images are cut out according to the defect areas for defect classification, and the defect detection result of the target image is obtained according to the defect categories of the target image and the prompt image. In the case of a small number of labeled training samples, the defect detection accuracy of industrial scenarios can be effectively improved, which helps to improve the production efficiency and product quality of industrial production.

[0035] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the defect detection method as described in the first aspect above when executing the computer program.

[0036] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the defect detection method as described in the first aspect above.

[0037] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the defect detection method as described in the first aspect above.

[0038] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0040] Figure 1 This is one of the flow charts of the defect detection method provided in the embodiment of the present application;

[0041] Figure 2 This is the second flow chart of the defect detection method provided in the embodiment of the present application;

[0042] Figure 3 This is the third flow chart of the defect detection method provided in the embodiment of the present application;

[0043] Figure 4 This is the fourth flow chart of the defect detection method provided in the embodiment of the present application;

[0044] Figure 5 This is one of the flow charts of data enhancement of image samples provided in the embodiments of the present application;

[0045] Figure 6 This is the second flow chart of data enhancement of image samples provided in the embodiment of the present application;

[0046] Figure 7 This is the third flow chart of data enhancement of image samples provided in the embodiment of the present application;

[0047] Figure 8 This is the fourth flow chart of data enhancement of image samples provided in the embodiment of the present application;

[0048] Fig. 9 This is the fifth flow chart of data enhancement of image samples provided in the embodiment of the present application;

[0049] Fig.10This is the sixth flow chart of data enhancement of image samples provided in the embodiment of the present application;

[0050] Fig.11 is a schematic diagram of the structure of a defect detection device provided in an embodiment of the present application;

[0051] Fig.12 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0053] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0054] The defect detection method, defect detection device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0055] The defect detection method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the defect detection method. The electronic devices mentioned in the embodiment of the present application include but are not limited to computers, industrial computers, etc. The defect detection method provided in the embodiment of the present application is described below using the electronic device as an example of the execution subject.

[0056] It should be noted that the defect detection method of the embodiment of the present application can be applied to industrial production scenarios to perform defect detection on products, improve the defect detection rate (the proportion of detected defects among real defects) and the yield rate (the proportion of detected defects that are real defects), which helps to improve product quality and production efficiency.

[0057] like Figure 1As shown, the defect detection method includes: step 110, step 120, step 130, step 140 and step 150.

[0058] Step 110: Obtain a labeled prompt image and a target image to be detected.

[0059] The label of the prompt image indicates the defect area of ​​the object in the prompt image.

[0060] It can be understood that the objects in the prompt image and the target image are of the same kind.

[0061] For example, in a cable production scenario, a labeled prompt image is obtained, the object in the prompt image is a cable, and a target image of the cable to be tested is obtained, and defects of the cable are detected based on the prompt image and the target image.

[0062] Step 120 : semantically segment the prompt image and the target image to obtain a first defect region corresponding to the prompt image and a second defect region corresponding to the target image.

[0063] It should be noted that when semantic segmentation is performed on the prompt image and the target image, no specific defect category is defined, and the defective areas of the prompt image and the target image are regarded as the general output of product abnormality.

[0064] In this embodiment, semantic segmentation is performed on the prompt image to obtain a first defect region corresponding to the prompt image, and semantic segmentation is performed on the target image to obtain a second defect region corresponding to the target image.

[0065] It can be understood that the first defect area obtained by performing semantic segmentation on the prompt image is the same as the defect area indicated by the label of the prompt image.

[0066] Step 130 : extracting a first defect image corresponding to the first defect area from the prompt image, and extracting a second defect image corresponding to the second defect area from the target image.

[0067] In this step, after determining the first defect area corresponding to the prompt image and the second defect area corresponding to the target image, the prompt image is cropped according to the first defect area to obtain the first defect image corresponding to the first defect area, and the target image is cropped according to the second defect area to obtain the second defect image corresponding to the second defect area.

[0068] In actual implementation, the first defect area can be a semantic segmentation annotation on the prompt image, and the circumscribed rectangular bounding box of the first defect area is obtained. The corresponding small image is cut on the prompt image according to the circumscribed rectangular bounding box, that is, the first defect image corresponding to the first defect area.

[0069] The second defect area can be a semantic segmentation annotation on the target image, and an circumscribed rectangular bounding box of the second defect area is obtained. The corresponding small image is cut on the target image according to the circumscribed rectangular bounding box, that is, the second defect image corresponding to the second defect area.

[0070] Step 140: Obtain a first defect category corresponding to the first defect image and a second defect category corresponding to the second defect image.

[0071] In this step, the defects in the first defect image are classified to obtain a first defect category corresponding to the first defect image, and the defects in the second defect image are classified to obtain a second defect category corresponding to the second defect image.

[0072] It can be understood that, in addition to indicating the defect region of the object in the prompt image, the label of the prompt image may also indicate the defect category of the object in the prompt image.

[0073] In this embodiment, the first defect category obtained by classifying the defects in the first defect image is the same as the defect category indicated by the label of the prompt image.

[0074] Step 150: Obtain a defect detection result of the target image based on the first defect category and the second defect category.

[0075] In this step, it is determined whether the second defect category is the same as the first defect category, whether the target image has defects of the same defect category as those in the prompt image, defect detection is performed on the object in the target image, and the defect detection result of the target image is output.

[0076] In actual implementation, the prompt image may include one or more defects, each defect has a corresponding label, and semantic segmentation of the prompt image can be performed to obtain one or more first defect areas, and one or more first defect images can be cropped out, each first defect image has a corresponding first defect category.

[0077] The target image may also include one or more defects. By performing semantic segmentation on the target image, one or more second defect regions may be obtained, and one or more second defect images may be cut out. Each second defect image has a corresponding second defect category.

[0078] In this embodiment, each first defect image and each second defect image may be traversed, and the first defect category and the second defect category may be compared to obtain a defect detection result of the target image.

[0079] The following takes a cable scenario as an example to introduce a specific implementation example.

[0080] like Figure 2As shown, a prompt image of the cable and a label of the prompt image are obtained, and a target image to be detected is obtained.

[0081] The embodiment of the present application designs a deep learning defect detection algorithm, which first performs deep learning semantic segmentation on the target image and the prompt image without category distinction, detects the defect areas of the target image and the prompt image, and then extracts the defect images corresponding to all defect areas in the target image and the defect images corresponding to all defect areas in the prompt image based on the predicted defect areas, and then classifies the defect images to obtain specific defect categories, and determines whether the category of each defect in the target prediction result of the target image is the same as the defect category of the prompt image. If they belong to the same defect category, the relevant information of the defect is retained. If they belong to different defect categories, the information is deleted, and the defect detection result of the target image is output.

[0082] At present, in the actual production process, the number of defective products is often small, and in the early stages of product production, it is unrealistic to obtain a large number of labeled training samples, resulting in low defect detection accuracy in industrial scenarios.

[0083] In the embodiment of the present application, semantic segmentation without category distinction is first performed to detect the defect areas of the target image and the prompt image. After the defect image is cut out according to the defect area, defect classification is performed to compare whether the defect categories of the target image and the prompt image are the same, and the defect detection result of the target image is output. Without the need to use a large number of labeled training samples for training, the defect detection accuracy in industrial scenarios can be effectively improved. It has strong generalization ability, high defect detection rate and yield rate, which helps to improve the product quality and production efficiency of industrial production, and can achieve cost reduction and efficiency improvement in defect detection in industrial small-sample scenarios.

[0084] According to the defect detection method provided in the embodiment of the present application, by performing semantic segmentation without category distinction, the defect areas of the target image and the prompt image are detected, the defect image is cut out according to the defect area for defect classification, and the defect detection result of the target image is obtained according to the defect categories of the target image and the prompt image. When the number of labeled training samples is small, the defect detection accuracy of industrial scenarios can be effectively improved, which helps to improve the production efficiency and product quality of industrial production.

[0085] In some embodiments, step 120, performing semantic segmentation on the prompt image and the target image to obtain a first defect area corresponding to the prompt image and a second defect area corresponding to the target image, may include:

[0086] The prompt image is input into the semantic segmentation model to obtain a first defect area output by the semantic segmentation model, and the target image is input into the semantic segmentation model to obtain a second defect area output by the semantic segmentation model.

[0087] The semantic segmentation model is obtained by training based on the first image sample set and the sample defect area corresponding to the first image sample set.

[0088] In this embodiment, the prompt image and the target image are respectively input into the semantic segmentation model to perform semantic segmentation without category distinction, detect the defect areas of the target image and the prompt image, and obtain the first defect area corresponding to the prompt image and the second defect area corresponding to the target image output by the semantic segmentation model.

[0089] It should be noted that the actual defect categories may exceed the definition of existing defects. When the semantic segmentation model is trained based on the first image sample set and the sample defect area corresponding to the first image sample set, all defects are regarded as the same defect category for training. That is, no specific defect category is defined in the actual training, and the defect area is regarded as the general category output of product abnormality, which can effectively improve the defect detection rate of the semantic segmentation model.

[0090] In actual implementation, the semantic segmentation model can be obtained by training the Mask DINO architecture. Mask DINO is a Transformer-based object detection and segmentation framework that can handle image segmentation tasks including instance segmentation, panoptic segmentation, and semantic segmentation, and improve the accuracy of the semantic segmentation model.

[0091] In some embodiments, the first image sample set includes a first image sample and a second image sample, and the second image sample is obtained by performing data enhancement on the first image sample.

[0092] The first image sample may be obtained by capturing a defective object using an image capture device.

[0093] In this embodiment, when training the semantic segmentation model, considering that the number of first image samples may be small, data enhancement can be performed on the first image samples, and the second image samples obtained by data enhancement can be included in the first image sample set to expand the scale of training data and improve the detection rate and generalization of the semantic segmentation model.

[0094] In some embodiments, the second image sample is obtained by the following steps:

[0095] Acquiring defect morphological features of the first image sample;

[0096] Based on the defect morphological features, at least one of color transformation, defect mapping, image scaling, image translation, illumination transformation, image blurring and image inversion is performed on the first image sample to obtain a second image sample.

[0097] The defect morphological feature is a morphological feature that characterizes the defect existing in the object in the first image sample.

[0098] It is understandable that different categories of objects correspond to different defect morphological characteristics.

[0099] For example, cables may have defects such as burrs and debonding, tablets may have defects such as chips and deformations, and wooden boards may have defects such as printing deformations and voids. The defect morphological characteristics corresponding to cables, tablets and wooden boards are different.

[0100] In this embodiment, according to the defect morphological characteristics, one or more of color transformation, defect mapping, image scaling, image translation, illumination transformation, image blurring and image inversion are selected to perform data enhancement on the first image sample to obtain a second image sample. Different first image samples may use different data enhancement methods.

[0101] Image scaling is to change the size of an image, which can be enlarging or reducing the image. The scaling can be proportional or non-proportional, that is, only changing the width or height. Using the second image sample obtained by image scaling to train the semantic segmentation model can help the semantic segmentation model learn the features of objects of different sizes.

[0102] Image translation can be achieved by moving the pixels of the image. Using the second image sample obtained by image translation to train the semantic segmentation model can help the semantic segmentation model learn the characteristics of objects at different positions and improve the robustness of the semantic segmentation model to changes in object positions.

[0103] Illumination transformation simulates images under different lighting conditions. It can be achieved by changing the brightness and contrast of the image or adding random noise. Using the second image sample obtained by illumination transformation to train the semantic segmentation model can help the semantic segmentation model recognize objects and defects on objects under different lighting conditions.

[0104] Image blurring refers to reducing the clarity of an image. Using the second image sample obtained by image blurring to train the semantic segmentation model can help the semantic segmentation model learn the features of objects under different clarity conditions and improve the semantic segmentation model's ability to recognize blurred images.

[0105] Image inversion is to convert each pixel value in an image into its complementary color value, which can simulate the effect of a black and white negative film. Using the second image sample obtained by image inversion to train the semantic segmentation model can help the semantic segmentation model learn the features of objects under color inversion.

[0106] In this embodiment, a data enhancement method for defect maps is also designed. By combining defects and backgrounds, an affine transformation is performed on the defect area to increase the data scale and defect morphology. This is suitable for situations where defect samples are scarce and can help the semantic segmentation model improve its ability to recognize defects.

[0107] For the first image sample where the color interference in the foreground area seriously affects the result, data augmentation can be performed through customized color transformation, which can change the color distribution of the detection area and reduce the impact of color interference on the semantic segmentation model. By changing the color attributes of the image, the diversity of the data set can be increased, which helps to improve the generalization ability of the semantic segmentation model.

[0108] The following uses three scenarios of cable, pill, and wood as examples to describe the embodiments of the present application in detail.

[0109] like Figure 5 As shown, for the Cable training data, data enhancements of custom color transformation, defect mapping, image scaling, image translation, illumination transformation, image blurring, and image inversion are implemented in sequence.

[0110] The color interference in the foreground area of ​​the Cable training data seriously affects the stability of the results, such as Figure 6 As shown, a custom color transformation is designed to change the color distribution of the detection area for data enhancement.

[0111] like Figure 7 As shown, for the Pill training data, data enhancement of defect mapping, custom color transformation, image scaling, image translation, illumination transformation, and image blur are implemented in sequence.

[0112] The color interference in the foreground area of ​​Pill training data will also affect the stability of the results, such as Figure 8 As shown, a custom color transformation is designed to change the color distribution of the detection area for data enhancement.

[0113] like Fig. 9 As shown in FIG. 1 , for Pill training data, by combining defects and background, affine transformation is performed on the defect area, and the scale of training data is expanded through defect maps.

[0114] like Fig.10 As shown, for the Wood training data, data augmentation of defect mapping, image scaling, image translation, illumination transformation, and image blurring are implemented in sequence.

[0115] In this embodiment, three scenes are classified according to defect morphological features, and corresponding data enhancement algorithms are designed for specific defect morphological features in each scene to expand the amount of training data and effectively improve the prediction accuracy and generalization ability of the semantic segmentation model.

[0116] In some embodiments, step 140, obtaining a first defect category corresponding to the first defect image and a second defect category corresponding to the second defect image, may include:

[0117] The first defect image is input into the defect classification model to obtain a first defect category output by the defect classification model, and the second defect image is input into the defect classification model to obtain a second defect category output by the defect classification model.

[0118] Among them, the defect classification model is trained based on the defect image sample set and the defect category labels corresponding to the defect image sample set.

[0119] In this embodiment, the first defect image and the second defect image are respectively input into the defect classification model to identify the defect categories of the defect images, and the first defect category corresponding to the first defect image and the second defect category corresponding to the second defect image output by the defect classification model are obtained.

[0120] It should be noted that the defect classification model is trained based on the defect image sample set and the defect category labels corresponding to the defect image sample set, and the images in the defect image sample set can be defect images cropped from the defect area in the first image sample set.

[0121] In actual implementation, the defect classification model can be obtained by training the ResNet18 architecture. The ResNet18 architecture can solve the gradient vanishing problem in deep network training through residual connections, and improve the generalization ability and training efficiency of the defect classification model.

[0122] In some embodiments, the defect image sample set includes multiple defect image sample subsets, and each defect image sample subset corresponds to a different defect category label.

[0123] In this embodiment, the defect image sample set includes multiple defect image samples, which are divided into multiple defect image sample subsets according to defect categories. Defect image samples of the same defect category belong to the same defect image sample subset, and different defect image sample subsets correspond to different defect category labels.

[0124] In some embodiments, the defect image sample set further includes a background image sample subset, and the defect category label corresponding to the background image sample subset is an unknown category.

[0125] In this embodiment, in addition to multiple defect image samples, the defect image sample set also includes a background image sample subset, which includes multiple background image samples. The background image samples can be background images after the defect image is cut out of the defect area in the first image sample set.

[0126] It should be noted that the defect category label corresponding to the background image sample subset is set to an unknown category, and the defect classification model is trained using a defect image sample set including the background image sample subset. In actual applications, if an unknown defect category that is not in the training data appears, the defect classification model can output the corresponding unknown category information.

[0127] In some embodiments, step 150, obtaining a defect detection result of the target image based on the first defect category and the second defect category, may include:

[0128] When the first defect category and the second defect category are the same, defect region information and defect category information of the target image are obtained.

[0129] In some other embodiments, step 150, obtaining a defect detection result of the target image based on the first defect category and the second defect category, may include:

[0130] In a case where the first defect category and the second defect category are different, it is determined that no defect is detected in the target image.

[0131] The first defect category corresponding to the prompt image is compared with the second defect category corresponding to the target image. When the second defect category corresponding to a second defect area in the target image belongs to the same category as the first defect category of the prompt image, the defect area information and defect category information corresponding to the second defect area in the target image are output.

[0132] When the second defect category corresponding to a second defect area in the target image belongs to a different category from the first defect category of the prompt image, the output defect detection result is empty, and no defect is detected in the second defect area in the target image.

[0133] In actual execution, the second defect category corresponding to each second defect area in the target image is traversed to determine whether it is the same as the first defect category in the prompt image. If they are the same, the defect area information and defect category information are retained and output; if they are different, they are deleted to obtain the defect detection result of the target image.

[0134] A specific embodiment is described below.

[0135] like Figure 3 As shown, a target image and a prompt image of the cable are obtained, and the prompt image has a corresponding defect label.

[0136] The semantic segmentation model is used to perform semantic segmentation on the target image and the prompt image respectively to obtain the defect areas corresponding to the target image and the prompt image respectively.

[0137] Among them, the semantic segmentation model is trained through the first image sample set. According to the defect morphological characteristics in different visual scenes such as cables, pills and wood, an exclusive data enhancement method can be set to perform data enhancement and expand the training data volume of the semantic segmentation model.

[0138] like Figure 4 As shown, the prompt image label corresponds to the first defect area of ​​the prompt image, and the target image segmentation prediction result corresponds to the second defect area of ​​the target image. Defect extraction is performed, and the corresponding small images, the target defect small image (i.e., the second defect image) and the prompt defect small image (i.e., the first defect image) are obtained by cutting the image according to the circumscribed rectangular bounding box of the defect area.

[0139] Defect images are classified into defect categories (Verify by classification), and a classification model family may be used for classification. The classification model family may include a group of defect classification models having similar characteristics or based on the same theoretical basis.

[0140] After identifying the defects of the prompt image and the target image, the defect category is verified. The label in the prompt image points to the defect area. If the defect belongs to the same category as the defect in the target image, the defect area, category and other information of the target image are output. If the defect belongs to a different category from the defect in the target image, the defect prediction result is output as empty, and the target prediction result of the target image is obtained.

[0141] In this embodiment, defect detection is performed by first semantic segmentation and then classification verification. There is no need to use a large number of labeled training samples for training, which effectively improves the defect detection accuracy in industrial scenarios. It has strong generalization ability, high defect detection rate and yield rate, and helps to improve the product quality and production efficiency of industrial production. It can achieve cost reduction and efficiency improvement in defect detection in industrial small sample scenarios.

[0142] Verification was carried out on the vision dataset, with a training data scale of 75 Cable images, 65 Pill images, and 83 Wood images, and 1000 test samples in each of the three scenes. The defect detection method of the embodiment of the present application was compared with the defect detection results of the ordinary one-shot solution in terms of detection rate and yield, and the final score was calculated with the average of the two to measure the performance of the solution.

[0143] Table 1

[0144] method Detection rate Yield Comprehensive score One-shot solution 84.67％ 84.88％ 84.77％ This application 91.50％ 91.00％ 91.25％

[0145] Table 1 shows the comparison results of the defect detection method of the embodiment of the present application and the ordinary one-shot solution on the vision data set. Compared with the one-shot solution, the defect detection method of the embodiment of the present application, which performs semantic segmentation first and then classification verification, can effectively improve the detection rate and yield rate. The algorithm performance of the embodiment of the present application is comprehensively much higher than that of the one-shot solution.

[0146] In addition, the defect detection method of the embodiment of the present application that performs semantic segmentation followed by classification verification was verified on four sets of industrial scene data sets, using 53 images of an electronic manufacturing scene data set A, 71 images of a lithium battery scene data set B, 70 images of a photovoltaic scene data set C, and 80 images of a printing scene data set D, with 1,200 test samples in each of the four scenes. The defect detection results of the defect detection method of the embodiment of the present application were also compared with those of the ordinary one-shot solution in terms of detection rate and yield, and the final score was calculated with the average of the two to measure the performance of the solution.

[0147] Table 2 shows the accuracy comparison results of the electronic manufacturing scenario methods, Table 3 shows the accuracy comparison results of the lithium battery scenario methods, Table 4 shows the accuracy comparison results of the photovoltaic scenario methods, and Table 5 shows the accuracy comparison results of the printing scenario methods.

[0148] Table 2

[0149] method Detection rate Yield Comprehensive score One-shot solution 92.63％ 90.95％ 91.79％ This application 96.09％ 91.85％ 93.97％

[0150] Table 3

[0151] method Detection rate Yield Comprehensive score One-shot solution 83.40％ 68.82％ 76.11％ This application 86.89％ 75.26％ 81.08％

[0152] Table 4

[0153] method Detection rate Yield Comprehensive score One-shot solution 85.82％ 73.06％ 79.44％ This application 86.69％ 74.64％ 80.66％

[0154] Table 5

[0155] method Detection rate Yield Comprehensive score One-shot solution 83.38％ 68.23％ 75.81％ This application 84.52％ 70.37％ 77.45％

[0156] The defect detection method of first segmentation and then classification verification in the embodiment of the present application can improve the detection rate and yield in the semantic segmentation problem of industrial few-sample scenarios in which the prompt image determines whether the target image defect is output, obtain performance that is better than the conventional few-sample semantic segmentation method, and can help industrial few-sample defect segmentation applications to reduce costs and increase efficiency.

[0157] The defect detection method provided in the embodiment of the present application can be executed by a defect detection device. In the embodiment of the present application, the defect detection device provided in the embodiment of the present application is described by taking the defect detection method executed by the defect detection device as an example.

[0158] The embodiment of the present application also provides a defect detection device.

[0159] like Fig.11 As shown, the defect detection device comprises:

[0160] The first processing module 1110 is used to obtain a labeled prompt image and a target image to be detected;

[0161] The second processing module 1120 is used to perform semantic segmentation on the prompt image and the target image to obtain a first defect area corresponding to the prompt image and a second defect area corresponding to the target image;

[0162] The third processing module 1130 is used to extract a first defect image corresponding to the first defect area from the prompt image, and to extract a second defect image corresponding to the second defect area from the target image;

[0163] A fourth processing module 1140 is used to obtain a first defect category corresponding to the first defect image and a second defect category corresponding to the second defect image;

[0164] The fifth processing module 1150 is used to obtain a defect detection result of the target image based on the first defect category and the second defect category.

[0165] According to the defect detection device provided in the embodiment of the present application, by performing semantic segmentation without category distinction, the defect areas of the target image and the prompt image are detected, the defect image is cut out according to the defect area for defect classification, and the defect detection result of the target image is obtained according to the defect categories of the target image and the prompt image. When the number of labeled training samples is small, the defect detection accuracy of industrial scenarios can be effectively improved, which helps to improve the production efficiency and product quality of industrial production.

[0166] In some embodiments, the second processing module 1120 is used to perform semantic segmentation on the prompt image and the target image to obtain a first defect area corresponding to the prompt image and a second defect area corresponding to the target image, including:

[0167] Inputting the prompt image into the semantic segmentation model to obtain a first defect area output by the semantic segmentation model, and inputting the target image into the semantic segmentation model to obtain a second defect area output by the semantic segmentation model;

[0168] The semantic segmentation model is obtained by training based on the first image sample set and the sample defect area corresponding to the first image sample set.

[0169] In some embodiments, the first image sample set includes a first image sample and a second image sample, and the second image sample is obtained by performing data enhancement on the first image sample.

[0170] In some embodiments, the second image sample is obtained by the following steps:

[0171] Acquiring defect morphological features of the first image sample;

[0172] Based on the defect morphological features, at least one of color transformation, defect mapping, image scaling, image translation, illumination transformation, image blurring and image inversion is performed on the first image sample to obtain a second image sample.

[0173] In some embodiments, the fourth processing module 1140 is used to obtain the first defect category corresponding to the first defect image and the second defect category corresponding to the second defect image, including:

[0174] Inputting the first defect image into the defect classification model to obtain a first defect category output by the defect classification model, and inputting the second defect image into the defect classification model to obtain a second defect category output by the defect classification model;

[0175] Among them, the defect classification model is trained based on the defect image sample set and the defect category labels corresponding to the defect image sample set.

[0176] In some embodiments, the defect image sample set includes multiple defect image sample subsets, and each defect image sample subset corresponds to a different defect category label.

[0177] In some embodiments, the defect image sample set further includes a background image sample subset, and the defect category label corresponding to the background image sample subset is an unknown category.

[0178] In some embodiments, the fifth processing module 1150 is used to obtain a defect detection result of the target image based on the first defect category and the second defect category, including:

[0179] When the first defect category and the second defect category are the same, defect region information and defect category information of the target image are obtained.

[0180] In some embodiments, the fifth processing module 1150 is used to obtain a defect detection result of the target image based on the first defect category and the second defect category, including:

[0181] In a case where the first defect category and the second defect category are different, it is determined that no defect is detected in the target image.

[0182] The defect detection device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0183] The defect detection device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0184] The defect detection device provided in the embodiment of the present application can implement each process implemented in the above-mentioned defect detection method embodiment. To avoid repetition, it will not be repeated here.

[0185] In some embodiments, Fig.12 As shown, an embodiment of the present application also provides an electronic device 1200, including a processor 1201, a memory 1202, and a computer program stored in the memory 1202 and executable on the processor 1201. When the program is executed by the processor 1201, each process of the above-mentioned defect detection method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0186] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0187] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned defect detection method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0188] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0189] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned defect detection method when executed by a processor.

[0190] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0191] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned defect detection method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0192] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0193] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0194] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0195] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0196] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0197] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A defect detection method, characterized in that: include: Obtain a labeled prompt image and a target image to be detected; Performing semantic segmentation on the prompt image and the target image to obtain a first defect area corresponding to the prompt image and a second defect area corresponding to the target image; Extracting a first defect image corresponding to the first defect area from the prompt image, and extracting a second defect image corresponding to the second defect area from the target image; Acquire a first defect category corresponding to the first defect image and a second defect category corresponding to the second defect image; Based on the first defect category and the second defect category, a defect detection result of the target image is obtained.

2. The defect detection method according to claim 1, characterized in that: The performing semantic segmentation on the prompt image and the target image to obtain a first defect area corresponding to the prompt image and a second defect area corresponding to the target image includes: Inputting the prompt image into a semantic segmentation model to obtain the first defect region output by the semantic segmentation model, and inputting the target image into the semantic segmentation model to obtain the second defect region output by the semantic segmentation model; The semantic segmentation model is obtained by training based on the first image sample set and the sample defect area corresponding to the first image sample set.

3. The defect detection method according to claim 2, characterized in that: The first image sample set includes a first image sample and a second image sample, where the second image sample is obtained by performing data enhancement on the first image sample.

4. The defect detection method according to claim 3, characterized in that: The second image sample is obtained by the following steps: Acquiring defect morphological features of the first image sample; Based on the defect morphological features, at least one of color transformation, defect mapping, image scaling, image translation, illumination transformation, image blurring and image inversion is performed on the first image sample to obtain the second image sample.

5. The defect detection method according to claim 1, characterized in that: The obtaining of a first defect category corresponding to the first defect image and a second defect category corresponding to the second defect image includes: Inputting the first defect image into a defect classification model to obtain the first defect category output by the defect classification model, and inputting the second defect image into the defect classification model to obtain the second defect category output by the defect classification model; The defect classification model is obtained by training based on a defect image sample set and defect category labels corresponding to the defect image sample set.

6. The defect detection method according to claim 5, characterized in that: The defect image sample set includes a plurality of defect image sample subsets, and each of the defect image sample subsets corresponds to a different defect category label.

7. The defect detection method according to claim 6, characterized in that: The defect image sample set also includes a background image sample subset, and the defect category label corresponding to the background image sample subset is an unknown category.

8. The defect detection method according to any one of claims 1 to 7, characterized in that: The obtaining a defect detection result of the target image based on the first defect category and the second defect category includes: When the first defect category and the second defect category are the same, defect area information and defect category information of the target image are obtained.

9. The defect detection method according to any one of claims 1 to 7, characterized in that: The obtaining a defect detection result of the target image based on the first defect category and the second defect category includes: When the first defect category and the second defect category are different, it is determined that no defect is detected in the target image.

10. A defect detection device, characterized in that: include: A first processing module is used to obtain a labeled prompt image and a target image to be detected; A second processing module is used to perform semantic segmentation on the prompt image and the target image to obtain a first defect area corresponding to the prompt image and a second defect area corresponding to the target image; A third processing module, configured to extract a first defect image corresponding to the first defect area from the prompt image, and to extract a second defect image corresponding to the second defect area from the target image; A fourth processing module, configured to obtain a first defect category corresponding to the first defect image and a second defect category corresponding to the second defect image; A fifth processing module is used to obtain a defect detection result of the target image based on the first defect category and the second defect category.

Images