Method and system for generating virtual defect image for image-based defect detection by using artificial intelligence

Abstract

The present invention relates to a method and a system for generating a virtual defect image for image-based defect detection by using artificial intelligence. According to the present invention, the method for generating a virtual defect image for image-based defect detection comprises the steps of: annotating at least one defect region present in a target region in an actual defect image; inpainting the annotated at least one defect region; extracting at least one defect patch by subtracting the inpainted image from the actual defect image; and generating at least one virtual defect image by synthesizing the at least one defect patch at a particular position in the target region in an actual normal image.

Description

Method and system for generating virtual defect images for image-based defect detection using artificial intelligence

[0001] The present invention relates to a method and device for generating a virtual defect image for image-based defect detection using artificial intelligence, and more particularly, to a method and device for generating a virtual defect image for image-based defect detection using artificial intelligence, which can generate a virtual defect image by synthesizing a normal image and an abnormal image containing a defect, and construct an image-based defect detection model from the virtual defect image.

[0002] Nondestructive testing (NDT) image analysis plays a crucial role in various industries, particularly in defect detection and quality improvement. While NDT image analysis relies on experts to directly detect defects and assess quality, human error and subjective judgment can lead to erroneous results.

[0003] Therefore, traditional defect detection methods rely on the expertise of skilled experts, leading to problems such as inconsistency, subjective judgment, time consumption, and high costs. These limitations highlight the need for more efficient, objective, and scalable solutions.

[0004] Recently, various techniques have been developed to detect defects or faults in objects by analyzing images taken of the object using artificial intelligence and deep learning.

[0005] However, training a deep learning model requires a large number of normal and abnormal (defective) image data. While collecting a large number of normal images is easy in typical manufacturing environments, collecting a large number of abnormal (abnormal) images is challenging due to the nature of the process, which typically results in only a small number of defects.

[0006] As such, most image data obtained from industrial sites is normal, with only a small percentage containing abnormal data. Consequently, securing sufficient training data for building defect detection models is difficult, and the limited number of available defective images leads to class imbalance problems.

[0007] The technology behind the invention is disclosed in Korean Patent Publication No. 10-2023-0122030 (published on August 22, 2023).

[0008] The purpose of the present invention is to provide a method and device for generating a virtual defect image for image-based defect detection using artificial intelligence, which can augment data by generating a virtual defect image by synthesizing an actual defect pattern into a normal image and can solve a class imbalance problem by utilizing the generated virtual defect image for model learning.

[0009] The present invention relates to a method for generating a virtual defect image for image-based defect detection, comprising: a step of annotating at least one defect region existing on a target region in an actual defect image; a step of inpainting the at least one annotated defect region; a step of extracting at least one defect patch by differentiating the inpainted image from the actual defect image; and a step of generating at least one virtual defect image by synthesizing the at least one defect patch at a portion of the target region in an actual normal image.

[0010] In addition, the annotating step can label the detection location and defect type value of each defect area that exists within the target area based on the actual ground truth.

[0011] In addition, the step of extracting the above-described defective patch can store each extracted defective patch in a database by matching it with its corresponding detection location and defect type value.

[0012] In addition, the step of generating the virtual defect image may include detecting a background area and a target area separately from the actual normal image, and then synthesizing the defect patch at a designated location within the detected target area.

[0013] Additionally, the step of generating the virtual defect image may increase the number of defect patches utilized in the synthesis by applying elastic deformation to the defect patch to randomly change the shape of the defect before the synthesis.

[0014] Additionally, the step of generating the virtual defect image can determine a different location within the target area where synthesis is performed depending on the defect type of the defect patch.

[0015] In addition, the step of generating the virtual defect image may include specifying a synthesis location within the target area according to the combination type of the defect patch, applying elastic deformation to the defect patch, and then synthesizing the defect patch at the specified synthesis location to generate the virtual defect image.

[0016] In addition, the method for generating a virtual defect image may further include a step of training a defect detection model based on a plurality of generated virtual defect images, applying an actual defect image to the defect detection model for which training has been completed to detect at least one of a defect area and a defect type, and performing a performance evaluation on the detection result to evaluate the quality of the virtual defect image.

[0017] And, the present invention is a virtual defect image generation system for image-based defect detection, comprising: an image preprocessing unit for annotating at least one defect region existing on a target region in an actual defect image; a defect patch extraction unit for inpainting the at least one annotated defect region, and then extracting at least one defect patch by differentially processing the inpainted image from the actual defect image; and an image generation unit for generating at least one virtual defect image by synthesizing the at least one defect patch at a portion of the target region in an actual normal image.

[0018] In addition, the virtual defect image generation system may further include a test unit that trains a defect detection model based on a plurality of generated virtual defect images, applies an actual defect image to the defect detection model for which training is completed, detects at least one of a defect area and a defect type, and performs a performance evaluation on the detection result to evaluate the quality of the virtual defect image.

[0019] According to the present invention, a virtual defect image similar to an actual defect can be generated by synthesizing an actual defect pattern extracted from an actual defect image into a normal image, and a class imbalance problem can be solved by constructing a defect detection model based on the generated virtual defect image.

[0020] Accordingly, it can provide important solutions for efficient and reliable defect detection in the manufacturing industry and contribute to the development of quality assurance and safety standards for manufactured products.

[0021] FIG. 1 is a diagram showing the configuration of a virtual defect image generation system according to an embodiment of the present invention.

[0022] FIG. 2 is a drawing explaining a method for generating a virtual defect image using the system of FIG. 1.

[0023] Figure 3 is a drawing to explain the method of Figure 2 in more detail.

[0024] FIG. 4 is a drawing exemplarily showing the result of enhancing a defective patch image using elastic deformation in an embodiment of the present invention.

[0025] FIG. 5 is a diagram showing the result of generating virtual defect data from actual defect data according to an embodiment of the present invention.

[0026] FIG. 6 is a diagram showing a process of testing the prediction performance of a defect detection model using an actual defect image after learning the defect detection model based on a virtual defect image according to an embodiment of the present invention.

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings so that those skilled in the art can easily practice the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description have been omitted to clearly explain the present invention, and similar parts have been designated with similar reference numerals throughout the specification.

[0028] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where the parts are "directly connected" but also the cases where the parts are "electrically connected" with other elements intervening. Furthermore, when a part is said to "include" a component, this does not exclude other components, but rather includes other components, unless otherwise stated.

[0029] The present invention relates to a system and method for generating a virtual defect image, and proposes a technique for generating a virtual defect image by synthesizing a normal image and an abnormal image containing a defect, constructing an image-based defect detection model by utilizing the virtual combined image generated in this way, and improving the prediction performance of the defect detection model.

[0030] FIG. 1 is a diagram showing the configuration of a virtual defect image generation system according to an embodiment of the present invention.

[0031] As shown in FIG. 1, a virtual defect image generation system (100) according to an embodiment of the present invention includes an image preprocessing unit (110), a defect patch extraction unit (120), an image generation unit (130), and may further include a test unit (140).

[0032] The operation of each part (110 to 140) and the data flow between each part (110 to 140) can be performed by a control unit (not shown) corresponding to the internal processor of the system.

[0033] Here, the virtual defect image generation system (100) may be implemented as a computer device or server that is physically configured and includes a processor, memory, a user interface input / output device and a storage device, a network input / output unit, etc., or may be implemented as an application program running on a computer device or a user terminal.

[0034] The image preprocessing unit (110) annotates at least one defect area existing on a target area within an actual defect image and inpaints at least one annotated defect area.

[0035] Here, the target region may refer to a region of interest within the image where defect detection is required. For example, if the goal is to detect welding defects, the target region may correspond to the weld area among the base material and weld area detected in the image of the manufactured product, with the remainder being treated as a background area.

[0036] The image preprocessing unit (110) can label the detection location and defect type value of each defect area existing in the target area based on the actual ground truth, and can inpaint the labeled area using a setting technique (e.g., LaMa technique).

[0037] The defect patch extraction unit (120) extracts at least one defect patch by subtracting (subtracting) the inpainted image from the actual defect image. In this way, by calculating the difference between two images, at least one defect patch corresponding to at least one defect area can be extracted.

[0038] In addition, the defect patch extraction unit (120) can store each extracted defect patch in a database by matching it with its corresponding defect location and defect type values.

[0039] The image generation unit (130) synthesizes at least one extracted defect patch onto a portion of a target area within a pre-prepared actual normal image to generate at least one virtual defect image. The generated virtual defect image can be utilized for training a deep learning-based defect detection model.

[0040] Here, the image generation unit (130) can separate and detect the background region and the target region from an actual normal image and then synthesize a defective patch at a designated location within the detected target region. At this time, the image generation unit (130) can determine the location within the target region where the synthesis is performed differently depending on the defect type of the defective patch.

[0041] In addition, the image generation unit (130) can increase the number of defect patches used for image synthesis by applying elastic deformation to the defect patches to randomly change the shape of the defects.

[0042] In this way, the image generation unit (130) can generate a virtual defect image by applying elastic deformation to the defect patch and then synthesizing the defect patch subjected to elastic deformation at a designated synthesis location within the target area according to the corresponding defect type.

[0043] The test unit (140) can train a deep learning-based defect detection model using multiple generated virtual defect images. The defect detection model can be trained through artificial intelligence using the virtual defect images and the area location and defect type of at least one defect pattern labeled within the images as training data.

[0044] The test unit (140) can detect at least one of a defect area and a defect type by applying an actual defect image to a defect detection model for which learning has been completed, and can evaluate the quality of a virtual defect image by performing a performance evaluation on the detection result.

[0045] The following describes in more detail a method for generating a virtual defect image according to an embodiment of the present invention.

[0046] For convenience of explanation, the proposed technique is illustrated below with a representative example of its use in learning a model for detecting welding defects in manufactured products (e.g., welded steel pipes). However, the technique of the present invention is not necessarily limited to the aforementioned applications and can be applied to a variety of objects, manufactured products, and other objects requiring image-based defect identification.

[0047] First, to detect welding defects based on digital radiography images using AI, a deep learning model for image classification and segmentation must be trained from images of manufactured products. Due to the nature of the welded steel pipe manufacturing process, defect patterns are diverse and the number of available defect images is limited, leading to class imbalance issues during the training process for the deep learning classification model.

[0048] In embodiments of the present invention, data can be augmented by generating virtual defect images based on defect patterns acquired from actual defect images. The proposed technique extracts defect patterns from actual defect images and then synthesizes them with actual normal images to generate a large number of virtual defect images that resemble actual defects. This can then be used to improve the predictive performance of a defect detection model.

[0049] In the following examples of the present invention, all images are grayscale images normalized to a set size. Of course, the present invention is not necessarily limited to this, and the images used for defect detection may be color images with RGB values, as well as grayscale images.

[0050] FIG. 2 is a drawing explaining a method for generating a virtual defect image using the system of FIG. 1, and FIG. 3 is a drawing explaining the method of FIG. 2 in more detail.

[0051] First, the virtual defective image generation system (100) can annotate at least one defective area existing on a target area within a real defective image (10) (S210).

[0052] Here, when the object for defect inspection is a welded steel pipe, the target area may mean the weld among the base material and the weld.

[0053] Afterwards, the virtual defect image generation system (100) can obtain an inpainted image (30) by inpainting at least one annotated defect area (S220).

[0054] Here, the virtual defect image generation system (100) collects an actual defect image (10), and based on the actual ground truth analyzed by the user in the collected actual defect image, can label the detection location and defect type value for each defect area existing within the target area within the actual defect image. For the detection location, the outline of the defect area, bounding box, etc. can be utilized.

[0055] Here, labeling information can be utilized to analyze defect occurrence locations by defect type. The defect occurrence locations (zones) within a target area (e.g., welds) may vary depending on the defect type. Based on the labeling information, information on the corresponding defect occurrence locations (zones) for each defect type can be stored and managed.

[0056] In the welding field, various types of welding defects have been previously known, and the locations of defect occurrence can vary depending on the type. The proposed technique automatically reflects the defect location for each type when synthesizing a defect pattern within a target area within a normal image, thereby generating a virtual defect image.

[0057] The virtual defect image generation system (100) can annotate (pixel-level ground truth mask annotation) multiple defect areas existing on a target area within an actual defect image (10) by masking using a ground truth mask (20), as shown in FIG. 3.

[0058] In addition, the virtual defect image generation system (100) can inpaint each defect area portion exposed when a ground truth mask (20) is applied on an actual defect image (10) using the LaMa technique. In this case, in addition to the LaMa technique, various known inpainting techniques can be utilized.

[0059] Looking at the inpainted image (30) in Fig. 3, it can be confirmed that all four defective areas that existed in the actual defective image (10) are filled with the background pattern at the corresponding location.

[0060] Next, the virtual defect image generation system (100) can extract at least one defect patch (40) by differentiating the inpainted image (30) from the actual defect image (10) (S230).

[0061] In the case of Fig. 3, it is shown that four different defect patches (40) are extracted by differentially processing the inpainted image (30) from the actual defect image (10).

[0062] In this way, by calculating the difference in grayscale values ​​between two images on a pixel-by-pixel basis, multiple defect patches (40) corresponding to multiple defect areas can be individually extracted. Here, since the image difference technique corresponds to a technique already known in the field of image analysis, a detailed description thereof is omitted.

[0063] The virtual defect image generation system (100) can store extracted defect patches in a database, and can manage a plurality of defect patch data (40) extracted from a plurality of actual defect images (10) by matching them with corresponding defect locations and defect type values. In addition, the virtual defect image generation system (100) can build a database of zones that match each defect patch's defect type among a plurality of zones within a target area.

[0064] Next, the virtual defective image generation system (100) can generate at least one virtual defective image (60) by synthesizing at least one extracted defective patch (40) to some location on a target area within at least one real normal image (50) (S240).

[0065] At this time, all or some of the multiple defective patches (40) may be randomly selected from the database and synthesized into a target area within an actual normal image (20). FIG. 3 shows the result of synthesizing at least one selected defective patch into a set location within three actual normal images (20).

[0066] Here, the virtual defect image generation step can be subdivided as follows.

[0067] First, the background region and the target region are separated and detected within an actual normal image (10). Then, a defective patch can be synthesized at a designated location within the detected target region.

[0068] Here, prior to synthesis, elastic deformation can be applied to each defect patch collected in step S230. In this way, applying elastic deformation to the defect patches, which randomly changes their shape, prior to image synthesis can increase the number of defect patches utilized for synthesis.

[0069] Fig. 4 is a diagram exemplarily showing the results of augmenting a defect patch image using elastic deformation in an embodiment of the present invention. Fig. 4 shows that a large number of elastically deformed defect patches are secured by applying various elastic deformations to an original defect image.

[0070] The number of defect patch images obtained in the previous S230 step is still limited, and the abnormal image (virtual defect image) obtained by synthesizing it with a normal image has the limitation of obtaining diversity only in terms of the location and intensity of the defect.

[0071] To train a more robust model, the defect shapes themselves must be enriched with diversity. Therefore, in an embodiment of the present invention, elastic deformation is applied, as shown in Figure 4, to enhance diversity in the defect shapes. Furthermore, by increasing the number of defect patches through elastic deformation, the diversity of virtual defect images can also be realized.

[0072] In addition, in this S240 step, the virtual defect image generation system (100) can designate a synthesis location within the target area according to the combination type of the defect patch, and after applying elastic deformation to the defect patch, can synthesize the defect patch at the designated synthesis location to generate a virtual defect image.

[0073] In this way, in an embodiment of the present invention, when synthesizing a defective patch into a normal image, the position where the synthesis is performed within the target area can be automatically changed according to the defect type of the defective patch, and the position can be randomly selected within a set area to reflect the characteristics of each defect type.

[0074] FIG. 5 is a diagram illustrating the results of generating virtual defect data from actual defect data according to an embodiment of the present invention. FIG. 5 shows only a target area extracted from an image, and various forms of virtual defect data can be obtained by synthesizing defect patches extracted from actual defect data with normal data after elastic modification.

[0075] Next, the test unit (140) can train a defect detection model using only a plurality of virtual defect images generated in this manner and then perform a defect detection performance evaluation by applying actual defect images (S250).

[0076] The test unit (140) can evaluate the quality and validity of the virtual defect images generated by the proposed method by performing model evaluation only with actual defect images after learning the defect detection model based on multiple virtual defect images generated by the proposed method.

[0077] FIG. 6 is a diagram showing a process of testing the prediction performance of a defect detection model using an actual defect image after learning the defect detection model based on a virtual defect image according to an embodiment of the present invention.

[0078] As shown in Fig. 6, the test unit (140) can first train a defect detection model using a virtual defect image and ground truth data labeled in the image as learning data. At this time, the U-Net model was used as the detection model to evaluate the similarity between the actual defect image and the virtual defect image.

[0079] Afterwards, actual normal images and actual abnormal images (images containing defects) can be input into the trained defect detection model to predict the detection results. The detection results can include the location and type of defects detected within the image.

[0080] Table 1 is a diagram showing the prediction performance of a defect detection model learned based on a virtual defect image according to an embodiment of the present invention.

[0081] ModelMethodAccuracyF1-ScoreRecallPrecisionCNNWeighted Loss0.85660.84090.82220.8605Conventional Augmentation0.84940.72000.60000.9000Virtual Defect0.91900.93480.95560.9149VGG16Weighted Loss0.86970.72000.60000.9000Conventional Augmentation0.92530.93950.75560.9444Virtual Defect0.95240.94380.93330.9545XceptionWeighted Loss0.56630.36920.26670.6000Conventional Augmentation0.81000.61970.48890.8462Virtual Defect0.80310.82980.86670.7959DenseNet121Weighted Loss0.38200.28570.22220.4000Conventional Augmentation0.66740.57140.48890.6875Virtual Defect0.96400.91760.86670.9750

[0082] Table 1 compares the performance of four models (CNN, VGG16, Xception, DenseNet121) and experiments were conducted using actual welded steel pipe data.

[0083] When model training was performed using virtual defect images, the F1 score increased by more than 10% compared to when training was performed by adjusting the class weights for actual normal and abnormal data by the inverse of the ratio of normal and abnormal data (weighted loss) and when model training was performed using conventional data augmentation techniques (conventional augmentation).

[0084] These points show that the virtual defect image generated according to the technique of the present invention can improve the learning and prediction performance of the defect detection model.

[0085] Therefore, when utilizing the virtual defect images generated using the technique of the present invention as training data in actual industrial settings, higher accuracy anomaly detection performance can be expected. These virtual defect images can be utilized for model training alongside actual normal / abnormal images.

[0086] According to the present invention as described above, a virtual defect image similar to an actual defect can be generated by synthesizing an actual defect pattern extracted from an actual defect image onto a normal image, and a class imbalance problem can be solved by constructing a defect detection model based on the generated virtual defect image.

[0087] While the present invention has been described with reference to the embodiments illustrated in the drawings, these are merely exemplary, and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible. Therefore, the true scope of technical protection of the present invention should be determined by the technical spirit of the appended claims.

Claims

1. A method for generating a virtual defect image for image-based defect detection, A step of annotating at least one defective region existing on a target region within an actual defective image; A step of inpainting at least one of the above annotated defect areas; A step of extracting at least one defect patch by differentiating the inpainted image from the actual defect image; and A method for generating a virtual defect image, comprising the step of generating at least one virtual defect image by synthesizing at least one defect patch at a certain location on a target area within an actual normal image.

2. In claim 1, The above annotation steps are: A method for generating a virtual defect image by labeling the detection location and defect type value of each defect area existing within the target area based on the actual ground truth.

3. In claim 2, The step of extracting the above defective patch is: A method for generating a virtual defect image by matching each of the extracted defect patches with its corresponding detection location and defect type value and storing them in a database.

4. In claim 1, The step of generating the above virtual defect image is: A method for generating a virtual defect image by separating and detecting a background area and a target area from the actual normal image and then synthesizing the defect patch at a designated location within the detected target area.

5. In claim 1, The step of generating the above virtual defect image is: A method for generating a virtual defect image by applying elastic deformation to the defect patch to randomly change the shape of the defect before the synthesis, thereby increasing the number of defect patches utilized in the synthesis.

6. In claim 1, The step of generating the above virtual defect image is: A method for generating a virtual defect image, wherein the position where synthesis is performed within the target area is determined differently depending on the defect type of the defect patch.

7. In claim 6, The step of generating the above virtual defect image is: A method for generating a virtual defect image, wherein a synthesis location within the target area is designated according to the combination type of the defect patch, elastic deformation is applied to the defect patch, and then the defect patch is synthesized at the designated synthesis location to generate a virtual defect image.

8. In claim 1, A method for generating a virtual defect image, further comprising the steps of: training a defect detection model based on a plurality of generated virtual defect images; applying an actual defect image to the trained defect detection model to detect at least one of a defect area and a defect type; and evaluating the quality of the virtual defect image by performing a performance evaluation on the detection result.

9. In a virtual defect image generation system for image-based defect detection, An image preprocessing unit for annotating at least one defective region present on a target region within an actual defective image; A defect patch extraction unit that extracts at least one defect patch by differentially processing the inpainted image from the actual defect image after inpainting at least one annotated defect area; and A virtual defect image generation system comprising an image generation unit that generates at least one virtual defect image by synthesizing at least one defect patch at a certain location on a target area within an actual normal image.

10. In claim 9, The above image generation unit, A virtual defect image generation system that generates a virtual defect image by specifying a synthesis location within the target area according to the combination type of the defect patch, applying elastic deformation to the defect patch, and then synthesizing the defect patch at the specified synthesis location.

11. In claim 9, A virtual defect image generation system further comprising a test unit that trains a defect detection model based on a plurality of generated virtual defect images, applies an actual defect image to the trained defect detection model to detect at least one of a defect area and a defect type, and performs a performance evaluation on the detection result to evaluate the quality of the virtual defect image.

Images