Generating minority class defect detection data from visual inspection dataset using self-supervised defect generator

Abstract

Methods, systems, and computer-readable storage media for a defect detection system that generates synthetic defect data representing real-world defects in a product. The synthetic defect data is included in training data for training a defect detection model. A defect detection model is deployed to a production environment to detect defects in a product.

Description

Background Art

[0001] Defect detection is performed during the manufacturing process in an effort to ensure that defective products do not reach the market. With the advancement of computer vision technology, automated visual inspection has been achieved through the use of machine learning (ML) models. For example, a defect detection model can be trained on a visual inspection (VI) dataset to identify the category (e.g., type) and location of defects on a product. However, the data-hungry nature of ML models requires a large VI dataset with a significant sample size for each defect class in order to train a usable ML model. The difficulty in obtaining a sufficient number of training images depicting defects for each of the multiple defect classes makes training a robust defect detection model challenging. Such a low defect sample size distribution makes the defect detection model prone to overfitting. Summary of the Invention

[0002] Embodiments of the present invention are directed to a defect detection system for training and deploying a defect detection model. More specifically, embodiments of the present invention are directed to a defect detection system that generates synthetic defect data representing real-world defects in products. The synthetic defect data is included in training data used to train the defect detection model. The defect detection model is deployed in a production environment to detect defects in products.

[0003] In some embodiments, the actions include: generating a first context image set from a set of positive images, each positive image representing one or more defects present in a product, each context image in the first context image set including at least one cropped region that crops out the defects represented in the corresponding positive image, using the context image set to train a defect generation model, the defect generation model being trained to inpaint synthetic defects within the cropped region, generating a second context image set from a set of negative images, each negative image representing a product without any defects, each context image in the second context image set including at least one cropped region that represents a target defect location, providing a synthetic defect dataset by processing the second context image set through the defect generation model, for each context image in the second context image set, the defect generation model inpainting the synthetic defects within the at least one cropped region to provide a synthetic defect image, training a defect detection model using training data including at least a portion of the synthetic defect dataset, and performing defect detection on the product by processing images of the product through the defect detection model. Other embodiments of this aspect include corresponding systems, apparatus, and computer programs configured to perform the actions of the method encoded on a computer storage device.

[0004] These and other embodiments may each optionally include one or more of the following features: generating context images in a first context image set based on text embeddings representing text indicating a target defect category; training a defect generation model based on text embeddings representing text indicating a target defect category; generating a second context image set from a negative image set comprising: providing at least one bounding box for each negative image, the at least one bounding box indicating a location within the negative image to be repaired, and cropping an area of the bounding box to provide at least one cropped area; providing a synthetic defect dataset by processing the second context image set through a defect generation model comprising: providing text embeddings representing text indicating a target defect category to the defect generation model, the defect generation model repairing a synthetic defect corresponding to the target defect category in each of at least one cropped area of the context images of the second context image set; the synthetic defect dataset comprises synthetic defect images representing multiple defect categories; and the training data also includes at least a portion of the negative image set.

[0005] The present disclosure also provides a computer-readable storage medium coupled to one or more processors and having instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform operations according to embodiments of the method provided herein.

[0006] The present disclosure also provides a system for implementing the methods provided herein. The system includes one or more processors and a computer-readable storage medium coupled to the one or more processors, on which instructions are stored, which, when executed by the one or more processors, cause the one or more processors to perform operations according to the embodiments of the methods provided herein.

[0007] It should be understood that the method according to the present disclosure may include any combination of aspects and features described herein. That is, the method according to the present disclosure is not limited to the combination of aspects and features specifically described herein, but also includes any combination of aspects and features provided.

[0008] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other features and advantages of the disclosure will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 Depicted is an example architecture that may be used to operate embodiments of the present disclosure.

[0010] Figure 2 Depicted is an example conceptual architecture for training according to embodiments of the present disclosure.

[0011] Figure 3Depicted is an example conceptual architecture for generating training data according to embodiments of the present disclosure.

[0012] Figure 4 Depicted are example processes that may be performed in accordance with implementations of the present disclosure.

[0013] Figure 5 is a schematic diagram of an example computer system that can be used to run embodiments of the present disclosure.

[0014] Like reference numbers in the various drawings represent like elements. DETAILED DESCRIPTION

[0015] Embodiments of the present invention relate to a defect detection system for training and deploying a defect detection model. More specifically, embodiments of the present invention relate to a defect detection system that generates synthetic defect data representing real-world defects in products. The synthetic defect data is included in the training data used to train the defect detection model. The defect detection model is deployed in a production environment to detect defects in products.

[0016] An embodiment may include the following actions: generating a first context atlas from a positive image set, wherein each positive image represents one or more defects present in a product, each context image in the first context image set includes at least one cropped region that crops out the defects represented in the corresponding positive image, training a defect generation model using the context image set, the defect generation model being trained to inpaint synthetic defects within the cropped region, generating a second context image set from a negative image set, wherein each negative image represents a product without any defects, each context image in the second context image set includes at least one cropped region that represents a target defect location, providing a synthetic defect dataset by processing the second context image set through the defect generation model, for each context image in the second context image set, the defect generation model inpaints synthetic defects within the at least one cropped region to provide a synthetic defect image, training a defect detection model using training data including at least a portion of the synthetic defect dataset, and running defect detection on the product by processing images of the product through the defect detection model.

[0017] To provide further context for embodiments of the present invention, and as described above, defect detection is performed during the manufacturing process in an effort to ensure that defective products do not reach the market. Defect detection can be described as the problem of identifying, locating, and classifying defective areas on a product, and is typically performed during the visual inspection stage of the supply chain. Visual inspection can be described as the process of inspecting products in a production line to identify defects for quality control.

[0018] With the advancement of computer vision technology, automated visual inspection has been enabled through the use of machine learning (ML) models, such as deep neural networks (DNNs). For example, defect detection models can be trained on visual inspection (VI) datasets to identify the category (e.g., type) and location of defects on products. VI datasets include images of products that can be labeled for training (e.g., defective, non-defective). However, the data-hungry nature of DNNs requires large VI datasets with significant sample sizes for each defect category in order to train DNNs. The inherent characteristics of VI datasets make training DNNs challenging. For example, most images of real-world products do not contain defects (e.g., a product may have a defect rate of less than 0.0001%). Consequently, the relatively small number of available images representing defective products leads to a low defect sample size distribution in VI datasets. The difficulty in obtaining training images depicting defects makes training robust defect detection models (e.g., DNNs) challenging. This low defect sample size distribution makes defect detection models prone to overfitting and computational inefficiency.

[0019] In view of the above context, embodiments of the present invention provide a defect detection system that generates synthetic defect data that is included in training data for training a defect detection model. More specifically, the defect detection system of the present invention uses self-supervised image inpainting on an image without defects to generate synthetic defect data as a labeled image of a product with defects. That is, an image of a real-world product without defects is modified to provide a synthetic defect image that depicts one or more defects using self-supervised image inpainting. The synthetic defect image is used as synthetic defect data. In some embodiments, and as described in further detail herein, inpainting is achieved using a denoised diffusion probability (DDPM)-based model. The synthetic defect data is used as training data to train a defect detection model. The synthetic defect data provides a defect sample size distribution (within the training data) that addresses technical issues (e.g., overfitting) of previous approaches.

[0020] Figure 1 An example system 100 is depicted in which embodiments of the present disclosure may operate. Example system 100 includes a computing device 102, a backend system 104, and a network 106. In some examples, network 106 includes a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, and connects websites, devices (e.g., computing device 102), and backend systems (e.g., backend system 108). In some examples, network 106 can be accessed via wired and / or wireless communication links.

[0021] In some examples, computing device 102 may include any suitable type of computing device, such as a desktop computer, a laptop computer, a handheld computer, a tablet computer, a personal digital assistant (PDA), a cellular telephone, a network device, a camera, a smart phone, an Enhanced General Packet Radio Service (EGPRS) mobile phone, a media player, a navigation device, an email device, a game console, or a suitable combination of any two or more of these devices or other data processing devices.

[0022] In the depicted example, backend system 104 includes at least one server system 108. In some examples, at least one server system 108 hosts one or more computer-implemented services that a user can interact with using a computing device. For example, server system 108 can host a defect detection system according to an embodiment of the present invention.

[0023] exist Figure 1 In the example of FIG. 1 , a camera 120 and an object 122 are depicted. Camera 120 can be any suitable type of camera (e.g., a video camera) that generates images representing an object, such as object 122. In the context of the present disclosure, camera 120 can generate images as digital data representing object 122. Camera 120 can capture images of each side of object 122, such as the front, back, left, right, top, and bottom sides of object 122. In some examples, multiple cameras 120 mounted at different angles can be provided to capture images of each side of object 122. In some examples, object 122 can be rotated so that camera 120 can capture images of each side of object 122.

[0024] According to embodiments, the images may be processed by a defect detection system to determine whether object 122 (as represented within the images) includes any defects. In some examples, the defect detection system runs on computing device 102. In some examples, the defect detection system runs in backend system 104. As described in further detail herein, the defect detection system runs a defect detection model (e.g., an ML model, such as a DNN) trained using synthetic defect data generated according to embodiments of the present invention. The defect detection model may process one or more images of object 122 to classify the images into one or more categories from a category group. Example categories may include, but are not limited to, no defect, defect type 1 (e.g., missing screw), defect type 2 (e.g., damaged component), and so on.

[0025] Figure 2An example conceptual architecture 200 for training according to an embodiment of the present disclosure is depicted. In the depicted example, the conceptual architecture 200 includes an input generator 202, a text encoder 204, a training system 206, and a positive datastore 208. The positive datastore 208 stores actual defect images that depict real-world defects in products. In the context of the present invention, the actual defect images are referred to as positive data because they are positive examples of defects. In some examples, the actual defect images are provided from a supply chain system 210 that includes a digital manufacturing subsystem 212. The example supply chain system can include, but is not limited to, SAP Supply Chain Management (SCM) provided by SAP SE of Walldorf, Germany.

[0026] According to an embodiment of the present disclosure, the defect generator model 220 is trained using positive data provided from the positive data store 208. In some examples, a positive data set is provided and can represent multiple defect categories (e.g., missing fasteners, damaged parts). For example, the positive data set includes actual defect images representing multiple defect categories. Each actual defect image, such as actual defect image 230, depicts a real-world product with a defect. The actual defect image 230 further includes a bounding box that defines the location of the defect. Figure 2 In the non-limiting example of , actual defect image 230 depicts a product that should have fasteners fastened in multiple locations, but one of the fasteners is missing. In this example, bounding box 234 defines the location of the missing fastener.

[0027] In some embodiments, text 232 is provided and indicates the defect category on which the defect generator model 220 is to be trained. For example, text 232 may be "missing fastener," indicating that the defect generator model 220 is to be trained to generate synthetic defect data (images) depicting a product with missing fastener(s). The text encoder 204 encodes the text 232. For example, the text encoder 204 may process the text 232 to generate an embedding, which is a multidimensional vector representing the text 232 in an embedding space. In some examples, the text encoder 204 may be provided as a pre-trained embedding model. Example embedding models may include, but are not limited to, Bidirectional Encoder Representations from Transformers (BERT) models. Example BERT models may include, but are not limited to, Sentence-BERT models, including Siamese BERT networks. However, it is contemplated that any suitable embedding model may be used.

[0028] In some embodiments, the input generator 202 receives positive data, such as actual defect images 230 and text embeddings. In some examples, the input generator 202 filters defects in the actual defect images to identify defects from a target defect category represented by the text embedding (e.g., based on text 232), and obtains the category and bounding box data (e.g., pixel coordinates corresponding to the bounding box). In some examples, each actual defect image stored in the positive data store 208 (such as the actual defect image 230) is a labeled image including one or more labels, each label indicating a target defect category, and including one or more bounding boxes, each bounding box corresponding to a label and including bounding box coordinates within the image. The input generator 202 can filter the labels to find images with the target defect category. In some embodiments, the input generator 202 crops defects from the actual defect images to provide training images. Figure 2 In the example of , input generator 202 crops the area defined by bounding box 234 to provide context image 236 including cropped area 238 .

[0029] According to an embodiment of the present disclosure, the defect generator model 220 is trained on a set of context images determined for a target defect category. For example, multiple context images, such as context image 236, can be generated for a target defect category represented by text 232 (e.g., missing fastener). In some embodiments, the defect generator model 220 is a DDPM-based model, which can be described as a parameterized Markov chain trained using variational inference to generate synthetic defect data in the context of the present disclosure. That is, for example, the defect generator model 220 is trained to inpaint a cropped region within a context image, such as cropped region 238 of context image 236, using the original target defect pixels. Figure 2 In the example of , defect generator model 220 is trained to inpaint pixels depicting missing fasteners within cropped region 238 of context image 236. Defect generator model 220 is deployed to generate synthetic defect data, as described in further detail herein.

[0030] In more detail, during training and along with other training data, the inputs to the defect generation model (in training) are the text embeddings (the output of text encoder 204) and the context image 236 (the image with the defect cropped out). The output of the defect generation model (in training) is an image with the cropped portion filled in to depict the defect. The goal is to make the output identical to the original image (e.g., the defect depicted in bounding box 234). In short, the defect generation model is trained to fill the cropped region 238 with the defect depicted in bounding box 234. This is accomplished through multiple training iterations using multiple training samples (e.g., tuples of [text embedding, context image, actual defect image]). In some examples, the defect generator model can be trained on multiple defect categories, with the text embeddings guiding training on the respective defect categories.

[0031] Figure 3 An example conceptual architecture 300 for generating synthetic defect data to be used as training data, according to embodiments of the present disclosure, is depicted. In the depicted example, conceptual architecture 300 includes a target defect location identification module 302, an input generation module 304, a synthetic defect data generation (SDDG) module 306, and a negative datastore 308. Conceptual architecture 300 further includes text encoder 204 and supply chain system 210. As described in further detail herein, SDDG module 306 runs defect generator model 220 to generate synthetic defect data from a negative dataset representing real-world products without defects. The synthetic defect data can be used to train a defect detection model, which is then deployed for inference to detect defects in products.

[0032] More specifically, negative data storage 308 stores actual images depicting real-world products without defects (e.g., non-defective products). In the context of the present invention, actual images are referred to as negative data because they do not depict defects. In some examples, the actual images are provided from supply chain system 210.

[0033] In some embodiments, for a target defect class, the target defect location identification module 302 automatically identifies a target defect location in the actual image from the negative data store 308. In some examples, the target defect class is represented by text 312, which indicates the class of defect that the synthetic defect data is to depict. For example, the text 312 may be "missing fastener," which indicates that the synthetic defect data is to depict a product with missing fastener(s). The text encoder 204 encodes the text 312. For example, and as described above, the text encoder 204 may process the text 312 to generate an embedding as a multi-dimensional vector representing the text 312 in an embedding space. In some examples, the target defect location identification module 302 processes the actual image in view of the text embedding to identify a target defect location, which is represented by a bounding box within which the synthetic defect is to be drawn. The bounding box variable is obtained from a dataset with existing positive data (e.g., from Figure 2 The positive data storage 208 (positive data) is sampled from a multivariate normal distribution with a mean value and a standard deviation.

[0034] exist Figure 3 In the example shown in FIG3 , target defect location identification module 302 receives actual image 330, which depicts a real-world product without any defects. Based on the text embeddings of text 312, target defect location identification module 302 identifies the target defect location within actual image 330 and provides a modified image 332 that includes a bounding box 334 at the target defect location. Modified image 332 is processed by input generation module 304, which generates a cropped image 336. For example, input generation module 304 crops an area of bounding box 334 to provide cropped image 336, which includes cropped area 338. In this manner, non-defective features (e.g., fasteners) are cropped from actual image 330.

[0035] In more detail, the multivariate normal distribution can be provided as four (4) dimensions (e.g., the x-coordinate, y-coordinate, width, and height of the target defect). Each defect type will have a normal distribution describing the possible locations of the defect, and this information is stored along with the corresponding defect name. Given the name of the defect, the normal distribution can be retrieved and sampled to obtain a multidimensional vector (e.g., x coord, y coord, width, height) representing the defect location (e.g., as a bounding box). This defect location is used to crop out the defect to obtain a cropped region 338 from the bounding box 334 (which contains the non-defective portion).

[0036] In some embodiments, the SDDG module 306 receives the text embedding and the cropped image 336. The SDDG module 306 processes the text embedding and the cropped image 336 via the defect generation model 220, where the defect generation model 220 generates a synthetic defect image 320 that depicts a defect 340 representing the defect class text 312. More specifically, the defect generation model 220 inpaints the defect corresponding to the text 312 within the cropped region 338 of the cropped image 336 to provide the synthetic defect image 320. Although Figure 3 The example depicts a bounding box around defect 340, but it is contemplated that synthetic defect image 320 need not include the bounding box depicted therein. In some implementations, multiple actual images (such as actual image 330) can be processed to generate multiple synthetic defect images (such as synthetic defect image 320) to represent the defect category indicated by text 312 (e.g., missing screw).

[0037] In some embodiments, the defect generation model 220 can be used to generate multiple synthetic defect datasets. For example, a first synthetic defect dataset can be generated based on first text indicating a first target defect category (e.g., missing fastener), and a second synthetic defect dataset can be generated based on second text indicating a second target defect category (e.g., damaged part). In this way, the synthetic defect data can represent multiple categories of defects that may occur in a product.

[0038] According to embodiments of the present invention, synthetic defect data (synthetic defect images) can be used to train defect detection models. For example, synthetic defect data can be included in the training data used to train the defect detection model. In some examples, the training data includes synthetic defect data and real-world non-defect data. In some examples, the real-world non-defect data can include actual images depicting negative examples (e.g., actual images of real-world products without defects). In some examples, the synthetic defect data can include multiple synthetic defect data sets to represent multiple categories of defects that may occur in the products within the training data. According to embodiments of the present invention, a sufficient amount of synthetic data can be included in the training data to provide an appropriate distribution of defect sample sizes within the training data.

[0039] In some examples, a defect detection model is iteratively trained on training data, wherein during each iteration, one or more parameters of the defect detection model are adjusted and an output is generated based on the training data. For each iteration, a loss value is determined based on a loss function. The loss value represents the accuracy of the output of the defect detection model for the corresponding iteration. The loss value can be described as a representation of the degree of difference between the input to the defect detection model and the expected output of the defect detection model. In some examples, if the loss value does not meet the expected value (e.g., is not equal to zero), the parameters of the defect detection model are adjusted in another iteration of training. In some instances, this process is repeated until the loss value meets the expected value or a certain number of training epochs (iterations) have been performed.

[0040] After training, the defect detection model is deployed for inference to detect defects in products. In some examples, the defect detection model can be incorporated into a defect detection system that processes images of real-world products to determine whether the product includes defects. For example, and with reference to Figure 1 One or more images of object 122 generated by camera 120 can be processed via a defect detection model of a defect detection system to determine whether the product includes any defect(s). In some examples, the defect detection model outputs a class from a class set, which can include, but is not limited to, no defect, missing fastener, and damaged component, among other possible classes.

[0041] Figure 4 An example process 400 is depicted that may be performed in accordance with implementations of the present disclosure. In some examples, the example process 400 is provided using one or more computer-executable programs executed by one or more computing devices.

[0042] Generate a context image from a positive dataset (402). For example, and as referenced herein Figure 2 As described in detail herein, context images, such as context image 236, are generated by input generator 202 for a target defect category represented in text 232. A defect generator model is trained (404). For example, and as described in detail herein, a context image set (e.g., a plurality of context images corresponding to the target defect category) is used to train defect generator model 220.

[0043] The context image is provided by cropping the negative image (406). For example, and as herein referred to Figure 3In detail, the target defect location identification module 302 receives an actual image 330 depicting a real-world product without any defects, and based on the text embedding of the text 312, the target defect location identification module 302 identifies the target defect location within the actual image 330 and provides a modified image 332 including a bounding box 334 at the target defect location. The modified image 332 is processed by the input generation module 304, which generates a cropped image 336.

[0044] Generate synthetic defect data using the defect generator model (408). For example, and as described herein with reference to Figure 3 As described in detail, the SDDG module 306 receives the text embedding and the cropped image 336 and processes the text embedding and the cropped image 336 via the defect generation model 220 , which generates a synthetic defect image 320 depicting a defect 340 representing the defect class indicated by the text 312 .

[0045] Training data is provided (410). For example, and as described in detail herein, synthetic defect data can be included in the training data used to train a defect detection model. In some examples, the training data includes the synthetic defect data and real-world non-defect data. In some examples, the synthetic defect data can include multiple synthetic defect data sets to represent multiple defect categories that may occur for products within the training data. A defect detection model is trained (412). For example, and as described in detail herein, the defect detection model is iteratively trained on the training data. A defect detection model is deployed (414). For example, and as described in detail herein, the defect detection model is deployed for inference to detect defects in the product. In some examples, the defect detection model can be incorporated into a defect detection system that processes images of real-world products to determine whether the product includes defects.

[0046] Embodiments of the present disclosure provide one or more technical advantages. For example, embodiments of the present invention generate synthetic defect data (positive images depicting defects) without any additional annotation cost. Synthetic defect images are labeled with the target defect category and sampled bounding box locations are used to derive text embeddings and context images. This addresses the data imbalance issue in the VI dataset, where positive data samples are significantly underrepresented. This also enables defect detection models to be trained in a manner that better learns to distinguish between positive images (defects) and negative images (no defects), enabling deployment for real-world defect detection. Furthermore, the defect generation model of the present invention is based on DDPM, providing the best-performing generative model for image generation. Results show that DDPM-based models for image inpainting can generate more semantically meaningful inpaint regions than other methods. As another advantage, the defect generation model is trained with image inpainting objectives rather than generating the full image. That is, image inpainting is performed within a cropped region of the image rather than generating the full image. Image inpainting ensures that the context region follows the distribution of the training set, and the inpainting task is easier to train than the image generation task. By training using both the generated positive images (synthetic defect images) and the corresponding original negative images, the defect detection model can focus on areas likely to contain defects (e.g., the location of fasteners). In this way, the defect detection model is more computationally efficient (saving technical resources) and more accurate than other methods of training the defect detection model.

[0047] Now refer to Figure 5 , provides a schematic diagram of an example computing system 500. System 500 can be used for the operations described in association with the embodiments described herein. For example, system 500 can be included in any or all of the server components discussed herein. System 500 includes a processor 510, memory 520, storage device 530, and input / output device 540. Components 510, 520, 530, 540 are interconnected using a system bus 550. Processor 510 is capable of processing instructions for execution within system 500. In some embodiments, processor 510 is a single-threaded processor. In some embodiments, processor 510 is a multi-threaded processor. Processor 510 is capable of processing instructions stored in memory 520 or on storage device 530 to display graphical information for a user interface on input / output device 540.

[0048] Memory 520 stores information within system 500. In some embodiments, memory 520 is a computer-readable medium. In some embodiments, memory 520 is a volatile memory unit. In some embodiments, memory 520 is a non-volatile memory unit. Storage device 530 can provide mass storage for system 500. In some embodiments, storage device 530 is a computer-readable medium. In some embodiments, storage device 530 can be a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device. Input / output device 540 provides input / output operations for system 500. In some embodiments, input / output device 540 includes a keyboard and / or a pointing device. In some embodiments, input / output device 540 includes a display unit for displaying a graphical user interface.

[0049] The described features can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. The apparatus can be implemented in a computer program product tangibly embodied in an information carrier (e.g., in a machine-readable storage device for execution by a programmable processor), and the method steps can be performed by the programmable processor executing the program of instructions to perform the functions of the described embodiments by operating on input data and generating output. The described features can advantageously be implemented in one or more computer programs executable on a programmable system comprising at least one programmable processor coupled to receive data and instructions from and send data and instructions to a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used, directly or indirectly, in a computer to perform an activity or bring about a result. A computer program can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0050] For example, suitable processors for executing a program of instructions include both general-purpose and special-purpose microprocessors, as well as the sole processor or one of multiple processors in any type of computer. Typically, a processor will receive instructions and data from read-only memory or random-access memory, or both. Elements of a computer may include a processor for executing instructions and one or more memories for storing instructions and data. Typically, a computer may also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of nonvolatile memory, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into an ASIC (Application Specific Integrated Circuit).

[0051] To provide for interaction with a user, these features may be implemented on a computer having a display device (such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (such as a mouse or trackball) through which the user can provide input to the computer.

[0052] These features can be implemented in a computer system that includes back-end components (such as a data server), or includes middleware components (such as an application server or an internet server), or includes front-end components (such as a client computer with a graphical user interface or an internet browser), or any combination thereof. The components of the system can be connected by any form or medium of digital data communication (such as a communication network). Examples of communication networks include, for example, a LAN, a WAN, and the computers and networks that form the Internet.

[0053] Computer systems can include clients and servers. A client and server are generally remote from each other and typically interact through a network, such as the one described. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0054] Furthermore, the logic flows depicted in the accompanying drawings do not require the particular order shown or sequential sequence to achieve the desired results. Furthermore, other steps may be provided, or steps may be eliminated from the described flows, and other components may be added to or removed from the described systems. Accordingly, other implementations are within the scope of the following claims.

[0055] A number of embodiments of the present disclosure have been described. However, it will be appreciated that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the appended claims.

Claims

1. A computer-implemented method for detecting defects in a product, the method being executed by one or more processors and comprising: generating a first context image set from a set of positive images, each positive image representing one or more defects present in a product, each context image in the first context image set including at least one cropping region that crops out the defects represented in the corresponding positive image; training a defect generation model using the context image set, wherein the defect generation model is trained to repair synthetic defects within the cropped region; generating a second context image set from the negative image set, each negative image representing a product without any defects, each context image in the second context image set including at least one cropped region, the at least one cropped region representing a target defect location; providing a synthetic defect dataset by processing the second set of context images through the defect generation model, for each context image in the second set of context images, the defect generation model inpainting a synthetic defect in the at least one cropped region to provide a synthetic defect image; training a defect detection model using training data comprising at least a portion of the synthetic defect dataset; as well as Defect detection of the product is performed by processing an image of the product through the defect detection model.

2. The method according to claim 1, wherein Context images in the first set of context images are generated based on text embeddings representing text indicating a target defect category.

3. The method according to claim 1, wherein The defect generation model is trained based on text embeddings that represent text that indicates a target defect class.

4. The method according to claim 1, wherein Generating a second set of context images from the set of negative images includes providing at least one bounding box for each negative image, the at least one bounding box indicating a location of a defect to be repaired within the negative image, and cropping a region of the bounding box to provide the at least one cropped region.

5. The method according to claim 1, wherein Providing a synthetic defect dataset by processing the second context image set through the defect generation model includes: providing a text embedding to the defect generation model, the text embedding representing text indicating a target defect category, and the defect generation model inpainting a synthetic defect corresponding to the target defect category in each of the at least one cropped region of the context image of the second context image set.

6. The method according to claim 1, wherein The synthetic defect dataset includes synthetic defect images representing a plurality of defect categories.

7. The method according to claim 1, wherein The training data also includes at least a portion of the set of negative images.

8. A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations for detecting defects in a product, the operations comprising: generating a first context image set from a set of positive images, each positive image representing one or more defects present in a product, each context image in the first context image set including at least one cropping region that crops out the defects represented in the corresponding positive image; training a defect generation model using the context image set, wherein the defect generation model is trained to repair synthetic defects within the cropped region; generating a second context image set from the negative image set, each negative image representing a product without any defects, each context image in the second context image set including at least one cropped region, the at least one cropped region representing a target defect location; providing a synthetic defect dataset by processing the second set of context images through the defect generation model, for each context image in the second set of context images, the defect generation model inpainting a synthetic defect in the at least one cropped region to provide a synthetic defect image; training a defect detection model using training data comprising at least a portion of the synthetic defect dataset; as well as Defect detection of the product is performed by processing an image of the product through the defect detection model.

9. The non-transitory computer-readable storage medium of claim 8, wherein: Context images in the first set of context images are generated based on text embeddings representing text indicating a target defect category.

10. The non-transitory computer-readable storage medium of claim 8, wherein: The defect generation model is trained based on text embeddings that represent text that indicates a target defect class.

11. The non-transitory computer-readable storage medium of claim 8, wherein: Generating a second set of context images from the set of negative images includes providing at least one bounding box for each negative image, the at least one bounding box indicating a location of a defect to be repaired within the negative image, and cropping a region of the bounding box to provide the at least one cropped region.

12. The non-transitory computer-readable storage medium of claim 8, wherein: Providing a synthetic defect dataset by processing the second context image set through the defect generation model includes: providing a text embedding to the defect generation model, the text embedding representing text indicating a target defect category, and the defect generation model inpainting a synthetic defect corresponding to the target defect category in each of the at least one cropped region of the context image of the second context image set.

13. The non-transitory computer-readable storage medium of claim 8, wherein: The synthetic defect dataset includes synthetic defect images representing a plurality of defect categories.

14. The non-transitory computer-readable storage medium of claim 8, wherein: The training data also includes at least a portion of the set of negative images.

15. A system comprising: computing devices; as well as a computer-readable storage device coupled to the computing device and having instructions stored thereon that, when executed by the computing device, cause the computing device to perform operations for detecting defects in a product, the operations comprising: generating a first context image set from a set of positive images, each positive image representing one or more defects present in a product, each context image in the first context image set including at least one cropping region that crops out the defects represented in the corresponding positive image; training a defect generation model using the context image set, wherein the defect generation model is trained to repair synthetic defects within the cropped region; generating a second context image set from the negative image set, each negative image representing a product without any defects, each context image in the second context image set including at least one cropped region, the at least one cropped region representing a target defect location; providing a synthetic defect dataset by processing the second set of context images through the defect generation model, for each context image in the second set of context images, the defect generation model inpainting a synthetic defect in the at least one cropped region to provide a synthetic defect image; training a defect detection model using training data comprising at least a portion of the synthetic defect dataset; and Defect detection of the product is performed by processing an image of the product through the defect detection model.

16. The system according to claim 15, wherein: Context images in the first set of context images are generated based on text embeddings representing text indicating a target defect category.

17. The system according to claim 15, wherein: The defect generation model is trained based on text embeddings that represent text that indicates a target defect class.

18. The system according to claim 15, wherein: Generating a second set of context images from the set of negative images includes providing at least one bounding box for each negative image, the at least one bounding box indicating a location of a defect to be repaired within the negative image, and cropping a region of the bounding box to provide the at least one cropped region.

19. The system of claim 15, wherein: Providing a synthetic defect dataset by processing the second context image set through the defect generation model includes: providing a text embedding to the defect generation model, the text embedding representing text indicating a target defect category, and the defect generation model inpainting a synthetic defect corresponding to the target defect category in each of the at least one cropped region of the context image of the second context image set.

20. The system of claim 15, wherein the synthetic defect dataset comprises synthetic defect images representing a plurality of defect classes.

Images