Controlled defect enhancement via text and image guided diffusion model

Abstract

The invention relates to controlled defect enhancement via text and image guided diffusion models. A machine learning (ML) system includes a visual language model (VLM) and a diffusion model. Before the diffusion model is trained by using the data pair, the VLM is finely tuned. The data pair includes image data displaying anomalies and text data describing the image data. The trimmed VLM includes an image encoder that generates an image embedding using image data and a text encoder that generates a text embedding using text data. Semantic sub-codes are generated using image embedding and text embedding. The diffusion model generates random sub-codes using the image data. The diffusion model generates a reconstructed image using the random sub-codes and the semantic sub-codes. The loss is optimized based on an expected value of a difference between a predicted noise of a noise instance of the image data at a particular time and an actual noise of the noise instance. The loss is used to update parameters of the diffusion model.

Description

Controlled Defect Enhancement of Diffusion Model Guided by Text and Images Technical Field

[0001] This disclosure generally relates to computer vision, and more particularly to controlled defect enhancement via diffusion models guided by text and images. Background Technology

[0002] A major challenge in training efficient anomaly detection models is the lack of balanced datasets that adequately represent both normal and defective images. For example, in manufacturing environments, defective images are less available and less diverse. This scarcity of defective images in manufacturing environments poses a challenge to training anomaly detection models in these settings.

[0003] Furthermore, traditional defect augmentation methods with generative models may be biased towards their training data. They often suffer from mode collapse, where they consistently generate overly similar outputs and fail to produce diverse, realistic images, which limits their utility in generating effective augmentation datasets for defective images. Summary of the Invention

[0004] The following is an overview of specific embodiments described in detail below. The described aspects are presented merely to provide the reader with a brief overview of these specific embodiments, and the description of these aspects is not intended to limit the scope of this disclosure. In fact, this disclosure may cover various aspects that may not be explicitly set forth below.

[0005] According to at least one aspect, the computer-implemented method involves training at least one diffusion model using a training dataset comprising data pairs. The data pairs include at least a first data pair. The first data pair includes at least (i) image data displaying anomalies and (ii) text data describing the corresponding image data containing the anomalies. The method includes generating image embeddings using pixels of the image data via an image encoder. The method includes generating text embeddings using text data via a text encoder. The method includes generating semantic subcodes using the image embeddings and text embeddings. The method includes generating random subcodes using pixels of the image data via the diffusion model. The method includes generating reconstructed image data using the random subcodes and semantic subcodes via the diffusion model. The reconstructed image data is a reconstruction of the image data via the diffusion model. The method includes optimizing a loss based on the expected value of the difference between the predicted noise of a noisy image at a specific time and the actual noise of the noisy image at that specific time during the generation of the reconstructed image data. The method includes updating the parameters of the diffusion model using the loss.

[0006] According to at least one aspect, the system includes at least one processor and at least one computer memory that communicates data with one or more processors. The one or more computer memories have computer-readable data stored thereon. The computer-readable data includes instructions that, when executed by the one or more processors, cause the one or more processors to perform a method for training at least a diffusion model using a training dataset comprising data pairs. The data pairs include at least a first data pair. The first data pair includes at least (i) image data displaying an anomaly and (ii) text data describing the corresponding image data including the anomaly. The method includes generating an image embedding using pixels of the image data via an image encoder. The method includes generating a text embedding using text data via a text encoder. The method includes generating a semantic subcode using the image embedding and the text embedding. The method includes generating a random subcode using pixels of the image data via a diffusion model. The method includes generating reconstructed image data using the random subcode and the semantic subcode via a diffusion model. The reconstructed image data is a reconstruction of the image data via the diffusion model. The method includes optimizing a loss based on an expected value of the difference between the predicted noise of a noisy image at a specific time and the actual noise of the noisy image at that specific time during the generation of the reconstructed image data. The method includes updating the parameters of the diffusion model using the loss.

[0007] According to at least one aspect, a computer-implemented method relates to generating a dataset for training a machine learning model. The method includes receiving a source image having non-anomaly source image data. The method includes receiving text input describing (i) an anomaly to be generated on the source image and (ii) at least one attribute of the anomaly. The method includes generating a source image embedding using pixels of the source image via an image encoder. The method includes generating a text input embedding using the text input via a text encoder. The method includes generating a semantic subcode using the source image embedding and the text input embedding. The method includes generating a random subcode using the source image via a diffusion model. The method includes generating a synthetic image using the random subcode and the semantic subcode via a diffusion model. The synthetic image displays the source image having the anomaly described as the text input. The dataset includes at least the synthetic image. The dataset is configured to train a machine learning model for anomaly detection. For example, the machine learning model could be an image classifier that classifies digital images as anomalous or non-anomaly.

[0008] These and other features, aspects, and advantages of the invention are discussed in the following detailed description with reference to the accompanying drawings. Similar characters denote similar or analogous parts throughout the drawings. Furthermore, the drawings are not necessarily drawn to scale, as some features may be enlarged or minimized to show detail of specific components. Attached Figure Description

[0009] Figure 1 is a diagram illustrating various aspects of fine-tuning a pre-trained visual language model according to at least one example embodiment of the present disclosure.

[0010] Figure 2 is a diagram illustrating various aspects of a training conditional diffusion model according to at least one example embodiment of the present disclosure.

[0011] Figure 3 is a diagram illustrating various aspects of an image synthesis process via a trained defect enhancement model according to at least one example embodiment of the present disclosure.

[0012] Figure 4 is a diagram of an example system with a controlled defect enhancer according to at least one exemplary embodiment of the present disclosure.

[0013] Figure 5 illustrates a flowchart of an example of a process for generating a dataset via a controlled defect enhancer according to at least one example embodiment of the present disclosure.

[0014] Figure 6 is a diagram of a system according to at least one example embodiment of the present disclosure, the system including a machine learning model trained on a dataset generated by a controlled defect enhancer.

[0015] Figure 7 is a diagram of the system of Figure 6 with respect to manufacturing technology according to at least one example embodiment of the present disclosure.

[0016] Figure 8 is a diagram of the system of Figure 6 with respect to imaging technology according to at least one example embodiment of the present disclosure. Detailed Implementation

[0017] From the foregoing description, one will understand the embodiments described herein (which have been shown and described as examples) and their many advantages, and it will be apparent that various changes can be made to the form, construction, and arrangement of the components without departing from the disclosed subject matter or sacrificing one or more of its advantages. In fact, the forms described in these embodiments are merely illustrative. These embodiments are susceptible to various modifications and alternatives, and the following claims are intended to cover and include such changes, and are not limited to the specific forms disclosed, but rather cover all modifications, equivalents, and alternatives falling within the spirit and scope of this disclosure.

[0018] This disclosure addresses the challenges associated with training efficient anomaly detection models due to the lack of balanced datasets that appropriately encompass (i) normal or “non-anomaly images” (e.g., digital images that do not show anomalies / defects) and (ii) “anomaly images” (e.g., digital images that show anomalies / defects). Furthermore, regarding generative models, there can be problems when sampling data unconditionally from generative models, as these models are prone to mode collapse, or they may be biased towards limited variations in the dataset and not always produce the desired diversity across sample instances.

[0019] Recognizing these technical challenges, the embodiments disclosed herein utilize data augmentation techniques to increase the number of defective samples (i.e., anomalous images) in a dataset, while also providing a more controlled manner for generating these defective samples. For example, the embodiments disclosed herein enable users to specify one or more distinct attributes (e.g., location, shape, severity level, etc.) of defects / anomalies to be generated in new samples or synthetic images. This control is achieved by modulating a generative model (such as diffusion model 140) to generate conditional samples. Specifically, the embodiments of this disclosure utilize both image embeddings and corresponding text embeddings derived from a pre-trained and fine-tuned base model (e.g., a contrastive language image pre-trained (CLIP) model or a prior art visual language base model). In doing so, the embodiments implement a guided diffusion model 140 while also providing a data augmentation method for generating new images (e.g., synthetic images) displaying desired defects.

[0020] Figures 1, 2, and 3 illustrate various aspects of a pipeline that conditioned the diffusion model 140 on a combination of image embeddings and corresponding text embeddings extracted from a finely tuned large visual language model (such as the CLIP model). This pipeline is advantageous in that it enables users to sample different images from various product types with desired defects. Furthermore, the embodiments are advantageous in that they construct a human-understandable natural language interface, thereby managing and manipulating various attributes of the generated anomalies with greater accuracy and flexibility, thus enabling and providing a more detailed and manageable way to create anomalous images for data augmentation.

[0021] Figures 1 and 2 illustrate the training process of a machine learning system according to an example embodiment. The machine learning system includes a defect augmentation model 200, which includes a visual language model (VLM) 100 and a diffusion model 140. As an example, in Figure 1, the training process is performed by one or more processors (e.g., the processing system 402 of Figure 4). The training process includes at least (i) fine-tuning the VLM 100 and (ii) training the guided diffusion model 140 for image synthesis. The fine-tuning of the VLM 100 occurs before the training of the guided diffusion model 140 for image synthesis.

[0022] Figure 1 is a diagram of various aspects of the first phase of the training process according to an example embodiment. The first phase of the training process includes fine-tuning a pre-trained VLM 100. For example, in Figure 1, VLM 100 includes a CLIP model. Specifically, VLM 100 includes a pre-trained image encoder 110 and a pre-trained text encoder 120. The pre-trained image encoder 110 is configured to generate an image embedding 14 using pixels from image data 10 of a digital image set. The pre-trained text encoder 120 is configured to generate a text embedding 16 using text data 12. The text data may be associated with digital images.

[0023] The training process uses a fine-tuning dataset, which may include a selected subset of image data with “detailed” annotations. The fine-tuning dataset is used to fine-tune a pre-trained VLM 100 (e.g., a pre-trained CLIP model). This fine-tuning dataset includes data pairs of: (i) digital images with image data 10 and (ii) text data 12 describing the corresponding digital images. As a non-limiting example, in a manufacturing environment, image data 10 may include production data (e.g., digital images of the product) at various stages of the manufacturing process. Regarding digital images, the fine-tuning dataset includes (i) a subset of digital images showing normal and non-abnormal image data (i.e., defect-free image data) and (ii) a subset of digital images showing abnormal image data (e.g., defective image data). For example, in Figure 1, the fine-tuning dataset includes at least a first digital image with first image data 10A, showing a top-plane view of a PEG product with a large defect on the left side. As another example, in Figure 1, the fine-tuning dataset includes at least a second digital image with second image data 10B, showing a top-plane view of a PEG product with a large defect at the top. In these non-limiting examples, the fine-tuning datasets involve digital images of PEG products and corresponding textual data associated with these PEG products. As shown in Figures 1, 2, and 3, the top view of the PEG product includes a sealed fluid ring illuminated by an LED ring.

[0024] Digital images consist of pixels. In digital imaging, a pixel is the smallest addressable element in a raster image, or in a dot-matrix display device. In most digital display devices, a pixel is the smallest element that can be manipulated by software. Each pixel is a sample of the original image, and thus more samples generally provide a more accurate representation of the original image. The intensity of each pixel is variable. For example, in a color imaging system, color is typically represented by three or four component intensities, such as red, green, and blue, or cyan, magenta, yellow, and black.

[0025] Simultaneously, the data pair includes text data 12. Text data 12 includes descriptive text describing the corresponding image data 10 of the data pair. Text data 12 includes details of one or more attributes, particularly those related to anomalies or defects present when present in the corresponding image data 10. For example, the descriptive text may specify the "data type" (i.e., defective or non-defective), the specific "defect" present, the "location" of the defect, the defect level indicating the "severity" of the defect, or any number and combination of the above. As a non-limiting example, in Figure 1, the fine-tuning dataset includes at least first text data 12A describing at least the first image data 10A. First text data 12A includes the following text: "Image of a PEG product with a large defect on the left side." As another example, in Figure 1, the fine-tuning dataset includes at least second text data 12B describing at least the second image data 10B. Second text data 12B includes the following text: "Image of a PEG product with a large defect on the top." In these examples, text data 12 is generated via a prompt such as “image of a PEG product with a [size] defect at [location]”, where [size] refers to a slot used to insert a size indication of the defect when displayed on image data 10, and [location] refers to a slot used to insert a location indication of the defect when displayed on image data 10.

[0026] The advantage of using a pre-trained VLM 100 (e.g., the CLIP model) lies in its ability to quickly adapt to subtle differences in these new product images, even with a limited dataset comprising, for example, only 32 data pairs (i.e., image-text pairs). Fine-tuning the pre-trained VLM 100 (e.g., the CLIP model) follows the same mechanism as the pre-training phase, through a contrastive learning objective. Fine-tuning involves maximizing the similarity between positive pairs (images and their associated text) while minimizing the similarity between negative pairs (text and non-corresponding images, and vice versa).

[0027] Furthermore, for an efficient and logical training process, the fine-tuning dataset is organized into multiple buckets. Each bucket contains data pairs of images and text that share strong similarity based on attributes, thus ensuring semantic similarity between the text and images. When batches are sampled for training, the training process involves selecting at most one sample from each bucket. This sampling method guarantees that the training process will not include two closely related pairs in the same batch when minimizing the similarity between text and its non-matching images (also known as negative pairs). For a given batch of image and text data pairs, the training process first computes image embeddings 14 via image encoder 110 and text embeddings 16 via text encoder 120. Next, dot products 18 are used to determine the similarity scores between all image-text pairs. As a non-limiting example, the image embeddings 14 of image data 10B (e.g., ) and text embeddings of text data 12B 16 (e.g. The data generated by these embeddings are dot products 18 ( Then, the softmax function calculates the probability that each text is paired with its corresponding image, and vice versa. The main objective is to maximize the log probability of the corresponding text-image pairings. This loss drives the VLM 100 to produce image and text embeddings that are close in the language-image embedding space (e.g., CLIP embedding space) for matching pairs and far apart for non-matching pairs.

[0028] Figure 2 is a diagram illustrating various aspects of the second phase of the training process. The second phase of the training process involves training a guided diffusion model 140 for image synthesis. In Figure 2, the guided diffusion model 140 is conditioned on a combination of text data and image data. For this second phase, the training process utilizes a broader and larger training dataset than the fine-tuning dataset. The training dataset consists of data pairs. Each data pair includes (i) a digital image with image data and (ii) text data describing the corresponding image data. The training dataset includes a subset of data pairs of (i) text data and corresponding digital images with non-abnormal image data (e.g., digital images with normal image data but without defects), and another subset of data pairs of (ii) text data and corresponding digital images with anomalous image data (e.g., digital images with image data showing defects).

[0029] The training dataset does not necessarily require detailed annotations for all data samples. At this point, the training process merges smaller, more detailed annotations (e.g., text data) from the fine-tuning dataset of the first stage (Figure 1) with a larger training dataset that can be automatically annotated using labels indicating normal, non-defective images (e.g., "OK" labels) or abnormal, defective images (e.g., "Not OK" labels). This minimal annotation describes the data type and indicates whether the image data of the digital image is normal or defective. As an example, Figure 2 shows a non-limiting example of data pairs comprising (i) a digital image with image data 20 showing a top-plane view of a defective PEG product, and (ii) text data 22 including "Image of a defective PEG product". In these examples, the text data is generated via a prompt such as "[Data Type] Image of a PEG product", where [Data Type] represents a slot used to insert an indication ("Normal" or "OK") that the PEG product is defective or non-defective. The conditional diffusion model 140 is trained using data pairs of "OK" or "Not OK" images and their corresponding text descriptions.

[0030] Image encoder 110 and text encoder 120, along with a tokenizer, are used to extract different semantic subcodes (e.g., embeddings in the CLIP embedding space) from image data and text data, respectively. For example, in Figure 2, image encoder 110 is configured to generate image embedding 24 using pixels from image data 20 of a digital image. Furthermore, in Figure 2, text encoder 120 is configured to generate text embedding 26 using text data 22. A tokenizer (not shown) is associated with text encoder 120. The tokenizer uses text data 22 to generate lexicalized text data for text encoder 120.

[0031] The training process also includes a semantic subcode generator 130. The semantic subcode generator 130 is configured to generate semantic subcode 28 using image embedding 24 and text embedding 26 via Equation 3. In other words, the semantic subcode generator 130 merges these two distinct subcodes (e.g., image embedding 24 and text embedding 26) to generate a unified “semantic” subcode 28, which uses coefficients ranging from 0 to 1. To balance image representation and text representation. In Equation 1, This represents an image input (e.g., a digital image with image data 20), and This represents image embedding, which is achieved through the image encoding function of image encoder 110. ) is generated. In Equation 2, This represents lexicalized text data (e.g., the lexicalized version of text data 22), and This indicates text embedding, which is achieved through the text encoding function of text encoder 120. The training process also includes transmitting the semantic subcode 28 to the diffusion model 140. .

[0032] The diffusion model 140 includes at least a diffusion encoder 140A, a diffusion decoder 140B, and a skip connection 140C. The diffusion encoder 140A includes a UNet architecture. The forward mechanism of the diffusion model 140 acts as a "random" encoder that captures random variations within the image data 20. In Equation 4, This represents random subcode 30, which is generated via the positive mechanism function of diffusion model 140. The denoising UNet in diffusion model 140 is used as a decoder to reconstruct the original image data 20 and generate the reconstructed image 32. The reconstructed image 32 includes the reconstructed image data. In Equation 5, This represents the reconstructed image 32, which receives the random subcode 30 ( ) and semantic subcode 28 ( When ) via the decoding function of diffusion model 140 ( )generate: .

[0033] The training process includes optimizing the loss function. Update the parameters of diffusion model 140 in real time and As expressed in Equation 6. Specifically, the training process includes updating the parameters of the diffusion model 140 while freezing the parameters of the VLM 100. At this point, Figure 2 illustrates the locks on the image encoder 110 and the text encoder 120 to indicate that the parameters are frozen (i.e., not updated) during this second phase of training. Furthermore, in Equation 6, It is a function that takes time. Noise image at time With semantic subcode And use UNet to predict its noise. In Equation 6, Indicates that it was added to To generate The actual noise. Specifically, in Equation 6, This represents the expectation function, which is used to calculate... exist and The expected value is as expressed in Equation 6. The expected value is calculated at least relative to the process of generating the reconstructed image data via diffusion model 140: .

[0034] Figure 3 is a diagram illustrating various aspects of the image synthesis process via the defect enhancement model 200 according to an example embodiment. During the image synthesis process, the image encoder 110, text encoder 120, and diffusion model 140 are locked, and their parameters are frozen, as indicated by the lock in Figure 3. The image synthesis process occurs during inference and after both the first phase (Figure 1) and the second phase (Figure 2) of the training process are completed. The image synthesis process is configured to be implemented by one or more processors of the processing system 402 (Figure 4) of system 400 or one or more processing devices of another computer system.

[0035] The defect enhancement model 200 is configured to receive input data pairs. In this regard, the data pairs include (i) a digital image with normal (“OK”) image data and (ii) text data with specific text describing at least one desired anomaly / defect to be recently generated on the normal image data. For example, Figure 3 illustrates a non-limiting example of a data pair including (i) a digital image with image data 34 showing a top-plane view of a defect-free normal (“non-abnormal”) PEG product labeled “OK”, and (ii) text data 36 describing a specific defect to be generated on image data 34. In this case, text data 36 describes that a small defect should be generated in the lower right corner of the image of the PEG product. Specifically, text data 36 is “Image of a PEG product with a small defect in the lower right corner”. In this example, text data is generated via the prompt: “Image of a PEG product with a defect of [defect size] at [location]”, where [defect size] represents a slot indicating the desired size of the defect to be generated on image data 34, and where [location] represents a slot indicating the desired location of the defect to be generated on image data 34. As demonstrated by this non-limiting example, a user can control the attributes (e.g., size and location) of at least one defect to be generated as new image data 46 by specifying attributes via slots in the prompt of text data 36.

[0036] In Figure 3, as an example, the fine-tuned image encoder 110 and the fine-tuned text encoder 120 are configured to receive the aforementioned data pairs as input data. Furthermore, the defect enhancement model 200 receives or obtains coefficients having values ​​of 0, values ​​between 0 and 1, or a value of 1. To balance the impact of both image and text embeddings on the output image, the image encoder 110 generates an image embedding 38 (“image semantic subcode”) using the pixels of the image data 34 of the digital image. Furthermore, in response to receiving text data 36, ​​the word segmenter and text encoder 120 generate a text embedding 40 (“text semantic subcode”) using the text data 36. The semantic subcode generator 130 is based on coefficients ( Semantic subcodes are generated using image embedding 38 and text embedding 40 via Equation 3.

[0037] Furthermore, diffusion model 140 includes a diffusion process that generates a random subcode 44 using image data 34. At this point, diffusion model 140 uses the pixels of image data 34 to generate the random subcode 44. Next, the generative process of diffusion model 140 creates new image data 46 using semantic subcode 42 and the random subcode 44. The resulting image with the new image data 46 retains several characteristics of the normal image input (e.g., image data 34) while also possessing defects with attributes specified in the text data input (e.g., text data 36). As a non-limiting example, the resulting image includes new image data 46 that displays most of the characteristics of the (OK) input image of image data 34, while being modified to display small defects as specified by text data 36 in the lower right corner of the resulting image. For ease of viewing the small defects, Figure 3 includes a small bounding box 48 surrounding the newly generated defects. That is, Figure 3 includes the bounding box only for the purpose of this discussion to highlight the newly generated defects relative to image data 46, but image data 46 may not include the bounding box 48.

[0038] Figure 4 is a diagram of an example system 400 with a controlled defect enhancer 300 according to an exemplary embodiment. System 400 includes at least a processing system 402. Processing system 402 includes one or more processing devices. For example, processing system 402 includes at least an electronic processor, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), any suitable processing technology, or any number and combination of the above. Processing system 402 is operable to provide the functionality described herein.

[0039] System 400 includes at least a memory system 404 operatively connected to processing system 402. Memory system 404 communicates data with processing system 402. In an example embodiment, memory system 404 includes at least one non-transitory computer-readable medium configured to store various types of data and provide access to that data, at least enabling processing system 402 to perform the operations and functions disclosed herein. In an example embodiment, memory system 404 includes a single device or multiple devices. Memory system 404 may include any suitable storage technology that is electrical, electronic, magnetic, optical, semiconductor, electromagnetic, or operable with system 400. For example, in an example embodiment, memory system 404 may include random access memory (RAM), read-only memory (ROM), flash memory, disk drive, memory card, optical storage device, magnetic storage device, memory module, any suitable type of memory device, or any combination thereof.

[0040] The memory system 404 includes at least a controlled defect enhancer 300, machine learning (ML) data 406, and other related data 408 stored thereon. The memory system 404 includes computer-readable data that, when executed by the processing system 402, is configured to provide the functionality described at least in Figures 1, 2, and 3. The computer-readable data may include instructions, code, routines, various related data, any software technology, or any number and combination of the foregoing. Specifically, the controlled defect enhancer 300 includes computer-readable data with instructions that, when executed by the processing system 402, are configured to train and employ a machine learning system (e.g., defect enhancement model 200) as described in this disclosure. The controlled defect enhancer 300 includes defect enhancement model 200, which includes VLM 100 and diffusion model 140, as well as a semantic subcode generator 130. Furthermore, ML data 406 includes various training data, various loss data, various weight data and / or parameter data, and any relevant machine learning data that enables system 400 to perform the functions disclosed herein. For example, the various training data include at least a fine-tuning dataset for fine-tuning VLM 100 and a training dataset for training the defect enhancement model 200. The various training data may also include new datasets that include at least synthetic images generated by the controlled defect enhancer 300 via the defect enhancement model 200. The various training data may also include source images used to generate the synthetic images. Meanwhile, other relevant data 408 provides various data (e.g., operating system, etc.) that enables system 400 to perform the functions discussed herein.

[0041] In an example embodiment, as shown in FIG. 4, system 400 is configured to include at least one sensor system 410. Sensor system 410 includes one or more sensors. For example, sensor system 410 includes an image sensor or a camera. Sensor system 410 may also include a radar sensor, a light detection and ranging (LIDAR) sensor, a thermal sensor, an ultrasonic sensor, an infrared sensor, a motion sensor, an audio sensor, an inertial measurement unit (IMU), any suitable sensor, or any combination of the above. Sensor system 410 is operable to communicate with one or more other components of system 400, such as processing system 402 and memory system 404. More specifically, for example, processing system 402 is configured to acquire sensor data directly or indirectly from at least one sensor. Sensor system 410 and / or processing system 402 are configured to generate digital images. Processing system 402 is configured to process the digital images in conjunction with controlled defect enhancer 300 and ML data 406.

[0042] In addition, system 400 includes other components that facilitate the controlled defect enhancer 300. For example, as shown in FIG4, memory system 404 is also configured to store additional relevant data 408 relating to the operation of one or more components, such as sensor system 410, input / output (I / O) system 412, and other functional modules 414. Furthermore, I / O system 412 includes I / O interfaces and may include one or more devices (e.g., display devices, keyboard devices, speaker devices, etc.). Furthermore, system 400 includes other functional modules 414, such as any suitable hardware technology, software technology, or combinations thereof that assist or contribute to the operation of system 400. For example, other functional modules 414 include communication technologies, as described herein, that enable the components of system 400 to communicate at least with each other. Communication technologies enable system 400 to communicate with other network devices (not shown) via a communication network. Utilizing at least the configuration discussed in the example of FIG4, system 400 is configured to enable the controlled defect enhancer 300 to perform the functions discussed in this disclosure.

[0043] Figure 5 illustrates an example flowchart of the process of a controlled defect enhancer 300 according to an exemplary embodiment. As shown in Figures 3 and 5, the controlled defect enhancer 300 is configured to receive a data pair comprising (i) source image data displaying a non-abnormal image (e.g., image data 34) and (ii) text input (e.g., text data 36) describing at least one desired anomalous / defect to be recently generated on normal image data. The controlled defect enhancer 300 is configured to use a defect enhancement model 200 to generate new image data 46 (synthetic image data). The controlled defect enhancer 300 is configured to generate synthetic image data using the source image data and text input via the process described in Figure 3. The controlled defect enhancer 300 is advantageous in that it enables the user to control the generation of anomalies and synthetic images by specifying information about these anomalies via text input. This is advantageous in that it establishes a balanced dataset for training a machine learning model 500 (e.g., anomaly detection model, image classifier, anomaly segmenter, etc.).

[0044] Furthermore, as shown in Figure 5, the process includes combining at least the synthetic image data and the source image data into a dataset 510. This process includes generating a sufficient number of synthetic images and a sufficient number of source images for dataset 510. Each synthetic image provides an anomalous image sample, while each source image provides a non-anomalous image sample. Dataset 510 may also include a corresponding text input. The text input can be used as a label for the corresponding synthetic image data, where the label can serve as ground-truth data. In constructing dataset 510 using a sufficient number of synthetic image data and a sufficient number of source image data, the process also includes any number and combination of pre-training, training, fine-tuning, or the above-mentioned processes relative to machine learning model 500 using dataset 510.

[0045] Figure 6 is a diagram of system 600, which is configured to include at least one trained machine learning model 500, which uses dataset 510 for pre-training, training, fine-tuning, or any number and combination of the above. In this respect, system 600 includes at least a sensor system 610, a control system 620, and an actuator system 630. System 600 is configured such that control system 620 controls actuator system 630 based on sensor data from sensor system 610. More specifically, sensor system 610 includes one or more sensors and / or corresponding devices to generate sensor data. For example, sensor system 610 includes at least one image sensor or camera. Sensor system 610 may also include radar sensors, light detection and ranging (LIDAR) sensors, thermal sensors, ultrasonic sensors, infrared sensors, motion sensors, any suitable sensors, or any combination of the above. When sensing its environment, the sensor system 610 is operable to communicate with the control system 620 via the input / output (I / O) system 660 and / or other functional modules 670 including communication technologies.

[0046] The control system 620 is configured to acquire sensor data directly or indirectly from one or more sensors of the sensor system 610. In this respect, the sensor data may include sensor data from a single sensor or sensor fusion data from multiple sensors. Upon receiving an input that includes at least the sensor data, the control system 620 is operable to process the sensor data via a processing system 640. In this respect, the processing system 640 includes at least one processor. For example, the processing system 640 includes an electronic processor, CPU, GPU, microprocessor, FPGA, ASIC, processing circuitry, any suitable processing technology, or any combination thereof. While processing at least the sensor data, the processing system 640 is operable to generate output data based on communication with the memory system 650. Furthermore, the processing system 640 is operable to provide actuator control data to the actuator system 630 based on the output data.

[0047] Memory system 650 is a computer or electronic storage system configured to store various types of data and provide access to such data to at least achieve the operations and functions disclosed herein. Memory system 650 includes a single device or multiple devices. Memory system 650 includes electrical, electronic, magnetic, optical, semiconductor, electromagnetic, any suitable memory technology, or any combination thereof. For example, memory system 650 may include RAM, ROM, flash memory, disk drive, memory card, optical storage device, magnetic storage device, memory module, any suitable type of memory device, or any combination thereof. Relative to control system 620 and / or processing system 640, memory system 650 is local, remote, or a combination thereof (e.g., partially local and partially remote). For example, memory system 650 may be configured to include at least a cloud-based storage system (e.g., a cloud-based database system) located remotely from processing system 640 and / or other components of control system 620.

[0048] The memory system 650 includes at least a computer vision application 680, a trained machine learning model 500, and other relevant data 690 stored thereon. The memory system 650 includes computer-readable data for the computer vision application 680, the trained machine learning model 500, and other relevant data, respectively. The computer-readable data may include instructions, code, routines, various relevant data, any software technology, or any number and combination of the foregoing. The computer vision application 680 and the trained machine learning model 500 are configured to be executed and / or implemented via a processing system 640. In this regard, the trained machine learning model 500 is configured to receive or acquire digital images directly as input, which are sometimes referred to herein as input images. The trained machine learning model 500 is configured to classify the digital images and output a single class label identifying the class to which the digital image and / or its image segments are believed to belong.

[0049] The trained machine learning model 500 benefits from having been trained using dataset 510, which is a balanced dataset of anomalous and non-anomalous images. The trained machine learning model 500 benefits from training on dataset 510, which is curate and / or controlled relative to the anomalies presented in the anomalous images. The trained machine learning model 500 exhibits improved anomaly detection / segmentation performance on computer vision tasks. The trained machine learning model 500 is configured to output at least one label indicating an "anomalous" classification and at least one other label indicating a "non-anomalous" classification based on the input image.

[0050] Furthermore, as shown in FIG. 6, system 600 includes other components that facilitate the operation of control system 620 in relation to sensor system 610 and actuator system 630. For example, as shown in FIG. 6, memory system 650 is also configured to store other relevant data 690 relating to the operation of system 600 and / or control of one or more of its components (e.g., sensor system 610, control system 620, actuator system 630, etc.). Additionally, as shown in FIG. 6, control system 620 includes I / O system 660, which includes one or more interfaces for one or more I / O devices relating to system 600. For example, I / O system 660 provides at least one interface to sensor system 610 and at least one interface to actuator system 630. Furthermore, control system 620 is configured to provide other functional modules 670, such as any suitable hardware technology, software technology, or any combination thereof that assists and / or contributes to the operation of system 600. For example, other functional modules 670 include an operating system and communication technology, which enables the components of system 600 to communicate with each other as described herein. Utilizing at least the configuration discussed in the example of Figure 6, system 600 can be applied to a variety of technologies, such as at least partially autonomous vehicles, robots, personal assistant technologies, manufacturing technologies, security technologies, medical imaging technologies, etc.

[0051] Figures 7 and 8 illustrate various non-limiting examples of applications of system 600. For a particular application, system 600 includes (i) a computer vision application 680 applied to that particular application, and (ii) a trained machine learning model 500 trained on a specific dataset 510 applied to that particular application. For example, manufacturing technology 700 includes an application of system 600, which includes a computer vision application 680 relating to manufacturing and a trained machine learning model 500 trained on dataset 510, which includes at least anomalous images 702 of instances of product 702 and non-anomalous images of instances of product 702. As another example, imaging technology 800 includes an application of system 600, which includes a computer vision application 680 relating to imaging (e.g., medical imaging) and a trained machine learning model 500 trained on dataset 510, which includes at least anomalous and non-anomalous images relating to a particular imaging (e.g., medical imaging) performed via imaging technology 800.

[0052] Figure 7 is a diagram of system 600 relative to manufacturing technology 700 according to an example embodiment. As a non-limiting example, manufacturing technology 700 includes any suitable type of manufacturing machine (e.g., a cutting machine, a sealing machine, a drilling machine, etc.). In Figure 7, sensor system 610 includes at least one image sensor or optical sensor. Control system 620 is configured to acquire image data from sensor system 610. A trained machine learning model 500 is configured to classify an input image or image segment as “abnormal” or “non-abnormal” given the state of a given product 702 (e.g., the PEG product in the previous example) (which is being manufactured or has been manufactured via manufacturing technology 700). Furthermore, control system 620 is configured to generate actuator control data based on the sensor data captured by sensor system 610, in response to the classification of the current state of product 702. For example, as a non-limiting example, in response to the actuator control data, actuator system 630 may be configured to actuate the next manufacturing step 704 of the manufacturing process based on the “non-abnormal” classification of product 702. Alternatively, in response to actuator control data, actuator system 630 can be configured to stop manufacturing technology 700 from performing the next action on immediate product 702 and / or stop immediate product 702 from continuing to the next manufacturing step 704 of the manufacturing process based on the “abnormal” classification of immediate product 702.

[0053] Figure 8 is a diagram of a system 600 for imaging technology 800 according to an exemplary embodiment. As a non-limiting example, imaging technology 800 includes a magnetic resonance imaging (MRI) apparatus, an X-ray imaging apparatus, an ultrasound apparatus, a medical imaging apparatus, or any suitable type of imaging apparatus. In Figure 8, sensor system 610 includes at least one image sensor. Control system 620 is configured to acquire image data from sensor system 610. Control system 620 is configured to classify the digital image data acquired from sensor system 610. For example, control system 620 may classify a digital image or a segment thereof as “abnormal” or “non-abnormal.” Control system 620 is configured to generate actuator control data in response to a classification (e.g., class label) provided by a trained machine learning model 500. For example, as a non-limiting example, actuator control data may cause control system 620 to at least partially highlight a digital image and display the highlighted digital image on display 802. As another example, actuator control data may cause control system 620 to automatically transmit a message notifying one or more entities of the classification of the digital image.

[0054] As described in this disclosure, the embodiments disclosed herein include several advantageous features and benefits. For example, the embodiments are advantageous in controlling the generation of defective images via a generative model. Specifically, the embodiments provide control within the generation process of a diffusion model (e.g., diffusion model 140). Furthermore, instead of relying on an autoencoder, the embodiments utilize a base model (e.g., a CLIP model) to more effectively guide the diffusion process of diffusion model 140. Specifically, the embodiments utilize an image encoder 110 and a text encoder 120 of a large, fine-tuned VLM 100, and incorporate relevant textual descriptions of each digital image in the process. The embodiments are advantageous in constructing a human-understandable natural language interface, thereby managing and manipulating the various attributes of the anomalies to be generated in new images with greater accuracy and flexibility, thereby enabling a more detailed and manageable way to create anomalous images for data augmentation.

[0055] Furthermore, the embodiment utilizes a finely tuned, large pre-trained VLM 100 to guide the generative process of the diffusion model 140, thereby creating a novel defect enhancement pipeline. The finely tuned VLM 100 is used to discover high-level semantics about a given digital image. These high-level semantics are used as conditions to train the diffusion model 140 to generate new images (e.g., new image data 46). The new image data 46 can be referred to as synthetic image data. Specifically, the input image is encoded into a bidirectional latent representation. The first latent representation is a semantic subcode, which is linear and has semantic content. The semantic subcode is extracted using the finely tuned VLM 100. The second latent representation is a random subcode, which represents random variations in the image data and is captured by the diffusion process of the diffusion model 140. The diffusion model 140 then acts as a decoder. The diffusion model 140 combines the high-level semantics with the random variations to reconstruct the original image. That is, the diffusion model 140 generates the reconstructed image 32 using the semantic subcode 28 and the random subcode 30. This mechanism enables attribute manipulation relative to a given digital image, enabling the generation of at least one new image from the source image. These embodiments include utilizing an image encoder 110 and a text encoder 120 of a finely tuned, pre-trained VLM 100 to extract high-level semantics from a combination of image input and its accompanying text input, which describes various attributes of the image input.

[0056] Furthermore, the above description is intended to be illustrative rather than restrictive, and is provided in the context of a particular application and its requirements. Those skilled in the art will appreciate from the foregoing description that the invention can be implemented in a wide variety of forms, and various embodiments can be implemented individually or in combination. Therefore, while embodiments of the invention have been described in conjunction with specific examples of the invention, the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of the described embodiments, and the true scope of the embodiments and / or methods of the invention is not limited to the shown and described embodiments, as various modifications will become apparent to those skilled in the art upon studying the drawings, specification, and the following claims. Additionally or alternatively, components and functions may be separated or combined in ways different from the various described embodiments, and may be described using different terms. These and other variations, modifications, additions, and improvements may fall within the scope of this disclosure as defined in the following claims.

Claims

1. A computer implementation method for a machine learning system including an image encoder, a text encoder, and a diffusion model, the method comprising: Receive a training dataset having data pairs, the data pairs including at least a first data pair having at least (i) image data displaying an anomaly and (ii) text data describing the corresponding image data including the anomaly; generate an image embedding using the pixels of the image data via the image encoder; generate a text embedding using the text data via the text encoder; generate a semantic subcode using the image embedding and the text embedding; generate a random subcode using the pixels of the image data via the diffusion model; Reconstructed image data is generated using the random subcode and the semantic subcode via the diffusion model; during the generation of the reconstructed image data, the loss is optimized based at least on the expected value of the difference between the predicted noise of the noisy image at a specific time and the actual noise of the noisy image at the specific time; and the parameters of the diffusion model are updated using the loss.

2. The computer implementation method according to claim 1, wherein: The semantic subcode is the sum of the image component and the text component; the text component is calculated by multiplying the text embedding by a first coefficient, the first coefficient being a value of 0, a value between 0 and 1, or a value of 1; and the image component is calculated by multiplying the image embedding by a second coefficient, the second coefficient being 1 minus the first coefficient.

3. The computer implementation method according to claim 1, wherein: The image data displays the object; and the anomaly is a defect on the object.

4. The computer implementation method according to claim 1, further comprising: A fine-tuning dataset is used to fine-tune a pre-trained visual language model (VLM), the fine-tuning dataset comprising (i) a first subset of digital images including non-abnormal image data and a first subset of corresponding text data describing the non-abnormal image data, and (ii) a second subset of digital images including anomalous image data and a second subset of corresponding text data describing the anomalous image data, wherein the image encoder is a fine-tuned image encoding component of the pre-trained VLM, and the text encoder is a fine-tuned text encoding component of the pre-trained VLM.

5. The computer implementation method according to claim 4, wherein: The fine-tuning dataset of the pre-trained VLM includes at least another data pair; the other data pair includes another digital image displaying another image data and another text data describing the other image data; and the other text data includes (i) a data type indicating whether the other image data displays an anomalous or non-anomalous object, and (ii) one or more attribute data indicating one or more attributes of the defect of the object when the data type is anomalous.

6. The computer implementation method of claim 5, wherein the other text data of the fine-tuning dataset for fine-tuning the VLM is more descriptive than the text data of the training dataset for training the diffusion model.

7. The computer implementation method according to claim 1, further comprising: Receive a source image with non-abnormal source image data; Receive text input describing (i) a desired anomaly to be generated relative to the source image and (ii) at least one attribute of the anomaly; and generate a synthetic image using the source image and the text input via the machine learning system, wherein the synthetic image displays the source image data having the desired anomaly as described by the text input.

8. The computer implementation method according to claim 7, further comprising: Create a new dataset that includes at least the source image and the synthesized image; And train an anomaly detector using the new dataset, the anomaly detector comprising at least one machine learning model.

9. A system comprising: One or more processors; One or more computer memories that communicate data with the one or more processors, the one or more computer memories having computer-readable data stored thereon, the computer-readable data including instructions that, when executed by the one or more processors, cause the one or more processors to perform a method of a machine learning system including an image encoder, a text encoder, and a diffusion model, the method comprising receiving a training dataset having data pairs, the data pairs including at least a first data pair having at least (i) image data displaying an anomaly and (ii) text data describing corresponding image data including the anomaly; generating an image embedding using pixels of the image data via the image encoder; generating a text embedding using the text data via the text encoder; generating a semantic subcode using the image embedding and the text embedding; and generating a random subcode using the pixels of the image data via the diffusion model; Reconstructed image data is generated using the random subcode and the semantic subcode via the diffusion model; and during the generation of the reconstructed image data, the loss is optimized based at least on the expected value of the difference between the predicted noise of the noisy image at a specific time and the actual noise of the noisy image at the specific time; and the parameters of the diffusion model are updated using the loss.

10. The system according to claim 9, wherein: The semantic subcode is the sum of the image component and the text component; the text component is calculated by multiplying the text embedding by a first coefficient, the first coefficient being a value of 0, a value between 0 and 1, or a value of 1; and the image component is calculated by multiplying the image embedding by a second coefficient, the second coefficient being 1 minus the first coefficient.

11. The system according to claim 9, wherein: The image data displays the object; and the anomaly is a defect on the object.

12. The system of claim 9, wherein the method further comprises: A fine-tuning dataset is used to fine-tune a pre-trained visual language model (VLM), the fine-tuning dataset comprising (i) a first subset of digital images including non-abnormal image data and a first subset of corresponding text data describing the non-abnormal image data, and (ii) a second subset of digital images including anomalous image data and a second subset of corresponding text data describing the anomalous image data, wherein the image encoder is a fine-tuning image encoding component of the VLM, and the text encoder is a fine-tuning text encoding component of the VLM.

13. The system according to claim 12, wherein: The fine-tuning dataset of the pre-trained VLM includes at least another data pair; the other data pair includes another digital image displaying another image data and another text data describing the other image data; and the other text data includes (i) a data type indicating whether the other image data is anomalous or non-anomalous, and (ii) one or more attribute data indicating one or more attributes of the defects displayed in the other image data when the data type is anomalous.

14. The system of claim 13, wherein the other text data of the fine-tuning dataset for fine-tuning the VLM is more descriptive than the text data of the training dataset for training the diffusion model.

15. The system of claim 9, wherein the method further comprises: The system receives a source image having source image data that is non-abnormal; receives text input describing (i) a desired anomalous element to be generated relative to the source image and (ii) at least one attribute of the anomalous element; and generates a synthetic image using the source image and the text input via the machine learning system, wherein the synthetic image displays the source image data having the desired anomalous element as described by the text input.

16. The system of claim 15, wherein the method further comprises: Create a new dataset that includes at least the source image and the synthesized image; And train an anomaly detector using the new dataset, the anomaly detector comprising at least one machine learning model.

17. A computer-implemented method for generating a dataset for training a machine learning model, the method comprising: Receive a source image containing source image data that is not abnormal; Receive text input describing (i) an anomaly to be generated relative to the source image data and (ii) one or more attributes of the anomaly; generate a source image embedding using pixels of the source image via an image encoder; A text input embedding is generated using the text input via a text encoder; The source image embedding and the text input embedding are used to generate semantic subcodes; Random subcodes are generated using the pixels of the source image via a diffusion model; And via the diffusion model, a synthetic image is generated using the random subcode and the semantic subcode, the synthetic image displaying the source image data having the anomaly as described by the text input, wherein the dataset includes at least the source image and the synthetic image, and the dataset is configured to train the machine learning model to perform anomaly detection tasks.

18. The computer implementation method according to claim 17, wherein: The semantic subcode is the sum of the image component and the text component; the text component is calculated by multiplying the text embedding by a first coefficient, the first coefficient being a value of 0, a value between 0 and 1, or a value of 1; and the image component is calculated by multiplying the image embedding by a second coefficient, the second coefficient being 1 minus the first coefficient.

19. The computer implementation method according to claim 17, wherein: The source image data displays the object; and the anomaly is a defect on the object.

20. The computer implementation method of claim 19, wherein the one or more attributes of the anomaly include (i) the size of the defect and (ii) the location of the defect.

Images