Detecting defects using a text-to-image diffusion model

Abstract

Methods for fine-tuning convolutional neural networks of text-to-image diffusion models in the context of identifying defects of manufactured products within images of those products are disclosed. Images of manufactured images having various scratches, dings, or other defects are provided to the model along with words or phrases indicating the presence of a defect. The model then learns to identify portions of the overall image that include the defect. Learning of this type of task is based on using segmentation masks corresponding to the images, which are then used along with cross-attention maps of the model in order to compute an average defect mask loss parameter of the model. By computing this parameter and applying it in updating the weights of the model, the model can be fine-tuned to detect defects of manufactured products.

Description

Technical Field

[0001] This disclosure relates to fine-tuning a text-to-image diffusion model. Background Technology

[0002] Diffusion models have been applied to various data modalities (such as point clouds, audio, and depth maps) and a wide range of tasks beyond generation (such as image inpainting, super-resolution, segmentation, and object detection), and have been used to solve various linear and nonlinear inversion problems. Because diffusion models can capture the underlying data distribution, they serve as good, high-capacity priors for data-driven applications. However, due to a lack of fine-tuning, the generalized application of diffusion models is not configured to perform specific tasks. Summary of the Invention

[0003] In an embodiment, a method for fine-tuning a text-to-image diffusion model (such as a text-to-image diffusion model) is provided. The method includes: receiving image-based data samples and embedded text samples, wherein: the image-based data samples are images of a defective manufactured product; and the embedded text samples are embeddings of text-based data samples indicating defects; performing a variational autoencoder to output a latent spatial representation of the image-based data samples; performing a noise model to output a noisy version of the latent spatial representation; providing the noisy version of the latent spatial representation and the embedded text samples to a convolutional neural network of the text-to-image latent diffusion model; performing the convolutional neural network to learn to predict noise in the image-based data samples using multiple cross-attention maps at different spatial resolutions; calculating an average defect mask loss parameter based on a cross-attention map at a given spatial resolution and a segmentation mask corresponding to the image-based data samples; updating one or more weights of the convolutional neural network at least in part based on the average defect mask loss parameter; and outputting a fine-tuned text-to-image latent diffusion model with the updated one or more weights for detecting defects in other image-based data samples of other manufactured products.

[0004] In another embodiment, a system includes a processor and a memory containing instructions that, when executed by the processor, cause the processor to perform these steps.

[0005] In another embodiment, a non-transitory computer-readable medium includes instructions that, when executed by a processor, cause the processor to perform these steps. Attached Figure Description

[0006] Figure 1 The illustration shows a system for training, fine-tuning, and utilizing machine learning models, such as convolutional neural networks, according to some embodiments.

[0007] Figure 2 The illustrations depict computer-implemented methods for training, fine-tuning, and utilizing machine learning models, such as convolutional neural networks, according to some embodiments.

[0008] Figure 3 The illustration shows the architecture of a text-to-image diffusion model according to some embodiments, which is configured to receive image-based data samples of a manufactured product and text-based data samples indicating defects in the manufactured product, and subsequently detect portions of the image corresponding to the defects.

[0009] Figure 4 The illustration shows a method for using, according to some embodiments, in Figure 3 The process of fine-tuning convolutional neural networks (e.g., U-Net) within a text-to-image diffusion model is described in the article.

[0010] Figure 5A The illustration shows a first example of an image of a defective manufactured product according to some embodiments, and a positive identification of the defective portion in the image by a finely tuned convolutional neural network of a text-to-image diffusion model.

[0011] Figure 5B The illustration shows that in Figure 5A The same image described in the text, as well as the failure of a certain other diffusion model to identify the defective portion of the image.

[0012] Figure 6A The illustration shows a second example of an image of a defective manufactured product according to some embodiments, and a positive identification of the defective portion in the image by a finely tuned convolutional neural network of a text-to-image diffusion model.

[0013] Figure 6B The illustration shows that in Figure 6A The same image described in the text, as well as the failure of a certain other diffusion model to identify the defective portion of the image.

[0014] Figure 7A The illustration shows a third example of an image of a defective manufactured product according to some embodiments, and a positive identification of the defective portion in the image by a finely tuned convolutional neural network of a text-to-image diffusion model.

[0015] Figure 7B The illustration shows that in Figure 7A The same image described in the text, as well as the failure of a certain other diffusion model to identify the defective portion of the image.

[0016] Figure 8AThe illustration shows a first example of an image of a defect-free manufactured product according to some embodiments, and a positive recognition of the absence of defects within the image by a finely tuned convolutional neural network of a text-to-image diffusion model.

[0017] Figure 8B The illustration shows a second example of an image of a defect-free manufactured product according to some embodiments, and a positive recognition of the absence of defects within the image by a finely tuned convolutional neural network of a text-to-image diffusion model.

[0018] Figure 8C The illustration shows a third example of an image of a defect-free manufactured product according to some embodiments, and a positive recognition of the absence of defects within the image by a finely tuned convolutional neural network of a text-to-image diffusion model.

[0019] Figure 9 This is a flowchart illustrating, according to some embodiments, the process of fine-tuning a convolutional neural network of a text-to-image diffusion model to detect portions in an image of a manufactured product that correspond to defects within the product.

[0020] Figure 10 A schematic diagram depicts the interaction between a computer-controlled machine and a control system according to some embodiments.

[0021] Figure 11 Depicting according to some embodiments Figure 10 A schematic diagram of a control system configured to control manufacturing machines in a manufacturing system, such as an automated optical inspection system for a production line. Detailed Implementation

[0022] Embodiments of this disclosure are described herein. However, it is to be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The drawings are not necessarily drawn to scale; some features may be enlarged or reduced to show detail of specific components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to employ the embodiments in various ways. As will be understood by those skilled in the art, the various features illustrated and described with reference to any of the drawings may be combined with features illustrated in one or more other drawings to produce embodiments not explicitly illustrated or described. Combinations of illustrated features provide representative embodiments of typical applications. However, for a particular application or implementation, various combinations and modifications of features consistent with the teachings of this disclosure may be required.

[0023] As used herein, “a,” “one,” and “the” refer to both the singular and plural referents of an object, unless the context clearly specifies otherwise. As an example, “processor” programmed to perform various functions refers to one processor programmed to perform each of those functions, or more than one processor programmed together to perform each of those functions.

[0024] Diffusion models have a wide and diverse range of applications. Because they can utilize large amounts of available and increasingly open-source training datasets, such models can be generalized to a variety of applications. However, until the development of this disclosure, past implementations of diffusion models lacked the ability to correlate minute defects within a large image of a manufactured product with the fact that a small portion of the image is indeed a defective area of ​​the manufactured product. The following few paragraphs detail the context of previous implementations of diffusion models, followed by an explanation of how this disclosure overcomes these limitations.

[0025] While diffusion models, generally trained on available training datasets, may already have other applications, they lack the ability to examine and determine the output of a specific dataset. For example, within a manufacturing setting, a company might use systems such as automated optical inspection systems to perform quality checks on products recently manufactured but not yet shipped out of the facility for purchase or use in other downstream manufacturing processes. While these quality checks are crucial to ensuring that defective products are not negligently shipped out of the manufacturing facility and sold to customers, the company might not be interested in incorporating images of its internal facilities into open-source training datasets that could help optimize such quality inspection procedures. Furthermore, if quality checks are performed at a midway point in the overall manufacturing process, the company might even be interested in keeping such images processed by an automated optical inspection system confidential.

[0026] Past implementations of this type of machine learning model application have attempted to apply diffusion models trained solely on images from the internet or other open-source domains to analyze manufactured products with minute manufacturing defects and various types of defects not visible in images from such open-source training datasets, but have inevitably failed. In particular, the head implementation of the diffusion model is trained to associate text with images, and therefore those simply, generally trained models fail to associate text (such as "the defective part") with the correct part of the image containing the defect because the model has not been trained for such a specialized task.

[0027] Furthermore, previous methods for attempting to fine-tune text-to-image diffusion models (such as stable diffusion) lack any mechanism to generate cross-attention maps, where defects, especially those occupying a very small number of pixels compared to the entire image, are the only non-zero pixels. Naïve fine-tuning of latent diffusion models similar to stable diffusion results in cross-attention maps where defect-specific pixels are indistinguishable from the rest of the image.

[0028] Furthermore, when fine-tuning the latent diffusion model on specialized images (such as those of the manufactured portion), simply fine-tuning the pre-trained text-to-image diffusion model using the original loss function provides insufficient information, where these deficiencies are minimal compared to the full resolution of the image and are not found in publicly available data on which such a model was originally trained.

[0029] This disclosure overcomes these challenges by reshaping the fine-tuning process, thereby allowing the model to be specifically trained to identify images and parts of images related to specific manufactured products of the company, as well as defects commonly seen or other manufacturing errors specific to those products. This disclosure goes even further when fine-tuning the model by incorporating novel loss parameters (also referred to herein as loss terms or loss functions).

[0030] This disclosure introduces an additional novel loss term (referred to herein as the average defect mask loss parameter) to force the cross-attention map (corresponding to a portion of the text input stating the defect type) at a specific resolution to resemble the defect in the original image. This is achieved by creating segmentation mask images using a deep segmentation network that isolates the defect from the rest of the image and forces the appropriate cross-attention map to be equal to these binary mask images. Thus, this disclosure enables the synthesis of defects in specialized and confidential manufactured products via a text-to-image diffusion model.

[0031] Furthermore, and in contrast to previous implementations of diffusion models, this disclosure does not require detailed annotations (also referred to herein as text-based data samples) from human experts capable of identifying defects on products on the production line, where human-based identification processes are both time-consuming and costly, as they describe every type and variation of defect. Instead, this disclosure applies general names or placeholder terms to defects, such as “scratches,” “stains,” or “dents,” and thus text-based data samples rely on simple words or phrases, such as “metallic surface with scratches.” The complex and nuanced details of defects (such as their location, orientation, shape, and size) are instead captured by cross-attention maps during model execution and trained to mimic binary segmentation mask images obtained by applying depth segmentation methods to the original image, allowing the model to isolate defects from the rest of the image.

[0032] The following description continues with a general introduction to machine learning techniques related to methods for training and fine-tuning diffusion models, such as those described herein. Next, various embodiments of the architecture and process flow for fine-tuning convolutional neural networks within a text-to-image diffusion model are discussed. Then, this disclosure demonstrates the versatility of the methods and systems described herein for incorporation into automated optical inspection systems within production line settings in manufacturing facilities.

[0033] Figure 1 A system 100 for training, fine-tuning, and utilizing neural networks, such as convolutional neural networks, is illustrated. It should be understood that, although this document discusses... Figure 1 and Figure 2 The example embodiments given in the following paragraphs relate to convolutional neural networks, but... Figure 1 and Figure 2 Additional embodiments can be applied to any other type of neural network-based or non-neural network-based machine learning model that is configured to be developed, trained, and fine-tuned for the various defect detection applications further described herein.

[0034] Furthermore, text-to-image diffusion models, such as those described in this paper within the defect detection context, can include at least a Large Language Model (LLM) text encoder, a variational autoencoder, and a convolutional neural network. The convolutional neural network can be configured with a U-Net architecture.

[0035] Therefore, and as described in this paper, a “convolutional” neural network configured with a U-Net architecture can be defined as having convolutional neural network blocks, self-attention blocks, cross-attention blocks, and ResNet blocks, arranged hierarchically above and below each other, and between the input and output layers of the model (see also in this paper). Figure 4Additional embodiments of this type of machine learning model are described herein with respect to the following: machine learning model 210; fine-tuned convolutional neural networks 316, 416; fine-tuned convolutional neural networks 500; fine-tuned convolutional neural networks 600; fine-tuned convolutional neural networks 700; fine-tuned convolutional neural networks 800; and block 908.

[0036] In some embodiments, system 100 may include an input interface for accessing a fine-tuning dataset 102 of a convolutional neural network. For example, such as Figure 1 As illustrated, the input interface can be comprised of a data storage interface 104, which can access the fine-tuning data 102 from the data storage device 106. For example, the data storage interface 104 can be a memory interface or a permanent storage interface (e.g., a hard disk or SSD interface), but it can also be a personal, local area network, or wide area network interface, such as a Bluetooth, ZigBee, or Wi-Fi interface, or an Ethernet or fiber optic interface. The data storage device 106 can be an internal data storage device of the system 100 (such as a hard disk drive or SSD), but it can also be an external data storage device, such as a network-accessible data storage device.

[0037] In some embodiments, data storage device 106 may further include a data representation 108 of an untrained version of the model (e.g., a version of a machine learning model that has not yet been trained), which may be accessed by system 100 from data storage device 106. However, it will be appreciated that the fine-tuned data 102 and the pre-trained convolutional neural network data representation 108 may also be accessed from different data storage devices, for example, via different subsystems of data storage interface 104. Each subsystem may be of the type of data storage interface 104 as described above. In other embodiments, the pre-trained convolutional neural network data representation 108 may be generated internally by system 100 based on the design parameters of the neural network, and therefore this data representation 108 may not be explicitly stored on data storage device 106. System 100 may also include a processor subsystem 110, which may be configured to provide an iterative function as a substitute for the stacked layers of the convolutional neural network to be fine-tuned during operation of system 100. Here, the corresponding layers in the replaced stacked layers can have mutually shared weights and can receive the output of the previous layer as input, or, for the first layer of the stacked layers, receive the initial activation and a portion of the input of the stacked layers. The processor subsystem 110 can also be configured to iteratively fine-tune the convolutional neural network using fine-tuning data 102 (e.g., thus generating an updated version of the machine learning model relative to a first "pre-trained" version of the model). Here, the iteration of fine-tuning by the processor subsystem 110 can include forward propagation and backward propagation parts. The backward propagation process can also be defined herein as a generative process.

[0038] System 100 may also include an output interface for outputting a fine-tuned convolutional neural network data representation 112, which may also be referred to as both the trained and fine-tuned model data 112. For example, as... Figure 1 As illustrated, the output interface can be comprised of a data storage interface 104, which in these embodiments is an input / output (“IO”) interface via which trained and fine-tuned model data 112 can be stored in data storage device 106. For example, the data representation 108 defining a “pre-trained” convolutional neural network can be at least partially replaced by the data representation 112 of the fine-tuned neural network during or after fine-tuning, because the parameters of the convolutional neural network, such as weights, hyperparameters, and other types of parameters, can be adapted to reflect the fine-tuning of the fine-tuned data 102. This also... Figure 1The figures are illustrated by reference numerals 108 and 112, which refer to the same data records on data storage device 106. In other embodiments, data representation 112 may be stored separately from data representation 108 defining a "pre-trained" convolutional neural network. In some embodiments, the output interface may be separate from data storage interface 104, but generally it can be of the type described above for data storage interface 104.

[0039] Figure 2 The illustration depicts computer-implemented methods for training, fine-tuning, and utilizing convolutional neural networks according to some embodiments. System 200 may include at least one computing system 202. Computing system 202 may include at least one processor 204 operatively connected to memory unit 208. Processor 204 may include one or more integrated circuits implementing the functions of central processing unit (CPU) 206, and in some embodiments, includes a graphics processing unit (GPU). CPU 206 may be a commercially available processing unit implementing an instruction set such as x86, ARM, Power, or MIPS instruction set families. During operation, CPU 206 may execute stored program instructions retrieved from memory unit 208. The stored program instructions may include software controlling the operation of CPU 206 to perform the operations described herein. In some examples, processor 204 may be a system-on-a-chip (SoC) that integrates the functions of CPU 206, memory unit 208, network interface, and input / output interface into a single integrated device. Computing system 202 may implement an operating system for managing various aspects of operation.

[0040] Memory cell 208 may include volatile and non-volatile memory for storing instructions and data. Non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is deactivated or loses power. Volatile memory may include static and dynamic random access memory (RAM) for storing program instructions and data. For example, memory cell 208 may store machine learning model 210 or algorithm, training and / or fine-tuning dataset 212 of machine learning model 210, original source dataset 214, etc.

[0041] The computing system 202 may include a network interface device 220 configured to provide communication with external systems and devices. For example, the network interface device 220 may include wired and / or wireless Ethernet interfaces defined by the Institute of Electrical and Electronics Engineers (IEEE) 902.11 standards family. The network interface device 220 may also include a cellular communication interface for communicating with cellular networks (e.g., 3G, 4G, 5G). The network interface device 220 may also be configured to provide a communication interface to an external network 222 or the cloud.

[0042] External network 222 may be referred to as the World Wide Web or the Internet. External network 222 can establish standard communication protocols between computing devices. External network 222 can allow information and data to be easily exchanged between computing devices and the network. One or more servers 224 can communicate with external network 222.

[0043] The computing system 202 may include an input / output (I / O) interface 218, which may be configured to provide digital and / or analog inputs and outputs. The I / O interface 218 may include an additional serial interface (e.g., a Universal Serial Bus (USB) interface) for communicating with external devices.

[0044] The computing system 202 may include a human-machine interface (HMI) device 216, and the HMI device 218 may include any device that enables the system 200 to receive control input. Examples of input devices may include human interface inputs such as a keyboard, mouse, touchscreen, voice input device, and other similar devices. The computing system 202 may include a display device 226. The computing system 202 may include hardware and software for outputting graphical and textual information to the display device 226. The display device 226 may include an electronic display screen, projector, printer, or other suitable device for displaying information to a user or operator. The computing system 202 may also be configured to allow interaction with remote HMIs and remote display devices via a network interface device 220.

[0045] System 200 can be implemented using one or more computing systems. While this example depicts a single computing system 202 implementing all the features described, it is intended that various features and functions can be decoupled and implemented by multiple computing units that communicate with each other. The specific system architecture chosen may depend on various factors.

[0046] System 200 can implement machine learning algorithm 210, which is configured to analyze raw source dataset 214. Raw source dataset 214 may include raw or unprocessed sensor data, which may represent the input dataset of the machine learning system. In some examples, machine learning algorithm 210 may be a convolutional neural network algorithm designed to perform a predetermined function. For example, the neural network algorithm may be configured to learn to detect defects in manufactured products present in image-based data samples.

[0047] Computer system 200 may store training and / or fine-tuning datasets 212 for machine learning algorithm 210. Training dataset 212 may represent a previously constructed set of data used to train machine learning algorithm 210. Machine learning algorithm 210 can use training dataset 212 to learn weighting factors associated with a convolutional neural network algorithm. Training dataset 212 may include a source dataset having corresponding results or outcomes that machine learning algorithm 210 attempts to replicate through the learning process.

[0048] The machine learning algorithm 210 can be operated in learning mode using training dataset 212 as input. Machine learning algorithm 210 can perform several iterations using data from training dataset 212. With each iteration, machine learning algorithm 210 can update its internal weighting factors based on the achieved results. For example, machine learning algorithm 210 can compare its output results (e.g., annotations) with those included in training dataset 212. Since training dataset 212 includes expected results, machine learning algorithm 210 can determine when performance is acceptable. After machine learning algorithm 210 achieves a predetermined performance level (e.g., 100% consistency with results associated with training dataset 212), machine learning algorithm 210 can be executed using data not in training dataset 212. The trained machine learning algorithm 210 can be applied to new datasets to generate annotated data.

[0049] Machine learning algorithm 210 can be configured to identify specific features in raw source data 214. Raw source data 214 may include multiple instances or input datasets from which annotation results are expected. Machine learning algorithm 210 can be programmed to process raw source data 214 to identify the presence of specific features. Machine learning algorithm 210 can be configured to identify features in raw source data 214 as predetermined features. Raw source data 214 can be derived from various sources. For example, raw source data 214 may be actual input data collected by a machine learning system. Raw source data 214 may be machine-generated for testing systems. As an example, raw source data 214 may include image-based data samples and text-based data samples of defective manufactured products.

[0050] In this example, machine learning algorithm 210 can then process the raw source data 214 and output an indication of where defects exist within the image. Machine learning algorithm 210 can generate confidence levels or factors for each generated output. For example, a confidence value exceeding a predetermined high confidence threshold can indicate that machine learning algorithm 210 is confident that the identified feature corresponds to a specific feature. A confidence value below a low confidence threshold can indicate that machine learning algorithm 210 has some uncertainty about the existence of a specific feature.

[0051] Figure 3 The illustration shows the architecture of a text-to-image diffusion model according to some embodiments, which is configured to receive image-based data samples of a manufactured product and text-based data samples indicating defects in the manufactured product, and subsequently detect portions of the image corresponding to the defects.

[0052] like Figure 3 The text-to-image latent diffusion model 300 shown may include three main components configured to interact with each other. The first component is an LLM text encoder 310, which receives text-based data samples 308 as input and, when executed, proceeds to transform the text-based data samples 308 into embeddings, as indicated by the embedded text 312. In some embodiments, the LLM text encoder 310 may resemble a contrastive language-image pre-trained (CLIP) encoder.

[0053] The second component is a variational autoencoder (VAE), where VAE encoder 304 receives image-based data samples 302 and generates a latent spatial representation 306 of the image. The third component is a convolutional neural network 316, which receives the noisy latent spatial representation 306 and embedded text 312 to first perform a denoising diffuse implicit model (DDIM) inversion process 314. The fine-tuned convolutional neural network 316 outputs a noisy DDIM latent spatial representation 318, which is then provided back to the fine-tuned convolutional neural network 316 during the execution of the DDIM generation process 320. During the DDIM generation process, the cross-attention map of each denoising step is stored in a memory buffer 322 and then used to compute cross-attention maps. The average cross-attention map 324 from each denoising step is then used to generate an output image-based data sample that defines the location of defects within the originally received image-based data sample 302. Figure 3 As illustrated, the detected defect 326 correctly locates the defect of the image-based data sample 302 as being located on the bottom right side of the image.

[0054] exist Figure 3 In the specific embodiment illustrated, the text-to-image latent diffusion model 300 falls within the latent diffusion model class because the convolutional neural network 316 is configured to operate in the latent space.

[0055] However, in other embodiments, in Figure 3 Throughout the process illustrated, the text-to-image diffusion model can remain within the image space. In such an embodiment, image-based data samples 302 are fed directly to the fine-tuned convolutional neural network 316 without being passed through the VAE encoder 304.

[0056] In the following Figure 4 The embodiments illustrated herein continue to describe the convolutional neural network 316 as implemented within a latent diffusion model version of the text-to-image latent diffusion model 300. However, it should be understood that a similar fine-tuning process 400 of the convolutional neural network 316 can be performed for embodiments in which the convolutional neural network 316 is implemented such that the text-to-image diffusion model remains in the image space rather than being transformed into the latent space.

[0057] Figure 4 The illustration shows, according to some embodiments, in Figure 3 The process of fine-tuning convolutional neural networks (e.g., the U-Net architecture for stabilizing diffusion models) within text-to-image diffusion models is described in the article.

[0058] In the Figure 4 When describing the moment in time, it should be understood that "convolutional neural network 416" refers to a pre-trained model that is currently undergoing fine-tuning via the methods described in this paper. This model is called "pre-trained" because it has already undergone one or more epochs of training using various training datasets and is therefore at a point where it can be used for generalization tasks. Figure 4 The moment described in the text refers to the "fine-tuning" of the pre-trained convolutional neural network 416 of the text-to-image latent diffusion model 300 in order to learn to detect defects within images of manufactured products. This "pre-trained" text-to-image diffusion model has not yet been trained for such a specialized task, and therefore... Figure 4 The architecture shown in this paper and in [the following text] Figure 9 The corresponding process described herein relates to fine-tuning the model so that it can then be performed for specialized tasks of this type (e.g., detecting portions of an image that contain defects, scratches, marks, or other quality issues).

[0059] The following paragraphs describe the common definition of the fine-tuning process 400 and its configuration for use. Figure 4The diagram illustrates the four process flows of the U-Net architecture operation. These paragraphs are formatted in a sequential manner, taking these steps to execute the pre-trained convolutional neural network 416 for fine-tuning the text-to-image latent diffusion model 300, enabling the model to learn to detect one or more portions of an image involving defects in a manufactured product. The first process flow involves... Figure 4 Blocks 402, 406, 408, 410, and 412. The second process involves... Figure 4 Blocks 416, 404, 412, and 422. The third process involves... Figure 4 Blocks 410, 420, and 422. The fourth process involves... Figure 4 Blocks 402, 424, 418, 428, and 430.

[0060] Referring now to the first process flow, the input to the convolutional neural network 416 of the text-to-image latent diffusion model includes both the noisy latent space representation 412 and the embedded text 404. For example... Figure 3 As described above, image-based data samples 402 are provided to a VAE encoder 406 to compress the image into a latent spatial representation 408. Then, before providing the samples to a convolutional neural network 416, the latent spatial representation 408 is provided to a noisy model 410 to output a noisy latent spatial representation 412. Also as... Figure 3 As described in the paper, text-based data samples are provided to an LLM text encoder, such as a CLIP encoder, to output an embedded text 404.

[0061] As shown in the figure, image-based data sample 402 resembles a manufactured product (e.g., a nut) with defects (e.g., scratches) on the surface of the bottom right side of an image. Since this disclosure relates to detecting defects within a manufacturing setting, image-based data samples can resemble images of a product captured while it is still within the manufacturing facility and has completed the manufacturing process but has not yet left the production facility (e.g., to be sold or shipped elsewhere). In some embodiments, the captured image may correspond to a moment when the manufactured product is undergoing quality inspection in an assembly line setting. Examples of such implementations are described herein. Figure 11 Further illustrations are provided.

[0062] Figure 4The specific image-based data sample shown is a manufactured product resembling a nut. However, it should be understood that images of other manufactured products are also intended to be included in the discussion herein. In some embodiments, the image may resemble a bolt or screw, or some other mechanical product part. In such embodiments, the image may include scratches, dents, defects, or other physical quality issues in a portion of the manufactured product. In other embodiments, the image may resemble a larger portion of a manufactured product. For example, the image may capture the hood of a car manufactured in an automotive manufacturing facility, and the image may also include a portion of the hood of a car with dents or scratches.

[0063] Also Figure 3 The text-based data samples shown include a short word, phrase, or sentence that provides a description for the image-based data sample 402. For example, the text-based data sample 308 corresponding to the image-based data sample 402 could contain the word "defect," the phrase "nut with scratches," or the sentence "The image is the manufactured product X marked on the right." It should be understood that any other short word or phrase that provides initial information to the convolutional neural network 416 indicating that the image-based data sample 402 contains manufacturing defects can also be used equally as the text-based data sample 304, including words and phrases such as "scratches," "dents," "defects," "discoloration," "warping," "bending," "failed quality inspection," etc.

[0064] Returning to the four process flows that collectively define the fine-tuning process 400, the first process flow is illustrated using blocks 402, 406, 408, 410, and 412, and involves preparing a noisy latent space representation 412, which is then used as input to a convolutional neural network 416. To fine-tune the convolutional neural network 416 to learn to detect defects within image-based data samples, an initial latent space representation 408 is provided to a noise model 410, which, when executed, adds random noise to the latent space representation of the image-based data sample 402 to output a noisy latent space representation 412. In some embodiments, the noise model is configured to have a predetermined noise schedule that progressively reduces the signal-to-noise ratio of the original image-based data sample 402. As described additionally below, the added noise is then used during the execution of the convolutional neural network 416 to learn to predict noise (see also the learned noise 422, described additionally below).

[0065] The second process flow in the four process flows involves Figure 4Blocks 416, 404, 412, and 422, and more specifically relating to the execution of convolutional neural network 416. In some embodiments, a noisy latent spatial representation 412 and embedded text 404 are provided to convolutional neural network 416, as described by... Figure 4 As indicated by the arrow in the diagram, the model is then executed to predict noise within the noisy latent spatial representation 412 using multiple cross-attention maps at different spatial resolutions within the U-Net architecture of the convolutional neural network 416. Cross-attention maps can be defined in this paper as the outputs or activations of cross-attention blocks within the U-Net architecture of the convolutional neural network 416 of a larger text-to-image latent diffusion model.

[0066] In some embodiments, the execution of the convolutional neural network 416 includes a forward process and a backward process. During the forward process, Gaussian noise is progressively added to the noisy latent spatial representation to destroy any structure in the image-based data samples, and ultimately transforms the information within the original image-based data samples into Gaussian noise. During the backward process, the convolutional neural network is trained to progressively remove the noise added to the image-based data samples during the forward process, such as via... Figure 4 The noise 422 learned in the process is indicated by this. "Gradually" refers to a process that is autoregressive and includes a large number of steps and / or iterations, in contrast to both the forward and backward processes. Once the given training and / or fine-tuning of the convolutional neural network 416 is complete, the model is thus able to use the backward process to generate image-based data samples, such as detected defects 326.

[0067] In some embodiments, the text-to-image latent diffusion model 300 utilizes an LLM 310, which has been trained on a massive amount of publicly available internet text data, to “guide” the generative process of the convolutional neural network 416 of the text-to-image latent diffusion model 300. The “guidance” of the model can be configured in part by modifying the model’s inverse process, where the inverse process is subject to small perturbations at each step to affect the overall evolution and thus the output of the inverse process. Modifications can be computed using conditional guidance, classifier guidance, or no-classifier guidance. For example, the text-to-image latent diffusion model 300 can be configured to use conditional guidance, and thus the inverse or generative process is “conditioned” on a text-based data sample 308 (e.g., the word “defect”).

[0068] Furthermore, and again by leveraging a large language model, a pre-trained large language model is executed to transform text-based data samples into a list of tokens, which are then further processed into embedding vectors, serving as a vector for each token. For example... Figure 4 As shown, the embedded vectors are then incorporated into the diffusion generation process using a cross-attention layer. The cross-attention layer uses an attention mechanism to ensure that different parts of the noisy latent space representation 412 are correctly influenced by the most relevant parts of the embedded text 404. In some embodiments, the U-Net architecture can be used to configure this connection between the cross-attention layer and the corresponding inputs to the text-to-image latent diffusion model 300.

[0069] Furthermore, the U-Net architecture can also be mathematically derived from (i) (in (referring to the text-to-image diffusion model), or (ii) (in This refers to a text-to-image latent diffusion model. In both cases, It can be defined as the text input for the embedding provided to the model, and These are the trainable weights of the model. At each step of the reverse process... The model is used to predict the amount of noise present in the current iteration of the generation process, for example, where It is in the steps place or The amount of noise predicted in the text. Therefore, conditional text guidance can be written. ,in For the corresponding steps in the generation process They are the same. According to some embodiments, the reverse process may include several steps corresponding to 1000-4000. In order to generate high-quality data. To prevent the reverse or generation process from becoming computationally expensive or slow, the architecture of the convolutional neural network 416 of the text-to-image latent diffusion model 300 can be further modified as follows.

[0070] In some embodiments, a "sampler" can be applied to a diffusion model, where such a configuration makes the reverse process faster without significantly compromising the quality of the generated data. For example, the DDIM sampler modifies the forward process to be non-Markovian, thereby enabling a modified reverse process with a significantly fewer steps. In some embodiments, the DDIM sampler can be written as... , in Together they represent the weights of the entire diffusion model, including the U-Net implementation of the convolutional neural network 416 and the large language model 310.

[0071] In other embodiments of the text-to-image latent diffusion model, the DDIM sampler can be written as... .

[0072] When any of the above equations are applied, the reverse or generation process of the DDIM sampler is deterministic and does not involve each step. Noise is added at the point. This allows data to be encoded into DDIM latent codes or DDIM latent noise vectors using DDIM.

[0073] Note that, in the case of the latent diffusion model, the DDIM latent code representation... Noise VAE potential However, after iteratively applying the given equations, in the case of the diffusion model, the DDIM latent code is Then, the DDIM latent code can be used as the starting point for a reversal or generation process to iteratively regenerate the original data using the equations above. In some embodiments, this protocol can be referred to as DDIM inversion. Mathematically, encoding is also an iterative forward process of transforming data into the DDIM latent code. This can be accomplished in a fixed number of steps. The following equation is used to achieve this: .

[0074] In some embodiments of applying the text-to-image latent diffusion model, a fixed number of steps can be performed. Iteratively apply the following equation: .

[0075] Return now Figure 4 The four process flows illustrated in the figure, the third and fourth process flows are about the calculation of the average diffusion loss parameter 420 and the average defect masking loss parameter 428, and then the average diffusion loss parameter 420 and the average defect masking loss parameter 428 are used to update the weights of the convolutional neural network 416 of the text to image latent diffusion model 300.

[0076] The third process flow of the overall fine-tuning process 400 involves blocks 410, 420, and 422. For example... Figure 4 As shown, the amount of noise applied during the execution of the noise model 410 can be compared with the learned noise 422 learned during the fine-tuning execution of the convolutional neural network 416 in order to calculate the average diffusion loss parameter 420 of the model. (See below for more information.) Figure 9 Block 912 provides additional descriptions of such calculations.

[0077] The overall fine-tuning process, specifically the fourth process flow of 400, involves... Figure 4Blocks 402, 424, 426, 428, and 430. To calculate the average defect mask loss parameter 428, a segmentation mask 426 corresponding to the image-based data sample 402 is first generated. In some embodiments, the image-based data sample 402 is provided to a depth segmentation model 424, and then the depth segmentation model 424 is executed to output the segmentation mask 426. For example, the depth segmentation model 424 can be similar to a Segment Anything Model (SAM).

[0078] In some embodiments, the segmentation mask 426 may resemble a binary image, wherein the pixel value of a subset of pixels in the image data sample 402 specifically corresponding to defects in the manufactured product is 255, while the pixel value of other pixels in the binary image is zero. Figure 4 As illustrated in the figure, the pixel count of the defect on the bottom right side of segmentation mask 426 is 255, while the pixel count of the rest of the image is zero.

[0079] Continuing with the description of the fourth process flow of the overall fine-tuning process 400, the sum of the cross-attention maps 430 at a given spatial resolution 418 is also used to calculate the average defect masking loss parameter 428. In some embodiments, and before performing the fine-tuning process 400, when calculating the average defect masking loss parameter 428, the user can determine which parameter to use. Figure 4 Which of the six spatial resolutions shown is being considered? This instruction on which specific spatial resolution to use can then be provided to the computing device performing the text-to-image latent diffusion model 300 and calculating the parameters 428, since the cross-attention map 430 and the segmentation mask 426 refer to the same spatial resolution 418, in order to perform such a calculation of the average defect mask loss parameter 428. The chosen spatial resolution can typically be one-eighth or one-sixteenth of the spatial resolution of the original image-based data sample 402. In the specific embodiment shown in the figure, spatial resolution 418 refers to a 64×64 resolution. In this document… Figure 5A , 6A Additional examples of cross-attention maps at that specific spatial resolution are also provided in 7A and 8A-8C.

[0080] like Figure 4 As shown, the sum of the cross-attention map 430 at a given spatial resolution 418 and the segmentation mask 426 at spatial resolution 418 based on the image data sample 402 is then used to calculate the average defect mask loss parameter 428. (See below for more information.) Figure 9 Block 910 provides additional descriptions of such calculations.

[0081] After calculating both the average diffusion loss parameter 420 and the average defect masking loss parameter 428, the fifth process flow of the fine-tuning process 400 can also be based on... Figure 4 The process involves using two parameters to update the weights of the convolutional neural network 416 of the text-to-image latent diffusion model 300. To update the model's weights, the average diffusion loss parameter 420 and the average defect mask loss parameter 428 are summed together to determine the total loss parameter of the convolutional neural network 416 of the text-to-image latent diffusion model 300. The total loss parameter is then optimized using any variant of stochastic gradient descent, such as by applying the Adam optimizer. The optimized total loss parameter is then used when updating one or more of the weights of the convolutional neural network 416 of the text-to-image latent diffusion model 300.

[0082] After updating one or more of the weights, a finely tuned convolutional neural network 416 of the text-to-image latent diffusion model 300 can be provided to detect the presence of defects in other image-based data samples of other manufactured products within the image. This is in Figure 5A , 6A Further illustrations are provided in 7A and 8A-8C.

[0083] Figures 5A-5B This refers to the methods and systems described herein (e.g., Figure 5A , 6A The use of 7A) and other past implementations of the diffusion model (e.g., Figure 5B , 6B This is compared to the use of 7B) to illustrate the success of this disclosure in correctly identifying portions of an image that refer to defects, and the overall failure of past implementations of the diffusion model in identifying such information. Furthermore, Figure 5A-7B The illustration continues with image-based data samples of manufactured products, including nut-like items. However, it should be understood that once the text-to-image latent diffusion model 300 has been fine-tuned to determine defects within the image of the manufactured product, it is possible to... Figure 5A-7B Other examples of image-based data samples used in the same way include images of bolts, screws, etc.

[0084] Figure 5A The illustration shows a first example of an image of a defective manufactured product according to some embodiments, and a positive identification of the defective portion in the image by a text-to-image diffusion model. In contrast, Figure 5B The illustration shows that in Figure 5A The same image described in the text, as well as the failure of a certain other diffusion model to identify the defective portion of the image.

[0085] In some embodiments, Figure 5AThis refers to a point in time after which the text-to-image latent diffusion model 300 has been both generally trained and then specifically fine-tuned to detect defects within images of manufactured products. This can also be described as the point in time during which the text-to-image latent diffusion model 300 operates in inference mode.

[0086] As shown in the figure, a fine-tuned convolutional neural network 500 of a larger text-to-image diffusion model (such as text-to-image latent diffusion model 300) is provided with image-based data samples 502 and corresponding text-based data samples including short words or phrases (such as "defect"). The image-based data sample 502 includes defects along the lower half of an image of a manufacturing portion, and the model then correctly identifies the defect in a cross-attention map 504, which corresponds to the output of the fine-tuned convolutional neural network 500 at a spatial resolution of 64×64.

[0087] As described above regarding segmentation mask 426, cross-attention map 504 can be similarly analogous to a binary image, where the pixel count of a subset of pixels in image-based data sample 502 specifically corresponding to defects in the manufactured product is 255, while the pixel count of other pixels in the binary image is zero. As illustrated in the figure, the pixel count of defects in the lower half of cross-attention map 504 is 255, while the pixel count of the remaining portion of the image is zero.

[0088] By calculating both the average diffusion loss parameter 420 and the average defect mask loss parameter 428 during the previous fine-tuning phase of the convolutional neural network, and by updating the model's weights using these two results, the resulting fine-tuned convolutional neural network 500 is then configured to... Figure 5A The model illustrated in the figure correctly identifies defects within image-based data sample 502 during execution.

[0089] In general comparison, Figure 5B The illustration shows a previously implemented diffusion model 550 that completely fails to identify defective portions of image-based data sample 552 when the calculation of the defect mask loss parameter is not incorporated into the weight update. This is illustrated using a cross-attention graph 554, where the diffusion model 550 fails to identify any portion of the image that corresponds to a “defect” and instead incorrectly indicates that none of the images correspond to a “defective” portion of the manufactured product.

[0090] Figure 6A The illustration shows a second example of an image of a defective manufactured product according to some embodiments, and the positive identification of the defective portion in the image by a text-to-image diffusion model. In contrast, Figure 6BThe illustration shows that in Figure 6A The same image described in the text, as well as the failure of a certain other diffusion model to identify the defective portion of the image.

[0091] and Figure 5A Similar to what is shown, Figure 6A This refers to a point in time after which the text-to-image latent diffusion model 300 has been both generally trained and then specifically fine-tuned to detect defects within images of manufactured products. This can also be described as the point in time during which the text-to-image latent diffusion model 300 operates in inference mode.

[0092] As shown in the figure, a fine-tuned convolutional neural network 600 of a larger text-to-image diffusion model (such as text-to-image latent diffusion model 300) is provided with image-based data samples 602 and corresponding text-based data samples including short words or phrases (such as "defect"). The image-based data sample 602 includes defects along the bottom right side of an image of a manufacturing part, and the model then correctly identifies the defects in a cross-attention map 604, which corresponds to the output of the fine-tuned convolutional neural network 600 at a spatial resolution of 64×64.

[0093] As described above regarding segmentation mask 426, cross-attention map 604 can be similarly analogous to a binary image, where the pixel value of a subset of pixels in image-based data sample 602 specifically corresponding to defects in the manufactured product is 255, while the pixel value of other pixels in the binary image is zero. As illustrated in the figure, the pixel value of defects in the bottom right-hand side of cross-attention map 604 is 255, while the pixel value of the rest of the image is zero.

[0094] By calculating both the average diffusion loss parameter 420 and the average defect mask loss parameter 428 during the previous fine-tuning phase of the convolutional neural network, and by updating the model's weights using these two results, the resulting fine-tuned convolutional neural network 600 is then configured to... Figure 6A The model illustrated in the figure correctly identifies defects within image-based data sample 602 during execution.

[0095] In general comparison, Figure 6B The illustration shows a previously implemented diffusion model 650 that completely fails to identify defective portions of image-based data sample 652 when the calculation of the defect mask loss parameter is not incorporated into the weight update. This is illustrated using a cross-attention graph 654, in which the diffusion model 650 fails to identify any portion of the image that corresponds to a “defect” and instead incorrectly indicates that none of the images correspond to a “defective” portion of the manufactured product.

[0096] Figure 7A The illustration shows a third example of an image of a defective manufactured product according to some embodiments, and a positive identification of the defective portion in the image by a text-to-image diffusion model. In contrast, Figure 7B The illustration shows that in Figure 7A The same image described in the text, as well as the failure of a certain other diffusion model to identify the defective portion of the image.

[0097] and Figure 5A and 6A Similar to what is shown, Figure 7A This refers to a point in time after which the text-to-image latent diffusion model 300 has been both generally trained and then specifically fine-tuned to detect defects within images of manufactured products. This can also be described as the point in time during which the text-to-image latent diffusion model 300 operates in inference mode.

[0098] As shown in the figure, a fine-tuned convolutional neural network 700 of a larger text-to-image diffusion model (such as text-to-image latent diffusion model 300) is provided with image-based data samples 702 and corresponding text-based data samples including short words or phrases (such as "defect"). The image-based data sample 702 includes defects along the lower half of an image of a manufacturing portion, and the model then correctly identifies the defect in a cross-attention map 704, which corresponds to the output of the fine-tuned convolutional neural network 700 at a spatial resolution of 64×64.

[0099] As described above regarding segmentation mask 426, cross-attention map 704 can be similarly analogous to a binary image, where the pixel count of a subset of pixels in image-based data sample 702 specifically corresponding to defects in the manufactured product is 255, while the pixel count of other pixels in the binary image is zero. As illustrated in the figure, the pixel count of defects in the lower half of cross-attention map 704 is 255, while the pixel count of the remaining portion of the image is zero.

[0100] By calculating both the average diffusion loss parameter 420 and the average defect mask loss parameter 428 during the previous fine-tuning phase of the convolutional neural network, and by updating the model's weights using these two results, the resulting fine-tuned convolutional neural network 700 is then configured to... Figure 7A The model illustrated in the figure correctly identifies defects within the image-based data sample 702 during execution.

[0101] In general comparison, Figure 7BThe illustration shows a previously implemented diffusion model 750 that completely fails to identify defective portions of image-based data sample 752 when the calculation of the defect mask loss parameter is not incorporated into the weight update. This is illustrated using a cross-attention graph 754, where the diffusion model 750 fails to identify any portion of the image that corresponds to a “defect” and instead incorrectly indicates that none of the images correspond to a “defective” portion of the manufactured product.

[0102] Figure 8A , 8B Figure 8C illustrates three examples of images of defect-free manufactured products according to some embodiments, and the affirmative recognition of the absence of defects within the image by a finely tuned convolutional neural network of a text-to-image diffusion model.

[0103] and Figure 5A , 6A Similar to that shown in 7A, Figure 8A , 8B The term 8C refers to a point in time after which the text-to-image latent diffusion model 300 has been both generally trained and then specifically fine-tuned to detect defects within images of manufactured products. This can also be referred to as the point in time during which the text-to-image latent diffusion model 300 operates in inference mode.

[0104] like Figure 8A As shown, a fine-tuned convolutional neural network 800 of a larger text-to-image diffusion model (such as text-to-image latent diffusion model 300) is provided with image-based data samples 802 and corresponding text-based data samples including short words or phrases (such as "defect"). The image-based data samples 802 do not include the defective portion of the image; the model then correctly identifies the defect in a cross-attention map 804, which corresponds to the output of the fine-tuned convolutional neural network 800 at a spatial resolution of 64×64. As illustrated, the overall pixel count of the cross-attention map 804 is zero, thus confirming that no defect exists in the image-based data samples 802.

[0105] like Figure 8BAs shown, a fine-tuned convolutional neural network 800 of a larger text-to-image diffusion model (such as text-to-image latent diffusion model 300) is provided with image-based data samples 812 and corresponding text-based data samples including short words or phrases (such as "defect"). The image-based data samples 812 do not include the defective portion of the image; the model then correctly identifies the defect in a cross-attention map 814, which corresponds to the output of the fine-tuned convolutional neural network 800 at a spatial resolution of 64×64. As illustrated, the overall pixel count of the cross-attention map 814 is zero, thus confirming that no defect exists in the image-based data samples 812.

[0106] like Figure 8C As shown, a fine-tuned convolutional neural network 800 of a larger text-to-image diffusion model (such as text-to-image latent diffusion model 300) is provided with image-based data samples 822 and corresponding text-based data samples including short words or phrases (such as "defect"). The image-based data samples 822 do not include the defective portion of the image; the model then correctly identifies the defect in a cross-attention map 824, which corresponds to the output of the fine-tuned convolutional neural network 800 at a spatial resolution of 64×64. As illustrated, the overall pixel count of the cross-attention map 824 is zero, thus confirming the absence of a defect in the image-based data samples 822.

[0107] By calculating both the average diffusion loss parameter 420 and the average defect mask loss parameter 428 during the previous fine-tuning phase of the convolutional neural network, and by updating the model's weights using these two results, the resulting fine-tuned convolutional neural network 800 is then configured to... Figure 8A , 8B During the execution of the model illustrated in 8C, image-based data samples 802, 812, and 822 were correctly identified as clean of defects.

[0108] Figure 9 This is a flowchart illustrating, according to some embodiments, the process of fine-tuning a convolutional neural network of a text-to-image diffusion model to detect portions in an image of a manufactured product that correspond to defects within the product.

[0109] The following description of process 900 relates to preprocessing steps (e.g., block 902) prior to the execution of the convolutional neural network for fine-tuning the text-to-image latent diffusion model, to the execution of the convolutional neural network for the text-to-image diffusion model (e.g., blocks 904, 906, 908, 910, 912, 914, 916, and 918), and to postprocessing steps (e.g., block 920) after the text-to-image convolutional neural network has been fine-tuned. Therefore, for the convenience of the discussion herein, reference may also be made to... Figure 3 The blocks within provide additional system-based context for the method steps described in the following paragraphs.

[0110] In block 902, both image-based and text-based data samples are received by a computing device that performs fine-tuning of a convolutional neural network for a text-to-image diffusion model. The image-based data samples include captured images of manufactured products that have been identified as "defective" for some reason, and the text-based data samples use short words or phrases such as "defect," "scratch," "stain," or "dent" to identify defects in the manufactured products. In some embodiments, the image-based data samples and corresponding text-based data samples may be received as a dataset and may also be referred to as a labeled dataset, since the text-based data samples can serve as ground truths that the images indeed contain defects somewhere within the corresponding images. As stated in the preceding paragraphs, and for the convenience of the discussion in the following paragraphs, examples of image-based data samples and corresponding text-based data samples may refer to image-based data sample 402 and text-based data sample 308.

[0111] Then, blocks 904, 906, 908, 910, 912, 914, 916, and 918 relate to the above regarding... Figure 4 The description outlines the various steps within the overall fine-tuning process 400. Block 904 broadly refers to the fine-tuning process 400, in which a pre-trained but not yet fine-tuned convolutional neural network (e.g., a U-Net architecture of a stable diffusion model) is executed to learn to predict noise in image-based data samples using multiple cross-attention maps at several spatial resolutions.

[0112] In block 908, a depth segmentation model (such as SAM) is provided with image-based data samples of a dataset and then executed to output a corresponding segmentation mask. This segmentation mask uses a pixel value of 255 to indicate portions of the image that include defects and uses a pixel value of zero to indicate portions of the image that do not include defects. In some embodiments, the segmentation mask may additionally be referred to as a defect mask. Furthermore, the depth segmentation model is configured to output the corresponding segmentation mask at the same spatial resolution as the spatial resolution, which will later be used to calculate the average defect mask loss parameter in block 910.

[0113] In block 906, the VAE autoencoder is used to output the latent space representation of the original image-based data samples, and then performs a noise model to output a noisy latent space representation, such as... Figure 4 The noisy latent space is represented in 412.

[0114] Then, the noisy latent spatial representations embedded using an LLM text encoder, along with their corresponding embedded text, are fed into a convolutional neural network for a text-to-image latent diffusion model, which is then executed. To learn to predict noise from image-based data samples, iterative steps involving multiple cross-attention maps at different spatial resolutions are computed. In some embodiments, this can also be interpreted using the following process flow.

[0115] Iterative execution of a convolutional neural network for a text-to-image latent diffusion model Next, among them, targeting arrive In each iteration of the iteration, the following process steps are completed: for mini-batch images and the corresponding segmentation mask Sampling is performed. Then, the U-Net-based architecture is initialized using the weights of a pre-trained convolutional neural network (e.g., U-Net) of a text-to-image diffusion model or a text-to-image latent diffusion model (such as stable diffusion). The weights are then determined. Next, a noise vector of the same size as the mini-batch is sampled. ,in It is the sample index within the mini-batch, and subsequently samples values ​​for a time step as large as the mini-batch size. Then, use Or, in the case of a text-to-image latent diffusion model, use , to noise Added to each image-based data sample, where It is a predetermined noise schedule for gradually reducing the signal-to-noise ratio during the forward pass of a convolutional neural network.

[0116] During the forward pass of a convolutional neural network, various cross-attention maps at different spatial resolutions can be saved to a buffer for future use when computing the parameters of the model (such as those described in blocks 910 and / or 912).

[0117] In block 910, the average defect masking loss parameter is calculated. As previously used Figure 4 As illustrated in spatial resolution 418, a given spatial resolution is selected from among several spatial resolutions to be used for calculating the average defect mask loss parameter. For example, the selected spatial resolution could be one-eighth or one-sixteenth of the original spatial resolution of the image-based data sample, as described in block 902. By adjusting the selected spatial resolution (e.g., ... Figure 4 The parameter is calculated by averaging the corresponding cross-attention maps at a spatial resolution of 418, where the average cross-attention map can then be represented as... Then, the average defect masking loss parameter can be calculated as follows: .

[0118] In block 912, the average diffusion loss parameter is calculated. In some embodiments, the average diffusion loss parameter can be calculated as follows, wherein, by adjusting the exponent... The expected value is calculated by averaging; for simplicity, this index... Not included in the following equations: .

[0119] In block 914, the average diffusion loss parameter... and average defect mask loss parameters Summation is performed to calculate the total loss parameter of the model: .

[0120] In block 916, the total loss parameter is then optimized using any variant of stochastic gradient descent (such as the Adam optimizer). .

[0121] In block 918, the weights of the convolutional neural network for the text-to-image latent diffusion model are updated. One or more of these can be used to generate both a pre-trained version and a fine-tuned version of the overall text-to-image latent diffusion model. In some embodiments, this can also be written as... .

[0122] Once fine-tuned, the text-to-image latent diffusion model can then be used to detect defects in other image-based data samples of manufactured products, as illustrated in block 920. For example, this model can be implemented in an automated optical inspection (AOI) system, which captures images of products that have just been manufactured and are now being inspected for quality control purposes. The fine-tuned text-to-image diffusion model is then executed to determine whether the images of the manufactured products undergoing quality inspection have defects, scratches, dents, etc., or whether they do not contain such defects and pass the quality control inspection. This is discussed in this paper. Figure 11 Further illustrations are provided.

[0123] The methods and systems disclosed in this paper can be used in many different applications. Using text-to-image latent diffusion models (such as those described in this paper) can better optimize the identification of production and / or manufacturing errors within a manufactured product before it leaves the manufacturing facility for sale. Figure 10 and 11 The diagram illustrates how such a scenario can be implemented.

[0124] Figure 10 A schematic diagram depicting the interaction between a computer-controlled machine 1000 and a control system 1002 is shown. The computer-controlled machine 1000 includes an actuator 1004 and a sensor 1006. The actuator 1004 may include one or more actuators, and the sensor 1006 may include one or more sensors. The sensor 1006 is configured to sense conditions of the computer-controlled machine 1000. Based on the teachings herein, the sensor 1006 may be configured to sense ID and / or OOD data, and a corresponding processor may be configured to determine whether the data is ID or OOD. The sensor 1006 may be configured to encode the sensed conditions into a sensor signal 1008 and transmit the sensor signal 1008 to the control system 1002. Non-limiting examples of the sensor 1006 include a camera, a video sensor, an optical sensor, and the like. In one embodiment, the sensor 1006 is an optical sensor configured to sense an optical image of the environment adjacent to the computer-controlled machine 1000.

[0125] The control system 1002 is configured to receive sensor signals 1008 from the computer-controlled machine 1000. As described below, the control system 1002 may also be configured to calculate actuator control commands 1010 based on the sensor signals and transmit the actuator control commands 1010 to the actuators 1004 of the computer-controlled machine 1000.

[0126] like Figure 10As shown, the control system 1002 includes a receiving unit 1012. The receiving unit 1012 can be configured to receive a sensor signal 1008 from the sensor 1006 and convert the sensor signal 1008 into an input signal. In an alternative embodiment, sensor signal 1008 is used as an input signal. It is received directly, without a receiving unit 1012. Each input signal This can be a part of each sensor signal 1008. The receiving unit 1012 can be configured to process each sensor signal 1008 to generate each input signal. Input signal This may include data corresponding to the image recorded by sensor 1006. For example, image-based data samples and text-based data samples may be received by receiving unit 1012.

[0127] The control system 1002 includes a fine-tuned text-to-image latent diffusion model 1014. The fine-tuned text-to-image latent diffusion model 1014 can be configured to determine whether an incoming image of a manufactured product from sensor 1006 contains defects. The fine-tuned text-to-image latent diffusion model 1014 is configured to be determined by parameters (such as those described above, e.g., parameters...). )) to parameterize. Parameters It can be stored in and provided by the non-volatile storage device 1016. The fine-tuned text-to-image latent diffusion model 1014 is configured to adjust according to the input signal. Determine the output signal Each output signal This includes assigning one or more labels to each input signal. The information. The finely tuned text-to-image latent diffusion model 1014 can output the signal. The signal is transmitted to the conversion unit 1018. The conversion unit 1018 is configured to convert the output signal... The signal is converted into actuator control command 1010. Control system 1002 is configured to transmit actuator control command 1010 to actuator 1004, which is configured to actuate computer-controlled machine 1000 in response to actuator control command 1010. In another embodiment, actuator 1004 is configured to directly actuate the output signal. To activate the computer-controlled machine 1000.

[0128] Upon receiving an actuator control command 1010, actuator 1004 is configured to perform an action corresponding to the associated actuator control command 1010. Actuator 1004 may include control logic configured to transform the actuator control command 1010 into a second actuator control command for controlling actuator 1004. In one or more embodiments, actuator control command 1010 may be used to control a display (instead of or attached to the actuator).

[0129] In another embodiment, the control system 1002 includes a sensor 1006 (instead of or attached to a computer-controlled machine 1000 including sensor 1006). The control system 1002 may also include an actuator 1004 (instead of or attached to a computer-controlled machine 1000 including actuator 1004).

[0130] like Figure 10 As shown, the control system 1002 also includes a processor 1020 and a memory 1022. The processor 1020 may include one or more processors. The memory 1022 may include one or more memory devices. A fine-tuned text-to-image latent diffusion model 1014 of one or more embodiments may be implemented by the control system 1002, which includes a non-volatile storage device 1016, a processor 1020, and a memory 1022.

[0131] The non-volatile storage device 1016 may include one or more persistent data storage devices, such as hard disks, optical drives, tape drives, non-volatile solid-state devices, cloud storage devices, or any other device capable of permanently storing information. The processor 1020 may include one or more devices selected from a high-performance computing (HPC) system, including high-performance cores, microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other device that manipulates signals (analog or digital) based on computer-executable instructions residing in memory 1022. Memory 1022 may include a single memory device or multiple memory devices, including but not limited to random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information. Furthermore, processor 1020 and memory 1022 can be configured to provide collected data to one or more other computing devices configured to execute a fine-tuned text-to-image latent diffusion model 1014 within a domain-specific embodiment. Figure 11 As shown in the diagram. Such collected data can be used to generate training and validation datasets for various stages of preparing and executing machine learning models in industrial-grade applications. In the scenario described herein concerning fine-tuning a text-to-image latent diffusion model 1014, processor 1020 and memory 1022 can be coupled to or otherwise remotely connected to a computing device, which can then perform fine-tuning processes such as those described above.

[0132] Processor 1020 may be configured to read into memory 1022 and execute computer-executable instructions residing in non-volatile storage device 1016 and embodying one or more machine learning algorithms and / or methods of one or more embodiments. Non-volatile storage device 1016 may include one or more operating systems and applications. Non-volatile storage device 1016 may store programs compiled and / or interpreted from computer programs created using various programming languages ​​and / or techniques, including but not limited to (and used individually or in combination) Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL.

[0133] When executed by processor 1020, computer-executable instructions of non-volatile storage device 1016 may cause control system 1002 to implement one or more of the machine learning algorithms and / or methods disclosed herein. Non-volatile storage device 1016 may also include machine learning data (including data parameters) supporting the functionality, features, and processes of one or more embodiments described herein.

[0134] Program code embodying the algorithms and / or methods described herein can be distributed individually or collectively as a program product in various different forms. The program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon, for causing a processor to execute aspects of one or more embodiments. The inherently non-transitory computer-readable storage medium can include tangible media that are volatile and non-volatile, removable and non-removable, implemented in any method or technology, for storing information such as computer-readable instructions, data structures, program modules, or other data. The computer-readable storage medium can also include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technologies, portable optical disc read-only memory (CD-ROM) or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be read by a computer. The computer-readable program instructions can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device, or another device, or downloaded via a network to an external computer or external storage device.

[0135] Computer-readable program instructions stored in a computer-readable medium can be used to instruct a computer, other type of programmable data processing apparatus, or other device that functions in a particular manner to produce an article of manufacture comprising instructions for implementing the functions, actions, and / or operations specified in a flowchart or diagram. In some alternative embodiments, consistent with one or more embodiments, the functions, actions, and / or operations specified in a flowchart or diagram can be reordered, processed sequentially, and / or processed concurrently. Furthermore, any flowchart or diagram may include more or fewer nodes or blocks than illustrated consistent with one or more embodiments.

[0136] The process, method, or algorithm may be embodied, in whole or in part, using suitable hardware components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0137] Figure 11 A schematic diagram of a control system 1002 is depicted, which is configured to control a system 1100 (e.g., an automated optical inspection system) of a manufacturing system 1102 (e.g., a production line). The control system 1002 may be configured to control an actuator 1004, which is configured to control the control system 1100.

[0138] The sensor 1006 of system 1100 (e.g., a manufacturing machine) may be an optical sensor configured to capture one or more attributes of the manufactured product 1104. A fine-tuned text-to-image latent diffusion model 1014 may be configured to determine the state of the manufactured product 1104 based on one or more of the captured attributes. The actuator 1004 may be configured to control system 1100 (e.g., the manufacturing machine) with respect to subsequent manufactured products 1106 based on the determined state of the manufactured product 1104. For example, if control system 1002 determines that a defect exists on or within the manufactured product 1104, the system may then command actuator 1004 to control system 1100 to remove the manufactured product 1104 from production line 1102 for further review. In another example, system 1100 can be used to halt the movement of production line 1102 while awaiting further review of manufactured product 1104. In such an example, the review of manufactured product 1106 can be suspended until the status of manufactured product 1104 is determined.

[0139] While exemplary embodiments have been described above, they are not intended to describe all possible forms encompassed by the claims. The language used in this specification is descriptive and not restrictive, and it is to be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously stated, features of various embodiments may be combined to form other embodiments of the invention that may not be explicitly described or illustrated. While various embodiments may have been described as providing an advantage or superiority over other embodiments or prior art implementations with respect to one or more desired features, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system properties depending on the particular application and implementation. These properties may include, but are not limited to, cost, strength, durability, lifecycle cost, merchantability, appearance, packaging, size, suitability, weight, manufacturability, ease of assembly, etc. Therefore, any embodiment described with respect to one or more features as not conforming as well as other embodiments or prior art implementations to the desired scope is not outside the scope of this disclosure and may be desirable for a particular application.

Claims

1. A computer-implemented method for fine-tuning a text-to-image latent diffusion model, comprising: Receive image-based data samples and embedded text samples, where: The image-based data samples are images of defective manufactured products; and The embedded text sample is an embedding of a text-based data sample that indicates the defect; Perform a variational autoencoder to output a latent spatial representation of the image-based data samples; Perform a noise model to output a noisy version of the latent space representation; The noisy version of the latent space representation and the embedded text samples are provided to the convolutional neural network of the text-to-image latent diffusion model; The convolutional neural network is executed to learn to predict noise in the image-based data samples using multiple cross-attention maps at different spatial resolutions; Based on the cross-attention map of a given spatial resolution among the different spatial resolutions and the segmentation mask corresponding to the image-based data sample, calculate the average defect mask loss parameter; Update one or more weights of the convolutional neural network, at least in part, based on the average defect mask loss parameter; and The output is a finely tuned text-to-image latent diffusion model with one or more updated weights, used to detect defects in other image-based data samples of other manufactured products.

2. The computer-implemented method according to claim 1 further includes: The average defect masking loss parameter and the average diffusion loss parameter are summed to determine the total loss parameter; The total loss parameter is optimized using stochastic gradient descent. as well as Additionally, the weights of the convolutional neural network are updated based on the optimized total loss parameter.

3. The computer-implemented method according to claim 2 further includes: The average diffusion loss parameter is calculated based on the noise model and the noise learned by the convolutional neural network. as well as The average diffusion loss parameter is provided to determine the total loss parameter.

4. The computer-implemented method of claim 2, wherein the stochastic gradient descent is the Adam optimizer.

5. The computer-implemented method according to claim 1, further comprising: The image-based data samples are provided to the deep segmentation model; as well as The depth segmentation model is executed to output the segmentation mask used to calculate the average defect mask loss parameters.

6. The computer-implemented method according to claim 5, wherein: The segmentation mask is a binary image; The pixel count of the subset of pixels in the binary image corresponding to the defect in the manufactured product is 255; and The pixel count of the other pixels in the binary image is zero.

7. The computer-implemented method of claim 1, wherein the noise model is configured to have a predetermined noise schedule that gradually reduces the signal-to-noise ratio of the latent spatial representation of the image-based data samples.

8. The computer-implemented method according to claim 1, further comprising: A given spatial resolution is determined from among the different spatial resolutions to be used to calculate the average defect mask loss parameter, wherein the given spatial resolution is one-eighth or one-sixteenth of the spatial resolution of the image-based data sample; as well as Before execution, the determined spatial resolution is provided to the convolutional neural network.

9. The computer-implemented method according to claim 1, wherein the text-to-image latent diffusion model is a stable diffusion model.

10. The computer-implemented method of claim 1, wherein the convolutional neural network is configured to have a U-Net architecture.

11. The computer-implemented method of claim 1, wherein the image-based data sample is an image of a bolt, screw, or nut.

12. A computer-implemented method for fine-tuning a text-to-image diffusion model, comprising: Receive image-based data samples and embedded text samples, where: The image-based data samples are images of defective manufactured products; and The embedded text sample is an embedding of a text-based data sample that indicates the defect; Perform a depth segmentation model to output a segmentation mask corresponding to the image-based data sample; The noisy version of the image-based data sample and the embedded text sample are provided to the convolutional neural network of the text-to-image diffusion model; The convolutional neural network is executed to learn to predict noise in the image-based data samples using multiple cross-attention maps at different spatial resolutions; Based on the cross-attention map of a given spatial resolution among the different spatial resolutions and the segmentation mask corresponding to the image-based data sample, calculate the average defect mask loss parameter; Update one or more weights of the convolutional neural network, at least in part, based on the average defect mask loss parameter; and The output is a finely tuned text-to-image diffusion model with one or more updated weights, used to detect defects in other image-based data samples of other manufactured products.

13. The computer-implemented method according to claim 12, wherein: The segmentation mask is a binary image; The pixel count of the subset of pixels in the binary image corresponding to the defect in the manufactured product is 255; and The pixel count of the other pixels in the binary image is zero.

14. The computer-implemented method of claim 12, further comprising executing a noise model to output the noisy version of the image-based data sample, wherein the noise model is configured to have a predetermined noise schedule that gradually reduces the signal-to-noise ratio of the image-based data sample.

15. The computer-implemented method according to claim 12, further comprising: The average defect masking loss parameter and the average diffusion loss parameter are summed to determine the total loss parameter; The total loss parameter is optimized using stochastic gradient descent. as well as Additionally, the one or more weights of the convolutional neural network are updated based on the optimized total loss parameter.

16. The computer-implemented method according to claim 15, further comprising: The average diffusion loss parameter is calculated based at least in part on the noise learned by the convolutional neural network. as well as The average diffusion loss parameter is provided to determine the total loss parameter.

17. A non-transitory computer-readable medium storing program instructions that, when executed on or across one or more processors, cause the one or more processors to: Receive embedded text samples, image-based data samples, and a segmentation mask corresponding to the image-based data samples, wherein:    The image-based data samples are images of defective manufactured products; as well as The embedded text sample is an embedding of a text-based data sample that indicates the defect; Generate a noisy version of the image-based data sample; Using the embedded text samples and the noisy version of the image-based data samples, a convolutional neural network of a text-to-image diffusion model is executed to learn to predict the noise of the image-based data samples using multiple cross-attention maps at different spatial resolutions; Based on the cross-attention map of a given spatial resolution among the different spatial resolutions and the segmentation mask, calculate the average defect mask loss parameter; One or more weights of the convolutional neural network are updated, at least in part, based on the average defect mask loss parameter. as well as The output is a finely tuned text-to-image diffusion model with one or more updated weights, used to detect defects in other image-based data samples of other manufactured products.

18. The non-transitory computer-readable medium according to claim 17, wherein, In order to generate the noisy version of the image-based data sample, the program instructions cause the one or more processors to execute a noise model that gradually reduces the signal-to-noise ratio of the image-based data sample.

19. The non-transitory computer-readable medium of claim 18, wherein the program instructions further cause the one or more processors to: The average defect masking loss parameter and the average diffusion loss parameter are summed to determine the total loss parameter; The total loss parameter is optimized using stochastic gradient descent; and Additionally, the one or more weights of the convolutional neural network are updated based on the optimized total loss parameter.

20. The non-transitory computer-readable medium of claim 19, wherein the program instructions cause the one or more processors to... Based on the noise model and the noise learned by the convolutional neural network, the average diffusion loss parameter is calculated; and The average diffusion loss parameter is provided to determine the total loss parameter.

Images