Method for generating abnormal image based on conditional diffusion model, and method and system for training artificial intelligence model on basis of same

Abstract

A method for generating an abnormal image based on a conditional diffusion model according to one embodiment of the present invention is a method for generating an abnormal image based on a conditional diffusion model by a computing system including a memory and a processor, the method comprising the steps of: inputting a normal image (in-distribution data) into a diffusion model; obtaining an incomplete diffusion-based noisy image on the basis of the diffusion model; obtaining, on the basis of the noisy image, a semantic mask which is a mask specifying a semantic element for the normal image, and a nuisance mask which is a mask specifying a nuisance element for the normal image; and generating an abnormal image (out-of-distribution data) based on semantic outlier generation via nuisance awareness (SONA) guidance which is a data processing guide that preserves the nuisance element and transforms the semantic element on the basis of the semantic mask and the nuisance mask.

Description

Abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method and system based thereon

[0001] The present invention relates to a method for generating abnormal images based on a conditional diffusion model, and a method and system for training an artificial intelligence model based thereon. More specifically, the present invention relates to a method for generating abnormal images based on a conditional diffusion model, which generates abnormal images that have similar nuisance elements and different semantic elements from normal images, and trains an artificial intelligence model based on the generated virtual abnormal images, and a method and system for training an artificial intelligence model based thereon.

[0002] Out-of-Distribution (OOD) detection is an important problem that accurately determines when an input does not belong to the distribution (hereinafter referred to as “In-Distribution (ID)”) that the AI ​​model was trained on.

[0003] These technologies are essential in a variety of real-world applications, including medical diagnosis, autonomous driving, and manufacturing process monitoring, and can contribute to ensuring model reliability and safety.

[0004] Recently, various studies based on the Outlier Exposure approach have been proposed, attempting to improve the ability to distinguish between normal and abnormal data.

[0005] However, existing anomaly detection methodologies have several major problems.

[0006] First, many studies focus on modeling the normal distribution using only normal samples and detecting anomalies based on whether new inputs fall within this distribution. While this approach enables effective anomaly detection based on a thorough understanding of normal data, it is often difficult to secure sufficient anomalous data (OOD samples) in real-world applications. In particular, anomalous data can be extremely rare in certain domains, or their collection and labeling can be costly and time-consuming, limiting their practical application.

[0007] Furthermore, while recent efforts to synthesize virtual abnormal images using diffusion models have attracted attention, existing pixel-level OOD synthesis methods still have challenges. First, existing synthesis methods using diffusion models generate images that are too different from the identification (ID), which may not effectively reflect subtle abnormal situations that may occur in real-world applications. Second, these methods fail to properly utilize ambient elements such as background or nuisance, limiting the ability to reflect subtle differences that may occur in situations where the generated abnormal images resemble actual ID data. Consequently, these limitations can make it difficult for anomaly detection models to distinguish OOD samples that are actually similar to the ID.

[0008] Therefore, there is a growing need for a new methodology that can effectively generate abnormal data while still using normal data.

[0009] One embodiment of the present invention is designed to solve the problems of the prior art as described above, and provides a method for generating abnormal images based on a conditional diffusion model that generates abnormal images having similar nuisance elements and different semantic elements to normal images, and trains an artificial intelligence model based on the generated virtual abnormal images, and a method and system for training an artificial intelligence model based thereon.

[0010] However, the technical problems to be solved by the present invention and embodiments of the present invention are not limited to the technical problems described above, and other technical problems may exist.

[0011] A method for generating an abnormal image based on a conditional diffusion model according to one embodiment of the present invention is a method for generating an abnormal image based on a conditional diffusion model by a computing system including a memory and a processor, the method comprising: inputting a normal image (In-Distribution data) into a diffusion model; obtaining a noisy image based on incomplete diffusion based on the diffusion model; obtaining a semantic mask, which is a mask specifying a semantic element for the normal image, and a nuisance mask, which is a mask specifying a nuisance element for the normal image, based on the noisy image; and generating an abnormal image (Out-Of-Distribution data) based on SONA (Semantic Outlier generation via Nuisance Awareness) guidance, which is a data processing guide that preserves the nuisance element and modifies the semantic element, based on the semantic mask and the nuisance mask.

[0012] In another aspect, the above diffusion model is a model that gradually injects noise into an input image based on a forward diffusion process and gradually removes the noise based on a reverse diffusion process.

[0013] In another aspect, the step of obtaining the incomplete diffusion-based noisy image includes the step of injecting the noise into the normal image up to a predetermined time step.

[0014] In another aspect, the step of obtaining the meaning mask and the surrounding mask includes a step of performing conditional diffusion that executes the de-diffusion process according to a specific condition based on the noisy image.

[0015] In another aspect, the step of obtaining the meaning mask and the surrounding mask further includes the step of performing unconditional diffusion, which executes the unconditional de-diffusion process based on the noisy image.

[0016] In another aspect, the step of obtaining the above-mentioned meaning mask and the above-mentioned surrounding mask further includes the step of calculating a conditional prediction value which is a noise prediction value according to the conditional diffusion and an unconditional prediction value which is a noise prediction value according to the unconditional diffusion.

[0017] In another aspect, the step of obtaining the meaning mask and the surrounding mask further includes the step of calculating a noise prediction error, which is a difference value between the conditional prediction value and the unconditional prediction value.

[0018] In another aspect, the step of obtaining the semantic mask and the peripheral mask further includes the step of generating the semantic mask based on pixels in which the absolute value of the noise prediction error belongs to a predetermined upper N(>0) percent, and generating the peripheral mask based on pixels in which the absolute value of the noise prediction error belongs to a predetermined lower M(>0) percent.

[0019] In another aspect, the step of generating an abnormal image based on the SONA guidance includes the step of setting a SONA guidance component, which is a component that specifies a method of applying the SONA guidance to the de-spreading process.

[0020] In another aspect, the SONA guidance component includes an original meaning removal component, which is a component that gradually removes the semantic elements, a peripheral information preservation component, which is a component that preserves the peripheral elements, and a new meaning addition component, which is a component that adds new semantic elements.

[0021] In another aspect, the new semantic addition component is a component that adds the new semantic element to an area excluding an area overlapping with the surrounding mask.

[0022] In another aspect, the step of setting the SONA guidance component includes the step of obtaining a SONA guidance integration component, which is a component that combines the original meaning removal component, the peripheral information preservation component, and the new meaning addition component.

[0023] In another aspect, the step of generating an abnormal image based on the SONA guidance further includes a step of applying the SONA guidance integration component to a noise prediction value for each time step according to the despreading process.

[0024] Meanwhile, one embodiment of the present invention includes a method for performing artificial intelligence model learning based on abnormal images based on a conditional diffusion model by a computing system including a memory and a processor.

[0025] On the other hand, a system for generating abnormal images based on a conditional diffusion model according to an embodiment of the present invention includes at least one memory; and at least one processor for reading at least one application stored in the memory to generate an abnormal image based on a conditional diffusion model; wherein instructions of the processor include instructions for performing the steps of: inputting a normal image (In-Distribution data) into a diffusion model; obtaining a noisy image (Noisy Image) based on incomplete diffusion based on the diffusion model; obtaining a semantic mask (Semantic Mask) which is a mask that specifies a semantic element for the normal image and a nuisance mask (Nuisance Mask) which is a mask that specifies a nuisance element for the normal image based on the noisy image; and generating an abnormal image (Out-Of-Distribution data) based on SONA (Semantic Outlier generation via Nuisance Awareness) guidance which is a data processing guide that preserves the nuisance element and modifies the semantic element based on the semantic mask and the nuisance mask.

[0026] In another aspect, a system for generating abnormal images based on a conditional diffusion model according to an embodiment of the present invention includes a plurality of neurons configured as an array including at least one register, at least one programmable logic, and at least one input interface; a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons; and at least one routing network that controls data flow between the plurality of neurons; wherein each of the plurality of neurons is connected to at least one other neuron through the routing network to set a transmission path of the weights, and further includes a field programmable gate array (FPGA) implementation for a predetermined artificial neural network.

[0027] In another aspect, a system for generating abnormal images based on a conditional diffusion model according to an embodiment of the present invention includes a plurality of neurons organized in an array including at least one register, at least one microprocessor, and at least one input; and a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons; wherein each of the plurality of neurons further includes an application specific integrated circuit (ASIC) for a predetermined artificial neural network connected to at least one other neuron through one of the plurality of synaptic circuits.

[0028] An abnormal image generation method based on a conditional diffusion model according to one embodiment of the present invention and an artificial intelligence model learning method and system based thereon generate abnormal images having similar nuisance elements and different semantic elements to normal images, thereby having the effect of expanding learning data of an anomaly detection model by utilizing virtual abnormal images even in an environment where it is difficult to collect actual abnormal images (i.e., OOD data).

[0029] Therefore, the abnormal image generation method based on a conditional diffusion model according to one embodiment of the present invention and the artificial intelligence model learning method and system based thereon can save the cost and time required for building learning data, and can easily increase the abnormality detection performance of a normal data-based learning model by generating high-quality abnormal images even in a situation where only normal images exist, thereby providing a model that exhibits high performance even for difficult anomaly detection problems such as near-OOD.

[0030] At this time, the abnormal image generation method based on a conditional diffusion model according to one embodiment of the present invention and the artificial intelligence model learning method and system based thereon clearly distinguish the peripheral (Nuisance) elements and semantic elements based on a normal image and provide a guide for inducing data processing (i.e., reverse diffusion method) optimized for each of them, thereby having the effect of directly and significantly improving the reliability and accuracy of the abnormal image generated thereby.

[0031] In addition, the abnormal image generation method based on a conditional diffusion model according to one embodiment of the present invention and the artificial intelligence model learning method and system based thereon train the artificial intelligence model based on the generated virtual abnormal image, thereby implementing model learning using high-quality abnormal data even in an environment where abnormal data is insufficient or difficult to secure, thereby having the effect of drastically improving model performance (e.g., anomaly detection and / or OOD (Out-Of-Distribution) performance, etc.).

[0032] Through this, the abnormal image generation method based on a conditional diffusion model according to one embodiment of the present invention and the artificial intelligence model learning method and system based thereon provide a high-performance artificial intelligence solution applicable to various domains such as manufacturing, medical, security, and / or autonomous driving, thereby having the effect of supporting the multifaceted construction of a robust system capable of identifying and responding to potential risk factors in advance.

[0033] In addition, the abnormal image generation method based on a conditional diffusion model according to one embodiment of the present invention and the artificial intelligence model learning method and system based thereon reduce the cost and time required for actual abnormal data collection, and at the same time support the construction of an artificial intelligence model that can flexibly respond to various abnormal situations using only normal data, thereby providing a data-based solution framework that is both efficient and reliable.

[0034] However, the effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood from the description below.

[0035] FIG. 1 illustrates an example of a block diagram of a computing system that implements an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon according to one embodiment of the present invention.

[0036] FIG. 2 illustrates an example of a block diagram of a computing device that implements an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon according to one embodiment of the present invention.

[0037] FIG. 3 illustrates an example block diagram from another aspect of a computing device that implements an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon according to one embodiment of the present invention.

[0038] FIG. 4 illustrates a flowchart for explaining an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon according to one embodiment of the present invention.

[0039] FIG. 5 illustrates a conceptual diagram for explaining an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon according to one embodiment of the present invention.

[0040] FIG. 6 illustrates examples of a semantic mask and a nuisance mask for an original image according to one embodiment of the present invention.

[0041] FIG. 7 illustrates examples of an original image and a virtual abnormal image according to one embodiment of the present invention.

[0042] The present invention is capable of various modifications and embodiments. Therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention, as well as the methods for achieving them, will become clear with reference to the embodiments described in detail below together with the drawings. However, the present invention is not limited to the embodiments disclosed below and can be implemented in various forms. In the following embodiments, the terms "first," "second," etc. are not used in a limiting sense but are used for the purpose of distinguishing one component from another. Furthermore, the singular expression includes the plural expression unless the context clearly indicates otherwise. Furthermore, terms such as "include" or "have" indicate the presence of a feature or component described in the specification, and do not preemptively exclude the possibility that one or more other features or components may be added. Furthermore, in the drawings, the sizes of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, and thus the present invention is not necessarily limited to what is shown.

[0043] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals and redundant descriptions thereof will be omitted.

[0044]

[0045] [An exemplary system implementing an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model training method based on this method]

[0046] Hereinafter, an exemplary system for implementing a method for generating abnormal images based on a conditional diffusion model, which generates abnormal images having similar nuisance elements and different semantic elements for normal images and trains an artificial intelligence model based on the generated virtual abnormal images, and a method for training an artificial intelligence model based on the abnormal image is described in detail with reference to the attached drawings.

[0047] FIG. 1 illustrates an example of a block diagram of a computing system that implements an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon according to one embodiment of the present invention.

[0048] Referring to FIG. 1, a computing system (1000) implementing an abnormal image generation method based on a conditional diffusion model of the present invention and an artificial intelligence model learning method based thereon includes a user computing device (110), a server computing system (130), and a training computing system (150), and each device and system are communicatively connected via a network (170).

[0049] The abnormal image generation method based on a conditional diffusion model according to one embodiment of the present invention and the artificial intelligence model learning method and system based thereon may 1) be implemented and provided locally by a user computing device (110), 2) be implemented and provided in the form of a web service by a server computing system (130) communicating with the user computing device (110), or 3) be implemented and provided by the user computing device (110) and the server computing system (130) in connection with each other.

[0050] At this time, in the embodiment, the user computing device (110) and / or the server computing system (130) can train the machine learning model (120 and / or 140) through interaction with a training computing system (150) communicatively connected via a network (170). The training computing system (150) may be separate from the server computing system (130) or may be part of the server computing system (130).

[0051] And at this time, the artificial intelligence model can be 1) trained directly locally by the user computing device (110), 2) trained by the server computing system (130) and the user computing device (110) interacting with each other through a network (170), and 3) trained by a separate training computing system (150) using various training techniques and learning techniques. And the artificial intelligence model trained by the training computing system (150) can be implemented in a manner of being provided / updated by transmitting the model to the user computing device (110) and / or the server computing system (130) through a network (170).

[0052] In some embodiments, the training computing system (150) may be part of a server computing system (130) or part of a user computing device (110).

[0053] -User Computing Device (110)

[0054] The user computing device (110) may include any type of computing device, such as a smart phone, a mobile phone, a digital broadcasting device, a personal digital assistant (PDA), a portable multimedia player (PMP), a desktop, a wearable device, an embedded computing device, and / or a tablet PC.

[0055] Additionally, in the embodiment, the user computing device (110) may further include a server computing device that provides an environment for generating abnormal images based on a conditional diffusion model and learning an artificial intelligence model based thereon.

[0056] Such a user computing device (110) includes at least one processor (111) and memory (112).

[0057] Here, the processor (111) of the user computing device (110) may be composed of at least one or a plurality of processors electrically connected among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors and / or other electrical units for performing functions.

[0058] In particular, depending on the embodiment, such a processor (111) may be configured based on a field programmable gate array (FPGA) implementation, which is a hardware technology for implementing a predetermined digital circuit, and / or an application specific integrated circuit (ASIC).

[0059] Here, a field programmable gate array (FPGA) can mean a flexible digital circuit that can be programmed according to user needs.

[0060] In an embodiment, the field programmable gate array implementation may include a register that temporarily stores data and controls the flow and timing of signals to maintain intermediate results of operations or state information to support synchronized operation of the FPGA; programmable logic that programs operations within the FPGA to perform specific functions or operations as logic circuits that can be configured according to user needs; and an input interface that receives signals from an external device or sensor as a path for receiving data from outside the FPGA and transmits them to the internal circuit.

[0061] By combining the above components, field programmable gate array implementations can provide flexible and diverse forms of digital circuits.

[0062] Meanwhile, an application-specific integrated circuit (ASIC) can refer to a custom integrated circuit that is specifically designed to perform a specific purpose or function.

[0063] As an example, the application-specific integrated circuit may include a register, which is a small memory device that temporarily stores and manages data and stores intermediate results of calculations or status information to support rapid processing of the operation of the ASIC; a microprocessor, which is a central processing unit that performs control and operations within the ASIC and, if necessary, performs various operations or generates control signals to coordinate the operation of the entire system; and an input block, which is an interface that receives data from the outside and receives data to be processed by the ASIC and transmits it to the inside, and receives various input data through a connection with a sensor or an external device.

[0064] By combining the above components, application-specific integrated circuits can perform specific tasks in an optimized manner.

[0065] Returning again, the memory (112) of the user computing device (110) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof, and may include web storage of a server that performs a memory storage function on the Internet. This memory (112) may store data (113) and commands (114) necessary for the at least one processor (111) to perform functional operations such as training an artificial intelligence model, generating abnormal images through an artificial intelligence model, and / or training an artificial intelligence model.

[0066] In one embodiment, the user computing device (110) can perform various deep learning for an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon in conjunction with a deep-learning neural network.

[0067] Here, the deep learning neural network according to the embodiment may include a convolutional neural network (CNN), an R-CNN (Regions with CNN features), a Fast R-CNN, a Faster R-CNN, a Mask R-CNN, etc., and may include any deep learning neural network that includes an algorithm capable of performing the embodiment described below, and the embodiment of the present invention does not limit or restrict such deep learning neural network itself.

[0068] At this time, depending on the embodiment, the deep learning neural network may be installed directly in the server computing system (130) or may operate as a separate device from the server computing system (130) to perform deep learning for the abnormal image generation method based on the conditional diffusion model and the artificial intelligence model learning method based thereon.

[0069] Additionally, in one embodiment, the user computing device (110) may store at least one machine learning model (120).

[0070] For example, the user computing device (110) may be various machine learning models such as a plurality of neural networks (e.g., Deep neural network) that perform an abnormal image generation method based on a conditional diffusion model based on structured / quantitative data and an artificial intelligence model learning method based thereon, or other types of machine learning models including nonlinear models and / or linear models, and may be configured as a combination thereof.

[0071] For example, the machine learning model may store linear regression, decision tree, random forest, gradient boosting, pre-trained language model, or / and deep learning model. The neural network may include at least one of feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or / and other types of neural networks.

[0072] In addition, according to an embodiment, the user computing device (110) may store a model to be used in each process and a prompt template that serves as the basis for input to the model in order to perform at least a part of the process of performing the abnormal image generation method based on the conditional diffusion model and the artificial intelligence model learning method based thereon through a large-scale language model (LLM).

[0073] In one embodiment, a user computing device (110) may receive at least one machine learning model (120) from a server computing system (130) via a network (170), store the received machine learning model (120) in a memory (112), and then execute the stored machine learning model (120) by a processor (111) to perform abnormal image generation based on a conditional diffusion model and artificial intelligence model learning based thereon.

[0074] In another embodiment, a user computing device (110) may perform an operation through a machine learning model (140) including at least one machine learning model (140) in conjunction with a server computing system (130), and may implement an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon by communicating data related thereto with the outside.

[0075] For example, a user computing device (110) can perform an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon in a manner that a server computing system (130) provides an output for a user's input using a machine learning model (140) via the web.

[0076] Additionally, the artificial intelligence model may be implemented in such a way that at least some of the machine learning models (120 and / or 140) are executed on a user computing device (110) and the rest are executed on a server computing system (130).

[0077] Additionally, the user computing device (110) may include at least one input component (121) that detects user input.

[0078] For example, the user input component (121) may include a touch sensor (e.g., a touch screen and / or touch pad, etc.) that detects the touch of a user's input medium (e.g., a finger or a stylus), an image sensor that detects the user's motion input, a microphone, button, mouse, and / or keyboard that detects the user's voice input, etc.

[0079] Here, the image sensor may include an image processing module. Specifically, the image sensor may process still images or moving images obtained by an image sensor device (e.g., CMOS or CCD).

[0080] Additionally, the image sensor can process still images or moving images acquired through the image sensor device using an image recognition process (e.g., OCR, etc.) and / or an image processing module to extract necessary information and transmit the extracted information to the processor.

[0081] Additionally, the input component (121) may receive input for an external controller (e.g., mouse, keyboard, etc.) based on the interface module, and in this case may include an external output device (e.g., speaker).

[0082] At this time, the interface module may be configured to include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, an earphone port, a power amplifier, an RF circuit, a transceiver, and other communication circuits.

[0083] In addition, the external output device may include a display system that outputs various information related to an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon as a graphic image.

[0084] Such a display system can be implemented by including at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0085] Meanwhile, the user computing device (110) including the above-described components may further perform at least some of the functional operations performed by the server computing system (130) described below.

[0086] -Server Computing System (130)

[0087] The server computing system (130) can perform a series of processes to provide an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon.

[0088] In detail, in an embodiment, the server computing system (130) can provide a method for generating abnormal images based on a conditional diffusion model and a method for learning an artificial intelligence model based thereon by exchanging data necessary for driving an abnormal image generation process based on a conditional diffusion model and an artificial intelligence model learning process based thereon with an external device, such as a user computing device (110).

[0089] In more detail, in an embodiment, a server computing system (130) may provide an environment in which an application can operate on a user computing device (110).

[0090] To this end, the server computing system (130) may include application programs, data and / or commands for the application to operate, and may transmit and receive various data based thereon with the external device.

[0091] Additionally, the server computing system (130) includes at least one processor (131) and memory (132).

[0092] Here, the processor (131) of the server computing system (130) may be composed of at least one or a plurality of processors electrically connected among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors and / or other electrical units for performing functions.

[0093] In particular, depending on the embodiment, such a processor (131) may be configured based on a Field Programmable Gate Array (FPGA) implementation, which is a hardware technology for implementing a predetermined digital circuit, and / or an Application Specific Integrated Circuit (ASIC). A detailed description thereof is omitted in accordance with the description of the FPGA and ASIC described above.

[0094] And the memory (132) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. This memory (132) may store data (133) and commands (134) necessary for the processor (131) to perform functional operations such as training an artificial intelligence model, generating abnormal images based on a conditional diffusion model through an artificial intelligence model, and training an artificial intelligence model based on the abnormal image.

[0095] In one embodiment, the server computing system (130) may be implemented to include at least one computing device. For example, the server computing system (130) may be implemented to operate multiple computing devices according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. Additionally, the server computing system (130) may include multiple computing devices connected via a network (170).

[0096] Additionally, the server computing system (130) may store at least one machine learning model (140). For example, the server computing system (130) may include a neural network and / or other multi-layer nonlinear model as the machine learning model (140). Exemplary neural networks may include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0097] In an embodiment, the server computing system (130) may further include a data store computing system (hereinafter, “data store”), which is a storage for continuously storing and managing raw data that serves as the basis for an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon.

[0098] These data stores can include various forms of data storage, ranging from file systems to cloud storage. For example, a data store can include at least one of the following: a relational database that uses a structured query language (SQL) to define and manipulate data; a NoSQL database designed for flexibility and scalability to handle unstructured and semi-structured data; a data warehouse, which is a system used for reporting and data analysis, centralizes large amounts of data from multiple sources and is optimized for querying and analysis; a data warehouse that stores large amounts of raw data in its native formats of structured, semi-structured, and unstructured data; and a database on a local storage device or Network Attached Storage (NAS) that stores data in files in a format typically accessible by a computer operating system.

[0099] -Training Computing System (150)

[0100] The training computing system (150) includes at least one processor (151) and memory (152).

[0101] Here, the processor (151) of the training computing system (150) may be composed of at least one or a plurality of processors electrically connected among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors and / or other electrical units for performing functions.

[0102] In particular, depending on the embodiment, such a processor (151) may be configured based on a Field Programmable Gate Array (FPGA) implementation, which is a hardware technology for implementing a predetermined digital circuit, and / or an Application Specific Integrated Circuit (ASIC). A detailed description thereof is omitted in accordance with the description of the FPGA and ASIC described above.

[0103] And the memory (152) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. This memory (152) may store data (153) and instructions (154) necessary for the processor (151) to perform learning of an artificial intelligence model, etc.

[0104] For example, the training computing system (150) may include a model trainer (160) that trains a machine learning model (120 and / or 140) stored in the user computing device (110) and / or server computing system (130) using various training or learning techniques, such as backpropagation of errors (according to the framework illustrated in FIG. 3).

[0105] For example, such a model trainer (160) may perform updates for one or more parameters of a machine learning model (120 and / or 140) for an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon, using a backpropagation method, based on a defined loss function.

[0106] In some implementations, performing backward propagation of errors may include performing truncated backpropagation through time. The model trainer (160) may perform a number of generalization techniques (e.g., weight reduction, dropout, and / or knowledge distillation) to improve the generalization ability of the trained machine learning model (120 and / or 140).

[0107] Additionally, the model trainer (160) may train the machine learning model (120 and / or 140) based on a series of training data (161). Here, the training data (161) may include data in different formats, such as, for example, images, audio samples, and / or text. Examples of image types that may be used may include video frames, LiDAR point clouds, X-ray images, computed tomography scans, hyperspectral images, and / or various other forms of images.

[0108] Such training data (161) may be provided by a user computing device (110) and / or a server computing system (130). When the training computing device trains a machine learning model (120 and / or 140) on specific data of the user computing device (110), the machine learning model (120 and / or 140) may be characterized as a personalized model.

[0109] And the model trainer (160) includes computer logic utilized to provide the desired function.

[0110] Additionally, the model trainer (160) may be implemented as hardware, firmware, and / or software that controls a general-purpose processor. In one implementation, the model trainer (160) includes a program file stored in a storage device, which may be loaded into a memory (152) and executed by one or more processors (151). In another implementation, the model trainer (160) includes one or more sets of computer-executable data (153) and instructions (154) stored in a tangible computer-readable storage medium, such as a RAM hard disk or an optical or magnetic medium.

[0111] The network (170) includes, but is not limited to, a 3rd Generation Partnership Project (3GPP) network, a Long Term Evolution (LTE) network, a World Interoperability for Microwave Access (WIMAX) network, the Internet, a Local Area Network (LAN), a Wireless Local Area Network (Wireless LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, and / or a Digital Multimedia Broadcasting (DMB) network.

[0112] In general, communication over a network (170) may be performed using any type of wired and / or wireless connection, using various communication protocols (e.g., TCP / IP, HTTP, SMTP, and / or FTP), encodings or formats (e.g., HTML and / or XML), and / or protection schemes (e.g., VPN, Secure HTTP, and / or SSL).

[0113] FIG. 2 illustrates an example of a block diagram of a computing device that implements an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon according to one embodiment of the present invention.

[0114] As shown in FIG. 2, the computing device (100) included in the user computing device (110), the server computing system (130), and the training computing system (150) includes a plurality of applications (e.g., Application 1 to Application N). Each application may include a machine learning library and one or more machine learning models. For example, the applications may include an image processing (e.g., Detection, Classification, and / or Segmentation, etc.) application, a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and / or a chat-bot application.

[0115] In an embodiment, the computing device (100) may include a model trainer (160) for training an artificial intelligence model, and may store and operate the trained artificial intelligence model to provide output data according to predetermined input data (for example, images and / or videos).

[0116] Each application of the computing device (100) may communicate with a number of other components of the computing device (100), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In one embodiment, each application may communicate with each device component using an API (e.g., a public API). In one embodiment, the API used by each application may be specific to that application.

[0117] FIG. 3 illustrates an example block diagram from another aspect of a computing device that implements an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon according to one embodiment of the present invention.

[0118] Referring to FIG. 3, a computing device (200) includes a plurality of applications (e.g., Application 1 to Application N). Each application may communicate with a central intelligence layer. For example, the applications may include an image processing application, a text messaging application, an email application, a dictation application, a virtual keyboard application, and / or a browser application. In one embodiment, each application may communicate with the central intelligence layer (and models stored therein) using an API (e.g., a common API across all applications).

[0119] The central intelligence layer may include multiple machine learning models. For example, as illustrated in FIG. 3, at least some of the machine learning models may be provided to each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model to all applications. In some implementations, the central intelligence layer may be included within the operating system of the computing device (200) or implemented differently.

[0120] The central intelligence layer may communicate with a central device data layer. The central device data layer may be a centralized data repository for the computing device (200). As illustrated in FIG. 3, the central device data layer may communicate with a number of other components of the computing device (200), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer may communicate with each device component using an API (e.g., a private API).

[0121] The techniques described herein may refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to or from such systems. It will be appreciated that the inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, and division of labor and functionality between and among components. For example, the processes described herein may be implemented using a single device or component, or multiple devices or components operating in combination. Databases and applications may be implemented on a single system or in a distributed system across multiple systems. Distributed components may operate sequentially or in parallel.

[0122]

[0123] [Abnormal image generation method based on a conditional diffusion model and an artificial intelligence model training method based on this method]

[0124] Hereinafter, a method of generating an abnormal image having a similar nuisance element and a different semantic element to a normal image and training an artificial intelligence model based on the generated virtual abnormal image by a computing system (1000) according to an embodiment of the present invention will be described in detail with reference to the attached drawings.

[0125] FIG. 4 illustrates a flowchart for explaining a method for generating abnormal images based on a conditional diffusion model and a method for learning an artificial intelligence model based thereon according to an embodiment of the present invention, and FIG. 5 illustrates a conceptual diagram for explaining a method for generating abnormal images based on a conditional diffusion model and a method for learning an artificial intelligence model based thereon according to an embodiment of the present invention.

[0126] Referring to FIGS. 4 and 5, an abnormal image generation method based on a conditional diffusion model and an artificial intelligence model learning method based thereon according to an embodiment of the present invention may include a step of obtaining a normal image data set (S101), a step of setting a diffusion model (S103), a step of obtaining an incomplete diffusion-based noisy image (S105), a step of obtaining a noisy image and a conditional diffusion-based ambient-meaning mask (S107), a step of setting an ambient-meaning mask-based SONA (Semantic Outlier generation via Nuisance Awareness) guidance component (S109), a step of obtaining an abnormal image based on SONA guidance (S111), a step of performing abnormal image-based artificial intelligence model learning (S113), and a step of providing a learned artificial intelligence model (S115).

[0127] Specifically, a computing system (1000) according to one embodiment of the present invention can obtain a normal image data set (In-Distribution Data Set). (S101)

[0128] In detail, in an embodiment, the computing system (1000) can collect and obtain a normal image data set that matches a specific domain, purpose, and / or environment, etc., in conjunction with a predetermined server and / or user input, etc.

[0129] For example, the computing system (1000) can obtain a normal image data set based on a normal product image data set generated in a specific manufacturing process and / or a specific class image data set of a previously published standard image data set (e.g., CIFAR-10 and / or ImageNet, etc.).

[0130] At this time, in the embodiment, the computing system (1000) can perform labeling corresponding to “normal” for each normal image (In-Distribution data) included in the normal image data set.

[0131] Additionally, according to an embodiment, the computing system (1000) can perform image preprocessing on the acquired normal image data set.

[0132] As an example, the computing system (1000) may perform preprocessing, such as normalization, standardization, and / or resolution adjustment, on each normal image.

[0133] Here, the image according to the embodiment of the present invention may include nuisance elements and semantic elements.

[0134] Specifically, in the embodiment, the peripheral (Nuisance) element may mean an element (e.g., background, texture, and / or lighting, etc.) that is not directly related to the core meaning of a specific object (e.g., a given object, etc.).

[0135] On the other hand, the semantic element in the embodiment may mean an element that plays a key role in image classification or recognition, such as a specific object (e.g., a given object, etc.) itself.

[0136] That is, in the embodiment, the image may include common and non-essential peripheral element information and core and essential semantic element information.

[0137] In addition, the computing system (1000) according to one embodiment of the present invention can set a diffusion model. (S103)

[0138] For reference, a diffusion model can mean a generative model that can gradually learn a probability distribution over the entire image (or data) space to generate an image from random noise, or conversely, to restore an original image from a noisy image.

[0139] In general, a diffusion model such as the above can enable image synthesis according to various conditions (e.g., text and / or class labels, etc.) by (i) sequentially noise-izing the input image in the forward stage, and (ii) reconstructing the original image through a learned probabilistic process in the reverse stage.

[0140] At this time, depending on the embodiment, the diffusion model may perform the reverse diffusion step described above using a predetermined method such as DDIM (Deterministic Diffusion Implicit Models).

[0141] In detail, in an embodiment, the computing system (1000) can build a predetermined pre-learned diffusion model.

[0142] In an embodiment, the computing system (1000) may build a pre-trained diffusion model (e.g., a stable diffusion model, a latent diffusion model, etc.) optimized for image generation and / or a diffusion model fine-tuned or pre-trained for a specific domain.

[0143] In the following examples, for the sake of effective explanation, the explanation will be based on the case where the diffusion model is a latent diffusion model (LDM), but is not limited thereto.

[0144] At this time, in the embodiment, the computing system (1000) can load the constructed diffusion model and set the model environment necessary for operation.

[0145] In an embodiment, the computing system (1000) can set the input format (e.g., text prompts, class labels and / or other conditions, etc.) and the output format (e.g., image form and / or latent representation, etc.) of the diffusion model.

[0146] Additionally, according to an embodiment, the computing system (1000) can be configured to generate and restore an image in a predetermined latent space by configuring an encoder and decoder module to be connected to a diffusion model.

[0147] Additionally, according to an embodiment, the computing system (1000) can adjust and set hyperparameters of the diffusion model (e.g., total number of diffusion steps, sampling schedule, guidance scale, etc.) according to domain-specific characteristics and / or purposes.

[0148] In addition, the computing system (1000) according to one embodiment of the present invention can obtain an incomplete diffusion-based noise image. (S105)

[0149] In detail, in an embodiment, the computing system (1000) can perform an incomplete diffusion process that noises only a certain portion of each normal image included in the normal image data set using the diffusion model set as described above.

[0150] In more detail, in an embodiment, the computing system (1000) can input a predetermined normal image into the diffusion model.

[0151] In this way, the diffusion model can perform an incomplete diffusion process that noises the input normal image only up to a specific time step.

[0152] Through this, the diffusion model can generate a noised image (hereinafter, a noisy image) only up to a certain level.

[0153] And the diffusion model can provide the generated noisy image to the computing system (1000).

[0154] Thus, in the embodiment, the computing system (1000) can obtain a noisy image, which is an image in which a certain amount of noise is injected only to a certain portion.

[0155] Thus, in the embodiment, the computing system (1000) can obtain a noisy image, which is an image into which a predetermined amount of noise has been injected.

[0156] In a more specific embodiment, the computing system (1000) inputs a normal image into an encoder of a diffusion model (in an embodiment, a pre-trained latent diffusion model (LDM) or the like) to generate a latent representation (latent representation). ) can be obtained.

[0157] Additionally, in the embodiment, the computing system (1000) may perform the forward diffusion process of the diffusion model as a part (e.g., arbitrarily determined) of the entire diffusion step (T). It can be performed up to step 1.

[0158] Accordingly, the computing system (1000) has an initial latent representation ( )at Noise-enhanced latent expression after the step ( ) can be obtained.

[0159] At this time, in the embodiment, the computing system (1000) is as described above. is greater than or equal to 1 and less than or equal to T (i.e., 1<= The degree of noise can be adjusted by setting it arbitrarily or according to specific rules within the range of <=T).

[0160] Here, the computing system (1000) is The smaller the value, the less noise is added, so a noisy image can be obtained in which relatively more semantic elements of the original normal image remain. The larger the value, the more noise is added, so a noisy image can be obtained in which relatively few semantic elements of the original normal image remain.

[0161] but, If it is too small, the original normal image will remain almost intact, which may result in degradation of the semantic elements in the future. If it is too large, the image will become completely random and close to noise, which may result in poor preservation of nuisance elements later on.

[0162] In an embodiment of the present invention, the computing system (1000) is intended to generate an abnormal image (Out-Of-Distribution data) having similar nuisance elements and different semantic elements to a normal image, and thus is optimized for the above-described functional operation according to a predetermined method. can be decided.

[0163] For example, a computing system (1000) may include: can be sampled uniformly over a certain range from 1 to T. As another example, a computing system (1000) is experimentally optimized in a specific domain. can be used permanently.

[0164] In this way, in the embodiment, the computing system (1000) can secure base data for a diffusion process performed to change semantic elements while maintaining peripheral elements by performing an incomplete diffusion process that denoises only a certain portion of a normal image.

[0165] FIG. 6 illustrates examples of a semantic mask and a nuisance mask for an original image according to one embodiment of the present invention.

[0166] Also, referring to FIG. 6, a computing system (1000) according to one embodiment of the present invention can obtain a noise image and a conditional diffusion-based surrounding-meaning mask. (S107)

[0167] In detail, in the embodiment, the computing system (1000) can perform a predetermined conditional diffusion and unconditional diffusion process based on the noisy image acquired as above.

[0168] For reference, conditional diffusion can refer to a method of adjusting the direction of the image (or data) to be generated through reverse diffusion by giving specific 'conditions' such as text prompts, class labels, and / or style information as input during the noise removal (reverse diffusion) process of a diffusion model. In other words, conditional diffusion can be a technique that induces the model to remove noise (Denoising) and generate data that meets the given conditions, rather than simply randomly restoring noise.

[0169] On the other hand, unconditional diffusion can refer to a method of generating (or restoring) an image without giving any external conditions such as text prompts, class labels, and / or style information during the noise removal (reverse diffusion) process of a diffusion model. In other words, unconditional diffusion can be a technique for generating or restoring an image by removing noise without additional conditions, relying on the data distribution learned by the model.

[0170] In more detail, as an embodiment, the computing system (1000) can perform noise prediction according to predetermined conditional diffusion and unconditional diffusion for a noisy image.

[0171] Specifically, in an embodiment, the computing system (1000) is configured to predict noise values ​​(hereinafter, conditional prediction values) given a specific condition (e.g., text prompt and / or class label, etc.) for a noisy image. )) can be produced.

[0172] In addition, in the embodiment, the computing system (1000) provides a noise prediction value in an unconditional state for a noisy image (hereinafter, an unconditional prediction value ( )) can be produced.

[0173] Additionally, in the embodiment, the computing system (1000) calculates the conditional prediction value ( ) and unconditional predicted values ​​( ) difference value (hereinafter, noise prediction error ( ) can be calculated.

[0174] If this is expressed as a formula, it can be as shown in [Mathematical Formula 1] below.

[0175] [Mathematical Formula 1]

[0176]

[0177] Here, ' in [Mathematical Formula 1] ' stands for noise prediction function, and ' ' means the latent expression of t step, and ' ' can mean a specific condition.

[0178] Through this, in the embodiment, the computing system (1000) can estimate which pixels in the noisy image are sensitive to a specific condition (i.e., a semantic element).

[0179] Specifically, in the embodiment, the computing system (1000) may calculate a noise prediction error ( ) is larger, the corresponding conditional prediction pixel can be judged as a pixel that reacts sensitively to the corresponding condition (i.e., a pixel that reflects a lot of semantic elements).

[0180] In other words, the computing system (1000) has a noise prediction error ( ) is larger, the conditional prediction pixel that is the source of this can be determined as a pixel that causes a large change for a given condition, i.e., a pixel that is heavily involved in semantic elements (hereinafter, semantic pixels).

[0181] Additionally, in the embodiment, the computing system (1000) can obtain a semantic mask and a nuisance mask based on the semantic pixels determined as above.

[0182]

[0183] *Here, the semantic mask according to the embodiment may mean a mask indicating an area that includes pixels (i.e., semantic pixels) that are sensitive to specific conditions within a given image.

[0184] That is, in the embodiment, the semantic mask may be a mask indicating a region (hereinafter, a semantic region) that includes a semantic element that plays a key role in image classification or recognition, such as a specific object (e.g., a predetermined object, etc.) itself.

[0185] In an embodiment, the computing system (1000) may be configured to: ) can be used to determine a semantic region based on semantic pixels, which are pixels whose absolute value belongs to the top N(>0) percent (e.g., top 5%).

[0186] And the computing system (1000) can perform masking processing to include pixels corresponding to the determined semantic area (i.e., semantic pixels).

[0187] As an example, the computing system (1000) can generate a binary mask that keeps each semantic pixel in an activated state and perform masking processing by applying the binary mask.

[0188] For example, the computing system (1000) can implement masking processing by applying a binary mask that sets a mask value corresponding to a semantic pixel to '1' and sets a mask value corresponding to other pixels to '0'.

[0189] Thus, in an embodiment, the computing system (1000) can obtain a semantic mask that identifies and represents pixels (i.e., semantic pixels) that are sensitive to specific conditions.

[0190] Meanwhile, the nuisance mask according to the embodiment may mean a mask indicating an area that includes pixels with low sensitivity to a specific condition (hereinafter, nuisance pixels) within a given image.

[0191] That is, in the embodiment, the peripheral mask may be a mask representing an area (hereinafter, the peripheral area) that includes peripheral elements that are not directly related to the core meaning of a specific object (e.g., a given object, etc.).

[0192] In an embodiment, the computing system (1000) may be configured to: ) can be used to determine the nuisance region based on the nuisance pixels, which are pixels whose absolute value falls within the lower M(>0) percent (e.g., lower 5%).

[0193] And the computing system (1000) can perform masking processing to include pixels corresponding to the determined nuisance area (i.e., nuisance pixels).

[0194] At this time, the computing system (1000) as an example can perform the above-described masking process by generating and applying a binary mask that keeps each nuisance pixel in an activated state.

[0195] Thus, in an embodiment, the computing system (1000) can obtain a peripheral mask that identifies and represents pixels with low sensitivity to a specific condition (i.e., peripheral pixels).

[0196] At this time, in the embodiment, the computing system (1000) can generate the semantic mask and the surrounding mask in a form that does not overlap each other.

[0197] That is, the computing system (1000) can cause the semantic mask and the surrounding mask to point to different parts of the image.

[0198] Through the masking process as described above, the computing system (1000) in the embodiment can clearly separate the nuisance elements and semantic elements in the noisy image, and through this, in a subsequent process, can easily implement a diffusion process that maintains the nuisance elements and transforms the semantic elements with high accuracy by utilizing the information.

[0199] In addition, a computing system (1000) according to one embodiment of the present invention can set a peripheral-semantic mask-based SONA (Semantic Outlier generation via Nuisance Awareness) guidance component. (S109)

[0200] Here, the SONA guidance according to the embodiment may mean a technical component that provides guidance for the reverse diffusion process of the diffusion model to selectively transform semantic elements while preserving nuisance elements as much as possible based on the semantic mask and peripheral mask as described above.

[0201] In embodiments, such SONA guidance may cause the diffusion model to perform differential updates for each region during the noise removal process, thereby generating abnormal images that meet certain conditions.

[0202] In addition, the SONA guidance component according to the embodiment may mean a SONA guidance structure that includes elements that determine how to apply SONA guidance to each pixel (in the embodiment, direction and size, etc.) in the reverse diffusion process of the diffusion model.

[0203] Specifically, in the embodiment, the SONA guidance component comprises an original meaning removal component ( ), peripheral information preservation component ( ) and new semantic addition components ( ) may be included.

[0204] In detail, the original meaning removal component according to the embodiment ( ) may be a SONA guidance component that gradually removes original semantic elements by reversing the original (normal) conditions for pixels included in the semantic mask.

[0205] As an example, the original meaning removal component ( ) is the original condition ( for each semantic pixel in the semantic mask. ) noise prediction error ( ) can be updated in the opposite direction to gradually lose the original meaning.

[0206] These original meaning removal components ( ) can be expressed as a formula as shown in [Mathematical Formula 2] below.

[0207] [Equation 2]

[0208]

[0209] Here, ' in [Mathematical Formula 2] ' is the original condition( ) represents the meaning mask for ' ' is the original condition( ) and the noise prediction error between the unconditional case ( ) can be expressed.

[0210] Meanwhile, the peripheral information preservation component according to the embodiment ( ) may be a SONA guidance component that preserves the surrounding (Nuisance) elements as much as possible by updating the pixels included in the surrounding mask in a direction that maintains (restores) the original (normal) conditions.

[0211] If this is expressed as a formula, it can be as shown in [Mathematical Formula 3] below.

[0212] [Equation 3]

[0213]

[0214] Here, ' in [Mathematical Formula 3] ' is the original condition( ) can represent the surrounding mask.

[0215] On the other hand, a new semantic addition component according to the embodiment ( ) is a new abnormal condition for pixels included in the semantic mask. ) may be a SONA guidance component that adds unusual semantic elements (i.e., new semantic elements).

[0216] At this time, a new meaning addition component in the embodiment ( ) is the original condition ( ) except for the area overlapping with the surrounding mask for the new abnormal condition described above ( ) can be used to perform updates.

[0217] Therefore, a new semantic addition component ( ) can easily insert new, unusual meanings without damaging (distorting) existing peripheral (nuisance) elements.

[0218] If this is expressed as a formula, it can be as shown in [Mathematical Formula 4] below.

[0219] [Equation 4]

[0220]

[0221] Here, ' in [Equation 4] ' is an abnormal condition ( ) represents a new semantic mask according to ' ' is a new abnormal condition( ) and the noise prediction error between the unconditional case ( ) can be expressed.

[0222] In detail, returning to FIG. 4, in an embodiment, the computing system (1000) can set the SONA guidance component described above based on the semantic mask and the surrounding mask.

[0223] In other words, in the embodiment, the computing system (1000) uses a semantic mask and a surrounding mask to remove the original semantic component ( ), peripheral information preservation component ( ) and new semantic addition components ( ) can be set individually.

[0224] At this time, in the embodiment, the computing system (1000) includes an original meaning removal component ( ), peripheral information preservation component ( ) and new semantic addition components ( ) can be calculated separately, and the guidance application method (for example, direction and size, etc.) for each pixel can be determined when performing reverse diffusion using a diffusion model.

[0225] Additionally, in the embodiment, the computing system (1000) comprises a set SONA guidance component (i.e., an original meaning removal component ( ), peripheral information preservation component ( ) and new semantic addition components ( ) to form the final SONA guidance component ( , you can obtain the SONA Guidance Integration Component.

[0226] If this is expressed as a formula, it can be as shown in [Mathematical Formula 5] below.

[0227] [Equation 5]

[0228]

[0229] In addition, the computing system (1000) according to one embodiment of the present invention can acquire an abnormal image based on SONA guidance. (S111)

[0230] Here, an abnormal image according to an embodiment may mean an image that includes a new semantic element other than the semantic element included in a normal image data set or has a feature that is semantically distinct from a normal image.

[0231] That is, in the embodiment, an abnormal image may be an image that has characteristics that deviate from the existing normal data distribution through semantic elements that have changed from a normal image.

[0232] At this time, in the embodiment, the abnormal image may be an image that maintains nuisance elements similar to a normal image, unlike semantic elements.

[0233] In detail, in an embodiment, the computing system (1000) comprises a SONA guidance integration component ( ) can be applied to the reverse diffusion process of the diffusion model.

[0234] In more detail, in the embodiment, the computing system (1000) performs each time step (t = , , … , 0) for noise prediction values ​​by SONA guidance integrated component ( ) can be applied.

[0235] Thus, in the embodiment, the computing system (1000) can perform reverse diffusion in a direction in which the diffusion model transforms semantic elements and preserves nuisance elements.

[0236] That is, the computing system (1000) integrates the SONA guidance integration component ( ) by implementing noise prediction for each time step, the original (normal) semantic elements gradually disappear, new (abnormal) semantic elements gradually appear, and the reverse diffusion process can be implemented in which the original (normal) nuisance elements are maintained as much as possible.

[0237] In addition, in the embodiment, the computing system (1000) obtains the latent representation (t = 0) at the final time step (t = 0) in the reverse process. ) through the decoder of the diffusion model to the pixel image ( ) can be restored.

[0238] Thus, the computing system (1000) can generate an abnormal image having a similar nuisance element and a different semantic element to a normal image.

[0239] FIG. 7 illustrates examples of an original image and a virtual abnormal image according to one embodiment of the present invention.

[0240] At this time, looking at Figure 7, it can be confirmed that the data corruption effect in the area corresponding to the semantic element compared to the peripheral (Nuisance) element according to the original normal image is more directly shown on the virtual abnormal image generated for each original normal image.

[0241] In this way, in the embodiment, the computing system (1000) can effectively generate a virtual abnormal image that deviates from the normal data distribution by modifying semantic elements or giving new abnormal meanings while maintaining the surrounding elements, such as the background, texture, and / or lighting of the normal image, as much as possible.

[0242] Through this, in the embodiment, the computing system (1000) can expand the learning data of the anomaly detection model by utilizing virtual abnormal images even in environments where it is difficult to collect actual abnormal images (OOD data) (e.g., manufacturing processes, medical diagnosis, etc.), and can save the cost and time required for this.

[0243] In addition, in the embodiment, the computing system (1000) can directly and significantly improve the reliability and accuracy of abnormal images generated through providing a guide that clearly distinguishes nuisance elements and semantic elements by utilizing SONA guidance and induces data processing (i.e., reverse diffusion method) optimized for each of them.

[0244] In particular, in the embodiment, the computing system (1000) can generate high-quality abnormal images even in a situation where only normal images exist, so that the abnormality detection performance of a normal data-based learning model can be easily improved, and accordingly, a model that exhibits high performance even for difficult abnormality detection problems such as near-OOD can be implemented and provided.

[0245] In addition, a computing system (1000) according to one embodiment of the present invention can perform abnormal image-based artificial intelligence model learning. (S113)

[0246] In detail, in an embodiment, the computing system (1000) can train a predetermined artificial intelligence model based on the virtual abnormal image acquired as described above.

[0247] As an example, the artificial intelligence model described above may include a predetermined anomaly detection model (e.g., One-Class SVM, Isolation Forest, Autoencoder-based model, etc.) and / or a deep learning classifier (e.g., Convolutional Neural Network (CNN), Transformer-based model, etc.).

[0248] In more detail, in an embodiment, the computing system (1000) can generate a learning data set (hereinafter, an integrated learning data set) that integrates existing normal images and virtual abnormal images by assigning abnormal images with an “abnormal” label and normal images with a “normal” label.

[0249] Additionally, in an embodiment, the computing system (1000) can perform learning for a predetermined artificial intelligence model using an integrated learning data set.

[0250] For example, the computing system (1000) can perform model learning based on an integrated learning data set to build a model that classifies normal and abnormal data or detects abnormal data.

[0251] Additionally, a computing system (1000) according to one embodiment of the present invention can provide a learned artificial intelligence model. (S115)

[0252] In detail, in an embodiment, the computing system (1000) can provide the learned artificial intelligence model as described above in a predetermined manner.

[0253] As an example, the computing system (1000) may provide an artificial intelligence model learned based on a virtual abnormal image according to an embodiment of the present invention in conjunction with a predetermined application service.

[0254] For example, the computing system (1000) can provide an artificial intelligence model as described above by integrating with various application services, such as a manufacturing process monitoring system, a medical diagnosis system, a security detection system, and / or an autonomous vehicle system.

[0255] In this way, the computing system (1000) in the embodiment can generate a virtual abnormal image that is close to but clearly different from the existing normal data distribution (In-Distribution, ID) by preserving the nuisance element as much as possible while transforming the semantic element based on the incomplete diffusion, conditional / unconditional diffusion, and SONA guidance strategies, and can support model learning based on the same.

[0256] Accordingly, the computing system (1000) can generate high-quality abnormal data even in an environment where abnormal data is insufficient or difficult to obtain, and utilize it for artificial intelligence model learning, thereby dramatically improving model performance (e.g., anomaly detection and / or OOD (Out-Of-Distribution) performance, etc.).

[0257] In addition, through this, the computing system (1000) can provide a high-performance artificial intelligence solution applicable to various domains such as manufacturing, medical, security, and / or autonomous driving, thereby supporting the multifaceted construction of a robust system capable of identifying and responding to potential risk factors in advance.

[0258] Furthermore, through this, the computing system (1000) in the embodiment can provide a data-based solution framework that is both efficient and reliable by reducing the cost and time required for collecting actual abnormal data and supporting the construction of an artificial intelligence model that can flexibly respond to various abnormal situations using only normal data.

[0259]

[0260] The embodiments of the present invention described above may be implemented in the form of program commands that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the computer-readable recording medium may be specially designed and configured for the present invention or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. Hardware devices may be changed into one or more software modules to perform processing according to the present invention, and vice versa.

[0261] The specific implementations described in the present invention are exemplary embodiments and do not limit the scope of the present invention in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the lines connecting or connecting members between components illustrated in the drawings are merely representative of functional connections and / or physical or circuit connections, and may be replaced or represented as various additional functional connections, physical connections, or circuit connections in an actual device. In addition, unless specifically mentioned as “essential,” “important,” etc., a component may not be absolutely necessary for the application of the present invention.

[0262] Although the detailed description of the present invention has been described with reference to preferred embodiments of the present invention, it will be understood by those skilled in the art or having ordinary knowledge in the art that various modifications and changes can be made to the present invention without departing from the spirit and technical scope of the present invention as set forth in the claims below. Accordingly, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be defined by the claims.

[0263]

[0264] The present invention relates to a method for generating abnormal images based on a conditional diffusion model and an artificial intelligence model learning method and system based thereon, and is applicable to the artificial intelligence industry, and thus has industrial applicability.

Claims

1. A method for generating an abnormal image based on a conditional diffusion model by a computing system including a memory and a processor, Step of inputting normal image (In-Distribution data) into diffusion model; A step of obtaining a noisy image based on incomplete diffusion based on the above diffusion model; A step of obtaining a semantic mask, which is a mask that specifies a semantic element for the normal image based on the noisy image, and a nuisance mask, which is a mask that specifies a nuisance element for the normal image; and A step of generating an abnormal image (Out-Of-Distribution data) based on the SONA (Semantic Outlier generation via Nuisance Awareness) guidance, which is a data processing guide that preserves the peripheral elements and transforms the semantic elements based on the above semantic mask and the peripheral mask. Abnormal image generation method based on conditional diffusion model.

2. In paragraph 1, The above diffusion model is, This is a model that gradually injects noise into the input image based on the forward diffusion process and gradually removes the noise based on the reverse diffusion process. Abnormal image generation method based on conditional diffusion model.

3. In paragraph 2, The step of obtaining the above incomplete diffusion-based noise image is as follows: A step of injecting the noise into the normal image up to a predetermined time step is included. Abnormal image generation method based on conditional diffusion model.

4. In paragraph 2, The step of obtaining the above meaning mask and the above surrounding mask is: A step of performing conditional diffusion, which executes the de-diffusion process according to a specific condition based on the above-mentioned noisy image. Abnormal image generation method based on conditional diffusion model.

5. In paragraph 4, The step of obtaining the above meaning mask and the above surrounding mask is: Further comprising a step of performing unconditional diffusion, which executes the unconditional de-diffusion process based on the above-mentioned noisy image. Abnormal image generation method based on conditional diffusion model.

6. In paragraph 5, The step of obtaining the above meaning mask and the above surrounding mask is: It further includes a step of calculating a conditional prediction value, which is a noise prediction value according to the conditional diffusion, and an unconditional prediction value, which is a noise prediction value according to the unconditional diffusion. Abnormal image generation method based on conditional diffusion model.

7. In paragraph 6, The step of obtaining the above meaning mask and the above surrounding mask is: Further comprising a step of calculating a noise prediction error, which is a difference value between the conditional prediction value and the unconditional prediction value. Abnormal image generation method based on conditional diffusion model.

8. In paragraph 7, The step of obtaining the above meaning mask and the above surrounding mask is: The method further comprises the step of generating the semantic mask based on pixels whose absolute value of the noise prediction error belongs to a predetermined upper N(>0) percent, and generating the surrounding mask based on pixels whose absolute value of the noise prediction error belongs to a predetermined lower M(>0) percent. Abnormal image generation method based on conditional diffusion model.

9. In paragraph 2, The step of generating an abnormal image based on the above SONA guidance is: A step of setting a SONA guidance component, which is a component that specifies how to apply the SONA guidance to the above de-spreading process, is included. Abnormal image generation method based on conditional diffusion model.

10. In paragraph 9, The above SONA guidance component is, It includes an original meaning removal component that is a component that gradually removes the above semantic elements, a peripheral information preservation component that is a component that preserves the peripheral elements, and a new meaning addition component that is a component that adds new semantic elements. Abnormal image generation method based on conditional diffusion model.

11. In paragraph 10, The above new meaning-adding component is, A component that adds the new semantic element to the area excluding the area overlapping with the surrounding mask. Abnormal image generation method based on conditional diffusion model.

12. In paragraph 10, The steps for setting the above SONA guidance component are: A step of obtaining a SONA guidance integration component that is a component that combines the above original meaning removal component, the above peripheral information preservation component, and the above new meaning addition component into one. Abnormal image generation method based on conditional diffusion model.

13. In paragraph 12, The step of generating an abnormal image based on the above SONA guidance is: Further comprising a step of applying the SONA guidance integration component to the noise prediction value for each time step according to the despreading process. Abnormal image generation method based on conditional diffusion model.

14. A method for performing artificial intelligence model learning based on abnormal images using a conditional diffusion model, in a computing system including memory and a processor.

15. At least one memory; and At least one processor for generating an abnormal image based on a conditional diffusion model by reading at least one application stored in the memory; The instructions of the above processor are: The step of inputting the normal image (In-Distribution data) into the diffusion model, A step of obtaining a noisy image based on incomplete diffusion based on the above diffusion model, A step of obtaining a semantic mask, which is a mask that specifies a semantic element for the normal image, and a nuisance mask, which is a mask that specifies a nuisance element for the normal image, based on the noisy image; A command that performs a step of generating an abnormal image (Out-Of-Distribution data) based on the SONA (Semantic Outlier generation via Nuisance Awareness) guidance, which is a data processing guide that preserves the peripheral elements and transforms the semantic elements based on the above semantic mask and the peripheral mask. An abnormal image generation system based on a conditional diffusion model.

16. In paragraph 15, A plurality of neurons comprising an array including at least one register, at least one programmable logic, and at least one input interface; a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons; and at least one routing network that controls data flow between the plurality of neurons; Each of the plurality of neurons further includes a field programmable gate array (FPGA) implementation for a predetermined artificial neural network that is connected to at least one other neuron through the routing network to set a transmission path of the weight. An abnormal image generation system based on a conditional diffusion model.

17. In paragraph 15, A plurality of neurons organized in an array including at least one register, at least one microprocessor, and at least one input; and a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons; Each of the plurality of neurons further includes an application specific integrated circuit (ASIC) for a given artificial neural network that is connected to at least one other neuron through one of the plurality of synaptic circuits. An abnormal image generation system based on a conditional diffusion model.

Images