Deep learning-based environmentally adaptive outlier detection method and system therefor

Abstract

A deep learning-based environmentally adaptive outlier detection method according to an embodiment of the present invention is a method by which a computing system comprising a memory and a processor performs deep learning-based environmentally adaptive outlier detection and comprises the steps of: acquiring a corresponding fair quality image dataset for each predetermined material; acquiring corresponding Gaussian application information for each material, on the basis of Gaussian density estimation based on the fair quality image dataset; constructing a database comprising the Gaussian application information; determining whether a predetermined inspection target image is suspected of being defective, on the basis of the database; and providing a result of whether the predetermined inspection target image is suspected of being defective, wherein the Gaussian application information is information comprising mean (μ) and covariance (Σ) data of feature vectors extracted from N (N > 1) fair quality images included in the fair quality image dataset.

Description

Deep learning-based environment-adaptive outlier detection method and system

[0001] The present invention relates to a deep learning-based environment-adaptive outlier detection method and system. More specifically, it relates to a deep learning-based environment-adaptive outlier detection method and system that adaptively operates on the characteristics of various materials constituting a vision inspection target.

[0002] Generally, vision inspection equipment is intended to automatically detect defects present on the appearance of a product by utilizing computer vision technology, including deep learning.

[0003] Based on data obtained from sensor equipment such as cameras, good products and defective products containing defects must be classified with very high accuracy, and for product quality control, equipment in the inspection process must operate uniformly without problems.

[0004] In this case, since processing time during the inspection process affects product output, it is very important to process quickly along with high accuracy.

[0005] However, since most vision inspections are performed at high resolution to detect even minute defects, using a deep learning model to inspect the entire product in detail can achieve high accuracy, but it presents limitations in terms of processing time.

[0006] Thus, in the conventional field of this technology, efforts are being made to satisfy accuracy and processing time by using a method that detects suspected defect areas through rule-based algorithms utilizing prior knowledge and makes judgments through deep learning models as post-processing.

[0007] However, rule-based algorithms based on various image processing and image-based quantitative measurement have the disadvantage that their performance is determined by the developer's capabilities and know-how, and that it is difficult to readjust the rules when the inspection environment changes, resulting in high costs and time.

[0008] For example, considering the characteristic that scratches on the exterior of a product reflect light and have a high brightness value, there is a rule-based algorithm that performs image processing to binarize the area where scratches occur when the brightness value is above a certain value, defines the area through Connected Component Labeling, and detects it as a suspected defect area by measuring its size. However, if the intensity of the light weakens or the material of the product changes and the reflection of light increases, the performance of this rule-based algorithm in defining the scratch area inevitably deteriorates.

[0009] Therefore, the development and introduction of new technologies are necessary to solve the aforementioned problems.

[0010] One embodiment of the present invention is designed to solve the problems of the prior art as described above, and aims to provide a deep learning-based environment-adaptive outlier detection method and system that adaptively operate on the characteristics of various materials constituting a vision inspection target.

[0011] In addition, one embodiment of the present invention aims to provide a deep learning-based environment-adaptive outlier detection method and system that detects suspected defect areas through a deep learning-based pre-trained model without separate training using a small number of good product images.

[0012] However, the technical problems that the present invention and the embodiments of the present invention aim to solve are not limited to the technical problems described above, and other technical problems may exist.

[0013] A deep learning-based environment-adaptive outlier detection method according to one embodiment of the present invention is a method in which a computing system including memory and a processor performs deep learning-based environment-adaptive outlier detection, comprising: a step of acquiring a good product image dataset corresponding to a predetermined material; a step of acquiring Gaussian application information corresponding to the material based on Gaussian density estimation based on the good product image dataset; a step of constructing a database including the Gaussian application information; a step of determining whether there is a suspected defect for a predetermined inspection target image based on the database; and a step of providing the result of the suspected defect, wherein the Gaussian application information is information including mean (μ) and covariance (Σ) data of feature vectors extracted from N (N>1) good product images included in the good product image dataset.

[0014] In another aspect, the step of obtaining the Gaussian application information corresponding to each of the above materials includes the step of dividing each of the N (N>1) good product images included in the good product image dataset into patches, the step of obtaining a multi-scale feature based on the divided patches using a predetermined pre-training model, and the step of obtaining the Gaussian application information based on the Gaussian density estimation based on the multi-scale feature.

[0015] In another aspect, the step of determining whether there is a suspected defect in the inspection target image comprises the step of dividing the inspection target image into patches and the step of obtaining multi-scale features based on the divided patches using a predetermined pre-trained model.

[0016] In another aspect, the step of determining whether there is a suspected defect in the inspection target image further includes the step of obtaining product material information, which is information specifying the material constituting the inspection target within the inspection target image.

[0017] In another aspect, the step of determining whether there is a suspected defect in the inspection target image further includes the step of detecting the Gaussian application information corresponding to the product material information from the database.

[0018] In another aspect, the step of determining whether there is a suspected defect in the inspection target image further includes, upon detecting Gaussian matching information, which is Gaussian application information corresponding to the product material information, the step of determining the suspected defect based on the Gaussian matching information and the multi-scale feature.

[0019] In another aspect, the step of determining whether there is a suspected defect based on the Gaussian matching information and the multi-scale feature includes the step of obtaining the probability of a good product per patch based on the Gaussian density estimation, the feature vector per patch, and the Gaussian matching information per patch.

[0020] In another aspect, the step of determining whether there is a suspected defect based on the Gaussian matching information and the multi-scale feature further comprises: a step of comparing the probability of a good product per patch with a predetermined threshold value; a step of determining each of the patches as a good product patch or a suspected defect patch according to the comparison result; and a step of determining whether there is a suspected defect based on at least one of the good product patches and the suspected defect patches.

[0021] In another aspect, the step of determining whether there is a suspected defect regarding the inspection target image further includes the step of providing a predetermined notification indicating the absence of the Gaussian matching information if the Gaussian matching information, which is the Gaussian application information corresponding to the product material information, is not detected.

[0022] Meanwhile, a deep learning-based environment-adaptive outlier detection system according to one embodiment of the present invention comprises: at least one memory; and at least one processor that reads at least one application stored in the memory and performs deep learning-based environment-adaptive outlier detection. The instructions of the processor include the steps of: acquiring a good product image data set corresponding to a predetermined material; acquiring Gaussian application information corresponding to the material based on Gaussian density estimation based on the good product image data set; constructing a database containing the Gaussian application information; determining whether there is a suspected defect for a predetermined inspection target image based on the database; and providing the result of the suspected defect. The Gaussian application information is information that includes mean (μ) and covariance (Σ) data of feature vectors extracted from N (N>1) good product images included in the good product image data set.

[0023] The deep learning-based environment-adaptive outlier detection method and system according to one embodiment of the present invention adaptively operates on the characteristics of various materials constituting a vision inspection target, thereby reducing inspection errors caused by different material characteristics in environments such as multi-product small-batch production, simultaneously improving inspection accuracy and efficiency, and enabling rapid and flexible adaptation to material changes of various products, which has the effect of significantly improving product quality control performance.

[0024] In addition, the deep learning-based environment-adaptive outlier detection method and system according to one embodiment of the present invention has the effect of reducing the burden of data collection for model training and flexibly responding to changes in the inspection environment by detecting suspected defect areas through a deep learning-based pre-trained model without separate training using a small number of good product images.

[0025] In addition, the deep learning-based environment-adaptive outlier detection method and system according to one embodiment of the present invention has the effect of reducing the time and resources required for inspection compared to a method of processing the entire image area in batches by dividing and processing the image at the patch level.

[0026] However, the effects obtainable from the present invention are not limited to those mentioned above, and other unmentioned effects can be clearly understood from the description below.

[0027] FIG. 1 illustrates an example of a block diagram of a computing system implementing an environment-adaptive outlier detection service according to one embodiment of the present invention.

[0028] FIG. 2 illustrates an example of a block diagram of a computing device implementing an environment-adaptive outlier detection service according to an embodiment of the present invention.

[0029] FIG. 3 illustrates an example of a block diagram in another aspect of a computing device implementing an environment-adaptive outlier detection service according to one embodiment of the present invention.

[0030] FIG. 4 illustrates a flowchart for explaining a method for detecting suspected defective areas according to an embodiment of the present invention.

[0031] FIG. 5 illustrates a conceptual diagram for explaining a method of aligning inspection target images according to an embodiment of the present invention.

[0032] FIG. 6 illustrates examples for explaining a patch-unit image segmentation method according to an embodiment of the present invention.

[0033] FIG. 7 illustrates a conceptual diagram for explaining a method of extracting image features through a pre-trained model according to one embodiment of the present invention.

[0034] FIG. 8 illustrates a conceptual diagram for explaining a method for acquiring multi-scale features according to an embodiment of the present invention.

[0035] FIG. 9 illustrates a flowchart for explaining a multi-stage classification method for patch images based on the probability of good quality according to an embodiment of the present invention.

[0036] FIG. 10 illustrates an example of a drawing for explaining a method for obtaining a minimum probability area image of a good product according to an embodiment of the present invention.

[0037] FIG. 11 illustrates a flowchart for explaining a deep learning-based environment-adaptive suspected defect area detection method according to an embodiment of the present invention.

[0038] FIG. 12 illustrates a conceptual diagram for explaining a method of constructing a Gaussian application information database for each product material according to an embodiment of the present invention.

[0039] The present invention is capable of various modifications and may have various embodiments; therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention, and the methods for achieving them, will become clear by referring to the embodiments described in detail below together with the drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various forms. In the following embodiments, terms such as "first," "second," etc., are used not in a limiting sense but for the purpose of distinguishing one component from another. Furthermore, singular expressions include plural expressions unless the context clearly indicates otherwise. Also, terms such as "include" or "have" mean that the features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added. Additionally, in the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily depicted for convenience of explanation, so the present invention is not necessarily limited to what is illustrated.

[0040] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0041]

[0042] [Exemplary system providing environment-adaptive outlier detection services]

[0043] Hereinafter, an exemplary system for implementing a deep learning-based environment-adaptive outlier detection service (hereinafter, environment-adaptive outlier detection service) that operates adaptively with respect to the characteristics of various materials constituting a vision inspection target will be described in detail with reference to the attached drawings.

[0044] FIG. 1 illustrates an example of a block diagram of a computing system implementing an environment-adaptive outlier detection service according to one embodiment of the present invention.

[0045] Referring to FIG. 1, a computing system (1000) implementing an environment-adaptive outlier detection service of the present invention includes a user computing device (110), a server computing system (130), and a training computing system (150), and each device and system is connected to communicate through a network (170).

[0046] A deep learning-based environment-adaptive outlier detection method and system according to one embodiment of the present invention may be implemented and provided locally by a user computing device (110), implemented and provided in the form of a web service by a server computing system (130) communicating with the user computing device (110), and implemented and provided by the user computing device (110) and the server computing system (130) in conjunction with each other.

[0047] In this embodiment, the user computing device (110) and / or the server computing system (130) can train a machine learning model (120 and / or 140) through interaction with a training computing system (150) that is communicatedly connected via a network (170). The training computing system (150) may be separate from the server computing system (130) or may be part of the server computing system (130).

[0048] And at this time, the artificial intelligence model can be 1) trained directly locally by a user computing device (110), 2) trained by the server computing system (130) and the user computing device (110) interacting with each other through a network (170), and 3) trained by a separate training computing system (150) using various training and learning techniques. It may also be implemented by transmitting the artificial intelligence model trained by the training computing system (150) to the user computing device (110) and / or the server computing system (130) through the network (170) to provide / update it.

[0049] In some embodiments, the training computing system (150) may be part of the server computing system (130) or part of the user computing device (110).

[0050] - User Computing Device (110: User Computing Device)

[0051] The user computing device (110) may include all other types of computing devices, such as a smartphone, a mobile phone, a digital broadcasting device, a PDA (personal digital assistants), a PMP (portable multimedia player), a desktop, a wearable device, an embedded computing device and / or a tablet PC.

[0052] Additionally, in the embodiment, the user computing device (110) may further include a predetermined server computing device that provides an environment-adaptive anomaly detection service environment.

[0053] This user computing device (110) includes at least one processor (111) and memory (112).

[0054] Here, the processor (111) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, or a plurality of electrically connected processors.

[0055] The memory (112) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof, and may include web storage of a server that performs memory storage functions on the internet. This memory (112) may store data (113) and instructions (114) necessary for the at least one processor (111) to perform functional operations, such as training an artificial intelligence model or executing a process to detect suspected defective areas through an artificial intelligence model.

[0056] In one embodiment, the user computing device (110) can perform various deep learnings for an environment-adaptive outlier detection service by linking with a deep-learning neural network.

[0057] Here, the deep learning neural network according to the embodiment may include a Convolutional Neural Network (CNN), R-CNN (Regions with CNN features), Fast R-CNN, Faster R-CNN, Mask R-CNN, etc., and may include any deep learning neural network that includes an algorithm capable of performing the embodiments described below, and the embodiments of the present invention do not limit or restrict such deep learning neural networks themselves.

[0058] At this time, according to the embodiment, the deep learning neural network may be installed directly in the server computing system (130) or operate as a separate device from the server computing system (130) to perform deep learning for the environment-adaptive outlier detection service.

[0059] Additionally, in one embodiment, the user computing device (110) can store at least one machine learning model (120).

[0060] For example, the user computing device (110) may be various machine learning models, such as multiple neural networks (e.g., deep neural networks) that perform a method for detecting suspected defect areas based on structured / quantitative data, or other types of machine learning models including non-linear models and / or linear models, and may be composed of a combination thereof.

[0061] For example, machine learning models may include linear regression, decision trees, random forests, gradient boosting pre-trained language models or / and deep learning models. And neural networks may include at least one of feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or / and other forms of neural networks.

[0062] Additionally, according to an embodiment, the user computing device (110) may store a model to be used in each process and a prompt template that serves as the basis for input to the model in order to perform at least part of the process for the method of detecting a suspected defective area through a large-scale language model (LLM).

[0063] In one embodiment, a user computing device (110) receives at least one machine learning model (120) from a server computing system (130) through a network (170), stores it in memory (112), and then executes the stored machine learning model (120) by a processor (111) to perform a process such as detecting a suspected defective area.

[0064] In another embodiment, the user computing device (110) may provide an environment-adaptive anomaly detection service to the user by performing operations through a machine learning model (140) including at least one machine learning model (140) in conjunction with a server computing system (130) and communicating related data to the outside.

[0065] For example, a user computing device (110) can perform an environment-adaptive anomaly detection service by having a server computing system (130) provide an output for the user's input using a machine learning model (140) via the web.

[0066] Additionally, the artificial intelligence model can be implemented in such a way that at least some of the machine learning models (120 and / or 140) are executed on a user computing device (110) and the rest are executed on a server computing system (130).

[0067] Additionally, the user computing device (110) may include at least one input component (121) that detects user input.

[0068] For example, the user input component (121) may include a touch sensor (e.g., a touch screen and / or a touch pad, etc.) that detects a touch of the user's input medium (e.g., a finger or a stylus), an image sensor that detects the user's motion input, a microphone that detects the user's voice input, a button, a mouse and / or a keyboard, etc.

[0069] Here, the image sensor may include an image processing module. Specifically, the image sensor may process still images or video obtained by an image sensor device (e.g., CMOS or CCD).

[0070] In addition, the image sensor can process a still image or video acquired through the image sensor device using an image recognition process (e.g., OCR, etc.) and / or an image processing module to extract necessary information and transmit the extracted information to a processor.

[0071] Additionally, the input component (121) can receive input from an external controller (e.g., mouse, keyboard, etc.) based on an interface module, and in this case, may include an external output device (e.g., speaker).

[0072] At this time, the interface module may be configured to include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, an earphone port, a power amplifier, an RF circuit, a transceiver, and other communication circuits.

[0073] In addition, the external output device may include a display system that outputs various information related to an environment-adaptive anomaly detection service as a graphic image.

[0074] Such a display system may be implemented by including at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0075] Meanwhile, the user computing device (110) including the above-described components may further perform at least some of the functional operations performed by the server computing system (130) described later.

[0076] -Server Computing System (130: Server Computing System)

[0077] The server computing system (130) can perform a series of processes to provide an environment-adaptive outlier detection service.

[0078] In detail, in an embodiment, the server computing system (130) can provide the environment-adaptive outlier detection service by exchanging data necessary to enable the environment-adaptive outlier detection service process to run on an external device, such as a user computing device (110), with said external device.

[0079] More specifically, in an embodiment, the server computing system (130) can provide an environment in which an application can run on a user computing device (110).

[0080] To this end, the server computing system (130) may include an application program, data and / or instructions, etc. for the application to operate, and may transmit and receive various data based thereon with the external device.

[0081] Additionally, the server computing system (130) includes at least one processor (131) and memory (132). Here, the processor (131) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions.

[0082] And the memory (132) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc. and combinations thereof. This memory (132) may store data (133) and instructions (134) necessary for the processor (131) to perform functional operations, such as training an artificial intelligence model or executing a process to detect suspected defective areas through the artificial intelligence model.

[0083] In one embodiment, the server computing system (130) may be implemented to include at least one computing device. For example, the server computing system (130) may be implemented to operate a plurality of computing devices according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. Additionally, the server computing system (130) may include a plurality of computing devices connected to a network (170).

[0084] Additionally, the server computing system (130) may store at least one machine learning model (140). For example, the server computing system (130) may include a neural network and / or other multi-layer non-linear model as the machine learning model (140). Exemplary neural networks may include a feed-forward neural network, a deep neural network, a recurrent neural network, and a convolutional neural network.

[0085] In an embodiment, the server computing system (130) may further include a data store computing system (hereinafter, data store) which is a storage for continuously storing and managing raw data that forms the basis of an environment-adaptive outlier detection service.

[0086] Such data stores may include various forms of data storage, ranging from file systems to cloud storage. For example, a data store may include at least one database among a relational database that uses a structured query language (SQL) to define and manipulate data, a NoSQL database designed for flexibility and scalability to process unstructured and semi-structured data, a data warehouse optimized for querying and analysis by centralizing large volumes of data from multiple sources as a system used for reporting and data analysis, a data warehouse that stores large volumes of raw data in basic formats such as structured data, semi-structured data, and unstructured data, and a local storage device or Network Attached Storage (NAS) that stores data in files in a format generally accessible by a computer operating system.

[0087] - Training Computing System (150: Training Computing System)

[0088] The training computing system (150) includes at least one processor (151) and memory (152). Here, the processor (151) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, or a plurality of electrically connected processors.

[0089] And the memory (152) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc. and combinations thereof. This memory (152) may store data (153) and instructions (154) necessary for the processor (151) to perform learning of an artificial intelligence model, etc.

[0090] For example, the training computing system (150) may include a model trainer (160) that trains a machine learning model (120 and / or 140) stored in a user computing device (110) and / or a server computing system (130) using various training or learning techniques, such as back propagation of error (according to the framework illustrated in FIG. 3).

[0091] For example, such a model trainer (160) can perform updates to one or more parameters of a machine learning model (120 and / or 140) for an environment-adaptive outlier detection service based on a defined loss function in a backpropagation manner.

[0092] In some embodiments, performing backpropagation of the error may include performing truncated backpropagation through time. The model trainer (160) may perform a number of generalization techniques (e.g., weight devaluation, dropout and / or knowledge distillation, etc.) to improve the generalization ability of the machine learning model (120 and / or 140) being trained.

[0093] Additionally, the model trainer (160) can train a machine learning model (120 and / or 140) based on a series of training data (161). Here, the training data (161) may include data of different forms, such as images, audio samples and / or text, for example. Examples of image types that may be used may include video frames, LiDAR point clouds, X-ray images, computed tomography scans, hyperspectral images and / or various other forms of images.

[0094] These training data (161) may be provided by a user computing device (110) and / or a server computing system (130). When the training computing device trains a machine learning model (120 and / or 140) on specific data of the user computing device (110), the machine learning model (120 and / or 140) may be characterized as a personalized model.

[0095] And the model trainer (160) includes computer logic that is utilized to provide the desired function.

[0096] Additionally, the model trainer (160) may be implemented as hardware, firmware, and / or software that controls a general-purpose processor. In one embodiment, the model trainer (160) may include a program file stored in a storage device, be loaded into memory (152), and be executed by one or more processors (151). In another embodiment, the model trainer (160) includes one or more sets of computer-executable data (153) and instructions (154) stored in a tangible computer-readable storage medium, such as a RAM hard disk or an optical or magnetic medium.

[0097] Network (170) includes, but is not limited to, 3GPP (3rd Generation Partnership Project) network, LTE (Long Term Evolution) network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth network, satellite broadcasting network, analog broadcasting network and / or DMB (Digital Multimedia Broadcasting) network.

[0098] Generally, communication through the network (170) can be performed using any type of wired and / or wireless connection through various communication protocols (e.g., TCP / IP, HTTP, SMTP and / or FTP, etc.), encodings or formats (e.g., HTML and / or XML, etc.), and / or protection schemes (e.g., VPN, Secure HTTP and / or SSL, etc.).

[0099] FIG. 2 illustrates an example of a block diagram of a computing device implementing an environment-adaptive outlier detection service according to an embodiment of the present invention.

[0100] Including FIG. 2, the computing device (100) included in the user computing device (110), server computing system (130), and training computing system (150) includes a plurality of applications (e.g., Application 1 to Application N). Each application may include a machine learning library and one or more machine learning models. For example, the applications may include an image processing application (e.g., Detection, Classification and / or Segmentation, etc.), a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application and / or a chat-bot application, etc.

[0101] In an embodiment, the computing device (100) may include a model trainer (160) for training an artificial intelligence model, and by storing and operating the trained artificial intelligence model, it may provide output data according to a predetermined input data (e.g., an inspection target image).

[0102] Each application of the computing device (100) can communicate with a number of other components of the computing device (100), such as, for example, at least one sensor, a context manager, a device state component, and / or additional components. In one embodiment, each application can communicate with each device component using an API (e.g., a public API). In one embodiment, the API used by each application may be specific to that application.

[0103] FIG. 3 illustrates an example of a block diagram in another aspect of a computing device implementing an environment-adaptive outlier detection service according to one embodiment of the present invention.

[0104] Referring to FIG. 3, the computing device (200) includes a plurality of applications (e.g., Application 1 to Application N). Each application can communicate with a central intelligence layer. For example, applications may include an image processing application, a text messaging application, an email application, a dictation application, a virtual keyboard application and / or a browser application. In one embodiment, each application can communicate with the central intelligence layer (and a model stored therein) using an API (e.g., a common API across all applications).

[0105] The central intelligence layer may include a number of machine learning models. For example, as illustrated in FIG. 3, at least some of the machine learning models may be provided for each application and managed by the central intelligence layer. In another embodiment, two or more applications may share a single machine learning model. For example, in some embodiments, the central intelligence layer may provide a single model for all applications. In some embodiments, the central intelligence layer may be included within the operating system of the computing device (200) or otherwise implemented.

[0106] The central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized data store for the computing device (200). As illustrated in FIG. 3, the central device data layer can communicate with a number of other components of the computing device (200), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0107] The technology described herein may refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to or from said systems. It will be recognized that the inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, division of tasks, and functionality between and from components. For example, the processes described herein may be implemented using a single device or component or multiple devices or components operating in combination. Databases and applications may be implemented in a single system or in a distributed system across multiple systems. Distributed components may operate sequentially or in parallel.

[0108] [Method for Detecting Suspected Defective Areas]

[0109] Hereinafter, a method for a computing system (1000) according to one embodiment of the present invention to detect a suspected defect area through a deep learning-based pre-trained model without separate training using a small number of good product images will be described in detail with reference to the attached drawings.

[0110] FIG. 4 illustrates a flowchart for explaining a method for detecting suspected defective areas according to an embodiment of the present invention.

[0111] Referring to FIG. 4, a method for detecting a suspected defect area according to one embodiment of the present invention may include the steps of: collecting a predetermined good product image (S101); acquiring a predetermined inspection target image (S103); aligning the inspection target image based on the good product image (S105); dividing the aligned inspection target image into patches (S107); acquiring a multi-scale feature based on the divided patches (S109); estimating the probability of a good product per patch image based on the acquired multi-scale feature (S111); determining whether there is a suspected defect in the inspection target image based on the estimated probability of a good product (S113); and providing a result of determining whether there is a suspected defect (S115).

[0112] Specifically, a computing system (1000) according to one embodiment of the present invention can collect a predetermined good product image. (S101)

[0113] Here, the good product image according to the embodiment may mean an image of a product that is of excellent quality and free of defects.

[0114] In detail, in the embodiment, the computing system (1000) can acquire and collect N (N>1) images of good products by linking with a predetermined sensor system and / or an external device, etc.

[0115] In addition, a computing system (1000) according to one embodiment of the present invention can acquire a predetermined inspection target image. (S103)

[0116] Here, the inspection target image according to the embodiment may refer to an image taken of a target (product) for which the presence or absence of defects is to be determined.

[0117] In detail, in an embodiment, the computing system (1000) can acquire the inspection target image described above by linking with a predetermined sensor system (e.g., a digital camera, etc.) and / or an external device (e.g., vision inspection equipment, etc.).

[0118] In addition, a computing system (1000) according to one embodiment of the present invention can align inspection target images based on good product images. (S105)

[0119] FIG. 5 illustrates a conceptual diagram for explaining a method of aligning inspection target images according to an embodiment of the present invention.

[0120] Referring specifically to FIG. 5, in an embodiment, the computing system (1000) can select one of N good product images as a good product representative image (GRI) according to a predetermined method (e.g., user input and / or a pre-established image selection algorithm, etc.).

[0121] In this embodiment, the computing system (1000) can store a selected representative image (GRI) of a good product in a non-volatile memory and use it for image alignment described later.

[0122] In addition, in the embodiment, the computing system (1000) can align the position of the inspection target image (ITI) based on the good product representative image (GRI).

[0123] This may be intended to prevent performance degradation caused by positional errors that occur when the target product cannot always be photographed from the same position when capturing the inspection target image (ITI).

[0124] In an example, the computing system (1000) can calculate a homography matrix based on a representative image of a good product (GRI) and an inspection target image (ITI).

[0125] For reference, homography is a matrix that represents a geometric transformation between two images in a two-dimensional plane, and can be a matrix capable of transforming feature points of one image into corresponding points of another image. Through this, even if two images are taken at different viewpoints, they can be aligned so that they exist on the same plane.

[0126] In detail, in an embodiment, the computing system (1000) can extract feature points (Feature Detection) of each of the good product representative image (GRI) and the inspection target image (ITI).

[0127] More specifically, the computing system (1000) can extract feature points of a good product representative image (GRI) (hereinafter referred to as good product feature points) and feature points of an inspection target image (ITI) (hereinafter referred to as target feature points) using a predetermined feature point extraction algorithm (e.g., algorithms such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), FAST9, SuperPoint, Munkres and / or LightGlue).

[0128] In addition, in the embodiment, the computing system (1000) can perform feature matching between extracted feature points.

[0129] In detail, the computing system (1000) can detect mutually corresponding feature points between a good product feature point and a target feature point using a predetermined feature point matching algorithm (e.g., Brute-Force Matching and / or FLANN (Fast Library for Approximate Nearest Neighbors), etc.).

[0130] And the computing system (1000) can match the detected feature points to form a pair.

[0131] That is, the computing system (1000) can acquire multiple feature point pairs (hereinafter, corresponding point pairs) that mutually match feature points at the closest positions between a good product feature point and a target feature point.

[0132] At this time, according to the embodiment, the computing system (1000) can remove a pair of feature points (hereinafter referred to as an outlier pair) having a matching rate of less than or equal to a predetermined threshold among a plurality of feature point pairs by using a predetermined outlier removal algorithm (e.g., RANSAC (Random Sample Consensus), etc.).

[0133] For reference, RANSAC may be a homography matrix estimation method that iteratively calculates homography using a set of feature point pairs and has the minimum error for all feature point pairs.

[0134] That is, the computing system (1000) can perform filtering to remove noise components within a plurality of feature point pairs using an outlier removal algorithm such as RANSAC.

[0135] In addition, in the embodiment, the computing system (1000) can calculate a homography matrix (Homography Matrix Estimation) based on the corresponding point pairs obtained as above.

[0136] Specifically, the computing system (1000) can calculate a homography matrix according to a predetermined pair of corresponding points according to the following [Equation 1].

[0137] [Mathematical Formula 1]

[0138]

[0139] That is, the computing system (1000) can convert the first feature point coordinates (x, y) into new feature point coordinates (x', y') through a homography matrix (H) in the form of a 3*3 transformation matrix according to a predetermined pair of corresponding points.

[0140] Thus, the computing system (1000) can estimate a homography matrix (H) based on a plurality of corresponding point pairs based on a good product representative image (GRI) and an inspection target image (ITI).

[0141] Additionally, in the embodiment, the computing system (1000) can perform image warping based on the calculated homography matrix (H).

[0142] Specifically, the computing system (1000) can use the calculated homography matrix (H) to convert the coordinate system of the inspection target image (ITI) to match the coordinate system of the good product representative image (GRI).

[0143] Through this, in the embodiment, the computing system (1000) can align the shooting time (location) of the inspection target image (ITI) with the shooting time (location) of the good product representative image (GRI).

[0144] Thus, the computing system (1000) can divide the aligned inspection target image (AII: hereinafter, aligned inspection image) based on the good product representative image (GRI).

[0145] Therefore, the computing system (1000) can effectively prevent performance degradation caused by position errors that occur when the target product cannot always be photographed at the same location when the inspection target image (ITI) is captured (e.g., when a change in position is induced with every movement by performing product movement automation using a machine).

[0146] In addition, a computing system (1000) according to one embodiment of the present invention can divide an aligned inspection target image (ITI) into patches. (S107)

[0147] Generally, since images used in vision inspection are captured at high resolution to detect even minute defects, processing the entire image area at once can be inefficient in terms of processing time and resource utilization.

[0148] Thus, in the embodiment, the computing system (1000) intends to process a given image (in the embodiment, a good product image and / or an inspection target image (ITI), etc.) by dividing it into predetermined patch units.

[0149] FIG. 6 illustrates examples for explaining a patch-unit image segmentation method according to an embodiment of the present invention.

[0150] Referring specifically to FIG. 6, in the embodiment, the computing system (1000) can divide the inspection target image aligned as above (i.e., the alignment inspection image (AII)) into patches of a preset size.

[0151] In this case, as an example, the computing system (1000) can divide the image into a plurality of patches in a form where there are no mutually overlapping regions, as in (a) of FIG. 6.

[0152] In another embodiment, the computing system (1000) can divide the image into multiple patches having an overlapping area of ​​a preset size of m (m > 0), as shown in FIG. 6 (b). In this case, the computing system (1000) can perform computational processing to minimize errors that may occur at the edges between the multiple patches.

[0153] Thus, in the embodiment, the computing system (1000) can divide the alignment inspection image (AII) into k (k>1) patches as in (c) of FIG. 6.

[0154] In addition, a computing system (1000) according to one embodiment of the present invention can acquire a multi-scale feature based on a divided patch. (S109)

[0155] Here, a multi-scale feature according to an embodiment may refer to features (feature vectors) extracted from an image at various sizes and / or resolutions.

[0156] That is, in the embodiment, the computing system (1000) can implement higher quality data processing that is robust to scale changes by comprehensively analyzing features extracted at various sizes and / or resolutions based on divided patches.

[0157] Referring further to FIG. 6, in the embodiment, the computing system (1000) can obtain a plurality of patch images (hereinafter, a plurality of inspection patch images) by dividing an alignment inspection image (AII) into patch units.

[0158] FIG. 7 illustrates a conceptual diagram for explaining a method of extracting image features through a pre-trained model according to an embodiment of the present invention, and FIG. 8 illustrates a conceptual diagram for explaining a method of acquiring multi-scale features according to an embodiment of the present invention.

[0159] Additionally, referring to FIGS. 7 and 8, in the embodiment, the computing system (1000) can acquire multi-scale features based on a plurality of inspection patch images (IPI) by linking with a model (PTM: hereinafter, pre-trained model) trained through various methodologies such as deep learning and / or machine learning.

[0160] Here, the pre-trained model (PTM) according to the embodiment may include a deep learning model that has been pre-trained with a large dataset such as ImageNet, or a foundation model that has been trained on hundreds of millions of images through self-supervised representation learning. However, these are merely examples, and any deep learning model capable of extracting features of each patch image may be included.

[0161] In another aspect, the pre-trained model (PTM) according to the embodiment may be a deep learning model including algorithms such as a Convolutional Neural Network (CNN) and / or a Transformer.

[0162] In this way, by utilizing a pre-trained model (PTM), the computing system (1000) can easily perform the image feature extraction process required for detecting suspected defect areas without separate training.

[0163] Returning to FIG. 8, in the embodiment, the computing system (1000) can acquire multi-scale features of various sizes and / or resolutions in units of a predetermined inspection patch image (IPI).

[0164] For example, a computing system (1000) can acquire multi-scale features according to a first size corresponding to the size of one inspection patch image (IPI), a second size corresponding to the size of four inspection patch images (IPI), and a third size corresponding to the size of nine inspection patch images (IPI).

[0165] At this time, according to the embodiment, the computing system (1000) can divide a predetermined inspection patch image (IPI) into predetermined sub-patch units.

[0166] In the embodiment, the specific method for dividing the inspection patch image (IPI) into a predetermined sub-patch unit is omitted by applying the description of the method for dividing the inspection target image (ITI) into patch units in the aforementioned step S107.

[0167] In addition, in the embodiment, the computing system (1000) can perform multi-scale feature extraction using the aforementioned prior learning model (PTM) based on a plurality of patch images (hereinafter, a plurality of sub-patch images) obtained by performing secondary partitioning.

[0168] That is, the computing system (1000) can divide the given patch image itself into sections to consider more detailed features and obtain multi-scale features based thereon.

[0169] In this way, in the embodiment, the computing system (1000) can extract features based on various scales rather than simply extracting features based on one size from a plurality of patch images.

[0170] Thus, the computing system (1000) can extract more detailed features from the image, ranging from complex patterns in a wide area to fine patterns in a small area.

[0171] In addition, a computing system (1000) according to one embodiment of the present invention can estimate the probability of a good product per patch image according to the acquired multi-scale feature. (S111)

[0172] Here, the probability of a good product according to the example may refer to a value obtained by quantifying, through Gaussian density estimation, how similar each patch of the inspection target image (ITI) is to the features of the good product image.

[0173] That is, in the embodiment, the computing system (1000) can calculate the probability of how similar the features of the alignment inspection image (AII) are to the features of the good product image by using the multi-scale features obtained in step S109 (i.e., feature vectors based on patches of the alignment inspection image (AII)).

[0174] In detail, in the embodiment, the computing system (1000) can calculate the probability of a good product (p(x)) of each patch image according to a Gaussian density estimation formula such as [Equation 2] below.

[0175] [Mathematical Formula 2]

[0176]

[0177] Here, 'x' in [Equation 2] represents the feature vector of the corresponding patch, 'μ' represents the average of the feature vectors extracted from the good product image, 'Σ' represents the covariance matrix of the good product image feature vectors, and 'd' represents the dimension of the feature vector.

[0178] More specifically, in an embodiment, the computing system (1000) can obtain the average (μ) of feature vectors extracted from N (N>1) good product images according to the following [Equation 3].

[0179] [Mathematical Formula 3]

[0180]

[0181] Here, 'N' in [Equation 3] represents the number of good product images used, and 'X + ' can mean a good quality image.

[0182] In addition, in the embodiment, the computing system (1000) can obtain the covariance (Σ) of feature vectors extracted from N (N>1) good product images according to the following [Equation 4].

[0183] [Mathematical Formula 4]

[0184]

[0185] Specifically, in the embodiment, the computing system (1000) can acquire multi-scale features for each good product image by applying the process according to steps S105 to S109 described above based on each good product image.

[0186] And the computing system (1000) can obtain the average (μ) described above based on the multi-scale features per acquired good product image and [Equation 3].

[0187] In addition, the computing system (1000) can obtain the covariance (Σ) described above based on the multi-scale features per acquired good product image and [Equation 4].

[0188] In addition, in the embodiment, the computing system (1000) can apply the mean (μ) and covariance (Σ) obtained as above to the [Equation 2] described above to calculate the probability of a good product (p(x)) for each patch image (i.e., each patch image of the alignment inspection image (AII)) based on Gaussian density estimation.

[0189] Thus, the computing system (1000) can estimate how similar the features of each patch of the inspection target image (ITI) (here, alignment inspection image (AII)) are to the average features of N good product images.

[0190] In addition, a computing system (1000) according to one embodiment of the present invention can determine whether there is a suspected defect in an inspection target image (ITI) based on an estimated probability of a good product (p(x)). (S113)

[0191] In detail, in an embodiment, the computing system (1000) can compare the probability of a good product (p(x)) for each patch image with a predetermined threshold (hereinafter, classification threshold).

[0192] And the computing system (1000) can determine each patch image as a good patch image or a patch image suspected of being defective based on the comparison result.

[0193] More specifically, as an example, the computing system (1000) can compare the probability of a good product (p(x)) for each patch image with a preset first classification threshold.

[0194] At this time, the computing system (1000) can determine that a patch image corresponding to a predetermined good product probability (p(x)) is a good product patch image if the predetermined good product probability (p(x)) is greater than or equal to a first classification threshold.

[0195] On the other hand, the computing system (1000) can determine that a patch image corresponding to a predetermined probability of a good product (p(x)) is a patch image suspected of being defective if the probability of a good product (p(x)) is less than a predetermined first classification threshold.

[0196] That is, in the embodiment, the computing system (1000) can determine whether a product is good or suspected of being defective on a patch-by-patch basis of an image.

[0197] Therefore, the computing system (1000) can detect areas suspected of defects more quickly and efficiently than processing the entire image at once, and can improve accuracy by analyzing each area within the image individually to detect specific areas where defects have occurred more precisely.

[0198] At this time, according to the embodiment, the computing system (1000) can obtain a probability of good quality (p(x)) for each of a plurality of sub-patch images corresponding to a predetermined first patch image by applying the description of step S111 described above.

[0199] And the computing system (1000) can detect a good product probability (p(x)) (hereinafter, minimum good product probability value) having a minimum value among the multiple good product probabilities (p(x)) for each acquired sub-patch image.

[0200] Additionally, the computing system (1000) can compare the detected minimum probability value of a good product with a preset first classification threshold.

[0201] At this time, the computing system (1000) can determine that a first patch image corresponding to a minimum probability value of a good product is a good product patch image if the minimum probability value of a good product is greater than or equal to a first classification threshold that has been set.

[0202] On the other hand, the computing system (1000) can determine that a first patch image corresponding to a minimum probability value of a good product is a patch image suspected of being defective if the minimum probability value of a good product is less than a preset first classification threshold.

[0203] That is, according to the embodiment, the computing system (1000) can further improve the accuracy by determining whether each patch image is a good product or suspected of being defective using the probability of a good product (p(x)) based on multi-scale features analyzed in more detail.

[0204] Meanwhile, according to an embodiment, the computing system (1000) can determine each patch image as a good patch image or a patch image suspected of being defective based on a multi-stage classification based on a plurality of preset thresholds (hereinafter, a plurality of classification thresholds).

[0205] FIG. 9 illustrates a flowchart for explaining a multi-stage classification method for patch images based on the probability of quality (p(x)) according to one embodiment of the present invention.

[0206] Referring specifically to FIG. 9, in the embodiment, the computing system (1000) can compare the probability of a good product (p(x)) for each patch image with a preset first distinction threshold. (S201)

[0207] Here, the first distinction threshold according to the embodiment can correspond to the minimum probability condition for being classified as a good product patch image.

[0208]

[0209] In addition, in the embodiment, the computing system (1000) can determine that a patch image corresponding to a certain probability of a good product (p(x)) is a good product patch image if the probability of a good product (p(x)) is greater than or equal to a preset first distinction threshold. (S203)

[0210] On the other hand, the computing system (1000) can set a patch image corresponding to a predetermined probability of a good product (p(x)) as a suspected defect candidate patch image if the probability of a good product (p(x)) is less than a predetermined first distinction threshold.

[0211] Additionally, in the embodiment, the computing system (1000) can compare the probability of a good product (p(x)) of a set suspected defect candidate patch image (probability of a good product (p(x)) less than a set first distinction threshold, hereinafter referred to as the probability of a suspected defect candidate) with a set second distinction threshold. (S205)

[0212] Here, the second distinction threshold according to the embodiment can correspond to the maximum probability condition for being classified as a patch image suspected of being defective.

[0213] Additionally, in the embodiment, the computing system (1000) can determine a patch image corresponding to a probability of a suspected defect candidate as a suspected defect patch image if the probability of a suspected defect candidate is less than or equal to a preset second distinction threshold. (S207)

[0214] FIG. 10 illustrates an example of a drawing for explaining a method for obtaining a minimum probability area image of a good product according to an embodiment of the present invention.

[0215] On the other hand, referring to FIG. 10, in the embodiment, the computing system (1000) can obtain a probability of a defective candidate (p(x)) for a plurality of sub-patch images corresponding to the defective candidate probability when the probability of a defective candidate exceeds a preset second distinction threshold (i.e., when a predetermined probability of a good product (p(x)) is less than the first distinction threshold and exceeds the second distinction threshold) by applying the description of step S111 described above.

[0216] And the computing system (1000) can detect a good product probability (p(x)) having a minimum value among the multiple good product probabilities (p(x)) for each sub-patch image obtained (i.e., minimum good product probability value).

[0217] In addition, in the embodiment, the computing system (1000) can extract an area having a predetermined size (H(Height)*W(Width)) centered on an area matching the detected minimum probability value of a good product (hereinafter, the minimum probability point of a good product).

[0218] At this time, in the embodiment, the computing system (1000) can extract an area having a predetermined size centered on the minimum probability point of a good product (in the embodiment, a pixel having a minimum probability value of a good product) when there is only one minimum probability point of a good product.

[0219] On the other hand, referring further to FIG. 10, the computing system (1000) can calculate the center of gravity for the location of the corresponding multiple minimum probability points of good products when there are multiple minimum probability points of good products.

[0220] In an embodiment of the present invention, a specific method for a computing system (1000) to calculate the center of gravity described above may be performed by utilizing various disclosed algorithms that implement the same, and the embodiments of the present invention do not limit or restrict the algorithm itself.

[0221] In addition, in the embodiment, the computing system (1000) can extract an area having a predetermined size centered on an area corresponding to the calculated center of gravity (hereinafter, center of gravity point).

[0222] That is, the computing system (1000) can extract a pixel having the lowest probability value among the multiple sub-patch image-specific probability of good product (p(x)) or an area of ​​a predetermined size centered on the centroid of those pixels (hereinafter, minimum probability area of ​​good product).

[0223] Thus, in the embodiment, the computing system (1000) can obtain an image including the extracted minimum probability area of ​​a good product (hereinafter, the minimum probability area of ​​a good product image). (S209)

[0224] Additionally, in the embodiment, the computing system (1000) can perform a determination of good product or suspected defect for each patch image of the good product minimum probability zone image obtained as above by linking with a model learned through various methodologies such as a predetermined deep learning and / or machine learning (i.e., a pre-trained model (PTM)). (S211)

[0225] In other words, the computing system (1000) can determine each patch image of the good product minimum probability area image as a good product patch image or a patch image suspected of being defective by linking with a predetermined prior learning model (PTM).

[0226] Here, the pre-trained model (PTM) according to the embodiment may include a deep learning classification model trained through a relevant dataset, or may include a model that extracts feature values ​​within an image based on a predetermined image processing methodology and then implements a machine learning methodology such as Random Forest, and various other embodiments are possible.

[0227] In this way, in the embodiment, the computing system (1000) can immediately determine patch samples with a probability of being good above a predetermined threshold as good, and immediately classify patch samples with a probability of being good below a predetermined threshold as suspected defective patches, thereby shortening the overall processing time.

[0228] Additionally, the computing system (1000) can determine whether there is an additional suspicion of defect by cutting out an image centered on the area with the lowest probability of being a good product (i.e., the area with the highest probability of a defect) for patch samples in an intermediate state where it is ambiguous whether they are good or suspected of being defective, and performing a re-inspection process based thereon.

[0229] Through this multi-stage judgment method, the computing system (1000) can significantly improve the efficiency and accuracy of detecting suspected defect areas by minimizing misjudgments that may occur in uncertain areas while maintaining the processing speed in the inspection process.

[0230] Returning to Fig. 4, the computing system (1000) that determines each patch image as a good patch image or a patch image suspected of being defective can determine whether there is a suspected defect in the corresponding inspection target image (ITI) according to a predetermined method.

[0231] In an example, the computing system (1000) can determine whether there is a suspected defect in the inspection target image (ITI) based on the number of good patch images (hereinafter, number of good patches) and the number of patch images suspected of being defective (hereinafter, number of patches suspected of being defective).

[0232] For example, a computing system (1000) can calculate the ratio between the number of good patches and the number of suspected defective patches, and detect the side with the higher ratio.

[0233] At this time, the computing system (1000) can determine that the inspection target image (ITI) is a good product image if the ratio of the number of good product patches is higher.

[0234] On the other hand, the computing system (1000) can determine that the inspection target image (ITI) is a suspected defective image if the ratio of the number of suspected defective patches is higher.

[0235] In another example, the computing system (1000) may determine that the inspection target image (ITI) is a good image if the number of good patches is greater than or equal to a predetermined threshold, or determine that the inspection target image (ITI) is a defective image if the number of suspected defective patches is greater than or equal to a predetermined threshold.

[0236] As described above, the computing system (1000) according to an embodiment of the present invention can efficiently detect suspected defective areas within an inspection target image (ITI) by estimating a patch-level good product probability (p(x)) using a small number of good product images without separate prior learning.

[0237] Through this, the computing system (1000) can quickly and accurately detect areas suspected of having defects through advanced analysis using a deep learning model.

[0238] In particular, the computing system (1000) can reduce the burden of data collection for model training by using a pre-trained model, and at the same time, flexibly adapt to various product and / or inspection environment changes without undergoing a separate training process.

[0239] In addition, the computing system (1000) can reduce the time and resources required for vision inspection compared to a method of processing the entire image area in batches by dividing and processing the image at the patch level.

[0240] Therefore, the computing system (1000) can implement a defect suspected area detection function that maintains high accuracy while maximizing the speed and efficiency of the vision inspection process.

[0241] In addition, a computing system (1000) according to one embodiment of the present invention can provide a result of determining whether there is a suspected defect. (S115)

[0242] That is, in the embodiment, the computing system (1000) can provide a result of determining whether there is a suspected defect in the inspection target image (ITI) (i.e., a result of determining whether it is a good product image or a suspected defect image) according to a predetermined method.

[0243] In an example, the computing system (1000) may provide a result of suspected defect determined according to an embodiment of the present invention by linking with a predetermined application service (e.g., a component inspection service, a semiconductor wafer inspection service, a food packaging inspection service and / or a smartphone manufacturing inspection service, etc.).

[0244] In summary, the deep learning-based environment-adaptive outlier detection method and system according to one embodiment of the present invention has the effect of reducing the burden of data collection for model training and flexibly responding to changes in the inspection environment by detecting suspected defect areas through a deep learning-based pre-trained model (PTM) without separate training using a small number of good product images.

[0245] In addition, the deep learning-based environment-adaptive outlier detection method and system according to one embodiment of the present invention has the effect of reducing the time and resources required for inspection compared to a method of processing the entire image area in batches by dividing and processing the image at the patch level.

[0246]

[0247] [Method for Providing Environment-Adaptive Anomaly Detection Services]

[0248] Hereinafter, a method for providing a deep learning-based environment-adaptive outlier detection service (i.e., an environment-adaptive outlier detection service) that operates adaptively on various material-specific characteristics constituting a vision inspection target, according to one embodiment of the present invention, will be described in detail with reference to the attached drawings.

[0249] In conventional vision inspection methods, performance degradation may occur because the same inspection standards are applied to products composed of different materials without considering differences in reflectance and / or patterns resulting from differences in the characteristics of the various materials constituting the product.

[0250] In particular, the existing method faces difficulties such as having to develop a new rule-based algorithm for vision inspection or modify the existing algorithm whenever the product material changes.

[0251] Thus, the computing system (1000) according to an embodiment of the present invention aims to provide a vision inspection (i.e., detection of suspected defect areas) method that operates adaptively to the characteristics of various product materials.

[0252] In other words, the computing system (1000) according to the embodiment of the present invention aims to implement a vision inspection method that effectively adapts to dynamic environmental changes, such as a multi-product, small-batch production environment, and detects suspected defect areas with high accuracy.

[0253] FIG. 11 illustrates a flowchart for explaining a deep learning-based environment-adaptive suspected defect area detection method according to an embodiment of the present invention.

[0254] Referring to FIG. 11, a method for providing an environment-adaptive outlier detection service according to an embodiment of the present invention comprises: a step of constructing a Gaussian application information database for each predetermined product material (S301); a step of collecting a predetermined good product image (S303); a step of acquiring a predetermined inspection target image (S305); a step of aligning the inspection target image based on the good product image (S307); a step of dividing the aligned inspection target image into patch units (S309); a step of acquiring multi-scale features based on the divided patches (S311); a step of acquiring product material information for the inspection target image (S313); a step of searching for Gaussian application information corresponding to the acquired product material information (S315); a step of determining whether there is a suspected defect in the inspection target image based on the searched Gaussian application information (S317); a step of providing a result of determining whether there is a suspected defect (S319); and an absent Gaussian It may include a step (S321) of providing a notification regarding application information.

[0255] In detail, a computing system (1000) according to one embodiment of the present invention can build a database of Gaussian application information for each predetermined product material. (S301)

[0256] Here, the Gaussian application information according to the embodiment may be information including the mean (μ) and covariance (Σ) applied to the Gaussian density estimation according to the aforementioned [Equation 2].

[0257] That is, the Gaussian application information according to the embodiment may be information including the mean (μ) of feature vectors extracted from N (N>1) good product images and the covariance (Σ) of feature vectors extracted from N (N>1) good product images applied to [Equation 2].

[0258] FIG. 12 illustrates a conceptual diagram for explaining a method of constructing a Gaussian application information database for each product material according to an embodiment of the present invention.

[0259] In detail, referring to FIG. 12, in an embodiment, the computing system (1000) can collect a data set of good product images by various product materials (GDS: hereinafter, a data set of good product images by material).

[0260] In addition, in the embodiment, the computing system (1000) can select a representative image of a good product corresponding to a predetermined material good product image data set (GDS).

[0261] In an example, the computing system (1000) may select any one of N good product images in the a material good product image data set (GDS) as a representative good product image for the a material good product image data set (GDS) according to a predetermined method (e.g., user input and / or a pre-established image selection algorithm).

[0262] Additionally, in the embodiment, the computing system (1000) can align the positions of the remaining good product images in the a material good product image data set (GDS) based on a selected good product representative image.

[0263] In addition, in the embodiment, the computing system (1000) can divide the aligned good product images into patches.

[0264] Additionally, in the embodiment, the computing system (1000) can acquire multi-scale features for each patch image based on the divided patches.

[0265] At this time, the specific method by which the computing system (1000) according to the embodiment of the present invention performs image alignment, performs patch unit segmentation, and acquires multi-scale features on the a material good product image data set (GDS) is omitted by applying the description described in steps S105 to S109 above.

[0266] In addition, in the embodiment, the computing system (1000) can calculate the mean (μ) and covariance (Σ) according to the aforementioned [Equation 3] and [Equation 4] based on the acquired multi-scale features.

[0267] Thus, in the embodiment, the computing system (1000) can obtain Gaussian application information of the a material good product image data set (GDS), including the mean (μ) and covariance (Σ) calculated for the a material good product image data set (GDS).

[0268] In the same way, in the embodiment, the computing system (1000) can obtain Gaussian application information for each of the remaining materials (e.g., material b, etc.) good product image data sets (GDS).

[0269] And in the embodiment, the computing system (1000) can store the acquired Gaussian application information for a plurality of materials (i.e., information on the mean (μ) and covariance (Σ) for a plurality of materials) in non-volatile memory to form a database.

[0270] In addition, a computing system (1000) according to one embodiment of the present invention can collect a predetermined good product image (S303), acquire a predetermined inspection target image (S305), align the inspection target image based on the good product image (S307), divide the aligned inspection target image into patch units (S309), and acquire a multi-scale feature based on the divided patch (S311).

[0271] At this time, the specific method by which the computing system (1000) according to the embodiment of the present invention performs steps S303 to S311 is omitted by applying the description of steps S101 to S109 described above.

[0272] In addition, a computing system (1000) according to one embodiment of the present invention can obtain product material information for an inspection target image. (S313)

[0273] Here, product material information according to the embodiment may mean information specifying the material constituting a predetermined product.

[0274] That is, in the embodiment, the computing system (1000) can obtain product material information for an inspection target image, which is information specifying the material constituting the target (product) included in the inspection target image.

[0275] In an example, the computing system (1000) can obtain product material information of an inspection target image through a predetermined user input and / or an external device (e.g., production process management system equipment, etc.).

[0276] In addition, a computing system (1000) according to one embodiment of the present invention can search for Gaussian application information corresponding to the acquired product material information. (S315)

[0277] In detail, in an embodiment, the computing system (1000) can search whether there is Gaussian application information corresponding to the product material information obtained as above in the product material-specific Gaussian application information database built in the aforementioned step S301.

[0278] In other words, the computing system (1000) can check whether the mean (μ) and covariance (Σ) information for the material corresponding to the product material information for the inspection target image is stored in the database.

[0279] In addition, a computing system (1000) according to one embodiment of the present invention can determine whether there is a suspected defect in an inspection target image based on the searched Gaussian application information. (S317)

[0280] Specifically, in an embodiment, if Gaussian application information (hereinafter, Gaussian matching information) corresponding to product material information of an inspection target image exists in a database, the computing system (1000) can estimate the probability of a good product for the inspection target image by applying the description of step S111 described above.

[0281] At this time, in the embodiment, the computing system (1000) can apply the Gaussian matching information described above to [Equation 2] to estimate the probability of a good product of an inspection target image based on Gaussian density estimation.

[0282] Additionally, in the embodiment, the computing system (1000) can determine whether there is a suspected defect in the inspection target image by applying the description of step S113 described above based on the estimated probability of a good product.

[0283] That is, in the embodiment, the computing system (1000) can determine that the inspection target image is a good product image or a suspected defective product image.

[0284] In addition, a computing system (1000) according to one embodiment of the present invention may provide a result of determining whether there is a suspected defect. (S319)

[0285] That is, in the embodiment, the computing system (1000) can provide a result of determining whether there is a suspected defect in the inspection target image (i.e., a result of determining whether it is a good product image or a suspected defect image) according to a predetermined method.

[0286] In an example, the computing system (1000) may provide a result of suspected defect determined according to an embodiment of the present invention by linking with a predetermined application service (e.g., a component inspection service, a semiconductor wafer inspection service, a food packaging inspection service and / or a smartphone manufacturing inspection service, etc.).

[0287] Meanwhile, a computing system (1000) according to one embodiment of the present invention can provide a notification regarding missing Gaussian application information. (S321)

[0288] Specifically, in an embodiment, the computing system (1000) can classify the inspection target image as an unprocessed image when Gaussian application information (i.e., Gaussian matching information) corresponding to the product material information of the inspection target image is not present in the database.

[0289] In addition, in the embodiment, the computing system (1000) may provide a predetermined notification (hereinafter, notification of absence of Gaussian application information) that indicates the absence of the corresponding Gaussian matching information.

[0290] At this time, in the embodiment, the computing system (1000) may provide a Gaussian application information absence notification that further includes a predetermined notification guiding the calculation of mean (μ) and covariance (Σ) information corresponding to the product material information.

[0291] In an example, the computing system (1000) may provide a notification of absence of Gaussian application information in the form of a predetermined message, sound and / or data signal to a predetermined user (e.g., a vision inspection operator, etc.) and / or a device (e.g., a production process management system equipment, etc.).

[0292] As described above, the computing system (1000) according to an embodiment of the present invention can implement a flexible inspection system that efficiently detects suspected defect areas by reflecting the characteristics of the product's materials.

[0293] That is, in the embodiment, the computing system (1000) performs accurate detection of suspected defect areas using mean and covariance information tailored to each of various materials, and can support the rapid collection and reflection of mean and covariance information for new materials in the absence of such information.

[0294] Thus, the computing system (1000) can reduce inspection errors caused by different material characteristics in environments such as multi-product small-batch production, simultaneously improve inspection accuracy and efficiency, and rapidly and flexibly adapt to material changes of various products, thereby greatly improving product quality control performance.

[0295]

[0296] Meanwhile, the embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present invention or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. Hardware devices may be modified into one or more software modules to perform processing according to the present invention, and vice versa.

[0297] The specific embodiments described in this invention are examples and do not limit the scope of the invention in any way. For the sake of brevity of the specification, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted. Additionally, the connections of lines or connecting members between components shown in the drawings are illustrative of functional connections and / or physical or circuit connections, and may be replaced or additionally represented as various functional connections, physical connections, or circuit connections in actual devices. Furthermore, unless specifically stated as “essential,” “importantly,” etc., a component may not be strictly necessary for the application of the invention.

[0298] Furthermore, although the detailed description of the present invention has been explained with reference to preferred embodiments of the invention, those skilled in the art or those with ordinary knowledge in the relevant technical field will understand that various modifications and changes can be made to the invention without departing from the spirit and technical scope of the invention as set forth in the claims below. Accordingly, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be determined by the claims.

[0299]

[0300] The present disclosure relates to a deep learning-based environment-adaptive outlier detection method and system, and since it is applicable to the artificial intelligence industry, it has industrial applicability.

Claims

A method for a computing system including memory and a processor to perform deep learning-based environment-adaptive outlier detection, A step of acquiring a data set of images of good products corresponding to each predetermined material; A step of obtaining corresponding Gaussian application information for each material based on Gaussian density estimation based on the above good product image dataset; A step of constructing a database including the above-mentioned Gaussian application information; and A step of determining whether there is a suspected defect in a predetermined inspection target image based on the above database; and It includes a step of providing the result of whether the above defect is suspected, and The above Gaussian application information is, Information including mean (μ) and covariance (Σ) data of feature vectors extracted from N (N>1) good product images included in the above good product image dataset Deep learning-based environment-adaptive outlier detection method. In Article 1, The step of obtaining corresponding Gaussian application information for each of the above materials is, The step of dividing each of the N (N>1) good product images included in the above good product image data set into patch units, and A step of obtaining a multi-scale feature based on the segmented patch using a predetermined pre-training model, and A step comprising obtaining the Gaussian application information based on the Gaussian density estimation based on the multi-scale feature. Deep learning-based environment-adaptive outlier detection method. In Article 1, The step of determining whether there is suspicion of defect in the above-mentioned inspection target image is, The step of dividing the above inspection target image into patch units, and A step comprising acquiring multi-scale features based on the segmented patches using a predetermined pre-trained model. Deep learning-based environment-adaptive outlier detection method. In Paragraph 3, The step of determining whether there is suspicion of defect in the above-mentioned inspection target image is, The method further includes the step of obtaining product material information, which is information specifying the material constituting the inspection target within the inspection target image. Deep learning-based environment-adaptive outlier detection method. In Paragraph 4, The step of determining whether there is suspicion of defect in the above-mentioned inspection target image is, The method further includes the step of detecting the Gaussian application information corresponding to the product material information from the above database. Deep learning-based environment-adaptive outlier detection method. In Article 5, The step of determining whether there is suspicion of defect in the above-mentioned inspection target image is, When detecting Gaussian matching information, which is Gaussian application information corresponding to the above product material information, the method further includes a step of determining whether there is a suspected defect based on the Gaussian matching information and the multi-scale feature. Deep learning-based environment-adaptive outlier detection method. In Article 6, The step of determining whether there is a suspected defect based on the Gaussian matching information and the multi-scale feature is: The step of obtaining the probability of a good product for each patch based on the Gaussian density estimation, the feature vector for each patch, and the Gaussian matching information for each patch. Deep learning-based environment-adaptive outlier detection method. In Article 7, The step of determining whether there is a suspected defect based on the Gaussian matching information and the multi-scale feature is: A step of comparing the probability of a good product for each patch with a predetermined threshold, and A step of determining each of the above patches as a good patch or a patch suspected of being defective based on the above comparison result, and The method further includes a step of determining whether there is a suspected defect based on at least one of the above good patches and above suspected defective patches. Deep learning-based environment-adaptive outlier detection method. In Article 5, The step of determining whether there is suspicion of defect in the above-mentioned inspection target image is, If Gaussian matching information, which is Gaussian application information corresponding to the above product material information, is not detected, the method further includes the step of providing a predetermined notification indicating the absence of the Gaussian matching information. Deep learning-based environment-adaptive outlier detection method. At least one memory; and It includes at least one processor that reads at least one application stored in the memory and performs deep learning-based environment-adaptive outlier detection; The instructions of the above processor are, A step of acquiring a corresponding good product image data set for each predetermined material, and A step of obtaining corresponding Gaussian application information for each material based on Gaussian density estimation based on the above good product image dataset, and The step of constructing a database including the above-mentioned Gaussian application information, and A step of determining whether there is suspicion of defect in a predetermined inspection target image based on the above database, and Includes a command that performs the step of providing the result of the suspected defect, and The above Gaussian application information is, Information including mean (μ) and covariance (Σ) data of feature vectors extracted from N (N>1) good product images included in the above good product image dataset Deep learning-based environment-adaptive outlier detection system.

Images