Suspected defect region detection method, and system therefor

Abstract

A suspected defect region detection method according to one embodiment of the present disclosure is a method by which a computing system comprising a memory and a processor detects a suspected defect region, the method comprising the steps of: acquiring N (N>1) non-defective product images; acquiring an inspection target image, which is an image obtained by photographing an object for which the presence or absence of a defect is to be determined; aligning the inspection target image on the basis of the non-defective product images; dividing the aligned inspection target image into patch units; using a predetermined pre-training model so as to acquire a multi-scale feature based on the divided patches; estimating a non-defective product probability for each patch on the basis of the acquired multi-scale feature; determining, on the basis of the estimated non-defective product probability for each patch, whether a defect is suspected for the inspection target image; and providing a result of the suspected defect determination.

Description

Method and system for detecting suspected defective areas

[0001] The present disclosure relates to a method and system for detecting suspected defect areas. More specifically, it relates to a vision inspection method and system for detecting suspected defect areas through a deep learning-based pre-trained model without separate training, using a small number of images of good products.

[0002] Generally, vision inspection equipment is intended to automatically detect defects present on the appearance of a product by utilizing computer vision technology, including deep learning.

[0003] Based on data obtained from sensor equipment such as cameras, good products and defective products containing defects must be classified with very high accuracy, and for product quality control, equipment in the inspection process must operate uniformly without problems.

[0004] In this case, since processing time during the inspection process affects product output, it is very important to process quickly along with high accuracy.

[0005] However, since most vision inspections are performed at high resolution to detect even minute defects, using a deep learning model to inspect the entire product in detail can achieve high accuracy, but it presents limitations in terms of processing time.

[0006] Thus, in the conventional field of this technology, efforts are being made to satisfy accuracy and processing time by using a method that detects suspected defect areas through rule-based algorithms utilizing prior knowledge and makes judgments through deep learning models as post-processing.

[0007] However, rule-based algorithms based on various image processing and image-based quantitative measurement have the disadvantage that their performance is determined by the developer's capabilities and know-how, and that it is difficult to readjust the rules when the inspection environment changes, resulting in high costs and time.

[0008] For example, considering the characteristic that scratches on the exterior of a product reflect light and have a high brightness value, there is a rule-based algorithm that performs image processing to binarize the area where scratches occur when the brightness value is above a certain value, defines the area through Connected Component Labeling, and detects it as a suspected defect area by measuring its size. However, if the intensity of the light weakens or the material of the product changes and the reflection of light increases, the performance of this rule-based algorithm in defining the scratch area inevitably deteriorates.

[0009] Therefore, the development and introduction of new technologies are necessary to solve the aforementioned problems.

[0010] One embodiment of the present disclosure is designed to solve the problems of the prior art as described above, and aims to provide a vision inspection method and a system that detects suspected defect areas through a deep learning-based pre-trained model without separate training using a small number of good product images.

[0011] However, the technical problems to be solved by the present disclosure and the embodiments thereof are not limited to the technical problems described above, and other technical problems may exist.

[0012] A method for detecting a suspected defect area according to one embodiment of the present disclosure is a method for a computing system including memory and a processor to detect a suspected defect area, comprising: a step of acquiring N (N>1) images of good products; a step of acquiring an inspection target image, which is an image of a target to be determined to be defective; a step of aligning the inspection target image based on the good product images; a step of dividing the aligned inspection target image into patches; a step of acquiring a multi-scale feature based on the divided patches using a predetermined pre-training model; a step of estimating the probability of a good product per patch based on the acquired multi-scale feature; a step of determining whether there is a suspected defect in the inspection target image based on the estimated probability of a good product per patch; and a step of providing the determined result of the suspected defect.

[0013] In another aspect, the multi-scale feature is data obtained by extracting feature vectors for a given image based on various sizes or resolutions.

[0014] In another aspect, the step of acquiring the multi-scale feature includes forming the size in patch units and acquiring the multi-scale feature.

[0015] In another aspect, the step of estimating the probability of a good product per patch includes the step of calculating the similarity between the feature vector of each patch and the feature vector of the good product image based on Gaussian density estimation.

[0016] In another aspect, the step of calculating the similarity includes the step of calculating the mean and covariance of the feature vectors of the good product image, and the step of calculating the similarity based on the calculated mean and covariance and the feature vector of each patch.

[0017] In another aspect, the step of determining whether there is a suspected defect in the inspection target image comprises comparing each of the probability of a good product for each patch with a predetermined first threshold, and determining each of the patches as a good product patch or a patch suspected of being defective based on the comparison result.

[0018] In another aspect, the step of determining whether there is a suspected defect in the inspection target image comprises comparing each of the probability of a good product for each patch with a predetermined second threshold, determining a patch corresponding to a probability of a good product that is greater than or equal to the second threshold as a good product patch and comparing a probability of a good product that is less than the second threshold with a predetermined third threshold, and determining a patch corresponding to a probability of a good product that is less than or equal to the third threshold as a patch suspected of being defective.

[0019] In another aspect, the step of determining whether there is a suspected defect in the inspection target image further comprises: a step of obtaining a plurality of sub-patches based on a patch corresponding to a good product probability that is less than the second threshold and greater than the third threshold; a step of obtaining a minimum good product probability value, which is a good product probability having the minimum value among the multiple good product probabilities for each sub-patch; a step of extracting an area having a predetermined size based on an area matching the obtained minimum good product probability value; and a step of determining whether there is a suspected defect based on an image including the extracted area.

[0020] In another aspect, the step of determining whether there is a suspected defect in the inspection target image further comprises, if there are multiple minimum probability values ​​for good products, a step of calculating the center of gravity of the locations of regions matching the multiple minimum probability values ​​for good products, a step of extracting a region having a predetermined size based on the region corresponding to the calculated center of gravity, and a step of determining whether there is a suspected defect based on an image including the extracted region.

[0021] In another aspect, the step of determining whether there is a suspected defect in the inspection target image further includes the step of determining the suspected defect based on at least one of the good patch and the suspected defective patch.

[0022] In another aspect, the step of aligning the inspection target image based on the good product image comprises: selecting one of the N (N>1) good product images as a representative good product image; extracting feature points of each of the representative good product image and the inspection target image (Feature Detection); mutually matching corresponding feature points between each of the feature points (Feature Matching); calculating a homography matrix based on the matched feature points (Homography Matrix Estimation); and performing image warping on the inspection target image based on the calculated homography matrix.

[0023] Meanwhile, a defect suspected area detection system according to one embodiment of the present disclosure comprises: at least one memory; and at least one processor that detects a defect suspected area by reading at least one application stored in the memory; and the instruction of the processor comprises the steps of: acquiring N (N>1) good product images; acquiring an inspection target image, which is an image of a target to be determined for defects; aligning the inspection target image based on the good product images; dividing the aligned inspection target image into patches; acquiring a multi-scale feature based on the divided patches using a predetermined pre-training model; estimating the probability of a good product per patch based on the acquired multi-scale feature; determining whether there is a suspected defect in the inspection target image based on the estimated probability of a good product per patch; and providing the result of the determined defect suspected.

[0024] A method and system for detecting suspected defect areas according to one embodiment of the present disclosure has the effect of reducing the burden of data collection for model training and flexibly responding to changes in the inspection environment by detecting suspected defect areas through a deep learning-based pre-trained model without separate training using a small number of good product images.

[0025] In addition, the method and system for detecting suspected defective areas according to one embodiment of the present disclosure have the effect of reducing the time and resources required for inspection compared to a method of processing the entire image area in batches by dividing and processing the image at the patch level.

[0026] Accordingly, the method and system for detecting suspected defect areas according to one embodiment of the present disclosure can be usefully applied to a multi-product, small-batch production environment and has the effect of simultaneously improving inspection accuracy and efficiency.

[0027] However, the effects obtainable in this disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly understood from the description below.

[0028] FIG. 1 illustrates an example of a block diagram of a computing system implementing a suspected defect area detection service according to one embodiment of the present disclosure.

[0029] FIG. 2 illustrates an example of a block diagram of a computing device implementing a suspected defect area detection service according to one embodiment of the present disclosure.

[0030] FIG. 3 illustrates an example of a block diagram in another aspect of a computing device implementing a suspected defect area detection service according to one embodiment of the present disclosure.

[0031] FIG. 4 illustrates a flowchart for explaining a method for detecting a suspected defective area according to one embodiment of the present disclosure.

[0032] FIG. 5 illustrates a conceptual diagram for explaining a method of aligning inspection target images according to one embodiment of the present disclosure.

[0033] FIG. 6 illustrates examples for explaining a patch-unit image segmentation method according to one embodiment of the present disclosure.

[0034] FIG. 7 illustrates a conceptual diagram for explaining a method of extracting image features through a pre-trained model according to one embodiment of the present disclosure.

[0035] FIG. 8 illustrates a conceptual diagram for explaining a method for acquiring multiple scale features according to one embodiment of the present disclosure.

[0036] FIG. 9 illustrates a flowchart for explaining a multi-stage classification method for patch images based on the probability of good quality according to one embodiment of the present disclosure.

[0037] FIG. 10 illustrates an example of a drawing for explaining a method for obtaining a minimum probability area image of a good product according to one embodiment of the present disclosure.

[0038] As the present disclosure is capable of various modifications and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present disclosure, and the methods for achieving them, will become clear by referring to the embodiments described below in detail together with the drawings. However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various forms. In the following embodiments, terms such as "first," "second," etc., are used not in a limiting sense but for the purpose of distinguishing one component from another. Furthermore, singular expressions include plural expressions unless the context clearly indicates otherwise. Also, terms such as "include" or "have" mean that the features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added. Additionally, in the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily depicted for convenience of explanation, so the present disclosure is not necessarily limited to what is depicted.

[0039] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0040]

[0041] [Exemplary system providing a service for detecting suspected defective areas]

[0042] Hereinafter, an exemplary system for implementing a suspected defect area detection service that detects suspected defect areas through a deep learning-based pre-trained model without separate training using a small number of images of good products will be described in detail with reference to the attached drawings.

[0043] FIG. 1 illustrates an example of a block diagram of a computing system implementing a suspected defect area detection service according to one embodiment of the present disclosure.

[0044] Referring to FIG. 1, a computing system (1000) implementing the defective area detection service of the present disclosure includes a user computing device (110), a server computing system (130), and a training computing system (150), and each device and system is connected to communicate through a network (170).

[0045] A method and system for detecting a suspected defective area according to one embodiment of the present disclosure may be implemented and provided locally by a user computing device (110), implemented and provided in the form of a web service by a server computing system (130) communicating with the user computing device (110), and implemented and provided by the user computing device (110) and the server computing system (130) in conjunction with each other.

[0046] In this embodiment, the user computing device (110) and / or the server computing system (130) can train a machine learning model (120 and / or 140) through interaction with a training computing system (150) that is communicatedly connected via a network (170). The training computing system (150) may be separate from the server computing system (130) or may be part of the server computing system (130).

[0047] And at this time, the artificial intelligence model can be 1) trained directly locally by a user computing device (110), 2) trained by the server computing system (130) and the user computing device (110) interacting with each other through a network (170), and 3) trained by a separate training computing system (150) using various training and learning techniques. It may also be implemented by transmitting the artificial intelligence model trained by the training computing system (150) to the user computing device (110) and / or the server computing system (130) through the network (170) to provide / update it.

[0048] In some embodiments, the training computing system (150) may be part of the server computing system (130) or part of the user computing device (110).

[0049] - User Computing Device (110: User Computing Device)

[0050] The user computing device (110) may include all other types of computing devices, such as a smartphone, a mobile phone, a digital broadcasting device, a PDA (personal digital assistants), a PMP (portable multimedia player), a desktop, a wearable device, an embedded computing device and / or a tablet PC.

[0051] Additionally, in the embodiment, the user computing device (110) may further include a predetermined server computing device that provides a service environment for detecting suspected defective areas.

[0052] This user computing device (110) includes at least one processor (111) and memory (112).

[0053] Here, the processor (111) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, or a plurality of electrically connected processors.

[0054] The memory (112) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof, and may include web storage of a server that performs memory storage functions on the internet. This memory (112) may store data (113) and instructions (114) necessary for the at least one processor (111) to perform functional operations, such as training an artificial intelligence model or executing a process to detect suspected defective areas through an artificial intelligence model.

[0055] In one embodiment, the user computing device (110) can perform various deep learnings for a defect suspected area detection service by linking with a deep-learning neural network.

[0056] Here, the deep learning neural network according to the embodiment may include a Convolutional Neural Network (CNN), R-CNN (Regions with CNN features), Fast R-CNN, Faster R-CNN, Mask R-CNN, etc., and may include any deep learning neural network that includes an algorithm capable of performing the embodiments described below, and the embodiments of the present disclosure do not limit or restrict such deep learning neural networks themselves.

[0057] At this time, according to the embodiment, the deep learning neural network may be directly installed in the server computing system (130) or may operate as a separate device from the server computing system (130) to perform deep learning for the defect suspected area detection service.

[0058] Additionally, in one embodiment, the user computing device (110) can store at least one machine learning model (120).

[0059] For example, the user computing device (110) may be various machine learning models, such as multiple neural networks (e.g., deep neural networks) that perform a method for detecting suspected defect areas based on structured / quantitative data, or other types of machine learning models including non-linear models and / or linear models, and may be composed of a combination thereof.

[0060] For example, machine learning models may include linear regression, decision trees, random forests, gradient boosting pre-trained language models or / and deep learning models. And neural networks may include at least one of feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or / and other forms of neural networks.

[0061] Additionally, according to an embodiment, the user computing device (110) may store a model to be used in each process and a prompt template that serves as the basis for input to the model in order to perform at least part of the process for the method of detecting a suspected defective area through a large-scale language model (LLM).

[0062] In one embodiment, a user computing device (110) receives at least one machine learning model (120) from a server computing system (130) through a network (170), stores it in memory (112), and then executes the stored machine learning model (120) by a processor (111) to perform a process such as detecting a suspected defective area.

[0063] In another embodiment, the user computing device (110) may provide a defective area detection service to the user by performing an operation through a machine learning model (140) including at least one machine learning model (140) in conjunction with a server computing system (130) and communicating related data to the outside.

[0064] For example, a user computing device (110) can perform a suspected defect area detection service by having a server computing system (130) provide an output for the user's input using a machine learning model (140) via the web.

[0065] Additionally, the artificial intelligence model can be implemented in such a way that at least some of the machine learning models (120 and / or 140) are executed on a user computing device (110) and the rest are executed on a server computing system (130).

[0066] Additionally, the user computing device (110) may include at least one input component (121) that detects user input.

[0067] For example, the user input component (121) may include a touch sensor (e.g., a touch screen and / or a touch pad, etc.) that detects a touch of the user's input medium (e.g., a finger or a stylus), an image sensor that detects the user's motion input, a microphone that detects the user's voice input, a button, a mouse and / or a keyboard, etc.

[0068] Here, the image sensor may include an image processing module. Specifically, the image sensor may process still images or video obtained by an image sensor device (e.g., CMOS or CCD).

[0069] In addition, the image sensor can process a still image or video acquired through the image sensor device using an image recognition process (e.g., OCR, etc.) and / or an image processing module to extract necessary information and transmit the extracted information to a processor.

[0070] Additionally, the input component (121) can receive input from an external controller (e.g., mouse, keyboard, etc.) based on an interface module, and in this case, may include an external output device (e.g., speaker).

[0071] At this time, the interface module may be configured to include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, an earphone port, a power amplifier, an RF circuit, a transceiver, and other communication circuits.

[0072] In addition, the external output device may include a display system that outputs various information related to a suspected defect area detection service as a graphic image.

[0073] Such a display system may be implemented by including at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0074] Meanwhile, the user computing device (110) including the above-described components may further perform at least some of the functional operations performed by the server computing system (130) described later.

[0075] -Server Computing System (130: Server Computing System)

[0076] The server computing system (130) can perform a series of processes to provide a service for detecting suspected bad areas.

[0077] In detail, in an embodiment, the server computing system (130) can provide the suspected defect area detection service by exchanging data necessary to enable the suspected defect area detection service process to be executed on an external device, such as a user computing device (110).

[0078] More specifically, in an embodiment, the server computing system (130) can provide an environment in which an application can run on a user computing device (110).

[0079] To this end, the server computing system (130) may include an application program, data and / or instructions, etc. for the application to operate, and may transmit and receive various data based thereon with the external device.

[0080] Additionally, the server computing system (130) includes at least one processor (131) and memory (132). Here, the processor (131) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions.

[0081] And the memory (132) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc. and combinations thereof. This memory (132) may store data (133) and instructions (134) necessary for the processor (131) to perform functional operations, such as training an artificial intelligence model or executing a process to detect suspected defective areas through the artificial intelligence model.

[0082] In one embodiment, the server computing system (130) may be implemented to include at least one computing device. For example, the server computing system (130) may be implemented to operate a plurality of computing devices according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. Additionally, the server computing system (130) may include a plurality of computing devices connected to a network (170).

[0083] Additionally, the server computing system (130) may store at least one machine learning model (140). For example, the server computing system (130) may include a neural network and / or other multi-layer non-linear model as the machine learning model (140). Exemplary neural networks may include a feed-forward neural network, a deep neural network, a recurrent neural network, and a convolutional neural network.

[0084] In an embodiment, the server computing system (130) may further include a data store computing system (hereinafter, data store) which is a storage for continuously storing and managing raw data that forms the basis of a suspected bad area detection service.

[0085] Such data stores may include various forms of data storage, ranging from file systems to cloud storage. For example, a data store may include at least one database among a relational database that uses a structured query language (SQL) to define and manipulate data, a NoSQL database designed for flexibility and scalability to process unstructured and semi-structured data, a data warehouse optimized for querying and analysis by centralizing large volumes of data from multiple sources as a system used for reporting and data analysis, a data warehouse that stores large volumes of raw data in basic formats such as structured data, semi-structured data, and unstructured data, and a local storage device or Network Attached Storage (NAS) that stores data in files in a format generally accessible by a computer operating system.

[0086] - Training Computing System (150: Training Computing System)

[0087] The training computing system (150) includes at least one processor (151) and memory (152). Here, the processor (151) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, or a plurality of electrically connected processors.

[0088] And the memory (152) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc. and combinations thereof. This memory (152) may store data (153) and instructions (154) necessary for the processor (151) to perform learning of an artificial intelligence model, etc.

[0089] For example, the training computing system (150) may include a model trainer (160) that trains a machine learning model (120 and / or 140) stored in a user computing device (110) and / or a server computing system (130) using various training or learning techniques, such as back propagation of error (according to the framework illustrated in FIG. 3).

[0090] For example, such a model trainer (160) can perform updates to one or more parameters of a machine learning model (120 and / or 140) for a defective area detection service in a backpropagation manner based on a defined loss function.

[0091] In some embodiments, performing backpropagation of the error may include performing truncated backpropagation through time. The model trainer (160) may perform a number of generalization techniques (e.g., weight devaluation, dropout and / or knowledge distillation, etc.) to improve the generalization ability of the machine learning model (120 and / or 140) being trained.

[0092] Additionally, the model trainer (160) can train a machine learning model (120 and / or 140) based on a series of training data (161). Here, the training data (161) may include data of different forms, such as images, audio samples and / or text, for example. Examples of image types that may be used may include video frames, LiDAR point clouds, X-ray images, computed tomography scans, hyperspectral images and / or various other forms of images.

[0093] These training data (161) may be provided by a user computing device (110) and / or a server computing system (130). When the training computing device trains a machine learning model (120 and / or 140) on specific data of the user computing device (110), the machine learning model (120 and / or 140) may be characterized as a personalized model.

[0094] And the model trainer (160) includes computer logic that is utilized to provide the desired function.

[0095] Additionally, the model trainer (160) may be implemented as hardware, firmware, and / or software that controls a general-purpose processor. In one embodiment, the model trainer (160) may include a program file stored in a storage device, be loaded into memory (152), and be executed by one or more processors (151). In another embodiment, the model trainer (160) includes one or more sets of computer-executable data (153) and instructions (154) stored in a tangible computer-readable storage medium, such as a RAM hard disk or an optical or magnetic medium.

[0096] Network (170) includes, but is not limited to, 3GPP (3rd Generation Partnership Project) network, LTE (Long Term Evolution) network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth network, satellite broadcasting network, analog broadcasting network and / or DMB (Digital Multimedia Broadcasting) network.

[0097] Generally, communication through the network (170) can be performed using any type of wired and / or wireless connection through various communication protocols (e.g., TCP / IP, HTTP, SMTP and / or FTP, etc.), encodings or formats (e.g., HTML and / or XML, etc.), and / or protection schemes (e.g., VPN, Secure HTTP and / or SSL, etc.).

[0098] FIG. 2 illustrates an example of a block diagram of a computing device implementing a suspected defect area detection service according to one embodiment of the present disclosure.

[0099] Including FIG. 2, the computing device (100) included in the user computing device (110), server computing system (130), and training computing system (150) includes a plurality of applications (e.g., Application 1 to Application N). Each application may include a machine learning library and one or more machine learning models. For example, the applications may include an image processing application (e.g., Detection, Classification and / or Segmentation, etc.), a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application and / or a chat-bot application, etc.

[0100] In an embodiment, the computing device (100) may include a model trainer (160) for training an artificial intelligence model, and by storing and operating the trained artificial intelligence model, it may provide output data according to a predetermined input data (e.g., an inspection target image).

[0101] Each application of the computing device (100) can communicate with a number of other components of the computing device (100), such as, for example, at least one sensor, a context manager, a device state component, and / or additional components. In one embodiment, each application can communicate with each device component using an API (e.g., a public API). In one embodiment, the API used by each application may be specific to that application.

[0102] FIG. 3 illustrates an example of a block diagram in another aspect of a computing device implementing a suspected defect area detection service according to one embodiment of the present disclosure.

[0103] Referring to FIG. 3, the computing device (200) includes a plurality of applications (e.g., Application 1 to Application N). Each application can communicate with a central intelligence layer. For example, applications may include an image processing application, a text messaging application, an email application, a dictation application, a virtual keyboard application and / or a browser application. In one embodiment, each application can communicate with the central intelligence layer (and a model stored therein) using an API (e.g., a common API across all applications).

[0104] The central intelligence layer may include a number of machine learning models. For example, as illustrated in FIG. 3, at least some of the machine learning models may be provided for each application and managed by the central intelligence layer. In another embodiment, two or more applications may share a single machine learning model. For example, in some embodiments, the central intelligence layer may provide a single model for all applications. In some embodiments, the central intelligence layer may be included within the operating system of the computing device (200) or otherwise implemented.

[0105] The central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized data store for the computing device (200). As illustrated in FIG. 3, the central device data layer can communicate with a number of other components of the computing device (200), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0106] The technology described herein may refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to or from said systems. It will be recognized that the inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, division of tasks, and functionality between and from components. For example, the processes described herein may be implemented using a single device or component or multiple devices or components operating in combination. Databases and applications may be implemented in a single system or in a distributed system across multiple systems. Distributed components may operate sequentially or in parallel.

[0107]

[0108] [Method for Providing Service for Detecting Suspected Defective Areas]

[0109] Hereinafter, a method for implementing a suspected defect area detection service, in which a computing system (1000) according to one embodiment of the present disclosure detects suspected defect areas through a deep learning-based pre-trained model without separate training using a small number of good product images, will be described in detail with reference to the attached drawings.

[0110] FIG. 4 illustrates a flowchart for explaining a method for detecting a suspected defective area according to one embodiment of the present disclosure.

[0111] Referring to FIG. 4, a method for providing a service for detecting suspected defect areas according to one embodiment of the present disclosure may include: a step of collecting a predetermined good product image (S101); a step of acquiring a predetermined inspection target image (S103); a step of aligning the inspection target image based on the good product image (S105); a step of dividing the aligned inspection target image into patches (S107); a step of acquiring a multi-scale feature based on the divided patches (S109); a step of estimating the probability of a good product per patch image based on the acquired multi-scale feature (S111); a step of determining whether there is a suspected defect in the inspection target image based on the estimated probability of a good product (S113); and a step of providing a result of determining whether there is a suspected defect (S115).

[0112] In detail, a computing system (1000) according to one embodiment of the present disclosure can collect a predetermined good product image. (S101)

[0113] Here, the good product image according to the embodiment may mean an image of a product that is of excellent quality and free of defects.

[0114] In detail, in the embodiment, the computing system (1000) can acquire and collect N (N>1) images of good products by linking with a predetermined sensor system and / or an external device, etc.

[0115] In addition, a computing system (1000) according to one embodiment of the present disclosure can acquire a predetermined inspection target image. (S103)

[0116] Here, the inspection target image according to the embodiment may refer to an image taken of a target (product) for which the presence or absence of defects is to be determined.

[0117] In detail, in an embodiment, the computing system (1000) can acquire the inspection target image described above by linking with a predetermined sensor system (e.g., a digital camera, etc.) and / or an external device (e.g., vision inspection equipment, etc.).

[0118] In addition, a computing system (1000) according to one embodiment of the present disclosure can align inspection target images based on good product images. (S105)

[0119] FIG. 5 illustrates a conceptual diagram for explaining a method of aligning inspection target images according to one embodiment of the present disclosure.

[0120] Referring specifically to FIG. 5, in an embodiment, the computing system (1000) can select one of N good product images as a good product representative image (GRI) according to a predetermined method (e.g., user input and / or a pre-established image selection algorithm, etc.).

[0121] In this embodiment, the computing system (1000) can store a selected representative image (GRI) of a good product in a non-volatile memory and use it for image alignment described later.

[0122] In addition, in the embodiment, the computing system (1000) can align the position of the inspection target image (ITI) based on the good product representative image (GRI).

[0123] This may be intended to prevent performance degradation caused by positional errors that occur when the target product cannot always be photographed from the same position when capturing the inspection target image (ITI).

[0124] In an example, the computing system (1000) can calculate a homography matrix based on a representative image of a good product (GRI) and an inspection target image (ITI).

[0125] For reference, homography is a matrix that represents a geometric transformation between two images in a two-dimensional plane, and can be a matrix capable of transforming feature points of one image into corresponding points of another image. Through this, even if two images are taken at different viewpoints, they can be aligned so that they exist on the same plane.

[0126] In detail, in an embodiment, the computing system (1000) can extract feature points (Feature Detection) of each of the good product representative image (GRI) and the inspection target image (ITI).

[0127] More specifically, the computing system (1000) can extract feature points of a good product representative image (GRI) (hereinafter referred to as good product feature points) and feature points of an inspection target image (ITI) (hereinafter referred to as target feature points) using a predetermined feature point extraction algorithm (e.g., algorithms such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), FAST9, SuperPoint, Munkres and / or LightGlue).

[0128] In addition, in the embodiment, the computing system (1000) can perform feature matching between extracted feature points.

[0129] In detail, the computing system (1000) can detect mutually corresponding feature points between a good product feature point and a target feature point using a predetermined feature point matching algorithm (e.g., Brute-Force Matching and / or FLANN (Fast Library for Approximate Nearest Neighbors), etc.).

[0130] And the computing system (1000) can match the detected feature points to form a pair.

[0131] That is, the computing system (1000) can acquire multiple feature point pairs (hereinafter, corresponding point pairs) that mutually match feature points at the closest positions between a good product feature point and a target feature point.

[0132] At this time, according to the embodiment, the computing system (1000) can remove a pair of feature points (hereinafter referred to as an outlier pair) having a matching rate of less than or equal to a predetermined threshold among a plurality of feature point pairs by using a predetermined outlier removal algorithm (e.g., RANSAC (Random Sample Consensus), etc.).

[0133] For reference, RANSAC may be a homography matrix estimation method that iteratively calculates homography using a set of feature point pairs and has the minimum error for all feature point pairs.

[0134] That is, the computing system (1000) can perform filtering to remove noise components within a plurality of feature point pairs using an outlier removal algorithm such as RANSAC.

[0135] In addition, in the embodiment, the computing system (1000) can calculate a homography matrix (Homography Matrix Estimation) based on the corresponding point pairs obtained as above.

[0136] Specifically, the computing system (1000) can calculate a homography matrix according to a predetermined pair of corresponding points according to the following [Equation 1].

[0137] [Mathematical Formula 1]

[0138]

[0139] That is, the computing system (1000) can convert the first feature point coordinates (x, y) into new feature point coordinates (x', y') through a homography matrix (H) in the form of a 3*3 transformation matrix according to a predetermined pair of corresponding points.

[0140] Thus, the computing system (1000) can estimate a homography matrix (H) based on a plurality of corresponding point pairs based on a good product representative image (GRI) and an inspection target image (ITI).

[0141] Additionally, in the embodiment, the computing system (1000) can perform image warping based on the calculated homography matrix (H).

[0142] Specifically, the computing system (1000) can use the calculated homography matrix (H) to convert the coordinate system of the inspection target image (ITI) to match the coordinate system of the good product representative image (GRI).

[0143] Through this, in the embodiment, the computing system (1000) can align the shooting time (location) of the inspection target image (ITI) with the shooting time (location) of the good product representative image (GRI).

[0144] Thus, the computing system (1000) can divide the aligned inspection target image (AII: hereinafter, aligned inspection image) based on the good product representative image (GRI).

[0145] Therefore, the computing system (1000) can effectively prevent performance degradation caused by position errors that occur when the target product cannot always be photographed at the same location when the inspection target image (ITI) is captured (e.g., when a change in position is induced with every movement by performing product movement automation using a machine).

[0146] In addition, a computing system (1000) according to one embodiment of the present disclosure can divide an aligned inspection target image (ITI) into patches. (S107)

[0147] Generally, since images used in vision inspection are captured at high resolution to detect even minute defects, processing the entire image area at once can be inefficient in terms of processing time and resource utilization.

[0148] Thus, in the embodiment, the computing system (1000) intends to process a given image (in the embodiment, a good product image and / or an inspection target image (ITI), etc.) by dividing it into predetermined patch units.

[0149] FIG. 6 illustrates examples for explaining a patch-unit image segmentation method according to one embodiment of the present disclosure.

[0150] Referring specifically to FIG. 6, in the embodiment, the computing system (1000) can divide the inspection target image aligned as above (i.e., the alignment inspection image (AII)) into patches of a preset size.

[0151] In this case, as an example, the computing system (1000) can divide the image into a plurality of patches in a form where there are no mutually overlapping regions, as in (a) of FIG. 6.

[0152] In another embodiment, the computing system (1000) can divide the image into multiple patches having an overlapping area of ​​a preset size of m (m > 0), as shown in FIG. 6 (b). In this case, the computing system (1000) can perform computational processing to minimize errors that may occur at the edges between the multiple patches.

[0153] Thus, in the embodiment, the computing system (1000) can divide the alignment inspection image (AII) into k (k>1) patches as in (c) of FIG. 6.

[0154] Additionally, a computing system (1000) according to one embodiment of the present disclosure can acquire a multi-scale feature based on a divided patch. (S109)

[0155] Here, a multi-scale feature according to an embodiment may refer to features (feature vectors) extracted from an image at various sizes and / or resolutions.

[0156] That is, in the embodiment, the computing system (1000) can implement higher quality data processing that is robust to scale changes by comprehensively analyzing features extracted at various sizes and / or resolutions based on divided patches.

[0157] Referring further to FIG. 6, in the embodiment, the computing system (1000) can obtain a plurality of patch images (hereinafter, a plurality of inspection patch images) by dividing an alignment inspection image (AII) into patch units.

[0158] FIG. 7 illustrates a conceptual diagram for explaining a method of extracting image features through a pre-trained model according to one embodiment of the present disclosure, and FIG. 8 illustrates a conceptual diagram for explaining a method of acquiring multi-scale features according to one embodiment of the present disclosure.

[0159] Additionally, referring to FIGS. 7 and 8, in the embodiment, the computing system (1000) can acquire multi-scale features based on a plurality of inspection patch images (IPI) by linking with a model (PTM: hereinafter, pre-trained model) trained through various methodologies such as deep learning and / or machine learning.

[0160] Here, the pre-trained model (PTM) according to the embodiment may include a deep learning model that has been pre-trained with a large dataset such as ImageNet, or a foundation model that has been trained on hundreds of millions of images through self-supervised representation learning. However, these are merely examples, and any deep learning model capable of extracting features of each patch image may be included.

[0161] In another aspect, the pre-trained model (PTM) according to the embodiment may be a deep learning model including algorithms such as a Convolutional Neural Network (CNN) and / or a Transformer.

[0162] In this way, by utilizing a pre-trained model (PTM), the computing system (1000) can easily perform the image feature extraction process required for the defect suspected area detection service without separate training.

[0163] Returning to FIG. 8, in the embodiment, the computing system (1000) can acquire multi-scale features of various sizes and / or resolutions in units of a predetermined inspection patch image (IPI).

[0164] For example, a computing system (1000) can acquire multi-scale features according to a first size corresponding to the size of one inspection patch image (IPI), a second size corresponding to the size of four inspection patch images (IPI), and a third size corresponding to the size of nine inspection patch images (IPI).

[0165] At this time, according to the embodiment, the computing system (1000) can divide a predetermined inspection patch image (IPI) into predetermined sub-patch units.

[0166] In the embodiment, the specific method for dividing the inspection patch image (IPI) into a predetermined sub-patch unit is omitted by applying the description of the method for dividing the inspection target image (ITI) into patch units in the aforementioned step S107.

[0167] In addition, in the embodiment, the computing system (1000) can perform multi-scale feature extraction using the aforementioned prior learning model (PTM) based on a plurality of patch images (hereinafter, a plurality of sub-patch images) obtained by performing secondary partitioning.

[0168] That is, the computing system (1000) can divide the given patch image itself into sections to consider more detailed features and obtain multi-scale features based thereon.

[0169] In this way, in the embodiment, the computing system (1000) can extract features based on various scales rather than simply extracting features based on one size from a plurality of patch images.

[0170] Thus, the computing system (1000) can extract more detailed features from the image, ranging from complex patterns in a wide area to fine patterns in a small area.

[0171] In addition, a computing system (1000) according to one embodiment of the present disclosure can estimate the probability of a good product per patch image according to the acquired multi-scale feature. (S111)

[0172] Here, the probability of a good product according to the example may refer to a value obtained by quantifying, through Gaussian density estimation, how similar each patch of the inspection target image (ITI) is to the features of the good product image.

[0173] That is, in the embodiment, the computing system (1000) can calculate the probability of how similar the features of the alignment inspection image (AII) are to the features of the good product image by using the multi-scale features obtained in step S109 (i.e., feature vectors based on patches of the alignment inspection image (AII)).

[0174] In detail, in the embodiment, the computing system (1000) can calculate the probability of a good product (p(x)) of each patch image according to a Gaussian density estimation formula such as [Equation 2] below.

[0175] [Mathematical Formula 2]

[0176]

[0177] Here, 'x' in [Equation 2] represents the feature vector of the corresponding patch, 'μ' represents the average of the feature vectors extracted from the good product image, 'Σ' represents the covariance matrix of the good product image feature vectors, and 'd' represents the dimension of the feature vector.

[0178] More specifically, in an embodiment, the computing system (1000) can obtain the average (μ) of feature vectors extracted from N (N>1) good product images according to the following [Equation 3].

[0179] [Mathematical Formula 3]

[0180]

[0181] Here, 'N' in [Equation 3] represents the number of good product images used, and 'X + ' can mean a good quality image.

[0182] In addition, in the embodiment, the computing system (1000) can obtain the covariance (Σ) of feature vectors extracted from N (N>1) good product images according to the following [Equation 4].

[0183] [Mathematical Formula 4]

[0184]

[0185] Specifically, in the embodiment, the computing system (1000) can acquire multi-scale features for each good product image by applying the process according to steps S105 to S109 described above based on each good product image.

[0186] And the computing system (1000) can obtain the average (μ) described above based on the multi-scale features per acquired good product image and [Equation 3].

[0187] In addition, the computing system (1000) can obtain the covariance (Σ) described above based on the multi-scale features per acquired good product image and [Equation 4].

[0188] In addition, in the embodiment, the computing system (1000) can apply the mean (μ) and covariance (Σ) obtained as above to the [Equation 2] described above to calculate the probability of a good product (p(x)) for each patch image (i.e., each patch image of the alignment inspection image (AII)) based on Gaussian density estimation.

[0189] Thus, the computing system (1000) can estimate how similar the features of each patch of the inspection target image (ITI) (here, alignment inspection image (AII)) are to the average features of N good product images.

[0190] In addition, a computing system (1000) according to one embodiment of the present disclosure can determine whether there is a suspected defect in an inspection target image (ITI) based on an estimated probability of a good product (p(x)). (S113)

[0191] In detail, in an embodiment, the computing system (1000) can compare the probability of a good product (p(x)) for each patch image with a predetermined threshold (hereinafter, classification threshold).

[0192] And the computing system (1000) can determine each patch image as a good patch image or a patch image suspected of being defective based on the comparison result.

[0193] More specifically, as an example, the computing system (1000) can compare the probability of a good product (p(x)) for each patch image with a preset first classification threshold.

[0194] At this time, the computing system (1000) can determine that a patch image corresponding to a predetermined good product probability (p(x)) is a good product patch image if the predetermined good product probability (p(x)) is greater than or equal to a first classification threshold.

[0195] On the other hand, the computing system (1000) can determine that a patch image corresponding to a predetermined probability of a good product (p(x)) is a patch image suspected of being defective if the probability of a good product (p(x)) is less than a predetermined first classification threshold.

[0196] That is, in the embodiment, the computing system (1000) can determine whether a product is good or suspected of being defective on a patch-by-patch basis of an image.

[0197] Therefore, the computing system (1000) can detect areas suspected of defects more quickly and efficiently than processing the entire image at once, and can improve accuracy by analyzing each area within the image individually to detect specific areas where defects have occurred more precisely.

[0198] At this time, according to the embodiment, the computing system (1000) can obtain a probability of good quality (p(x)) for each of a plurality of sub-patch images corresponding to a predetermined first patch image by applying the description of step S111 described above.

[0199] And the computing system (1000) can detect a good product probability (p(x)) (hereinafter, minimum good product probability value) having a minimum value among the multiple good product probabilities (p(x)) for each acquired sub-patch image.

[0200] Additionally, the computing system (1000) can compare the detected minimum probability value of a good product with a preset first classification threshold.

[0201] At this time, the computing system (1000) can determine that a first patch image corresponding to a minimum probability value of a good product is a good product patch image if the minimum probability value of a good product is greater than or equal to a first classification threshold that has been set.

[0202] On the other hand, the computing system (1000) can determine that a first patch image corresponding to a minimum probability value of a good product is a patch image suspected of being defective if the minimum probability value of a good product is less than a preset first classification threshold.

[0203] That is, according to the embodiment, the computing system (1000) can further improve the accuracy by determining whether each patch image is a good product or suspected of being defective using the probability of a good product (p(x)) based on multi-scale features analyzed in more detail.

[0204] Meanwhile, according to an embodiment, the computing system (1000) can determine each patch image as a good patch image or a patch image suspected of being defective based on a multi-stage classification based on a plurality of preset thresholds (hereinafter, a plurality of classification thresholds).

[0205] FIG. 9 illustrates a flowchart for explaining a multi-stage classification method for patch images according to a probability of quality (p(x)) of quality according to one embodiment of the present disclosure.

[0206] Referring specifically to FIG. 9, in the embodiment, the computing system (1000) can compare the probability of a good product (p(x)) for each patch image with a preset first distinction threshold. (S201)

[0207] Here, the first distinction threshold according to the embodiment can correspond to the minimum probability condition for being classified as a good product patch image.

[0208] Additionally, in the embodiment, the computing system (1000) can determine that a patch image corresponding to the probability of a good product (p(x)) is a good product patch image if the probability of a good product (p(x)) is greater than or equal to a preset first distinction threshold. (S203)

[0209] On the other hand, the computing system (1000) can set a patch image corresponding to a predetermined probability of a good product (p(x)) as a suspected defect candidate patch image if the probability of a good product (p(x)) is less than a predetermined first distinction threshold.

[0210] Additionally, in the embodiment, the computing system (1000) can compare the probability of a good product (p(x)) of a set suspected defect candidate patch image (probability of a good product (p(x)) less than a set first distinction threshold, hereinafter referred to as the probability of a suspected defect candidate) with a set second distinction threshold. (S205)

[0211] Here, the second distinction threshold according to the embodiment can correspond to the maximum probability condition for being classified as a patch image suspected of being defective.

[0212] Additionally, in the embodiment, the computing system (1000) can determine a patch image corresponding to a probability of a suspected defect candidate as a suspected defect patch image if the probability of a suspected defect candidate is less than or equal to a preset second distinction threshold. (S207)

[0213] FIG. 10 illustrates an example of a drawing for explaining a method for obtaining a minimum probability area image of a good product according to one embodiment of the present disclosure.

[0214] On the other hand, referring to FIG. 10, in the embodiment, the computing system (1000) can obtain a probability of a defective candidate (p(x)) for a plurality of sub-patch images corresponding to the defective candidate probability when the probability of a defective candidate exceeds a preset second distinction threshold (i.e., when a predetermined probability of a good product (p(x)) is less than the first distinction threshold and exceeds the second distinction threshold) by applying the description of step S111 described above.

[0215] And the computing system (1000) can detect a good product probability (p(x)) having a minimum value among the multiple good product probabilities (p(x)) for each sub-patch image obtained (i.e., minimum good product probability value).

[0216] In addition, in the embodiment, the computing system (1000) can extract an area having a predetermined size (H(Height)*W(Width)) centered on an area matching the detected minimum probability value of a good product (hereinafter, the minimum probability point of a good product).

[0217] At this time, in the embodiment, the computing system (1000) can extract an area having a predetermined size centered on the minimum probability point of a good product (in the embodiment, a pixel having a minimum probability value of a good product) when there is only one minimum probability point of a good product.

[0218] On the other hand, referring further to FIG. 10, the computing system (1000) can calculate the center of gravity for the location of the corresponding multiple minimum probability points of good products when there are multiple minimum probability points of good products.

[0219] In an embodiment of the present disclosure, a specific method for a computing system (1000) to calculate the center of gravity described above may be performed by utilizing various disclosed algorithms that implement the same, and the embodiments of the present disclosure do not limit or restrict the algorithms themselves.

[0220] In addition, in the embodiment, the computing system (1000) can extract an area having a predetermined size centered on an area corresponding to the calculated center of gravity (hereinafter, center of gravity point).

[0221] That is, the computing system (1000) can extract a pixel having the lowest probability value among the multiple sub-patch image-specific probability of good product (p(x)) or an area of ​​a predetermined size centered on the centroid of those pixels (hereinafter, minimum probability area of ​​good product).

[0222] Thus, in the embodiment, the computing system (1000) can obtain an image including the extracted minimum probability area of ​​a good product (hereinafter, the minimum probability area of ​​a good product image). (S209)

[0223] Additionally, in the embodiment, the computing system (1000) can perform a determination of good product or suspected defect for each patch image of the good product minimum probability zone image obtained as above by linking with a model learned through various methodologies such as a predetermined deep learning and / or machine learning (i.e., a pre-trained model (PTM)). (S211)

[0224] In other words, the computing system (1000) can determine each patch image of the good product minimum probability area image as a good product patch image or a patch image suspected of being defective by linking with a predetermined prior learning model (PTM).

[0225] Here, the pre-trained model (PTM) according to the embodiment may include a deep learning classification model trained through a relevant dataset, or may include a model that extracts feature values ​​within an image based on a predetermined image processing methodology and then implements a machine learning methodology such as Random Forest, and various other embodiments are possible.

[0226] In this way, in the embodiment, the computing system (1000) can immediately determine patch samples with a probability of being good above a predetermined threshold as good, and immediately classify patch samples with a probability of being good below a predetermined threshold as suspected defective patches, thereby shortening the overall processing time.

[0227] Additionally, the computing system (1000) can determine whether there is an additional suspicion of defect by cutting out an image centered on the area with the lowest probability of being a good product (i.e., the area with the highest probability of a defect) for patch samples in an intermediate state where it is ambiguous whether they are good or suspected of being defective, and performing a re-inspection process based thereon.

[0228] Through this multi-stage judgment method, the computing system (1000) can significantly improve the efficiency and accuracy of detecting suspected defect areas by minimizing misjudgments that may occur in uncertain areas while maintaining the processing speed in the inspection process.

[0229] Returning to Fig. 4, the computing system (1000) that determines each patch image as a good patch image or a patch image suspected of being defective can determine whether there is a suspected defect in the corresponding inspection target image (ITI) according to a predetermined method.

[0230] In an example, the computing system (1000) can determine whether there is a suspected defect in the inspection target image (ITI) based on the number of good patch images (hereinafter, number of good patches) and the number of patch images suspected of being defective (hereinafter, number of patches suspected of being defective).

[0231] For example, a computing system (1000) can calculate the ratio between the number of good patches and the number of suspected defective patches, and detect the side with the higher ratio.

[0232] At this time, the computing system (1000) can determine that the inspection target image (ITI) is a good product image if the ratio of the number of good product patches is higher.

[0233] On the other hand, the computing system (1000) can determine that the inspection target image (ITI) is a suspected defective image if the ratio of the number of suspected defective patches is higher.

[0234] In another example, the computing system (1000) may determine that the inspection target image (ITI) is a good image if the number of good patches is greater than or equal to a predetermined threshold, or determine that the inspection target image (ITI) is a defective image if the number of suspected defective patches is greater than or equal to a predetermined threshold.

[0235] As described above, the computing system (1000) according to the embodiment of the present disclosure can efficiently detect suspected defective areas within an inspection target image (ITI) by estimating a patch-level good product probability (p(x)) using a small number of good product images without separate prior learning.

[0236] Through this, the computing system (1000) can quickly and accurately detect areas suspected of having defects through advanced analysis using a deep learning model.

[0237] In particular, the computing system (1000) can reduce the burden of data collection for model training by using a pre-trained model, and at the same time, flexibly adapt to various product and / or inspection environment changes without undergoing a separate training process.

[0238] In addition, the computing system (1000) can reduce the time and resources required for vision inspection compared to a method of processing the entire image area in batches by dividing and processing the image at the patch level.

[0239] Therefore, the computing system (1000) can implement a defect suspected area detection service that maintains high accuracy while maximizing the speed and efficiency of the vision inspection process.

[0240] In addition, a computing system (1000) according to one embodiment of the present disclosure may provide a result of determining whether there is a suspected defect. (S115)

[0241] That is, in the embodiment, the computing system (1000) can provide a result of determining whether there is a suspected defect in the inspection target image (ITI) (i.e., a result of determining whether it is a good product image or a suspected defect image) according to a predetermined method.

[0242] In an example, the computing system (1000) may provide a result of suspected defect determined according to an embodiment of the present disclosure by linking with a predetermined application service (e.g., a component inspection service, a semiconductor wafer inspection service, a food packaging inspection service and / or a smartphone manufacturing inspection service, etc.).

[0243] Above, the method and system for detecting suspected defect areas according to one embodiment of the present disclosure have the effect of reducing the burden of data collection for model training and flexibly responding to changes in the inspection environment by detecting suspected defect areas through a deep learning-based pre-trained model (PTM) without separate training using a small number of good product images.

[0244] In addition, the method and system for detecting suspected defective areas according to one embodiment of the present disclosure have the effect of reducing the time and resources required for inspection compared to a method of processing the entire image area in batches by dividing and processing the image at the patch level.

[0245]

[0246] Meanwhile, the embodiments according to the present disclosure described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present disclosure or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. Hardware devices may be modified into one or more software modules to perform processing according to the present disclosure, and vice versa.

[0247] The specific embodiments described in this disclosure are examples and do not limit the scope of this disclosure in any way. For the sake of brevity of the specification, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted. Additionally, the connections of lines or connecting members between components shown in the drawings are illustrative of functional connections and / or physical or circuit connections, and may be replaced or additionally represented as various functional connections, physical connections, or circuit connections in actual devices. Furthermore, unless specifically stated as “essential,” “importantly,” etc., a component may not be strictly necessary for the application of this disclosure.

[0248] Furthermore, although the detailed description of the present disclosure has been explained with reference to preferred embodiments of the present disclosure, those skilled in the art or those with ordinary knowledge in the art will understand that the present disclosure can be modified and changed in various ways without departing from the spirit and technical scope of the present disclosure as set forth in the claims below. Accordingly, the technical scope of the present disclosure should not be limited to the contents described in the detailed description of the specification but should be determined by the claims.

[0249]

[0250] The present disclosure relates to a method and system for detecting suspected defect areas and is applicable to the artificial intelligence industry, thus possessing industrial applicability.

Claims

1. A method for a computing system including memory and a processor to detect a suspected faulty area, wherein A step of acquiring N (N>1) images of good products; A step of acquiring an inspection target image, which is an image of an object to be judged for defects; A step of aligning the inspection target image based on the above good product image; A step of dividing the above-mentioned aligned inspection target image into patches; A step of obtaining a multi-scale feature based on the segmented patch using a predetermined pre-training model; A step of estimating the probability of a good product per patch based on the multi-scale features obtained above; A step of determining whether there is suspicion of defect in the inspection target image based on the above-mentioned probability of a good product per patch; and A step comprising providing the result of the suspected defect determined above. Method for detecting suspected defective areas.

2. In Paragraph 1, The above multi-scale feature is, Data extracted from feature vectors of a given image based on various sizes or resolutions Method for detecting suspected defective areas.

3. In Paragraph 2, The step of acquiring the above multi-scale feature is, A step comprising forming the size in the patch units and acquiring the multi-scale feature Method for detecting suspected defective areas.

4. In Paragraph 1, The step of estimating the probability of good products per patch above is, A step comprising calculating the similarity between the feature vector of each of the patches and the feature vector of the good product image based on Gaussian density estimation. Method for detecting suspected defective areas.

5. In Paragraph 4, The step of calculating the above similarity is, The step of calculating the mean and covariance of the feature vectors of the above-mentioned good product image, and The step of calculating the similarity based on the calculated mean and covariance and the feature vector of each of the patches Method for detecting suspected defective areas.

6. In Paragraph 1, The step of determining whether there is suspicion of defect in the above-mentioned inspection target image is, A step of comparing each of the above patch-specific good product probabilities with a predetermined first threshold, and A step of determining each of the above patches as a good patch or a patch suspected of being defective based on the above comparison result. Method for detecting suspected defective areas.

7. In Paragraph 1, The step of determining whether there is suspicion of defect in the above-mentioned inspection target image is, A step of comparing each of the above patch-specific good product probabilities with a predetermined second threshold, and A step of determining a patch corresponding to a good product probability greater than or equal to the second threshold as a good product patch, and comparing a good product probability less than the second threshold with a predetermined third threshold, and A step of determining a patch corresponding to a probability of good quality below the third threshold as a patch suspected of being defective. Method for detecting suspected defective areas.

8. In Paragraph 7, The step of determining whether there is suspicion of defect in the above-mentioned inspection target image is, The step of obtaining a plurality of sub-patches based on a patch corresponding to a probability of a good product that is less than the second threshold and greater than the third threshold, and A step of obtaining a minimum probability value of a good product, which is a good product probability having a minimum value among the above plurality of good product probabilities for each sub-patch, and A step of extracting an area having a predetermined size based on an area matching the minimum probability value of a good product obtained above, and A step of further including determining whether there is a suspected defect based on an image including the extracted region. Method for detecting suspected defective areas.

9. In Paragraph 8, The step of determining whether there is suspicion of defect in the above-mentioned inspection target image is, If there are multiple minimum probability values ​​for good products, a step of calculating the center of gravity for the locations of regions matching the multiple minimum probability values ​​for good products, and A step of extracting an area having a predetermined size based on an area corresponding to the center of gravity calculated above, and A step of further including determining whether there is a suspected defect based on an image including the extracted region. Method for detecting suspected defective areas.

10. In either Paragraph 6 or Paragraph 7, The step of determining whether there is suspicion of defect in the above-mentioned inspection target image is, The method further includes a step of determining whether there is a suspected defect based on at least one of the above good patches and above suspected defective patches. Method for detecting suspected defective areas.

11. In Paragraph 1, The step of aligning the inspection target image based on the above good product image is, A step of selecting one of the above N (N>1) good product images as a representative good product image, and A step of extracting feature points (Feature Detection) for each of the above-mentioned representative image of a good product and the above-mentioned inspection target image, and A step of mutually matching corresponding feature points between each of the above feature points (Feature Matching), and A step of calculating a homography matrix (Homography Matrix Estimation) based on the above-mentioned matched feature points, and The method includes the step of performing image warping on the inspection target image based on the calculated homography matrix. Method for detecting suspected defective areas.

12. At least one memory; and It includes at least one processor that reads at least one application stored in the memory and detects a suspected defective area; The instructions of the above processor are, A step of acquiring N (N>1) images of good products, and A step of acquiring an inspection target image, which is an image of an object to be judged for defects, and The step of aligning the inspection target image based on the above good product image, and The step of dividing the above-mentioned aligned inspection target image into patches, and A step of obtaining a multi-scale feature based on the segmented patch using a predetermined pre-training model, and A step of estimating the probability of a good product per patch based on the multi-scale features obtained above, and A step of determining whether there is suspicion of defect in the inspection target image based on the above-mentioned probability of good quality per patch, and A command including a step of providing the result of the suspected defect determined above. Defect Suspected Area Detection System.

Images