Method for monitoring determination record for anomaly determination guide and system therefor

Abstract

A method for monitoring a determination record for an anomaly determination guide, according to an embodiment of the present invention, comprises: providing determination detailed information that is information specifying determination content regarding whether or not there is an anomaly, by detecting at least one similar image having a similarity greater than or equal to a predetermined reference with respect to a predetermined inspection image; acquiring first determination result information that is information specifying the result of determining whether or not there is an anomaly in the inspection image; and then generating the inspection image and the first determination result according a similarity between the inspection image and each of the at least one similar image.

Description

Method and system for monitoring judgment records for outlier judgment guide

[0001] The present invention relates to a method and system for monitoring judgment records for outlier judgment guidance. More specifically, the present invention relates to a method and system for providing guidance for real-time outlier judgment based on past outlier judgment records.

[0002] Vision inspection equipment aims to automatically detect visible defects in products through computer vision technology, including deep learning.

[0003] Based on data obtained through sensor equipment such as cameras, it is necessary to classify good products and defective products with a very high degree of accuracy, and in order to control product quality, the equipment in the inspection process must operate uniformly and without problems.

[0004] However, since vision inspection equipment cannot detect defects with 100% accuracy due to factors such as new types of defects that did not occur during the process of training the classification model or noise generated by process environmental conditions, and because defects must be prevented from leaking, some of the produced products ultimately go through a process of being monitored by workers.

[0005] Workers visually judge based on image data captured by equipment. However, in the case of vision inspection data, there is a problem that there is a lot of data where the boundary between normal products and products containing defects is ambiguous, and in the case of such ambiguous shapes, the accuracy of defect judgment is greatly reduced.

[0006] In particular, in the case of unskilled workers, there is a high possibility of making incorrect judgments, and the phenomenon of inspection results varying depending on the skill level of the inspector can frequently occur, which also causes the problem of not being able to maintain consistency in inspection quality.

[0007] We are trying to reduce the probability of mistakes made by beginners by learning certain rules such as guidebooks, but there is a limitation in that it is impossible to learn about all cases.

[0008] Meanwhile, anomaly detection can refer to a process of identifying abnormal patterns, outliers, and / or exceptional values ​​from given data.

[0009] That is, anomaly detection can be a process of detecting components that deviate from the properties of normal data.

[0010] Systems that implement such anomaly detection are actively used in various application fields where the identification of abnormal patterns is important, such as process monitoring, security intrusion detection, fraud identification, and / or medical diagnosis.

[0011] However, in cases where the data required for model learning for outlier detection is relatively rare or diverse and insufficient, such as when it is difficult to collect abnormal data containing a certain defect, when labeled data is limited, or when additional training is desired on a large amount of data without labels, there is a problem that there is a limit to the performance of task processing for anomaly detection based on this.

[0012] In addition, in the past, anomaly detection based on specific images was actively performed, especially in the field of vision inspection. However, in the case of such images, since they belong to high-dimensional data, there is a problem that data processing and computation costs are not efficient when all data for the entire image is used at once to detect outliers.

[0013] In addition, in most cases, outliers are observed as abnormal patterns that appear in various sizes and shapes in a small part of the image, and the conventional method has a problem in that the discrimination ability for such local patterns on the entire image is very low.

[0014] Therefore, there is a need to develop new technologies that can further improve the accuracy and efficiency of anomaly detection, even under limited environmental conditions, while also improving task processing performance accordingly.

[0015] (Prior patent document 1) KR 10-2296471 B1

[0016] One embodiment of the present invention seeks to implement a method and system for providing a guide for real-time outlier determination based on past outlier determination records.

[0017] At this time, one embodiment of the present invention can provide a guide for outlier determination by utilizing a deep learning model specialized for anomaly detection using a patch feature-based learning method.

[0018] A method for monitoring a judgment record for an outlier judgment guide according to an embodiment of the present invention comprises the steps of: obtaining a predetermined inspection image; detecting at least one similar image having a similarity higher than a predetermined standard with the obtained inspection image; providing judgment detailed information that is information specifying a judgment content for the presence or absence of an outlier for each of the detected similar images and the similar images; obtaining first judgment result information that is information specifying a result of the judgment for the presence or absence of an outlier for the inspection image; and storing the inspection image and the first judgment result information in a database according to the similarity between the inspection image and each of the similar images.

[0019] In another aspect, the step of providing the judgment details includes a step of providing at least one piece of information from among information on the person in charge, which is information specifying the person in charge who performed the judgment on the presence or absence of an outlier for the similar image, and information on the second judgment result, which is information specifying the result of the judgment on the presence or absence of an outlier for the similar image.

[0020] In another aspect, the step of detecting the similar image includes the step of measuring raw data similarity between at least one over-inspection image stored in the database and the inspection image.

[0021] In another aspect, the step of detecting the similar image further includes the step of obtaining a test feature vector, which is a feature vector for the test image, and a test feature vector, which is a feature vector for the over-test image, based on a predetermined image feature extraction model, and the step of measuring a similarity between feature vectors, which is a similarity between the obtained test feature vector and the test feature vector.

[0022] In another aspect, the image feature extraction model includes an outlier detection model, which is a deep learning model specialized in anomaly detection using a patch feature-based learning method.

[0023] In another aspect, the step of detecting the similar image further includes the step of calculating a decision similarity that specifies the final similarity between the test image and the over-test image based on the raw data similarity and the feature vector similarity.

[0024] In another aspect, the step of detecting the similar image further includes the step of detecting the similar image based on the determined similarity and a preset similarity criterion threshold.

[0025] In another aspect, the step of detecting the similar image further includes the step of determining whether the number of the detected similar images satisfies a preset number, and the step of additionally detecting the similar images if the number does not meet the preset number.

[0026] In another aspect, the step of additionally detecting the similar image includes the step of adjusting the similarity reference threshold, and the step of detecting the similar image based on the adjusted similarity reference threshold.

[0027] In another aspect, the step of providing the similar image and judgment details includes the step of further providing the decision similarity.

[0028] In another aspect, the step of providing the similar image and judgment details includes the step of additionally detecting the similar image according to a ratio between the second judgment result information for each of the at least one similar image, and the step of further providing an additional provided image which is the additionally detected similar image.

[0029] In another aspect, the step of providing the similar image and judgment details includes the step of aligning the detected similar images based on at least one of the information on the responsible worker and the similarity between the inspection image and each of the similar images.

[0030] In another aspect, the step of storing in the database further includes the step of storing the inspection image and the first judgment result information in the database depending on whether the first judgment result information and the second judgment result information are identical.

[0031] In another aspect, the judgment record monitoring method for an outlier judgment guide according to an embodiment of the present invention further includes a step of filtering the detected similar image according to preset conditions.

[0032] In another aspect, the step of obtaining the inspection image includes the step of obtaining the inspection image based on an outlier detection model, which is a deep learning model specialized for anomaly detection using a patch feature-based learning method.

[0033] On the other hand, a computing device according to an embodiment of the present invention includes at least one vision inspection unit; at least one inspection monitoring processing unit; at least one monitoring interface (I / F) unit; and at least one inspection database; and controls the vision inspection unit to obtain a predetermined inspection image, controls the inspection monitoring processing unit to detect at least one similar image having a similarity higher than a predetermined standard with the obtained inspection image, controls the monitoring interface (I / F) unit to provide judgment detailed information that is information specifying the content of a judgment on the presence or absence of an anomaly for each of the detected similar images and the similar images, controls the monitoring interface (I / F) unit to obtain first judgment result information that is information specifying a result of a judgment on the presence or absence of an anomaly for the inspection image, and controls the inspection monitoring processing unit to store the inspection image and the first judgment result information in the inspection database according to the similarity between the inspection image and each of the similar images.

[0034] A method and system for monitoring judgment records for an outlier judgment guide according to one embodiment of the present invention can support improvement of consistency and minimization of errors in real-time outlier judgment by providing a guide for real-time outlier judgment based on past outlier judgment records, thereby having the effect of improving the overall quality and performance of a vision inspection process.

[0035] At this time, the method for monitoring judgment records for an outlier judgment guide according to one embodiment of the present invention and its system can provide a guide for outlier judgment by utilizing a deep learning model specialized in anomaly detection in a patch feature-based learning method, thereby improving the quality of the provided guide and also improving the data processing efficiency for providing the guide.

[0036] FIG. 1 illustrates an example of a block diagram of a computing system that implements an outlier judgment guide and monitoring service according to one embodiment of the present invention.

[0037] FIG. 2 illustrates an example block diagram of a computing device implementing an outlier determination guide and monitoring service according to one embodiment of the present invention.

[0038] FIG. 3 illustrates an example block diagram of another aspect of a computing device implementing an outlier determination guide and monitoring service according to one embodiment of the present invention.

[0039] FIG. 4 illustrates an example block diagram of a computing system according to another embodiment of the present invention.

[0040] FIG. 5 is an example of a block flow diagram for explaining an outlier detection model according to one embodiment of the present invention.

[0041] FIG. 6 is a flowchart illustrating a patch feature learning method for anomaly detection according to one embodiment of the present invention.

[0042] FIG. 7 is a flowchart illustrating a feature representation learning method based on patch features according to one embodiment of the present invention.

[0043] FIG. 8 is an example of a drawing showing an example of measuring similarity between patch features according to one embodiment of the present invention.

[0044] FIG. 9 is an example of a drawing showing an application example of a ReConPatch Process according to one embodiment of the present invention.

[0045] FIG. 10 is a flowchart illustrating a method for monitoring judgment records for an outlier judgment guide according to one embodiment of the present invention.

[0046] Fig. 11 is a flowchart for explaining a similar image detection method according to one embodiment of the present invention.

[0047] Figure 12 is a conceptual diagram for explaining a similar image detection method according to one embodiment of the present invention.

[0048] FIG. 13 is an example of providing a similar image according to one embodiment of the present invention.

[0049] FIG. 14 is an example of providing an additional provision image according to one embodiment of the present invention.

[0050] FIG. 15 is an example of a drawing for explaining an alignment method when providing similar images according to one embodiment of the present invention.

[0051] The present invention is capable of various modifications and embodiments. Therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention, as well as the methods for achieving them, will become clear with reference to the embodiments described in detail below together with the drawings. However, the present invention is not limited to the embodiments disclosed below and can be implemented in various forms. In the following embodiments, the terms "first," "second," etc. are not used in a limiting sense but are used for the purpose of distinguishing one component from another. Furthermore, the singular expression includes the plural expression unless the context clearly indicates otherwise. Furthermore, terms such as "include" or "have" indicate the presence of a feature or component described in the specification, and do not preemptively exclude the possibility that one or more other features or components may be added. Furthermore, in the drawings, the sizes of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, and thus the present invention is not necessarily limited to what is shown.

[0052] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals and redundant descriptions thereof will be omitted.

[0053]

[0054] [Example system providing outlier determination guide and monitoring services]

[0055] Hereinafter, an exemplary system for implementing an outlier judgment guide and monitoring service that performs patch feature-based learning for a pretrained model based on an image data set for an outlier detection target is described in detail with reference to the attached drawings.

[0056] FIG. 1 illustrates an example of a block diagram of a computing system that implements an outlier judgment guide and monitoring service according to one embodiment of the present invention.

[0057] Referring to FIG. 1, a computing system (1000) implementing an outlier judgment guide and monitoring service of the present invention includes a user computing device (110), a server computing system (130), and a training computing system (150), and the devices are capable of communicating via a network (170).

[0058] The patch feature learning method for anomaly detection according to an embodiment of the present invention may 1) be implemented and provided locally by a user computing device (110), 2) be implemented and provided in the form of a web service by a server computing system (130) communicating with the user computing device (110), or 3) be implemented and provided by the user computing device (110) and the server computing system (130) in connection with each other.

[0059] At this time, in the embodiment, the user computing device (110) and / or the server computing system (130) can train the machine learning model (120 and / or 140) through interaction with a training computing system (150) communicatively connected via a network (170). The training computing system (150) may be separate from the server computing system (130) or may be part of the server computing system (130).

[0060] And at this time, the artificial intelligence model (in the embodiment, an outlier detection model, etc.) can be 1) trained directly locally by the user computing device (110), 2) trained by the server computing system (130) and the user computing device (110) interacting with each other through a network (170), and 3) trained by a separate training computing system (150) using various training techniques and learning techniques. And the artificial intelligence model trained by the training computing system (150) can be implemented in a manner of being provided / updated by transmitting it to the user computing device (110) and / or the server computing system (130) through the network (170).

[0061] In some embodiments, the training computing system (150) may be part of a server computing system (130) or part of a user computing device (110).

[0062] The user computing device (110) may include any type of computing device, such as a smart phone, a mobile phone, a digital broadcasting device, a personal digital assistant (PDA), a portable multimedia player (PMP), a desktop, a wearable device, an embedded computing device, and / or a tablet PC.

[0063] Such a user computing device (110) includes at least one processor (111) and memory (112). Here, the processor (111) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0064] The memory (112) may include one or more non-transitory / transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof, and may include web storage of a server that performs a memory storage function on the Internet. The memory (112) may store data (113) and commands (114) necessary for the at least one processor (111) to perform functional operations, such as training an artificial intelligence model or executing anomaly detection using an artificial intelligence model.

[0065] In one embodiment, the user computing device (110) may store at least one machine learning model (120).

[0066] In detail, the machine learning model (120) may be various machine learning models such as multiple neural networks (e.g., deep neural networks) or other types of machine learning models including nonlinear models and / or linear models, and may be composed of a combination thereof.

[0067] At this time, the neural network may include at least one of feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, and / or other types of neural networks.

[0068] In one embodiment, the user computing device (110) may receive at least one machine learning model (120) from a server computing system (130) via a network (170), store the model in a memory (112), and then execute the stored machine learning model (120) by a processor (111) to perform outlier detection, etc.

[0069] In another embodiment, the server computing system (130) may include at least one machine learning model (140) and perform operations through the machine learning model (140), and may provide an outlier determination guide and monitoring service to the user by communicating data related thereto with the user computing device (110) in conjunction with the user computing device (110).

[0070] For example, a user computing device (110) can perform an outlier judgment guide and monitoring service by having a server computing system (130) provide output for a user's input using a machine learning model (140) via the web.

[0071] Additionally, the artificial intelligence model may be implemented in such a way that at least some of the machine learning models (120 and / or 140) are executed on a user computing device (110) and the rest are executed on a server computing system (130).

[0072] In addition, the user computing device (110) may include at least one input component (121) that detects a user's input. For example, the user input component (121) may include a touch sensor (e.g., a touch screen and / or a touch pad, etc.) that detects a touch of a user's input medium (e.g., a finger or a stylus), an image sensor that detects a user's motion input, a microphone, a button, a mouse, and / or a keyboard that detects a user's voice input, etc. In addition, the user input component (121) may include an interface and an external controller when receiving an input to an external controller (e.g., a mouse and / or a keyboard, etc.) through an interface.

[0073] The server computing system (130) includes at least one processor (131) and memory (132). Here, the processor (131) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0074] And the memory (132) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. This memory (132) may store data (133) and instructions (134) necessary for the processor (131) to perform functional operations such as training an artificial intelligence model or executing outlier detection using an artificial intelligence model.

[0075] In one embodiment, the server computing system (130) may be implemented to include at least one computing device. For example, the server computing system (130) may be implemented to operate multiple computing devices according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. Additionally, the server computing system (130) may include multiple computing devices connected via a network (170).

[0076] Additionally, the server computing system (130) may store at least one machine learning model (140). For example, the server computing system (130) may include a neural network and / or other multi-layer nonlinear model as the machine learning model (140). Exemplary neural networks may include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0077] The training computing system (150) includes at least one processor (151) and memory (152). Here, the processor (151) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0078] And the memory (152) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. This memory (152) may store data (153) and instructions (154) necessary for the processor (151) to perform learning of an artificial intelligence model, etc.

[0079] For example, the training computing system (150) may include a model trainer (160) that trains a machine learning model (120 and / or 140) stored in the user computing device (110) and / or server computing system (130) using various training or learning techniques, such as backpropagation of errors (according to the framework illustrated in FIG. 3).

[0080] For example, such a model trainer (160) may perform updates to one or more parameters of a machine learning model (120 and / or 140) in a backpropagation manner based on a defined loss function.

[0081] In some implementations, performing backward propagation of errors may include performing truncated backpropagation through time. The model trainer (160) may perform a number of generalization techniques (e.g., weight reduction, dropout, and / or knowledge distillation) to improve the generalization ability of the trained machine learning model (120 and / or 140).

[0082] In particular, the model trainer (160) can train the machine learning model (120 and / or 140) based on a series of training data (161). Here, the training data (161) can include data in different formats, such as, for example, images, audio samples, and / or text. Examples of image types that can be used can include video frames, LiDAR point clouds, X-ray images, computed tomography scans, hyperspectral images, and / or various other forms of images.

[0083] Such training data (161) may be provided by a user computing device (110) and / or a server computing system (130). When the training computing device trains a machine learning model (120 and / or 140) on specific data of the user computing device (110), the machine learning model (120 and / or 140) may be characterized as a personalized model.

[0084] And the model trainer (160) includes computer logic utilized to provide the desired function.

[0085] Additionally, the model trainer (160) may be implemented as hardware, firmware, and / or software that controls a general-purpose processor. In one implementation, the model trainer (160) includes a program file stored in a storage device, which may be loaded into a memory (152) and executed by one or more processors (151). In another implementation, the model trainer (160) includes one or more sets of computer-executable data (153) and instructions (154) stored in a tangible computer-readable storage medium, such as a RAM hard disk or an optical or magnetic medium.

[0086] The network (170) includes, but is not limited to, a 3rd Generation Partnership Project (3GPP) network, a Long Term Evolution (LTE) network, a World Interoperability for Microwave Access (WIMAX) network, the Internet, a Local Area Network (LAN), a Wireless Local Area Network (Wireless LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, and / or a Digital Multimedia Broadcasting (DMB) network.

[0087] In general, communication over a network (170) may be performed using any type of wired and / or wireless connection, using various communication protocols (e.g., TCP / IP, HTTP, SMTP, and / or FTP), encodings or formats (e.g., HTML and / or XML), and / or protection schemes (e.g., VPN, Secure HTTP, and / or SSL).

[0088] FIG. 2 illustrates an example block diagram of a computing device implementing an outlier determination guide and monitoring service according to one embodiment of the present invention.

[0089] As shown in FIG. 2, the computing device (100) included in the user computing device (110), the server computing system (130), and the training computing system (150) includes a plurality of applications (e.g., Application 1 to Application N). Each application may include a machine learning library and one or more machine learning models. For example, the applications may include an image processing (e.g., Detection, Classification, and / or Segmentation, etc.) application, a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and / or a chat-bot application.

[0090] In an embodiment, the computing device (100) may include a model trainer (160) for training an artificial intelligence model, and may store and operate the trained artificial intelligence model to provide output data according to predetermined input data (e.g., image data, etc.).

[0091] Each application of the computing device (100) may communicate with a number of other components of the computing device (100), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In one embodiment, each application may communicate with each device component using an API (e.g., a public API). In one embodiment, the API used by each application may be specific to that application.

[0092] FIG. 3 illustrates an example block diagram of another aspect of a computing device (100) implementing an outlier judgment guide and monitoring service according to one embodiment of the present invention.

[0093] Referring to FIG. 3, a computing device (300) includes a plurality of applications (e.g., Application 1 to Application N). Each application may communicate with a central intelligence layer. For example, the applications may include an image processing application, a text messaging application, an email application, a dictation application, a virtual keyboard application, and / or a browser application. In one embodiment, each application may communicate with the central intelligence layer (and models stored therein) using an API (e.g., a common API across all applications).

[0094] The central intelligence layer may include multiple machine learning models. For example, as illustrated in FIG. 3, at least some of the machine learning models may be provided to each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model to all applications. In some implementations, the central intelligence layer may be included within the operating system of the computing device (300) or implemented differently.

[0095] The central intelligence layer may communicate with a central device data layer. The central device data layer may be a centralized data repository for the computing device (300). As illustrated in FIG. 3, the central device data layer may communicate with a number of other components of the computing device (300), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer may communicate with each device component using an API (e.g., a private API).

[0096] FIG. 4 illustrates an example block diagram of a computing system (1000) according to another embodiment of the present invention.

[0097] Referring to FIG. 4, in another aspect, a computing system (1000) according to an embodiment of the present invention may include at least one vision inspection unit (VIU), an inspection monitoring processing unit (MPU), an inspection database (IDU), a monitoring interface (I / F) unit (MIU), and / or an inspection post-processing unit (PPU).

[0098] In detail, a vision inspection unit (VIU) according to an embodiment can automatically determine whether there is an outlier in a given image (e.g., a specific product image, etc.) based on a model (hereinafter, a defect judgment model) learned through various methodologies such as a given deep learning and / or machine learning.

[0099] Here, the defect judgment model according to the embodiment may include an outlier detection model according to the embodiment of the present invention.

[0100] That is, in the embodiment, the vision inspection unit (VIU) can perform anomaly detection based on a given image using the anomaly detection model described below, and can automatically determine whether there is an anomaly in the image based on the result of the performed anomaly detection.

[0101] At this time, in the embodiment, if the reliability of the outlier determination result does not meet the preset standard, the vision inspection unit (VIU) can provide the corresponding image (i.e., the image whose reliability of the outlier determination result is below the standard) to the inspection monitoring processing unit (MPU) according to the embodiment of the present invention.

[0102] As an example, a vision inspection unit (VIU) can calculate an error range of an outlier determination result for a given first image.

[0103] And, if the calculated error range does not meet a preset standard (e.g., within an allowable error range, etc.), the vision inspection unit (VIU) can transmit the corresponding first image (inspection image in the embodiment) to the inspection monitoring processing unit (MPU).

[0104] In addition, the inspection monitoring processing unit (MPU) according to the embodiment can search and extract from the inspection database (IDU) according to the embodiment of the present invention at least one image (similar image, in the embodiment) having a similarity level higher than a preset standard with an image (in the embodiment, an inspection image) received from the vision inspection unit (VIU).

[0105] And the inspection monitoring processing unit (MPU) can provide at least one extracted similar image to a monitoring interface (I / F) unit (MIU) according to an embodiment of the present invention.

[0106] In addition, in the embodiment, the inspection monitoring processing unit (MPU) can receive and obtain various data (in the embodiment, data for determining the presence or absence of an outlier, etc.) according to input from a predetermined first user (in the embodiment, an operator performing an outlier determination, etc.) from a monitoring interface (I / F) unit (MIU).

[0107] And the inspection monitoring processing unit (MPU) can store and manage various data obtained from the monitoring interface (I / F) unit (MIU) and / or the vision inspection unit (VIU) in the inspection database (IDU).

[0108]

[0109] In addition, the inspection database (IDU) according to the embodiment can store and manage various data, information and / or algorithms required for the outlier judgment guide and monitoring service according to the embodiment of the present invention.

[0110] In addition, the monitoring interface (I / F) unit (MIU) according to the embodiment can display and output various data and / or information related to the outlier judgment guide and monitoring service as a predetermined graphic image.

[0111] In an embodiment, a monitoring interface (I / F) unit (MIU) may configure a screen to provide various data (e.g., inspection images, similar images, and / or judgment details, etc.) received from an inspection monitoring processing unit (MPU) so that a predetermined first user (hereinafter, an operator) can check them.

[0112] In addition, in the embodiment, the monitoring interface (I / F) unit (MIU) can provide various data according to the operator's input (e.g., input for determining the presence or absence of an abnormality in an inspection image, etc.) to the inspection monitoring processing unit (MPU).

[0113] In addition, the post-inspection processing unit (PPU) according to the embodiment can update predetermined data and / or information stored in the inspection database (IDU) based on additional data (e.g., judgment result feedback information, etc.) additionally acquired after the process for determining the presence or absence of an outlier (i.e., the vision inspection process) according to the embodiment of the present invention is performed on a predetermined inspection image.

[0114] In FIG. 4 of the present invention, in order to prevent the features according to the embodiment of the present invention from being blurred, the computing system (1000) is described as including components of the functional aspect as described above.

[0115]

[0116] However, it is obvious that a person skilled in the art can understand that, depending on the embodiment, other general components may be included in addition to the components illustrated in FIG. 4, or some of the components illustrated in FIG. 4 may be omitted.

[0117] Meanwhile, the techniques described herein may refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to or from such systems. It will be appreciated that the inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, and division of labor and functionality between and among components. For example, the processes described herein may be implemented using a single device or component, or multiple devices or components operating in combination. Databases and applications may be implemented on a single system or in a distributed system across multiple systems. Distributed components may operate sequentially or in parallel.

[0118]

[0119] [Outlier Detection Model (ODM)]

[0120] FIG. 5 is an example of a block flow diagram for explaining an outlier detection model (ODM) according to one embodiment of the present invention.

[0121] Referring to FIG. 5, an outlier detection model (ODM) according to an embodiment of the present invention may mean an image deep learning model that performs anomaly detection based on a given input image and classifies and / or recognizes the image based on the anomaly detection.

[0122] Here, for reference, anomaly detection can mean a process of identifying abnormal patterns, outliers, and / or exceptions from specific data.

[0123] That is, anomaly detection can be a process of detecting components that deviate from the properties of normal data.

[0124] As an example, anomaly detection can be implemented based on a method such as grouping certain data into clusters and considering points that deviate from the clusters as outliers.

[0125] Therefore, in an embodiment, an outlier detection model (ODM) can determine whether a given input image contains a specific abnormal attribute, and classify and / or recognize the image based on the determination result.

[0126] In detail, in an embodiment, the outlier detection model (ODM) may include a first network (TN: Teacher Network) and a second network (SN: Student Network).

[0127] In more detail, the first network (TN) according to the embodiment may mean a neural network that calculates similarity between predetermined features (in the embodiment, patch features).

[0128] In an embodiment, such a first network (TN) comprises a first feature representation layer ( ) and the first spatial projection layer ( ) may be included.

[0129] Here, the feature representation layer according to the embodiment may mean a layer that reconstructs (adjusts) the feature in a direction to improve the performance of the feature representation according to a given feature (in the embodiment, a patch feature).

[0130] To this end, the feature representation layer can be trained to extract meaningful features from a given feature (in the example, a patch feature) with higher accuracy and reconstruct (adjust) the feature based on the extracted meaningful features.

[0131] Additionally, the spatial projection layer according to the embodiment may mean a layer that projects a feature expression according to a given feature (in the embodiment, a patch feature) into a given feature expression space.

[0132] In an embodiment, such a spatial projection layer can be trained to project feature representations according to a given feature (in an embodiment, a patch feature) into a feature representation space that can more effectively apply the goal of model learning.

[0133] Meanwhile, the second network (SN) according to the embodiment may mean a neural network that implements feature representation learning.

[0134] For reference, feature representation learning can refer to the process by which a deep learning model automatically detects and learns useful features from given data.

[0135] Through feature representation learning, deep learning models can effectively encode useful information contained in data to generate meaningful features that can be used in various deep learning tasks, and can understand the complex structure and patterns of data and make more accurate predictions based on this.

[0136] In the embodiment, the second network (SN) implementing the above feature representation learning includes a second feature representation layer ( ) and the second spatial projection layer ( ) may be included.

[0137] At this time, the first feature expression layer according to the embodiment ( ) and the second feature representation layer ( ) and the first spatial projection layer ( ) and the second spatial projection layer ( ) is intended to distinguish between the feature representation layer and spatial projection layer included in the first network (TN) and the feature representation layer and spatial projection layer included in the second network (SN), so the description of the feature representation layer and spatial projection layer of the second network (SN) applies to the description of the feature representation layer and spatial projection layer of the first network (TN) described above.

[0138] In addition, a more detailed description of the first network (TN) and the second network (SN) according to the embodiment will be described in the patch feature learning method for anomaly detection described later.

[0139] On the other hand, in the embodiment, the outlier detection model (ODM) can perform various functional operations required for an outlier judgment guide and monitoring service and / or patch feature training service for anomaly detection in conjunction with a pre-trained model (hereinafter, “pre-trained model”) to perform concept learning.

[0140] Here, the pre-trained model according to the embodiment may be an image deep-learning model pre-trained to perform concept learning based on a predetermined training data set (e.g., a predetermined natural image data set and / or a predetermined normal image data set, etc.).

[0141] For reference, concept learning can mean the process of inferring general rules, concepts, or patterns from given data and classifying them.

[0142] That is, in the embodiment, the pre-trained model may be an image deep learning model that takes a predetermined image as input data, learns common features of objects or patterns in the input image, groups image components with similar features based on the common features, and supports classifying or recognizing a specific image based on the common features.

[0143] Specifically, the pre-trained model can: 1) extract a feature map for a given input image.

[0144] In detail, the pre-trained model can automatically extract feature maps based on the raw pixel data of the input image using a given image deep learning neural network (e.g., a convolutional neural network (CNN)).

[0145] These feature maps can represent various visual properties of the image, such as edges, colors, and / or textures.

[0146] Additionally, the pre-trained model can perform clustering based on 2) feature space.

[0147] In detail, the pre-trained model can classify input images and / or objects within the input images into groups with similar features based on the feature maps extracted as above.

[0148] As an example, the pre-trained model can classify the extracted feature maps according to the feature space using a predetermined clustering algorithm (e.g., K-means, DBSCAN, and / or a hierarchical clustering algorithm, etc.) or dimensionality reduction algorithm (e.g., t-SNE and / or UMAP, etc.).

[0149] Additionally, the pre-trained model can 3) assign a label to each cluster.

[0150] In other words, the pre-trained model can define a concept (hypothesis) representing each cluster and set it as a label for that cluster.

[0151] At this time, the pre-trained model can learn the features of images belonging to a specific concept or category by manually or semi-automatically assigning labels to each cluster.

[0152] Additionally, the pre-trained model can 4) verify and adjust the assigned hypothesis.

[0153] In detail, the pre-trained model can validate the initially defined concepts (clustered feature groups) as described above and adjust the corresponding hypotheses if necessary.

[0154] At this time, the pre-trained model can perform a process of detecting and improving incorrectly clustered data using new image data.

[0155] Additionally, the pre-trained model can repeat the process described above 5).

[0156] That is, the pre-trained model can be trained to continuously improve its clustered feature association concepts through new image data and additional feedback, and to classify or recognize images more accurately.

[0157] In this embodiment, the pre-trained model may be directly included in the outlier detection model (ODM), or may be implemented as a separate device and / or server from the outlier detection model (ODM).

[0158] In the following description, the pre-trained model is described as being implemented as an outlier detection model (ODM), but is not limited thereto.

[0159] In addition, in FIG. 5, the outlier detection model (ODM) is described as including the components described above to prevent blurring of features according to an embodiment of the present invention, but it is obvious to those skilled in the art that, depending on the embodiment, other general components may be further included in addition to the components illustrated in FIG. 5 or some of the components illustrated in FIG. 5 may be omitted.

[0160]

[0161] [Patch Feature Learning Method for Anomaly Detection]

[0162] Hereinafter, a method for implementing a patch feature training service for anomaly detection, in which a computing system (1000) according to an embodiment of the present invention performs patch feature-based learning for a predetermined pretrained model based on an image data set for an anomaly detection target, will be described in detail.

[0163] A patch feature learning method for anomaly detection of a computing system (1000) according to an embodiment of the present invention can improve the performance and quality of various anomaly detection-based services by using an outlier detection model (ODM) learned according to an embodiment of the present invention.

[0164] At this time, the patch feature learning method for anomaly detection of a computing system (1000) according to an embodiment of the present invention can effectively provide an outlier detection model (ODM) with improved performance by performing patch feature-based learning that reduces the variance of mutually similar patch features and increases the difference between mutually heterogeneous patch features.

[0165] Hereinafter, a patch feature learning method for anomaly detection according to an embodiment of the present invention will be described in more detail with reference to the attached drawings.

[0166] FIG. 6 is a flowchart illustrating a patch feature learning method for anomaly detection according to one embodiment of the present invention.

[0167] Referring to FIGS. 5 and 6, a patch feature learning method for anomaly detection according to an embodiment of the present invention comprises: a step (S101) of obtaining a feature map based on a pretrained model; a step (S103) of extracting a plurality of patch features based on the obtained feature map; a step (S105) of performing feature representation learning based on the extracted plurality of patch features; a step (S107) of obtaining a reconpatch feature according to the performed feature representation learning; a step (S109) of performing coreset sampling based on the obtained reconpatch feature; a step (S111) of obtaining a test sample image; a step (S113) of obtaining a reconpatch feature according to the obtained test sample image; and performing outlier detection based on the obtained reconpatch feature. It may include step (S115).

[0168] Specifically, the computing system (1000) according to an embodiment of the present invention can obtain a feature map based on a pretrained model. (S101)

[0169] In detail, in an embodiment, a computing system (1000) can obtain a feature map according to an image data set (hereinafter, target image data set) for a predetermined outlier detection target through a model (i.e., a pre-trained model) pre-trained to perform concept learning.

[0170] Here, in other words, the pre-trained model according to the embodiment may be an image deep-learning model that is pre-trained to perform concept learning based on a predetermined training data set (e.g., a predetermined natural image data set and / or a predetermined normal image data set, etc.).

[0171] In other words, in the embodiment, the computing system (1000) can obtain a feature map based on a predetermined target image data set (i.e., an image data set including multiple images for a predetermined outlier detection target) in conjunction with a pre-trained model as described above.

[0172] In detail, in an embodiment, the computing system (1000) can input a target image data set (e.g., an image data set including multiple images of a given electronic circuit element, etc.) into a pre-training model.

[0173] In this way, the above pre-trained model can output a feature map according to the input target image data set and provide it to the computing system (1000).

[0174] Thus, the computing system (1000) can obtain a feature map according to the target image data set.

[0175] Additionally, in the embodiment, the computing system (1000) can extract multiple patch features based on the acquired feature map. (S103)

[0176] Here, a patch feature according to an embodiment may mean a feature extracted from a patch representing a small portion of a given image.

[0177] Specifically, the above patch may be a rectangular area representing a specific portion within a given image. Such a patch may be viewed as a subset containing a portion of the information of the entire image, and may primarily include local information or texture information.

[0178] Additionally, the above features are feature information extracted from a given image or patch, and can summarize or express important properties of the image (e.g., pattern, texture, color, and / or shape, etc.).

[0179] Therefore, the above patch feature may be data representing local properties within a given image in a patch unit.

[0180] In detail, in an embodiment, the computing system (1000) can extract multiple patch features based on the feature map obtained as above.

[0181] In more detail, as an embodiment, the computing system (1000) may divide a target training image (hereinafter, a target training image) included in a target image data set into units of a predetermined patch size before inputting the target training image into the above-described pre-training model.

[0182] And the computing system (1000) can input each segmented patch into a pre-training model to obtain a corresponding feature map for each patch.

[0183] In other words, the computing system (1000) can obtain a feature map for each patch by dividing the target training image into predetermined patch size units and then inputting the divided target training image into a pre-training model.

[0184] At this time, according to the embodiment, the computing system (1000) can perform coreset sampling on the acquired patch-specific feature map.

[0185] Here, for reference, core set sampling is one of the methods for efficiently processing large data sets, and can mean the process of extracting a set of representative samples that preserve the statistical characteristics or structure of the original data set as much as possible while reducing the size of the data set.

[0186] As an example, the computing system (1000) can perform core set sampling using an approximate algorithm method that selects some samples that can represent the entire data set while maintaining the characteristics of the original data set within a predetermined error range by considering the distribution of the given data.

[0187] Alternatively, the computing system (1000) may perform coreset sampling using an importance sampling method that assigns a sampling probability based on the importance of each given data point and preferentially selects data points with high importance.

[0188] Thus, the computing system (1000) can obtain a plurality of patch features for which coreset sampling has been performed.

[0189] Meanwhile, in another embodiment, the computing system (1000) can obtain a feature map for the entire area of ​​the target training image (hereinafter, the entire feature map).

[0190] And the computing system (1000) can divide the entire acquired feature map into units of a predetermined patch size.

[0191] Thus, the computing system (1000) can extract multiple patch features from the target training image.

[0192] In this way, in the embodiment, the computing system (1000) can extract multiple patch features according to the target training image in at least one of the above-described methods.

[0193] At this time, according to an embodiment, the computing system (1000) can extract each patch feature by aggregating surrounding feature vectors within a specific patch size.

[0194] Alternatively, depending on the embodiment, the computing system (1000) may use the pixel values ​​themselves within each patch as features.

[0195] Alternatively, depending on the embodiment, the computing system (1000) may use a statistical summary (e.g., mean, variance, and / or histogram, etc.) of pixel values ​​within each patch as a feature.

[0196] Alternatively, according to an embodiment, the computing system (1000) may analyze texture patterns within each patch and use them as features. For example, the computing system (1000) may extract texture-based patch features using a Gabor filter, Local Binary Patterns (LBP), and / or Histogram of Oriented Gradients (HOG) technique.

[0197] Alternatively, according to an embodiment, the computing system (1000) may automatically learn and extract high-dimensional features within each patch using a deep learning algorithm such as a convolutional neural network (CNN) and use them as features.

[0198] In this way, the computing system (1000) in the embodiment can extract features at the patch level and support outlier detection using the same, thereby increasing processing efficiency in the data learning and analysis process and detecting abnormal local patterns that mainly appear in small parts within an image in more detail.

[0199] Additionally, in the embodiment, the computing system (1000) can perform feature representation learning based on a plurality of extracted patch features. (S105)

[0200] Here, in other words, feature representation learning can mean a process in which a deep learning model (in this example, an outlier detection model (ODM)) automatically detects and learns useful features from given data.

[0201] In detail, in an embodiment, the computing system (1000) can learn a feature representation for a plurality of patch features extracted as described above based on an outlier detection model (ODM) according to an embodiment of the present invention.

[0202] That is, the computing system (1000) can perform feature representation learning for an outlier detection model (ODM) that performs anomaly detection based on the extracted plurality of patch features.

[0203] Here, in other words, anomaly detection can mean a process of identifying abnormal patterns, outliers, and / or exceptions from specific data, i.e., a process of detecting components that deviate from the properties of normal data.

[0204] Accordingly, in an embodiment, the computing system (1000) can determine whether a given input image includes a specific abnormal property, and perform feature representation learning (e.g., concept learning, etc.) to classify and / or recognize the image based on the determination result based on the plurality of patch features described above.

[0205] At this time, in the embodiment, the computing system (1000) can perform the feature representation learning described above based on a semi-supervised learning method.

[0206] In other words, the computing system (1000) can build an outlier detection model (ODM) that implements semi-supervised anomaly detection based on semi-supervised learning.

[0207] Here, for reference, semi-supervised learning can mean a deep learning method that trains a model using not only data with labels (i.e., supervised data) but also data without labels (i.e., unsupervised data).

[0208] In general, when collecting basic data for building an anomaly detection system, it is difficult to obtain a sufficient amount of abnormal data (e.g., image data capturing the abnormal state of the target of anomaly detection) for smooth learning, and thus, there may be limitations in anomaly detection learning that aims to recognize abnormal states (outliers) of various shapes with high accuracy.

[0209] Therefore, in an embodiment of the present invention, a semi-supervised learning-based anomaly detection is implemented by constructing a pre-trained model (i.e., a pre-trained model) mainly using normal data (e.g., image data capturing the normal state of an anomaly detection target, etc.), and performing anomaly detection based on pseudo labels using the pre-trained model.

[0210] Here, for reference, pseudo-label can mean a label predicted by a model trained on unlabeled data.

[0211] These pseudo-labels can be used primarily when model training requires limited labeled data or additionally leverages large amounts of unlabeled data.

[0212] Through this, the computing system (1000) in the embodiment can easily achieve model learning and performance improvement for building an outlier detection process even when labeled data is relatively rare or diverse and thus limited.

[0213] In more detail, in an embodiment, the computing system (1000) can perform feature representation learning according to multiple patch features based on a first network (TN) and a second network (SN) of an outlier detection model (ODM).

[0214] FIG. 7 is a flowchart illustrating a feature representation learning method based on patch features according to one embodiment of the present invention.

[0215] Specifically, referring to FIG. 7, in the embodiment, the computing system (1000) can project any patch feature pair into a predetermined feature representation space. (S201)

[0216] In detail, in an embodiment, the computing system (1000) may include any first patch feature ( ) and the second patch feature ( ) can be projected into a feature representation space a patch feature pair (hereinafter, a first patch feature pair).

[0217] In an embodiment, a computing system (1000) projects a first patch feature ( ) is expressed as in [(a) of Mathematical Formula 1], and the second patch feature (projected into the feature representation space) is ) can be expressed as in [(b) of Mathematical Formula 1].

[0218] [Equation 1 - (a)]

[0219]

[0220] [Equation 1 - (b)]

[0221]

[0222] Additionally, in the embodiment, the computing system (1000) can calculate pairwise similarity based on patch feature pairs projected into a feature representation space. (S203)

[0223] Here, the pairwise similarity according to the embodiment means that the first patch feature ( ) and the second patch feature ( ) can mean data that measures the similarity between two groups.

[0224] That is, in the embodiment, the computing system (1000) includes a first patch feature ( ) and the second patch feature ( ) can be used to measure pairwise similarity, which indicates the similarity between them.

[0225] At this time, the computing system (1000) as an example can calculate the pairwise similarity described above according to [Mathematical Formula 2] below.

[0226] [Equation 2]

[0227]

[0228] FIG. 8 is an example of a drawing showing an example of measuring similarity between patch features according to one embodiment of the present invention.

[0229] However, referring to FIG. 8, the first patch feature ( included in any patch feature pair) ) and the second patch feature ( ) When measuring similarity only for the relationship, the pairwise similarity is the same, but the discrimination accuracy may be reduced in cases where the first feature and the second feature should be classified with different labels as in (a) of Fig. 8 (i.e., the more distant they are from each other, the closer they are to the correct label), and in cases where the first feature and the second feature should be classified with the same label as in (b) of Fig. 8 (i.e., the closer they are to each other, the closer they are to the correct label).

[0230] In other words, if we only measure pairwise similarity, the first patch feature ( ) for K nearest neighbors ( ) and the second patch feature ( ) for K nearest neighbors ( ) may result in poor accuracy as label prediction is performed without considering the mutual similarity in the group relationship.

[0231] Thus, in the embodiment, the computing system (1000) can calculate contextual similarity based on patch feature pairs projected into the feature representation space. (S205)

[0232] Here, the context similarity according to the embodiment means the first patch feature (included by any patch feature pair) ) for K nearest neighbors ( ) and the second patch feature ( ) for K nearest neighbors ( ) can mean data that measures the two-way similarity between two objects.

[0233] At this time, in the embodiment, the bidirectional similarity is the first patch feature ( ) for K nearest neighbors ( ) features and the second patch features ( ) for K nearest neighbors ( ) can be calculated based on the average similarity between features.

[0234] In detail, in an embodiment, the computing system (1000) can calculate the context similarity described above according to the K-Nearest Neighbors (K-NN) algorithm that performs prediction based on the distance between data points as in [Mathematical Formula 3] below and [Mathematical Formula 4].

[0235] [Equation 3]

[0236] if ; 0, otherwise

[0237] [Equation 4]

[0238]

[0239]

[0240] That is, in the embodiment, the computing system (1000) includes a first patch feature ( ) and the second patch feature ( ) can be used to calculate contextual similarity, which considers the more common nearest neighbors they share, the higher the contextual similarity.

[0241] In this way, the computing system (1000) has the first patch feature ( ) and the second patch feature ( ) can be learned and reflected in the pseudo-label prediction process.

[0242] That is, the computing system (1000) has the first patch feature ( ) and the second patch feature ( ) by extracting the closest K feature samples, calculating the context similarity that measures how many samples intersect, and using this together with the pairwise similarity to train an outlier detection model (ODM), the trained outlier detection model (ODM) can be made to extract better quality features.

[0243] Thus, the computing system (1000) can more accurately determine whether the pairwise similarity between the first feature and the second feature is the same but the first feature and the second feature should be classified into different labels (i.e., the more apart they are from each other, the closer they are to the correct label) or whether the first feature and the second feature should be classified into the same label (i.e., the closer they are to each other, the closer they are to the correct label), and reflect this in predicting the pseudo label. Accordingly, the computing system (1000) can directly improve the task processing quality and performance of an outlier detection model (ODM) that performs semi-supervised learning-based anomaly detection.

[0244] Returning again, in the embodiment, the computing system (1000) can also calculate integrated similarity based on the pairwise similarity and contextual similarity calculated as above. (S207)

[0245] Here, the integrated similarity according to the embodiment means the first patch feature ( ) and the second patch feature ( ) can mean data that combines pairwise similarity and contextual similarity.

[0246] In detail, in the embodiment, the computing system (1000) includes a first patch feature (included in the first patch feature pair) according to [Mathematical Formula 5] as follows: ) and the second patch feature ( ) can linearly combine pairwise similarity and contextual similarity.

[0247] [Equation 5]

[0248]

[0249]

[0250] At this time, the integrated similarity according to the embodiment is ' ' can be defined as a linear combination of two similarities satisfying .

[0251]

[0252] Additionally, in the embodiment, the computing system (1000) can train a second network (SN) of an outlier detection model (ODM) based on the calculated integrated similarity. (S209)

[0253] That is, in the embodiment, the computing system (1000) can perform second network (SN) learning that implements feature representation learning using integrated similarity.

[0254] In detail, in the embodiment, the computing system (1000) calculates the integrated similarity for the first patch feature pair against the loss of the relaxation ( ) can be applied to train the second network (SN).

[0255] Here, the loss of relaxation according to the embodiment ( ) is as shown in [Mathematical Formula 6] below.

[0256] [Equation 6]

[0257]

[0258] Here, ' in [Equation 6] 'silver ' ' is the embedding vector inferred by , and ' ' is the number of mini-batches, i.e. patch instances, and ' ' is the repelling margin, and ' ' may be a parameter that determines the weights of the attraction and repulsion loss terms.

[0259]

[0260] FIG. 9 is an example of a drawing showing an application example of a ReConPatch Process according to one embodiment of the present invention.

[0261] At this time, referring to FIG. 9, in the embodiment, the computing system (1000) uses the integrated similarity to determine the first patch feature ( ) and the second patch feature ( ) is judged to be a patch feature pair (hereinafter, a positive feature pair) that should be classified with different labels, the first patch feature ( ) and the second patch feature ( ) in the feature representation space and map them to each other in the feature representation space. ), embedding function) can be learned.

[0262] In contrast, in the embodiment, the computing system (1000) uses the integrated similarity to identify the first patch feature ( ) and the second patch feature ( ) are judged to be a patch feature pair (hereinafter, a voice feature pair) that should be classified with the same label, the first patch feature ( ) and the second patch feature ( ) to mutually access and map the second feature representation layer (() of the second network (SN) in the feature representation space. ), embedding function) can be learned.

[0263] In this way, in the embodiment, the computing system (1000) reduces the variance of similar patch features and increases the difference between heterogeneous patch features by using a second feature representation layer ( ) can be taught.

[0264] That is, the computing system (1000) has a second feature representation layer ( ) can be trained to extract arbitrary patch features in a form closer to the correct pseudo-label.

[0265] In this way, in the embodiment, the computing system (1000) includes a second feature representation layer ( ) can directly improve the feature representation performance of the outlier detection model (ODM) itself by learning to more accurately extract meaningful features from arbitrary patch features, and at the same time, improve the processing quality of various tasks (such as anomaly detection in the example) based on it.

[0266] In addition, in an embodiment, a computing system (1000) that trains a second network (SN) based on integrated similarity can train a first network (TN) of 6) an outlier detection model (ODM).

[0267] In detail, in the embodiment, the computing system (1000) calculates the parameters of the second network (SN) according to [Mathematical Formula 7] using the exponential moving average (EMA) method. ) according to the parameters of the first network (TN) ) can be gradually distilled into data.

[0268] [Equation 7]

[0269]

[0270] That is, in the embodiment, the computing system (1000) can perform learning of the first network (TN) of the outlier detection model (ODM) by gradually distilling information learned in the second network (SN) into the first network (TN) according to the above-described [Mathematical Formula 7].

[0271] At this time, in the embodiment, the computing system (1000) can perform the first network (TN) learning described above by further applying an update speed control variable, which is a variable that controls the speed of information distillation.

[0272] Thus, in the embodiment, the computing system (1000) can implement feature representation learning based on multiple patch features based on the first network (TN) and the second network (SN) of the outlier detection model (ODM).

[0273] In this way, in the embodiment, the computing system (1000) can perform a process (in the embodiment, a ReConPatch Process) of building a discriminant feature for outlier detection by distilling the main features of the data set for the outlier detection target into a pre-trained model based on semi-supervised learning as described above.

[0274] Thus, the computing system (1000) can build a high-performance outlier detection model (ODM) that is trained to classify features corresponding to a given patch level into correct pseudo-labels more accurately.

[0275] Accordingly, the computing system (1000) can directly and effectively improve the processing performance and quality of various tasks (anomaly detection, etc. in the embodiment) using the above-described outlier detection model (ODM).

[0276]

[0277] Additionally, in the embodiment, the computing system (1000) can perform coreset sampling based on the acquired reconpatch features. (S109)

[0278] Here, in other words, core set sampling according to the embodiment can mean a process of extracting a set of representative samples that preserve the statistical characteristics or structure of the original data set as much as possible while reducing the size of the data set, as one of the methods for efficiently processing a large data set.

[0279] In detail, as an embodiment, the computing system (1000) can perform core set sampling using an approximate algorithm method that selects some samples that can represent the entire reconpatch learning data set while maintaining the characteristics of the original reconpatch learning data set within a predetermined error range by considering the distribution of the acquired reconpatch learning data set.

[0280] In another embodiment, the computing system (1000) may perform coreset sampling using an importance sampling method that assigns a sampling probability based on the importance of each piece of acquired reconpatch feature data and preferentially selects data points with high importance.

[0281] Thus, the computing system (1000) can obtain a reconpatch learning data set (hereinafter, reconpatch sampling data set) on which coreset sampling has been performed.

[0282] Additionally, in the embodiment, the computing system (1000) can store and manage the acquired reconpatch sampling data set on a predetermined database.

[0283] Through this, the computing system (1000) can detect outliers with high accuracy while reducing data processing costs.

[0284]

[0285] Additionally, in the embodiment, the computing system (1000) can obtain a test sample image. (S111)

[0286] Here, the test sample image according to the embodiment may mean an image for which the presence or absence of an outlier is to be detected, i.e., image data that captures the target of outlier detection.

[0287] In detail, in an embodiment, the computing system (1000) may obtain a test sample image as described above based on a predetermined user input and / or connection with an external server.

[0288] Additionally, in the embodiment, the computing system (1000) can obtain a reconpatch feature according to the acquired test sample image. (S113)

[0289] In the following, any content that overlaps with the above description may be summarized or omitted.

[0290] In detail, in an embodiment, the computing system (1000) can input the acquired test sample images into a pre-trained model.

[0291] Additionally, in the embodiment, the computing system (1000) can obtain a feature map for a test sample image from a pre-trained model that has received the test sample image as input. A detailed description thereof follows the description of step S101 described above.

[0292] Additionally, in the embodiment, the computing system (1000) can extract multiple patch features based on the acquired feature map. A detailed description thereof follows the description of step S103 described above.

[0293] Additionally, in the embodiment, the computing system (1000) can input the extracted plurality of patch features into the reconpatch layer described above.

[0294] In this way, the reconpatch layer can output multiple reconpatch features according to multiple input patch features.

[0295] Thus, in the embodiment, the computing system (1000) can obtain a reconpatch feature data set (hereinafter, reconpatch target data set) in a form that reduces the dispersion of similar patch features and increases the difference of heterogeneous patch features for a plurality of input patch features.

[0296] Additionally, in the embodiment, the computing system (1000) can perform outlier detection based on the acquired reconpatch features. (S115)

[0297] That is, in the embodiment, the computing system (1000) can perform outlier detection for a test sample image based on a reconpatch sampling data set obtained based on each training image used for learning and a reconpatch target data set obtained based on a test sample image.

[0298] In other words, in the embodiment, the computing system (1000) can perform anomaly detection on a test sample image based on a reconpatch sampling data set and a reconpatch target data set.

[0299] In detail, in an embodiment, the computing system (1000) can calculate the similarity (hereinafter, outlier detection similarity) between at least a portion of the reconpatch sampling data set stored in the database and the reconpatch target data set.

[0300] Additionally, in the embodiment, the computing system (1000) can generate an anomaly score map based on the calculated outlier detection similarity.

[0301] Here, for reference, the anomaly score map may refer to an indicator that indicates how much a state deviates from a given normal state based on the value (score) assigned by the model.

[0302] Additionally, in the embodiment, the computing system (1000) can determine that the higher the score according to the generated anomaly score map, the closer the test sample image is to an abnormal state, and the lower the score according to the anomaly score map, the closer the test sample image is to a normal state.

[0303] Thus, in the embodiment, the computing system (1000) can perform outlier detection on a test sample image.

[0304] As described above, in the embodiment of the present invention, the computing system (1000) performs a process (in the embodiment, a ReConPatch Process) of building a discriminant feature for outlier detection by distilling the main features of the data set for the outlier detection target into a pre-trained model based on semi-supervised learning as described above, and can perform anomaly detection using the outlier detection model (ODM) learned through this.

[0305] Thus, in the embodiment, the computing system (1000) can directly and significantly improve the outlier detection performance and quality based on the outlier detection model (ODM) according to the embodiment of the present invention.

[0306] Above, the patch feature learning method and system for anomaly detection according to one embodiment of the present invention performs patch feature-based learning for a predetermined pretrained model based on an image data set for an anomaly detection target, thereby performing more efficient data processing and providing an anomaly detection model (ODM) that further improves task processing performance and quality for anomaly detection.

[0307] In addition, the patch feature learning method and system for anomaly detection according to one embodiment of the present invention performs patch feature-based learning that reduces the variance of mutually similar patch features and increases the difference between mutually heterogeneous patch features, thereby improving the accuracy and efficiency of anomaly detection even in a limited learning environment, and at the same time, enhancing task processing performance accordingly.

[0308]

[0309] [How to Monitor Judgment Records for Outlier Judgment Guide]

[0310] Hereinafter, a method for implementing an outlier judgment guide and monitoring service that provides a guide for real-time outlier judgment based on past outlier judgment records by a computing system (1000) according to an embodiment of the present invention will be described in detail.

[0311] A method for monitoring judgment records for an outlier judgment guide of a computing system (1000) according to an embodiment of the present invention can detect at least one image (in the embodiment, a similar image) similar to an image (in the embodiment, an inspection image) for which the presence or absence of an outlier is to be determined from past outlier judgment records, and can display and output various data related to the detected similar image and the presence or absence of an outlier previously determined for the image as a predetermined graphic image.

[0312] At this time, the method for monitoring judgment records for an outlier judgment guide of a computing system (1000) according to an embodiment of the present invention can utilize an outlier detection model (ODM) learned according to an embodiment of the present invention, thereby improving data processing efficiency for detection of the above-described inspection image and / or similar image, and at the same time increasing accuracy and reliability thereof.

[0313] Hereinafter, a judgment record monitoring method for an outlier judgment guide according to an embodiment of the present invention will be described in more detail with reference to the attached drawings.

[0314] FIG. 10 is a flowchart illustrating a method for monitoring judgment records for an outlier judgment guide according to one embodiment of the present invention.

[0315] Referring to FIG. 10, a judgment record monitoring method for an outlier judgment guide according to an embodiment of the present invention may include a step (S301) of acquiring a predetermined inspection image, a step (S303) of detecting a similar image for the acquired inspection image, a step (S305) of filtering the detected similar image, a step (S307) of providing the filtered similar image, a step (S309) of acquiring judgment result information for the inspection image, and a step (S311) of storing and managing the acquired judgment result information.

[0316] Specifically, the computing system (1000) according to an embodiment of the present invention can obtain a predetermined inspection image. (S301)

[0317] Here, the inspection image according to the embodiment of the present invention may mean a target image for which the presence or absence of an outlier is to be determined manually.

[0318] That is, the inspection image may be an image that requires manual judgment by a worker regarding the presence or absence of an outlier.

[0319] In detail, in the embodiment, the computing system (1000) can obtain the above-described inspection image in conjunction with a vision inspection unit (VIU) or the like.

[0320] In more detail, the vision inspection unit (VIU) according to the embodiment can automatically determine whether there is an outlier in a given image (e.g., a specific product image, etc.) using the defect judgment model described above.

[0321] Here, the defect judgment model according to the embodiment may include the above-described outlier detection model (ODM).

[0322] That is, in the embodiment, the vision inspection unit (VIU) can perform anomaly detection based on a given image using the above-described anomaly detection model (ODM), and can automatically determine whether there is an anomaly in the image based on the result of the performed anomaly detection.

[0323] At this time, in the embodiment, if the reliability of the outlier determination result does not meet the preset standard, the vision inspection unit (VIU) can provide the corresponding image (i.e., the image whose reliability of the outlier determination result is below the standard) as a separate output.

[0324] As an example, a vision inspection unit (VIU) can calculate an error range of an outlier determination result for a given first image.

[0325] And, if the calculated error range does not meet the preset standard (e.g., within the allowable error range, etc.), the vision inspection unit (VIU) can separately output the corresponding first image (i.e., inspection image) and provide it to the computing system (1000).

[0326] Thus, in the embodiment, the computing system (1000) can obtain an inspection image to be provided to a worker to manually determine the presence or absence of an outlier.

[0327] Additionally, in the embodiment, the computing system (1000) can detect a similar image for the acquired inspection image. (S303)

[0328] Here, a similar image according to an embodiment of the present invention may mean an image among the over-inspection images stored in the inspection database (IDU) that has a similarity level higher than a preset standard with the above-described inspection image.

[0329] At this time, the over-inspection image according to the embodiment may mean an inspection image for which an outlier presence or absence was determined manually in the past.

[0330] That is, the over-inspection image may be an inspection image for which a judgment on the presence or absence of an outlier was performed by a specific worker before the current point in time.

[0331] In embodiments, such over-screen images may include judgment details corresponding to the over-screen images.

[0332] Here, the judgment details according to the embodiment may mean information including the judgment content for the presence or absence of an outlier in a given inspection image (here, an over-inspection image), i.e., various metadata related to the judgment for the presence or absence of an outlier.

[0333] In an embodiment, the judgment details may include information on the person in charge, judgment result information, judgment result feedback information, and / or defect type information.

[0334] At this time, the information on the worker in charge according to the embodiment may be information that specifies the worker who performed the judgment on the presence or absence of an abnormality in a given inspection image (here, an over-inspection image).

[0335] In an embodiment, the worker information may include worker name, job title, career and / or skill level information.

[0336] Here, the skill information according to the embodiment may be information specifying the skill level of the worker in manual vision inspection based on the inspection image (i.e., the task of manually determining the presence or absence of an outlier in the inspection image).

[0337] That is, the skill information may be information that indicates the accuracy of determining whether or not the worker is an outlier in a predetermined manner.

[0338] As an example, the computing system (1000) can calculate an error rate for the judgment result of the worker in order to obtain the above-described skill information.

[0339] Specifically, the computing system (1000) can calculate an error rate (e.g., a predetermined percentage (%), etc.) for the judgment result of the worker based on the judgment result information of the worker and the judgment result feedback information matching the judgment result information.

[0340] And the computing system (1000) can specify the skill level information of the worker based on the calculated error rate.

[0341] At this time, the judgment result information according to the embodiment may be information specifying the result of the worker in charge judging whether there is an outlier in a given inspection image (here, an over-inspection image).

[0342] For example, the judgment result information may be OK (i.e., no outliers) or NG (i.e., outliers exist).

[0343] In addition, the judgment result feedback information according to the embodiment may be information that determines whether the judgment result information of a given inspection image (here, an over-inspection image) is correct or incorrect.

[0344] That is, the judgment result feedback information may be information on whether the judgment result of the worker in charge of a given inspection image (here, the over-inspection image) is correct or incorrect.

[0345] As an example, the computing system (1000) can obtain the above judgment result feedback information by linking with a predetermined user input and / or an external server (e.g., a defective product management server, etc.).

[0346] And the computing system (1000) can match the acquired judgment result feedback information to the corresponding inspection image and store and manage it in the inspection database (IDU).

[0347] In addition, the defect type information according to the embodiment may be information that specifies a specific defect type (e.g., absence of a specific component, etc.) included in a specific inspection image (here, an over-inspection image).

[0348] In an embodiment, the defect type information may be set based on input from a predetermined user (e.g., a worker who performed an outlier determination for the corresponding inspection image, etc.).

[0349] Returning to FIG. 10 again, in a detailed embodiment, the computing system (1000) can detect at least one similar image for the inspection image acquired as above from the inspection database (IDU) in conjunction with an inspection monitoring processing unit (MPU) or the like.

[0350] FIG. 11 is a flowchart for explaining a similar image detection method according to one embodiment of the present invention, and FIG. 12 is a conceptual diagram for explaining a similar image detection method according to one embodiment of the present invention.

[0351] In more detail, referring to FIGS. 11 and 12, in the embodiment, the computing system (1000) can measure raw data similarity (RDS) between an inspection image (IIM) and a post-inspection image (PII). (S401)

[0352] Here, for reference, raw data similarity (RDS) may refer to information that quantitatively measures the similarity between individual data points within a given data set (in an embodiment, a data set including an inspection image (IIM) and at least one over-inspection image (PII)).

[0353] In an embodiment, the computing system (1000) may measure the raw data similarity (RDS) between the inspection image (IIM) and the under-inspection image (PII) using a raw data similarity (RDS) measurement index such as a structural similarity index map (SSIM), a mean square error (MSE), a Euclidean distance, cosine similarity, Jaccard similarity, Pearson correlation coefficient, and / or a Manhattan distance.

[0354] Additionally, in the embodiment, the computing system (1000) can obtain feature vectors for each of the inspection image (IIM) and the inspection image (PII). (S403)

[0355] In detail, in an embodiment, a computing system (1000) can obtain a feature vector (IFV: hereinafter, inspection feature vector) for an inspection image (IIM) and a feature vector (PFV: hereinafter, over-inspection feature vector) for a pre-inspection image (PII) by linking with a deep learning model (hereinafter, image feature extraction model) that takes a predetermined image as input data and a feature vector according to the input image as output data.

[0356] Here, as an example, the image feature extraction model may be a deep learning model (hereinafter, the first feature extraction model) including a convolutional neural network (CNN) that is trained based on a predetermined image data set (e.g., a natural image data set such as ImageNet, COCO (Common Objects in Context), and / or PASCAL VOC) that is unrelated to the inspection image (IIM).

[0357] In another embodiment, the image feature extraction model may be a deep learning model (hereinafter, the second feature extraction model) including a convolutional neural network (CNN) trained on a predetermined image data set (e.g., an image data set including a plurality of over-inspection images (PII)) specialized in determining the presence or absence of an outlier in an inspection image (IIM).

[0358] At this time, according to an embodiment, the image feature extraction model may include an outlier detection model (ODM) (hereinafter, the third feature extraction model) that is learned through a predetermined image data set (in the embodiment, a target image data set) specialized in determining the presence or absence of an outlier in an inspection image (IIM).

[0359] In more detail, the computing system (1000) as an embodiment can obtain the inspection feature vector (IFV) and the over-inspection feature vector (PFV) described above in conjunction with the first feature extraction model and / or the second feature extraction model.

[0360] Specifically, the computing system (1000) can input an inspection image (IIM) and a post-inspection image (PII) into the first feature extraction model and / or the second feature extraction model.

[0361] In this way, the first feature extraction model and / or the second feature extraction model can extract and output feature vectors for each of the input inspection image (IIM) and the over-inspection image (PII).

[0362] Thus, the computing system (1000) can obtain an inspection feature vector (IFV) and a over-inspection feature vector (PFV) from the first feature extraction model and / or the second feature extraction model.

[0363] In another embodiment, the computing system (1000) can obtain the inspection feature vector (IFV) and the over-inspection feature vector (PFV) described above in conjunction with a third feature extraction model (i.e., an outlier detection model (ODM)).

[0364] Below, any content that overlaps with the above description may be summarized or omitted.

[0365] In detail, in this embodiment, the computing system (1000) can input an inspection image (IIM) and an inspection image (PII) into the third feature extraction model.

[0366] In this way, the third feature extraction model can obtain a feature map based on the aforementioned pre-training model for each of the input inspection image (IIM) and the over-inspection image (PII).

[0367] In addition, the third feature extraction model can input the feature map of the acquired inspection image (IIM) (hereinafter, inspection feature map) and the feature map of the over-inspection image (PII) (hereinafter, over-inspection feature map) into the aforementioned reconpatch layer.

[0368] Thus, the third feature extraction model can obtain a reconpatch feature data set (hereinafter, “reconpatch inspection data set”) according to the inspection feature map and a reconpatch feature data set (hereinafter, “reconpatch over-inspection data set”) according to the over-inspection feature map from the reconpatch layer.

[0369] At this time, depending on the embodiment, the third feature extraction model may perform core set sampling based on the reconpatch inspection data set and / or the reconpatch over-inspection data set.

[0370] Thus, the third feature extraction model can obtain an inspection feature vector (IFV) based on the acquired reconpatch inspection data set and an over-check feature vector (PFV) based on the reconpatch over-check data set.

[0371] In this way, according to an embodiment, the computing system (1000) can obtain a feature vector based on an outlier detection model (ODM) whose performance is improved through learning more optimized for anomaly detection.

[0372] Accordingly, the computing system (1000) can measure the similarity between the corresponding images by utilizing the feature vectors extracted with higher accuracy from the feature data of each of the inspection image (IIM) and the inspection image (PII).

[0373] Thus, the computing system (1000) can further improve the quality of similar images determined later based on the measured similarity.

[0374] Additionally, in the embodiment, the computing system (1000) can measure the similarity between the acquired feature vectors. (S405)

[0375] That is, the computing system (1000) can measure the similarity (distance) between the inspection feature vector (IFV) and the over-inspection feature vector (PFV) obtained as above.

[0376] In an embodiment, the computing system (1000) can measure the similarity (FVS: hereinafter, feature vector similarity) between the inspection feature vector (IFV) and the over-inspection feature vector (PFV) using measurement indicators such as cosine similarity, L1 distance, L2 distance, and / or Earth Mover distance.

[0377] Additionally, in the embodiment, the computing system (1000) can calculate decision similarity based on the measured feature vector similarity (FVS) and / or raw data similarity (RDS). (S407)

[0378] Here, the decision similarity according to the embodiment may mean the final similarity that specifies the degree of similarity between the inspection image (IIM) and the over-inspection image (PII).

[0379] In detail, in an embodiment, the computing system (1000) can calculate decision similarity by combining the above-described feature vector similarity (FVS) and raw data similarity (RDS) in a predetermined manner.

[0380] As an example, the computing system (1000) can calculate a predetermined weighted sum based on feature vector similarity (FVS) and raw data similarity (RDS), and derive decision similarity based on the weighted sum.

[0381] Alternatively, in an embodiment, the computing system (1000) may calculate decision similarity by selecting either feature vector similarity (FVS) or raw data similarity (RDS).

[0382] That is, depending on the embodiment, the computing system (1000) may calculate decision similarity using either feature vector similarity (FVS) or raw data similarity (RDS) alone.

[0383] Additionally, in the embodiment, the computing system (1000) can detect at least one similar image based on the calculated decision similarity. (S409)

[0384] In detail, in the embodiment, the computing system (1000) can compare the decision similarity calculated as above with a preset threshold (hereinafter, similarity criterion threshold).

[0385] At this time, as an example, the computing system (1000) can determine that the corresponding inspection image (IIM) and inspection image (PII) are similar to each other if the decision similarity is greater than or equal to the similarity criterion threshold.

[0386] That is, the computing system (1000) can detect a matching image (PII) as a similar image if the decision similarity is greater than a similarity criterion threshold.

[0387] On the other hand, as an example, the computing system (1000) can determine that the corresponding inspection image (IIM) and inspection image (PII) are mutually different if the decision similarity is less than the similarity criterion threshold.

[0388] In other words, the computing system (1000) can exclude the matching over-inspection image (PII) from the similar image if the decision similarity is less than the similarity criterion threshold.

[0389] Thus, in the embodiment, the computing system (1000) can detect at least one similar image based on decision similarity.

[0390] That is, in the embodiment, the computing system (1000) can detect at least one image among the inspection images (PII) stored in the inspection database (IDU) that has a similarity level higher than a preset standard with the inspection image (IIM) as a similar image.

[0391] Returning to Figure 10 again, in another embodiment, the computing system (1000) can filter the detected similar images. (S305)

[0392] In detail, in an embodiment, the computing system (1000) can obtain user input for setting whether to perform filtering on at least one detected similar image.

[0393] At this time, in the embodiment, the computing system (1000) may perform a filtering process for at least one similar image when obtaining a user input that sets execution of similar image filtering.

[0394] In more detail, in an embodiment, the computing system (1000) can perform the similar image filtering process described above based on preset filtering conditions.

[0395] Here, the filtering condition according to the embodiment may include a condition in which the skill of the worker who judges whether a given similar image has an outlier is below a preset standard (i.e., skill information matching the similar image is below a preset standard).

[0396] Additionally, the filtering condition may include a condition in which the presence or absence of an outlier judged for a given similar image is incorrect (i.e., the judgment result feedback information matching the similar image is incorrect).

[0397] Additionally, the filtering conditions may include conditions that do not satisfy similar image search conditions according to user settings.

[0398] At this time, similar image search conditions according to the embodiment may include conditions for matching a specific defect type, conditions for matching a specific worker, and / or conditions for matching a specific piece of equipment (e.g., a specific shooting sensor, etc.).

[0399] Additionally, in the embodiment, the computing system (1000) may perform filtering to remove a corresponding similar image if the filtering condition described above is satisfied.

[0400] In this way, in the embodiment, the computing system (1000) can perform filtering to exclude images that do not meet user needs or have a quality reliability below a standard from among similar images detected from the search database.

[0401] Through this, the computing system (1000) can implement an outlier judgment guide and monitoring service that efficiently utilizes more meaningful data by selecting it.

[0402] Meanwhile, in the embodiment, the computing system (1000) can determine whether the number of at least one similar image detected and / or filtered as above (hereinafter, the number of similar images detected) is equal to or greater than a preset number (N: hereinafter, the minimum number of images detected).

[0403] Here, the minimum number of image detections (N) according to the embodiment can be manually set according to user input or automatically set according to a similar reference threshold.

[0404] According to an embodiment, when the minimum number of image detections (N) is automatically set according to a similarity reference threshold, the computing system (1000) can set the minimum number of image detections (N) in inverse proportion to the similarity reference threshold.

[0405] That is, the computing system (1000) can acquire a larger amount of image data as the number of similar images detected is lower based on the similarity criterion.

[0406] Accordingly, the computing system (1000) can check more past judgment records as there are more ambiguous cases in which to determine outliers, thereby confirming multiple judgment opinions.

[0407] At the same time, the computing system (1000) can perform more efficient data processing by reducing the amount of data as the number of cases useful for determining outliers increases.

[0408] Returning again, the computing system (1000) that determines whether the number of similar image detections satisfies the minimum number of image detections (N) as above can perform a similar image additional detection process if the number of similar image detections is less than the minimum number of image detections (N) (i.e., if it does not satisfy the minimum number of image detections (N)).

[0409] Specifically, in the embodiment, the computing system (1000) may re-perform the above-described S303 and / or S305 step process based on the remaining over-inspection images (PII) excluding the previously detected similar images (i.e., the over-inspection images (PII) that are determined to be similar to the inspection image (IIM) and thus detected).

[0410] At this time, in the embodiment, the computing system (1000) can perform a similar image addition detection process by lowering the similarity reference threshold described above by a user-set and / or preset value.

[0411] Thus, in the embodiment, the computing system (1000) can obtain at least one additionally detected similar image (hereinafter, additionally detected image) according to step S303, and can further include it in the similar image corresponding to the inspection image (IIM) and repeat the filtering process according to step S305.

[0412] That is, if the number of detected similar images is less than the preset number, the computing system (1000) can relax the similar image detection criteria to obtain and use additional similar images.

[0413] Therefore, the computing system (1000) can easily secure a sufficient amount of data necessary for the smooth operation of the outlier judgment guide and monitoring service.

[0414] On the other hand, in the embodiment, the computing system (1000) can determine at least one similar guide image based on the determination similarity for each detected similar image if the number of similar image detections is greater than or equal to the minimum number of image detections (N) (i.e., if the minimum number of image detections (N) is satisfied).

[0415] Here, the similar guide image according to the embodiment may mean a similar image that is finally determined to be provided to a user (in the embodiment, a worker) through a display output among at least one similar image detected from a search database.

[0416] In detail, the computing system (1000) as an example can determine whether the similarity of the detected similar images satisfies a predetermined criterion (e.g., within the upper M (M>0)%).

[0417] Additionally, in the embodiment, the computing system (1000) can determine at least one similar image that satisfies the above-described criteria as a similar guide image.

[0418] And in the embodiment, the computing system (1000) can perform the process described below based on at least one determined similar guide image.

[0419] At this time, according to an embodiment, the computing system (1000) may automatically perform the filtering process described above only when the number of similar image detections is greater than or equal to the minimum number of image detections (N).

[0420] Additionally, in an embodiment, the computing system (1000) can provide a filtered similar image. (S307)

[0421] That is, in the embodiment, the computing system (1000) can provide at least one similar guide image determined as above.

[0422] In detail, in an embodiment, the computing system (1000) can display and output the similar guide image and the corresponding judgment details as a predetermined graphic image in conjunction with a monitoring interface (I / F) unit (MIU) or the like.

[0423] FIG. 13 is an example of providing a similar image according to one embodiment of the present invention.

[0424] At this time, referring to FIG. 13, in the embodiment, the computing system (1000) can provide at least one similar guide image (SGI) and the judgment detailed information (JDI) corresponding to each similar guide image (SGI) by matching them with each other.

[0425] Additionally, in the embodiment, the computing system (1000) can provide matching corresponding inspection images (IIMs).

[0426] Specifically, in an embodiment, the computing system (1000) can provide at least one pseudo guide image (SGI) corresponding to an inspection image (IIM) and at least some of the metadata information among various metadata (in an embodiment, information on a responsible worker, information on a judgment result, feedback information on a judgment result, and / or defect type information, etc.) related to the judgment of the presence or absence of an outlier in each pseudo guide image (SGI).

[0427] At this time, depending on the embodiment, the computing system (1000) may further match and provide a decision similarity corresponding to the corresponding similar guide image (SGI).

[0428] In other words, the computing system (1000) can further match and provide a specific decision similarity value (score) showing how similar the corresponding similar guide image (SGI) and the inspection image (IIM) are.

[0429] In this way, in the embodiment, the computing system (1000) can detect at least one over-inspection image (PII) similar to an inspection image (IIM) that a current worker wants to manually determine whether there is an outlier, and provide the detected similar image and data on whether there is an outlier previously determined for the image so that the current worker can easily check them.

[0430] In other words, the computing system (1000) can provide the current worker with the results of a defect judgment made by an existing worker for an image similar to the image whose defect must be currently judged.

[0431] Through this, the computing system (1000) can provide a guide that helps the current worker more easily perform a consistent judgment on the presence or absence of outliers based on past meaningful data and reduce errors.

[0432] Accordingly, the computing system (1000) can effectively improve the overall quality and performance of the vision inspection process.

[0433] FIG. 14 is an example of providing an additional provision image according to one embodiment of the present invention.

[0434] At this time, referring to FIG. 14, according to an embodiment, the computing system (1000) may provide at least one additional provision image (API) according to judgment decision information for each provided similar guide image (SGI).

[0435] Here, the additionally provided image (API) according to the embodiment may mean an image additionally provided when the ratio between the judgment results for each similar guide image (SGI) provided for the inspection image (IIM) does not meet a preset standard.

[0436] In detail, in an embodiment, the computing system (1000) can calculate a ratio between the judgment results (in an embodiment, OK (i.e., no outlier) or NG (i.e., presence of outlier)) for each provided similar guide image (SGI) for the inspection image (IIM).

[0437] For example, the computing system (1000) can calculate a ratio between judgment results, such as 'OK:NG=2:1'.

[0438] Additionally, in the embodiment, the computing system (1000) can determine whether the ratio between the produced judgment results satisfies a preset ratio (e.g., 'OK:NG=1:1', etc.).

[0439] At this time, if the computing system (1000) in the embodiment determines that the preset ratio (hereinafter, required judgment result ratio) is not satisfied, it can set the judgment result information and the number of images (hereinafter, required image attribute information) required to satisfy the required judgment result ratio.

[0440] For example, the computing system (1000) can set required image attribute information requesting one NG similar guide image (SGI) when the ratio between the above-described judgment results is 'OK:NG=2:1', the number of OK similar guide images (SGI) is '2', and the number of NG similar guide images (SGI) is '1', and the ratio of the required judgment results is 'OK:NG=1:1'.

[0441] Additionally, in the embodiment, the computing system (1000) can extract at least one similar image (hereinafter, a selected similar image) that has been filtered (i.e., removed) in the step S305 described above.

[0442] According to an embodiment, if the computing system (1000) determines that the preset ratio is insufficient, it may perform the similar image detection process described above in step S305. A detailed description thereof follows the description in step S305.

[0443] Additionally, in the embodiment, the computing system (1000) can determine an additional provision image (API) based on the extracted selection similar image and / or additional detection image obtained according to the similar image detection process as described above, and the required image attribute information set as described above.

[0444] In detail, in an embodiment, the computing system (1000) may determine at least one image among the selected similar images and / or additional detected images (i.e., next-ranked images) as an additional provided image (API) based on required image attribute information.

[0445] In more detail, in an embodiment, the computing system (1000) can select at least one image that matches the required image attribute information from among the selected similar images and / or additional detected images.

[0446] For example, when 'one NG similar guide image (SGI)' is required according to required image attribute information, the computing system (1000) can select 'one' image whose judgment result information is 'NG' from among the selected similar images and / or additional detected images.

[0447] At this time, if there are multiple images matching the required image attribute information, the computing system (1000) can select images in order of high decision similarity for each image.

[0448] Additionally, in an embodiment, the computing system (1000) may determine at least one selected image as an additional provision image (API).

[0449] And the computing system (1000) can provide at least one additional provided image (API) by including it in a similar guide image (SGI).

[0450] That is, the computing system (1000) can, in conjunction with a monitoring interface (I / F) unit (MIU) or the like, display and output at least one additional provision image (API) and corresponding judgment details (JDI) determined as above and provide them as a predetermined graphic image.

[0451] In other words, the computing system (1000) can further provide, through a display, at least one additional provision image (API) and the corresponding judgment detail information (JDI) for each additional provision image (API) by matching them with each other.

[0452] In this way, in the embodiment, the computing system (1000) can provide additional guide images to compensate for the judgment results of the similar guide images (SGI) provided to the worker when the judgment results are biased or do not meet the preset ratio.

[0453] Thus, the computing system (1000) can provide additional data that can supplement biased data provided, such as only images judged OK are provided, only images judged NG are provided, or the ratio of images judged OK is excessively high, thereby supporting the worker to determine whether there are outliers by referring to more diverse and reasonable data.

[0454] FIG. 15 is an example of a drawing for explaining an alignment method when providing similar images according to one embodiment of the present invention.

[0455] At this time, referring to FIG. 15, according to an embodiment, the computing system (1000) may provide at least one similar guide image (SGI) by listing it according to a predetermined sorting criterion.

[0456] As an embodiment, the computing system (1000) can provide at least one similar guide image (SGI) sorted in order of high decision similarity of the similar guide image (SGI).

[0457] In addition, as an embodiment, the computing system (1000) can provide at least one similar guide image (SGI) sorted in order of high proficiency information of the similar guide image (SGI).

[0458] Accordingly, the computing system (1000) can preferentially refer to a similar guide image (SGI) that is more closely related to the inspection image (IIM) or corresponds to the judgment result by a more skilled worker.

[0459] Thus, the computing system (1000) can provide a guide that can further improve the efficiency and accuracy of workers in determining the presence or absence of outliers.

[0460] Meanwhile, according to an embodiment, the computing system (1000) may further cooperate with a post-inspection processing unit (PPU) to provide judgment result feedback information corresponding to a similar guide image (SGI).

[0461] That is, the computing system (1000) can provide, together with the result of the judgment of whether or not an outlier is present, as judged by a past worker, for each similar guide image (SGI), whether the result is correct or incorrect.

[0462] Thus, the computing system (1000) can induce the current worker to perform a more careful judgment on the presence or absence of an outlier in the inspection image (IIM) when there is a similar guide image (SGI) that has a history of being misjudged in the past.

[0463] Additionally, in the embodiment, the computing system (1000) can obtain judgment result information for the inspection image (IIM). (S309)

[0464] Here, in other words, the judgment result information according to the embodiment may be information specifying the result of the judgment by the worker in charge of the presence or absence of an outlier in a given inspection image (IIM) (here, the current inspection image).

[0465] For example, the judgment result information may be OK (i.e., no outliers) or NG (i.e., outliers exist).

[0466] That is, in the embodiment, the computing system (1000) can obtain data on the presence or absence of an outlier judged by the current operator for the inspection image (IIM) for which the presence or absence of an outlier is to be judged.

[0467] Referring further to FIG. 13, as an embodiment, a computing system (1000) may provide a user interface (OJI: hereinafter, an outlier judgment input interface) that can input judgment result information for an inspection image (IIM) in conjunction with a monitoring interface (I / F) unit (MIU), etc.

[0468] Additionally, the computing system (1000) can obtain judgment result information for the inspection image (IIM) described above based on a user (here, a current worker) input based on a provided outlier judgment input interface (OJI).

[0469] Thus, the computing system (1000) can obtain data on the presence or absence of an outlier judgment result for the current inspection image (IIM) made by the current worker by referring to past judgment records in cases similar to the current inspection image (IIM).

[0470] Additionally, in the embodiment, the computing system (1000) can store and manage the acquired judgment result information. (S311)

[0471] In detail, in the embodiment, the computing system (1000) can store and manage the judgment result information (hereinafter, real-time judgment result information) obtained for the inspection image (IIM) as described above in the inspection database (IDU) by matching it with the corresponding inspection image (IIM).

[0472] At this time, in the embodiment, the computing system (1000) can determine whether to store the inspection image (IIM) based on the determination similarity between the inspection image (IIM) and the similar guide image (SGI).

[0473] In detail, in an embodiment, the computing system (1000) can compare the decision similarity between the inspection image (IIM) and each corresponding similar guide image (SGI) with a preset threshold (hereinafter, image recording threshold).

[0474] At this time, in the embodiment, the computing system (1000) may not store the inspection image (IIM) in the inspection database (IDU) if there is a similar guide image (SGI) having a decision similarity greater than the image recording threshold.

[0475] On the other hand, in the embodiment, the computing system (1000) can store the inspection image (IIM) in the inspection database (IDU) if the decision similarity between the inspection image (IIM) and each similar guide image (SGI) is both below the image recording threshold.

[0476] That is, in the embodiment, if there is a pre-inspection image (PII) having a similarity level higher than a certain threshold with respect to the inspection image (IIM), the computing system (1000) determines that the inspection image (IIM) is likely to be an image that has a tendency to overlap with an existing image already stored in the inspection database (IDU), and thus, in order to exclude duplication, the corresponding inspection image (IIM) may not be stored in the inspection database (IDU).

[0477] Thus, the computing system (1000) can implement an outlier judgment guide and monitoring service that performs data processing and operations more efficiently.

[0478] At this time, according to an embodiment, if there is a similar guide image (SGI) (hereinafter, a duplicate suspected image) having a decision similarity greater than an image recording threshold, the computing system (1000) may determine whether to store the corresponding inspection image (IIM) based on the judgment result information of each of the corresponding duplicate suspected image and the inspection image (IIM).

[0479] In detail, in the embodiment, the computing system (1000) can compare the judgment result information (hereinafter, judgment result inspection information) of the inspection image (IIM) with the judgment result information (hereinafter, judgment result suspicion information) of the duplicate suspected image.

[0480] At this time, in the embodiment, the computing system (1000) can store the corresponding inspection image (IIM) in the inspection database (IDU) even if there are duplicate suspicious images when the judgment result inspection information and the judgment result suspicious information are different from each other.

[0481] For example, if the judgment result inspection information is 'OK' and the judgment result suspicious information is 'NG', the computing system (1000) can store the corresponding inspection image (IIM) in the inspection database (IDU).

[0482] In this way, in the embodiment, even if there is a pre-inspection image (PII) having a similarity higher than a certain threshold with respect to the inspection image (IIM), if the determination of whether there is an outlier determined for the inspection image (IIM) and the determination of whether there is an outlier determined for the pre-inspection image (PII) having a similarity higher than the certain threshold (i.e., a suspected duplicate image) are different, the computing system (1000) may determine that the inspection image (IIM) contains meaningful information different from the pre-inspection image (PII) previously stored in the inspection database (IDU), and may store the inspection image (IIM) in the inspection database (IDU).

[0483] In this way, the computing system (1000) selectively determines whether to store the inspection image (IIM) based on the determination similarity between the inspection image (IIM) and the similar guide image (SGI), thereby minimizing the problem of infinite data increase in the inspection database (IDU) and constructing the inspection database (IDU) based on valuable data containing more meaningful information.

[0484] Above, the method and system for monitoring judgment records for an outlier judgment guide according to one embodiment of the present invention can support improvement of consistency and minimization of errors in real-time outlier judgment by providing a guide for real-time outlier judgment based on past outlier judgment records, thereby having the effect of improving the overall quality and performance of the vision inspection process.

[0485] At this time, the method for monitoring judgment records for an outlier judgment guide according to one embodiment of the present invention and its system can provide a guide for outlier judgment by utilizing a deep learning model specialized in anomaly detection in a patch feature-based learning method, thereby improving the quality of the provided guide and also improving the data processing efficiency for providing the guide.

[0486]

[0487] Meanwhile, the embodiments of the present invention described above may be implemented in the form of program commands that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the computer-readable recording medium may be specially designed and configured for the present invention or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. Hardware devices may be changed into one or more software modules to perform processing according to the present invention, and vice versa.

[0488] The specific implementations described in the present invention are exemplary embodiments and do not limit the scope of the present invention in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the lines connecting or connecting members between components illustrated in the drawings are merely representative of functional connections and / or physical or circuit connections, and may be replaced or represented as various additional functional connections, physical connections, or circuit connections in an actual device. In addition, unless specifically mentioned as “essential,” “important,” etc., a component may not be absolutely necessary for the application of the present invention.

[0489] Although the detailed description of the present invention has been described with reference to preferred embodiments of the present invention, it will be understood by those skilled in the art or having ordinary knowledge in the art that various modifications and changes can be made to the present invention without departing from the spirit and technical scope of the present invention as set forth in the claims below. Accordingly, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be defined by the claims.

[0490] The present invention relates to a method and device for executing a program that photographs a product produced using a machine learning model and determines whether there is an abnormality, and therefore has industrial applicability.

Claims

1. A method for a computing system to perform judgment record monitoring for an outlier judgment guide, A step of acquiring a predetermined inspection image; A step of detecting at least one similar image having a similarity (similarity) higher than a predetermined standard with the acquired inspection image; A step of providing judgment details, which are information specifying the presence or absence of an outlier in each of the detected similar images and the similar images; A step of obtaining first judgment result information, which is information specifying the result of judgment on the presence or absence of an outlier in the above inspection image; and A step of storing the inspection image and the first judgment result information in a database according to the similarity between the inspection image and the similar image, respectively. A method for monitoring judgment records for outlier judgment guides.

2. In paragraph 1, The step of providing the above judgment details is: A step of providing at least one piece of information among information on a responsible worker who performed the judgment on the presence or absence of an outlier in the similar image and information on a second judgment result that specifies the result of the judgment on the presence or absence of an outlier in the similar image. A method for monitoring judgment records for outlier judgment guides.

3. In paragraph 1, The step of detecting the above similar image is, A step of measuring raw data similarity between at least one over-examination image stored in the database and the inspection image. A method for monitoring judgment records for outlier judgment guides.

4. In paragraph 3, The step of detecting the above similar image is, A step of obtaining a test feature vector, which is a feature vector for the test image, and a test feature vector, which is a feature vector for the over-test image, based on a predetermined image feature extraction model; It further includes a step of measuring the similarity between feature vectors, which is the similarity between the acquired inspection feature vector and the over-inspection feature vector. A method for monitoring judgment records for outlier judgment guides.

5. In paragraph 4, The above image feature extraction model is, It includes an outlier detection model, a deep learning model specialized in anomaly detection using a patch feature-based learning method. A method for monitoring judgment records for outlier judgment guides.

6. In paragraph 4, The step of detecting the above similar image is, A step of calculating a decision similarity that specifies the final similarity between the inspection image and the over-inspection image based on the raw data similarity and the feature vector similarity is further included. A method for monitoring judgment records for outlier judgment guides.

7. In paragraph 6, The step of detecting the above similar image is, Further comprising a step of detecting the similar image based on the above decision similarity and a preset similarity criterion threshold. A method for monitoring judgment records for outlier judgment guides.

8. In paragraph 7, The step of detecting the above similar image is, A step of determining whether the number of similar images detected above satisfies a preset number; If the above preset number is insufficient, the step of additionally detecting the similar image is further included. A method for monitoring judgment records for outlier judgment guides.

9. In paragraph 8, The step of additionally detecting the above similar image is: A step of adjusting the above similar reference threshold, A step of detecting the similar image based on the adjusted similarity reference threshold. A method for monitoring judgment records for outlier judgment guides.

10. In paragraph 6, The step of providing the above similar image and judgment details is: comprising a step of further providing the above decision similarity; A method for monitoring judgment records for outlier judgment guides.

11. In paragraph 2, The step of providing the above similar image and judgment details is: A step of additionally detecting the similar image according to the ratio between the second judgment result information for at least one similar image, A step of further providing an additional provided image, which is the additionally detected similar image, is included. A method for monitoring judgment records for outlier judgment guides.

12. In paragraph 2, The step of providing the above similar image and judgment details is: A step of aligning the detected similar images based on at least one of the information on the responsible worker and the similarity between the inspection image and each of the similar images. A method for monitoring judgment records for outlier judgment guides.

13. In paragraph 2, The step of saving to the above database is: Further comprising a step of storing the inspection image and the first judgment result information in a database according to whether the first judgment result information and the second judgment result information are identical. A method for monitoring judgment records for outlier judgment guides.

14. In paragraph 1, Further comprising a step of filtering the detected similar images according to preset conditions. A method for monitoring judgment records for outlier judgment guides.

15. In paragraph 1, The step of obtaining the above inspection image is: A step of acquiring the inspection image based on an outlier detection model, which is a deep learning model specialized for anomaly detection using a patch feature-based learning method. A method for monitoring judgment records for outlier judgment guides.

16. At least one vision inspection unit; At least one Inspection Monitoring Processing Unit; At least one monitoring interface (I / F) unit; and Contains at least one Inspection Database; A step of controlling the above vision inspection unit to obtain a predetermined inspection image, A step of controlling the inspection monitoring processing unit to detect at least one similar image having a similarity (Similarity) higher than a predetermined standard with the acquired inspection image, A step of controlling the monitoring interface (I / F) unit to provide judgment details, which are information specifying the presence or absence of anomalies in each of the detected similar images and the judgment details for the presence or absence of anomalies in each of the similar images, A step of controlling the above monitoring interface (I / F) unit to obtain first judgment result information, which is information that determines whether there is an abnormality in the inspection image; A step of controlling the above inspection monitoring processing unit to store the inspection image and the first judgment result information in the inspection database according to the similarity between the inspection image and each of the similar images. A judgment record monitoring system for outlier judgment guidance.

17. At least one vision inspection unit; At least one Inspection Monitoring Processing Unit; At least one monitoring interface (I / F) unit; and Contains at least one Inspection Database; Control the above vision inspection unit to obtain a predetermined inspection image, Controlling the above inspection monitoring processing unit to detect at least one similar image having a similarity (Similarity) higher than a predetermined standard with the acquired inspection image, Controlling the above monitoring interface (I / F) unit to provide judgment details, which are information specifying the presence or absence of anomalies in the detected similar images and each of the similar images, Control the above monitoring interface (I / F) unit to obtain first judgment result information, which is information that determines whether there is an abnormality in the inspection image, Controlling the above inspection monitoring processing unit to store the inspection image and the first judgment result information in the inspection database according to the degree of similarity between the inspection image and each of the similar images. Computing device.

Images