Apparatus, method, and non-transitory computer-readable recording medium

Abstract

A method carried out by means of an apparatus comprising a memory including a plurality of banks, and a processor comprises: acquiring, by the processor, a plurality of anomaly scores corresponding to a plurality of image regions segmented from an inspection image of a target object by using the plurality of banks; acquiring, by the processor, an equipment anomaly value by performing auto-encoding on the plurality of anomaly scores for each of a predetermined number of inspection images accumulated in a time series; and detecting, by the processor, an anomaly of inspection equipment on the basis of the acquired equipment anomaly value.

Description

Device, method and non-transitory computer-readable recording medium

[0001] The embodiments relate to devices, methods and non-transitory computer-readable recording media.

[0002] Typically, in industrial fields such as manufacturing, inspection equipment is used to determine whether a target, such as a product, is defective.

[0003] Inspection equipment must be able to determine whether a target is defective under optimal environmental conditions to improve the reliability of its defect detection.

[0004] If the environmental conditions of the inspection equipment itself change and cause a malfunction in the inspection equipment, there is a problem that the accuracy of determining whether there is a defect is significantly reduced.

[0005] Accordingly, there is an urgent need to develop a device that can detect abnormalities in the inspection equipment itself.

[0006] The present invention aims to solve the above-mentioned and other problems.

[0007] Another object of the present invention is to provide a device, method and non-transitory computer-readable recording medium capable of detecting an abnormality in an inspection device itself using an inspection image of a target object.

[0008] The technical problems of the embodiment are not limited to those described in this article, but include those that can be understood through the description of the invention.

[0009] According to a first aspect of the embodiment to achieve the above or other purposes, a method by a device including a memory including a plurality of banks and a processor includes: a step in which the processor obtains a plurality of abnormality scores corresponding to a plurality of image areas segmented from an inspection image of a target object using the plurality of banks; a step in which the processor performs auto-encoding on the plurality of abnormality scores for each of a predetermined number of inspection images accumulated in time series, thereby obtaining an equipment abnormality; and a step in which the processor detects an abnormality of the inspection equipment based on the obtained equipment abnormality.

[0010] Each of the above multiple banks may include a plurality of feature data. The plurality of feature data may be extracted using deep learning for each of the plurality of image regions for each of the plurality of inspection images acquired from the target object when the inspection equipment is operating normally.

[0011] The step of obtaining the above plurality of abnormality scores may include a step of the processor performing wavelet transform to divide the inspection image for the object into a plurality of image regions.

[0012] The step of obtaining the above plurality of abnormality scores may include a step in which the processor extracts a plurality of feature data using deep learning for each of the plurality of divided image regions; and a step in which the processor compares the plurality of feature data extracted for each of the plurality of divided image regions with the plurality of feature data of each of the plurality of banks to obtain an abnormality score for each of the plurality of divided image regions.

[0013] The step of obtaining an abnormality score for each of the plurality of divided image regions may include a step of the processor obtaining an average value of distances between the plurality of feature data extracted for each of the plurality of divided image regions and the plurality of feature data of each of the plurality of banks as an abnormality score for each of the plurality of divided image regions.

[0014] The step of obtaining the above plurality of abnormal scores may include a step in which the processor assigns different weights to the abnormal scores for each of the plurality of segmented image regions.

[0015] The plurality of divided image regions may include different frequency components. The plurality of divided image regions may include a first image region including a first frequency component and a second image region including a second frequency component higher than the first frequency component. The step of obtaining the plurality of abnormality scores may include a step in which the processor assigns a higher weight to an abnormality score obtained corresponding to the first image region than to an abnormality score obtained corresponding to the second image region.

[0016] The step of obtaining the above equipment anomaly may include: a step of the processor compressing and restoring a plurality of anomaly scores for each of a predetermined number of time-series accumulated inspection images, and obtaining a plurality of restored values ​​obtained by restoring them; and a step of obtaining the difference between the accumulated value of the plurality of anomaly scores and the accumulated value of the plurality of restored values ​​as the equipment anomaly.

[0017] The step of acquiring the above equipment anomaly may include a step of the processor periodically acquiring the equipment anomaly by filling in the inspection images after the last rank as much as the first rank inspection images are missing from the predetermined number of inspection images accumulated in the time series.

[0018] According to a second aspect of the embodiment to achieve the above or other purposes, the device comprises: a memory including a plurality of banks; and a processor; wherein the processor obtains a plurality of abnormality scores corresponding to a plurality of image areas segmented from an inspection image of a target object using the plurality of banks, performs auto-encoding on the plurality of abnormality scores for each of a predetermined number of inspection images accumulated in time series, thereby obtaining an equipment abnormality, and detects an abnormality of the inspection equipment based on the obtained equipment abnormality.

[0019] Each of the above multiple banks may include a plurality of feature data. The plurality of feature data may be extracted using deep learning for each of the plurality of image regions for each of the plurality of inspection images acquired from the target object when the inspection equipment is operating normally.

[0020] The above processor can perform wavelet transform to divide the inspection image of the object into a plurality of image regions.

[0021] The processor extracts a plurality of feature data using deep learning for each of the plurality of divided image regions, and compares the plurality of feature data extracted for each of the plurality of divided image regions with the plurality of feature data of each of the plurality of banks to obtain an ideal score for each of the plurality of divided image regions.

[0022] The processor can obtain an average value of distances between the plurality of feature data extracted for each of the plurality of divided image regions and the plurality of feature data of each of the plurality of banks as an ideal score for each of the plurality of divided image regions.

[0023] The above processor can assign different weights to the abnormality scores for each of the plurality of divided image regions.

[0024] The plurality of divided image regions may include different frequency components. The plurality of divided image regions may include a first image region including a first frequency component and a second image region including a second frequency component higher than the first frequency component. The processor may assign a higher weight to an abnormality score obtained corresponding to the first image region than to an abnormality score obtained corresponding to the second image region.

[0025] The above processor obtains a plurality of restored values ​​by compressing and restoring the accumulated values ​​of a plurality of abnormal scores for each of a predetermined number of time-series accumulated inspection images, and can obtain the difference between the accumulated values ​​of the plurality of abnormal scores and the accumulated values ​​of the plurality of restored values ​​as the equipment abnormality value.

[0026] The above processor can periodically obtain the equipment abnormality by filling in inspection images after the last priority as much as the first priority inspection image is missing from the predetermined number of inspection images accumulated in the time series.

[0027] The above inspection equipment may include a camera for acquiring the inspection image of the target object. The processor may detect an abnormality in the camera.

[0028] According to a third aspect of the embodiment to achieve the above or other purposes, a non-transitory computer-readable recording medium storing commands includes: a step of obtaining a plurality of abnormality scores corresponding to a plurality of image areas segmented from an inspection image of a target object using a plurality of banks; a step of performing auto-encoding on a plurality of abnormality scores for each of a predetermined number of inspection images accumulated in time series, thereby obtaining an equipment abnormality; and a step of detecting an abnormality of inspection equipment based on the obtained equipment abnormality.

[0029] The effects of the device, method and non-transitory computer-readable recording medium according to the embodiment are described as follows.

[0030] According to at least one of the embodiments, a wavelet transform may be performed to segment an inspection image into multiple image regions containing different frequency components. Different weights may be assigned to the abnormality scores obtained for each of the multiple segmented image regions. Accordingly, the defect characteristics of the inspection equipment itself may be reflected more than the defect characteristics of the object, thereby preventing misrecognition of the abnormality detection of the inspection equipment, thereby improving the abnormality detection performance of the inspection equipment.

[0031] According to at least one of the embodiments, the average of the distances between multiple feature data extracted from real-time acquired inspection images and feature data in the bank can be obtained as an abnormality score. In this case, even if the closest feature data has the maximum distance from one of the multiple feature data, the obtained abnormality score does not fluctuate significantly, thereby preventing false positives in the inspection equipment.

[0032] According to at least one of the embodiments, auto-encoding may be performed on a plurality of abnormality scores corresponding to each of a plurality of image regions segmented from each of a predetermined number of time-series accumulated inspection images, thereby performing equipment anomaly detection. Accordingly, by performing auto-encoding on a plurality of abnormality scores corresponding to each of a plurality of image regions segmented from each of a single inspection image, a decrease in the accuracy of the obtained equipment anomaly can be prevented.

[0033] Further scope of applicability of the embodiments will become apparent from the detailed description below. However, since various changes and modifications within the spirit and scope of the embodiments will be readily apparent to those skilled in the art, it should be understood that the detailed description and specific embodiments, such as preferred embodiments, are given by way of example only.

[0034] Figure 1a illustrates an artificial intelligence (AI) device according to one embodiment of the present invention.

[0035] Figure 1b illustrates an AI server according to one embodiment of the present invention.

[0036] Figure 2 illustrates an image of a target object being captured using inspection equipment according to an embodiment.

[0037] Figure 3 illustrates a device according to an embodiment.

[0038] Figure 4 illustrates in detail the wavelet transform unit and the ideal score estimation unit of Figure 3.

[0039] Figure 5 is a flowchart illustrating a method according to an embodiment.

[0040] The sizes, shapes, and dimensions of components depicted in the drawings may differ from the actual components. Furthermore, even if the same components are depicted with different sizes, shapes, and dimensions across drawings, this is merely an example within the drawings, and the same components may have the same sizes, shapes, and dimensions across drawings.

[0041] Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers and redundant descriptions thereof will be omitted. The suffixes 'module' and 'part' used for components in the following description are given or used interchangeably in consideration of the ease of writing the specification, and do not have distinct meanings or roles in themselves. In addition, the attached drawings are intended to make it easier to understand the embodiments disclosed in the present specification, and the technical ideas disclosed in the present specification are not limited by the attached drawings. In addition, when an element such as a layer, region, or substrate is referred to as existing 'on' another element, this includes that it may be directly on the other element or that other intermediate elements may exist therebetween.

[0042]

[0043] Figure 1a illustrates an AI device according to one embodiment of the present invention.

[0044] The AI ​​device (100) may be a device that constructs a model capable of learning using artificial intelligence such as machine learning, trains the constructed model using previously collected data, distributes the trained model so that it can be used in the field, applies the distributed model to the field, trains using newly generated data, and performs all procedures for operating the model.

[0045] Referring to FIG. 1A, the AI ​​device (100) may include a communication unit (110), an input unit (120), a learning processor (130), a sensing unit (140), an output unit (150), a memory (170), and a processor (180).

[0046] The communication unit (110) can transmit and receive data with external devices such as other AI devices or AI servers (200 in FIG. 1B) using wired or wireless communication technology. For example, the communication unit (110) can transmit and receive sensor information, user input, learning models, control signals, etc. with external devices.

[0047] At this time, the communication technologies used by the communication unit (110) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth (Bluetooth™), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

[0048] The input unit (120) can obtain various types of data.

[0049] At this time, the input unit (120) may include a camera for inputting a video signal, a microphone for receiving an audio signal, a user input interface for receiving information from a user, etc. Here, the camera or microphone may be treated as a sensor, and the signal obtained from the camera or microphone may be referred to as sensing data or sensor information.

[0050] The input unit (120) can obtain input data to be used when obtaining output using learning data and learning models for model learning. The input unit (120) can also obtain unprocessed input data, in which case the processor (180) or learning processor (130) can extract input features as preprocessing for the input data.

[0051] The learning processor (130) can train a model composed of an artificial neural network using learning data. Here, the trained artificial neural network may be referred to as a learning model. The learning model can be used to infer result values ​​for new input data other than the learning data, and the inferred values ​​can be used as a basis for making decisions regarding certain actions.

[0052] At this time, the running processor (130) can perform AI processing together with the running processor (240) of the AI ​​server (200 in FIG. 1b).

[0053] At this time, the running processor (130) may include a memory integrated or implemented in the AI ​​device (100). Alternatively, the running processor (130) may be implemented using a memory (170), an external memory directly coupled to the AI ​​device (100), or a memory maintained in an external device.

[0054] The sensing unit (140) can obtain at least one of internal information of the AI ​​device (100), information about the surrounding environment of the AI ​​device (100), and user information using various sensors.

[0055] At this time, the sensors included in the sensing unit (140) include a proximity sensor, a light sensor, an acceleration sensor, a magnetic sensor, a gyro sensor, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, a light sensor, a microphone, a lidar, a radar, etc.

[0056] The output interface (150) can generate output related to visual, auditory, or tactile sensations.

[0057] At this time, the output unit (150) may include a display unit that outputs visual information, a speaker that outputs auditory information, a haptic module that outputs tactile information, etc.

[0058] The memory (170) can store data that supports various functions of the AI ​​device (100). For example, the memory (170) can store input data, learning data, learning models, learning history, etc. obtained from the input unit (120).

[0059] The processor (180) may determine at least one executable operation of the AI ​​device (100) based on information determined or generated using a data analysis algorithm or a machine learning algorithm. Then, the processor (180) may control components of the AI ​​device (100) to perform the determined operation.

[0060] To this end, the processor (180) can request, retrieve, receive or utilize data from the running processor (130) or memory (170), and control components of the AI ​​device (100) to execute at least one of the executable operations, a predicted operation or an operation determined to be desirable.

[0061] At this time, if the processor (180) requires connection of an external device to perform a determined operation, it can generate a control signal for controlling the external device and transmit the generated control signal to the external device.

[0062] The processor (180) can obtain intent information for user input and determine the user's requirements based on the obtained intent information.

[0063] The processor (180) can collect history information including the operation details of the AI ​​device (100) or the user's feedback on the operation, and store the information in the memory (170) or the learning processor (130), or transmit the information to an external device such as an AI server (200). The collected history information can be used to update the learning model.

[0064] The processor (180) can control at least some of the components of the AI ​​device (100) to drive an application program stored in the memory (170). Furthermore, the processor (180) can operate two or more of the components included in the AI ​​device (100) in combination to drive the application program.

[0065]

[0066] Figure 1b illustrates an AI server according to one embodiment of the present invention.

[0067] Referring to FIG. 1b, the AI ​​server (200) may refer to a device that trains an artificial neural network using a machine learning algorithm or utilizes a trained artificial neural network. Here, the AI ​​server (200) may be comprised of multiple servers to perform distributed processing, and may be defined as a 5G network. In this case, the AI ​​server (200) may be included as part of the AI ​​device (100) and may perform at least a portion of the AI ​​processing.

[0068] The AI ​​server (200) may include a communication unit (210), memory (230), a learning processor (240), and a processor (260).

[0069] The communication unit (210) can transmit and receive data with an external device such as an AI device (100).

[0070] The memory (230) may include a model storage unit (231). The model storage unit (231) may store a model (or artificial neural network, 231a) being learned or learned through the learning processor (240).

[0071] A learning processor (240) can train an artificial neural network (231a) using learning data. The learning model can be used while mounted on the AI ​​server (200) of the artificial neural network, or can be mounted on an external device such as an AI device (100).

[0072] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (230).

[0073] The processor (260) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0074]

[0075] Meanwhile, when using inspection equipment to detect defects in a target object (i.e., a product), there may be anomalies in the inspection equipment itself, such as changes in environmental conditions. The embodiment utilizes artificial intelligence technology to simply, quickly, and accurately detect defects in the inspection equipment itself from inspection images of the target object. Accordingly, immediate follow-up measures can prevent a decline in the accuracy of defect detection in the target object, thereby enhancing the reliability of product defect identification.

[0076]

[0077] Figure 2 illustrates an image of a target object being captured using inspection equipment according to an embodiment.

[0078] As illustrated in FIG. 2, the inspection equipment (300) may include a camera (301). The camera (3010) may be included in the input unit (120) illustrated in FIG. 1A, but is not limited thereto.

[0079] The camera (301) can capture an inspection image (400) of a target object (280) placed on a stage (270). The inspection equipment (300) can determine whether the target object (280) is defective through a predetermined processing procedure based on the inspection image (400). The stage (270) can be moved by a conveyor belt or a robot arm.

[0080] Meanwhile, while the operation of determining whether or not the target object (280) is defective using the inspection equipment (300) is being performed, an abnormality may occur in the inspection equipment (300) itself. For example, an abnormal phenomenon may occur in which the functional efficiency of the camera (301) is reduced, such as when the camera (301) is out of focus or the surrounding lighting environment changes. In this way, if there is an abnormality in the inspection equipment (300) itself, the accuracy of the defect in the target object (280) may decrease. For example, a target object (280) that is actually determined to be defective when the inspection equipment (300) is normal may be incorrectly determined to be a good product if there is an abnormality in the inspection equipment (300) itself.

[0081] In this embodiment, when there is a problem with the inspection equipment (300) itself, the problem with the inspection equipment (300) can be detected simply, quickly, and accurately. Hereinafter, the embodiment will be described in more detail with reference to FIGS. 3 to 5.

[0082] Figure 3 illustrates a device according to an embodiment.

[0083] A device according to an embodiment can detect an abnormality in inspection equipment. The device according to an embodiment can be equipped in the inspection equipment (300) illustrated in FIG. 2. The abnormality detection device of the inspection equipment (300) according to an embodiment can be the processor (180, 260) illustrated in FIGS. 1A and 1B. The abnormality detection device of the inspection equipment (300) according to an embodiment can be equipped separately from the processor (180, 260) illustrated in FIGS. 1A and 1B.

[0084] In the following description, if the operation is not by the component illustrated in FIG. 3, it may be operated by the processor (180, 260) illustrated in FIG. 1a and FIG. 1b.

[0085] Referring to FIGS. 1A to 3, the abnormality detection device of the inspection equipment (300) according to the embodiment may include an inspection image (400) acquisition unit (310), a low-pass filter (320), a wavelet transform unit (330), an abnormality score estimation unit (340), and an auto-encoder (350). The abnormality detection device of the inspection equipment (300) according to the embodiment may include a memory (360). Alternatively, the memory (360) may be provided separately from the abnormality detection device of the inspection equipment (300) according to the embodiment. The memory (360) may include the memories (170, 230) illustrated in FIGS. 1A and 1B.

[0086] The abnormality score estimation unit (340) and / or the autoencoder (350) may be artificial intelligence models trained using deep learning. For example, the abnormality score estimation unit (340) may be referred to as a first artificial intelligence model, and the autoencoder (350) may be referred to as a second artificial intelligence model. The abnormality score estimation unit (340) and / or the autoencoder (350) may be stored in the memory (360). In this case, the processor (180, 260) may load the first artificial intelligence model and / or the second artificial intelligence model from the memory (360), and obtain an abnormality score using the first artificial intelligence model and obtain an equipment abnormality value (d) using the second artificial intelligence model.

[0087] Meanwhile, the inspection image (400) acquisition unit (310) can acquire an inspection image (400) by capturing an image of a target object (280) from a camera (301) mounted on the inspection equipment (300). The inspection image (400) can be acquired by digitally processing the target object (280) captured by the camera (301). For example, the inspection image (400) for the target object (280) can be acquired every hour (t, …, t+n).

[0088] The low pass filter (320) can alleviate or remove noise from the inspection image (400). Noise may be introduced into the inspection image (400) obtained by the camera (301) capturing the object (280). Therefore, the noise included in the inspection image (400) can be alleviated or removed by the low pass filter (320). If the inspection image (400) has little or no noise, the low pass filter (320) may be omitted. For example, the low pass filter (320) may include a Gaussian filter, but is not limited thereto.

[0089]

[0090] Meanwhile, the inspection image (400) may include various frequency components. Typically, information for determining whether a target object (280) is defective is reflected relatively more in high-frequency components, and information for determining the characteristics or abnormalities of the inspection equipment (300) itself may be reflected relatively more in low-frequency components.

[0091] If an abnormality score is obtained from an inspection image (400) by an abnormality score estimation unit (340) without considering that the degree of reflection of a defect in the target object (280) or a defect in the inspection equipment (300) itself varies depending on the frequency component, the abnormality detection performance of the inspection equipment (300) may be significantly reduced.

[0092] Accordingly, the abnormality detection device of the inspection equipment (300) according to the embodiment may include a wavelet transform unit (330).

[0093] The wavelet transform unit (330) can perform wavelet transform to divide the inspection image (400) of the target object (280) into multiple image regions. In FIG. 3, the inspection image (400) is shown to be divided into four image regions (410 to 440), but it may be divided into fewer or more image regions. For example, the wavelet transform unit (330) may use the Haar transform method, but is not limited thereto.

[0094] The plurality of image areas (410 to 440) may include different frequency components. For example, the first image area (410) may include low frequency components, and the second image area (420) may include higher frequency components than the low frequency components of the first image area (410). The third image area (430) may include higher frequency components than the frequency components of the second image area (420), and the fourth image area (440) may include higher frequency components than the frequency components of the third image area (430), for example, high frequency components.

[0095] As will be explained later, the abnormal score estimation unit (340) estimates the abnormal score corresponding to each of the plurality of image areas (410 to 440), for example, the abnormal score at time t (s in FIG. 4). 00 , s 10 , s 20 , s 30 ) is obtained, the frequency components of each of the multiple image areas (410 to 440) are considered and the corresponding abnormality score (s 00 , s 10 , s 20 , s 30 ) may be given different weights (ω0, ω1, ω2, ω3 in Fig. 4). For example, the abnormality score (s) obtained corresponding to the first image area (410) with the lowest frequency component 00) may be given the highest weight (ω0). For example, the ideal score (s) obtained corresponding to the second image area (420) 10 ) is the ideal score (s) obtained corresponding to the first image area (410). 00 ) may be given a lower weight (ω1). For example, the abnormality score (s) obtained corresponding to the third image area (430) 20 ) is the ideal score (s) obtained corresponding to the second image area (420). 10 ) may be given a lower weight (ω2). For example, the abnormality score (s) obtained corresponding to the fourth image area (440) 30 ) can be given the lowest weight (ω3). The ideal score (s) obtained corresponding to each of the multiple image areas (410 to 440) 00 , s 10 , s 20 , s 30 ) can be changed. Therefore, the abnormality score (s) obtained corresponding to the first image area (410) including the low-frequency component 00 ) is given a relatively high weight (ω0) and an abnormality score (s) obtained corresponding to the fourth image area (440) containing high-frequency components 30 ) may be given a relatively low weight (ω3). Accordingly, the abnormality score (s) obtained corresponding to the first image area (410) including the low-frequency component 00 ) so that the defect characteristics of the inspection equipment (300) itself are reflected more than the defect characteristics of the target object (280), thereby preventing misrecognition of the defect detection of the inspection equipment (300), and thus improving the defect detection performance of the inspection equipment (300).

[0096]

[0097] Referring to FIGS. 3 and 4, the abnormality score estimation unit (340) can obtain an abnormality score corresponding to a plurality of image areas (410 to 440) segmented from the inspection image (400). For example, when the inspection image (400) is segmented into four image areas (410 to 440), the abnormality score estimation unit (340) can obtain four abnormality scores corresponding to the four image areas (410 to 440).

[0098] The number of abnormal score estimation units (340) may be equal to the number of image areas (410 to 440) segmented from the inspection image (400). For example, when the inspection image (400) is segmented into a first image area (410), a second image area (420), a third image area (430), and a fourth image area (440), the abnormal score estimation unit (340) may include a first abnormal score estimation unit (341), a second abnormal score estimation unit (342), a third abnormal score estimation unit (343), and a fourth abnormal score estimation unit (344). Accordingly, when five or more image areas are provided, five or more abnormal score estimation units (340) may also be provided.

[0099] The first abnormal score estimation unit (341) estimates the abnormal score (s) corresponding to the first image area (410). 00 ) is obtained, and the second abnormal score estimation unit (342) obtains the abnormal score (s) corresponding to the second image area (420). 10 ) can be obtained. The third abnormal score estimation unit (343) can obtain an abnormal score (s) corresponding to the third image area (430). 20 ) is obtained, and the fourth abnormal score estimation unit (344) obtains the abnormal score (s) corresponding to the fourth image area (440). 30 ) can be obtained.

[0100] The first abnormal score estimation unit (341), the second abnormal score estimation unit (342), the third abnormal score estimation unit (343), and the fourth abnormal score estimation unit (344) may each include the same components. That is, the first abnormal score estimation unit (341), the second abnormal score estimation unit (342), the third abnormal score estimation unit (343), and the fourth abnormal score estimation unit (344) may each include a feature extraction unit (341a, 342a, 343a, 344a), a neighbor search unit (341b, 342b, 343b, 344b), an abnormal score calculation unit (341c, 342c, 343c, 344c), and a weighting unit (341d, 342d, 343d, 344d).

[0101] The feature extraction units (341a, 342a, 343a, 344a) can extract a plurality of feature data for each of the plurality of image regions (410 to 440) using deep learning. For example, the feature extraction units (341a, 342a, 343a, 344a) can perform average pooling on each of the plurality of image regions (410 to 440) and extract a plurality of feature data from the average pooled feature data using deep learning. The extracted plurality of feature data can be referred to as patch feature data.

[0102] Deep learning can be used interchangeably with terms such as neural network, network function, and neural network. Neural networks can include convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), Q-networks, U-networks, and Siamese networks.

[0103] The neighbor search units (341b, 342b, 343b, 344b) can compare the extracted plurality of feature data with the plurality of feature data of the banks (361 to 364) to select the closest feature data. The banks (361 to 364) may be provided in the memory (360). The number of banks (361 to 364) may be equal to the number of image areas (410 to 440). For example, the memory (360) may include a first bank (361) corresponding to the first image area (410), a second bank (362) corresponding to the second image area (420), a third bank (363) corresponding to the third image area (430), and a fourth bank (364) corresponding to the fourth image area (440).

[0104] At least one feature data that has been pre-trained (or learned) using deep learning may be stored in each of the first bank (361), the second bank (362), the third bank (363), and the fourth bank (364). For example, when the inspection equipment (300) operates normally, numerous inspection images (400) are acquired, and each of the acquired inspection images (400) may be divided into a first image region (410), a second image region (420), a third image region (430), and a fourth image region (440) through wavelet transform. At this time, the inspection image (400) may include good or defective information about the target object (280). At least one feature data may be extracted and stored in the first bank (361) by training (or learning) each of the numerous first image regions (410) using deep learning. Using deep learning, each of the numerous second image areas (420) can be trained to extract at least one feature data and store it in the second bank (362). Using deep learning, each of the numerous third image areas (430) can be trained to extract at least one feature data and store it in the third bank (363). Using deep learning, each of the numerous fourth image areas (440) can be trained to extract at least one feature data and store it in the fourth bank (364).

[0105] Meanwhile, coreset subsampling may be performed to reduce the feature data stored in the first bank (361), the second bank (362), the third bank (363), and the fourth bank (364) and to enhance representativeness, but this is not limited thereto.

[0106] For example, the number of feature data extracted from each of the first image area (410), the second image area (420), the third image area (430), and the fourth image area (440) segmented for the inspection image (400) acquired through the camera (301) of the inspection equipment (300) in real time may be greater than the number of feature data stored in each of the first bank (361), the second bank (362), the third bank (363), and the fourth bank (364).

[0107] The ideal score calculation unit (341c, 342c, 343c, 344c) calculates the ideal score (s) based on the closest feature data. 00 , s 10 , s 20 , s 30 ) can be obtained. For example, the maximum distance among the distances between the closest feature data and the plurality of feature data of the banks (361 to 364) can be obtained as an ideal score.

[0108] However, if the maximum distance is obtained as an abnormal score, a misrecognition of an abnormality in the inspection equipment (300) may occur. If the maximum distance is obtained as an abnormal score, if auto-encoding is performed for the abnormal score, a misrecognition problem may occur in which the equipment abnormality exceeds the set value and is detected as an abnormality in the inspection equipment.

[0109] To solve this problem, the abnormality score calculation unit (341c, 342c, 343c, 344c) calculates the average value of the distances between the extracted plurality of feature data and the feature data of the banks (361 to 364) as the abnormality score (s 00 , s 10 , s 20 , s 30 ) can be obtained. The average value of the distances between the extracted plurality of feature data and the feature data of the banks (361 to 364) is an ideal score (s 00 , s 10 , s 20 , s 30), even if the closest feature data has the maximum distance from one of the multiple feature data, the obtained ideal score (s) 00 , s 10 , s 20 , s 30 ) does not have a large fluctuation, so that misrecognition of abnormality detection of the inspection equipment (300) can be prevented. That is, according to an embodiment, by artificially generating a strong artifact in a local area of ​​the inspection image randomly by the defect occurrence rate in the production process and performing data augmentation, even if the outlier score increases due to a defect in some inspection images, misrecognition of an abnormality of the inspection equipment (300) can be prevented from occurring.

[0110] The average value of the distances between the above extracted multiple feature data and the feature data of the banks (361 to 364) is an ideal score (s 00 , s 10 , s 20 , s 30 ) can also be obtained.

[0111] The weighting unit (341d, 342d, 343d, 344d) obtains the above-mentioned ideal score (s 00 , s 10 , s 20 , s 30 ) can be given weights (ω0, ω1, ω2, ω3) set according to the frequency components of the corresponding image area (410 to 440).

[0112]

[0113] Meanwhile, the auto encoder (350) can obtain an equipment outlier (d) by performing auto-encoding on a plurality of outlier scores corresponding to each of a plurality of image areas (410 to 440) segmented from each of a predetermined number of time-series accumulated inspection images (400).

[0114] For one inspection image (400), an abnormality score (s) corresponding to each of the first image area (410), the second image area (420), the third image area (430), and the fourth image area (440) 00 , s 10 , s 20 , s 30 ) can be acquired. These inspection images (400) can be accumulated every hour. When the accumulated inspection images reach a predetermined number or a predetermined section (t to t+n), auto-encoding is performed on a plurality of abnormality scores corresponding to each of a plurality of image areas (410 to 440) divided from each of the inspection images accumulated in the predetermined number, thereby acquiring an equipment abnormality value (d).

[0115] The autoencoder (350) may include an input interface (351), an encoder (352), a decoder (353), and an output interface (354).

[0116] The input interface (351) can act as a buffer and accumulate inspection images (400) until the number of inspection images (400) reaches a predetermined number. The encoder (352) can compress a plurality of abnormality scores corresponding to a plurality of image areas each divided from a predetermined number of inspection images accumulated in time series. The decoder (353) can restore the compressed plurality of abnormality scores. The output interface (354) can obtain a plurality of restoration values ​​restored and output by the decoder (353).

[0117] The processor (180, 260) can obtain the difference between the accumulated value (S) of multiple abnormal scores and the accumulated value (S') of multiple restored values ​​as the equipment abnormal value (d), as shown in mathematical expression 1.

[0118]

[0119] The processor (180, 260) can detect an abnormality in the inspection equipment (300) based on the equipment abnormality value (d). For example, if the equipment abnormality value (d) is greater than a set value, it is determined that there is an abnormality in the inspection equipment (300) itself, and the abnormality in the inspection equipment (300) can be notified. The notification of an abnormality in the inspection equipment (300) can be made using voice or text, but is not limited thereto.

[0120] Meanwhile, the auto-encoder (350) of the embodiment can learn based on inspection images obtained when the inspection equipment (300) is operating normally. In addition, the auto-encoder (350) of the embodiment can learn by including defective information, rather than good information, about the target object (280).

[0121] As described above, auto-encoding can be performed on multiple abnormality scores corresponding to multiple image regions segmented from each of a predetermined number of time-series accumulated inspection images, thereby obtaining an equipment abnormality value (d). Accordingly, auto-encoding can be performed on multiple abnormality scores corresponding to multiple image regions segmented from each single inspection image, thereby preventing a decrease in the accuracy of the obtained equipment abnormality value (d).

[0122]

[0123] Meanwhile, the processor (180, 260) can periodically obtain the equipment outlier (d) by filling in the lower priority inspection images (400) as the upper priority inspection images (400) are omitted from among a predetermined number of inspection images (400) accumulated in time series. For example, if the number of inspection images (400) used as input for the auto-encoder (350) is 10, if the first inspection image (400) that is the earliest in time is omitted, the 11th inspection image (400) after the 10th inspection image (400) is filled in, so that 10 inspection images (400) composed of the 2nd to 11th inspection images (400) can be maintained. For example, if the number of inspection images (400) used as inputs of the auto encoder (350) is 20, when the first to third inspection images (400) that are the earliest in time are omitted, the 21st to 23rd inspection images (400) after the 20th inspection image (400) are filled in, so that 20 inspection images (400) composed of the 4th to 23rd inspection images (400) can be maintained. The equipment anomaly (d) can be periodically acquired for a predetermined number of inspection images (400) accumulated in time series. For example, the equipment anomaly (d) can be acquired on an hourly basis for a predetermined number of inspection images (400) accumulated in time series.

[0124]

[0125] Figure 5 is a flowchart illustrating a method according to an embodiment.

[0126] Referring to FIGS. 1A to 5, the processor (180, 260) may obtain an inspection image (400) (S510), perform low-pass filtering (320) (S520), and perform wavelet transform (S530). The processor (180, 260) may obtain an abnormality score (S540), accumulate inspection images (400) in time series until a predetermined number is reached (S550), and obtain an equipment abnormality (d) (S560). The processor (180, 260) may compare the equipment abnormality (d) with a set value (TH) (S570) and notify an abnormality of the inspection equipment (300) (S580).

[0127] Specifically, the processor (180, 260) can acquire an inspection image (400) (S510). A camera (301) mounted on the inspection equipment (300) can capture an image of a target object (280) and digitally process the captured image of the target object (280) to acquire an inspection image (400).

[0128] The processor (180, 260) can perform a low-pass filter (320) (S520). Through the low-pass filter (320), noise included in the inspection image (400) can be alleviated or removed.

[0129] The processor (180, 260) can perform wavelet transformation (S530). Through wavelet transformation, the inspection image (400) can be divided into multiple image areas (410 to 440).

[0130] The processor (180, 260) can obtain an abnormality score (S540). The abnormality score can be obtained by comparing the multiple feature data extracted for each of the multiple images with the multiple feature data of the multiple banks (361 to 364). The method for obtaining the abnormality score has been previously described, so a detailed description will be omitted.

[0131] The processor (180, 260) can sequentially accumulate inspection images (400) until a predetermined number is reached (S550). The number of inspection images (400) to be input into the input interface (351) of the auto-encoder (350) can be set. The inspection images (400) can be accumulated until the preset number of inspection images (400) is reached.

[0132] The processor (180, 260) can acquire an equipment anomaly value (d) (S560). By performing auto-encoding on a plurality of anomaly scores corresponding to each of a plurality of image areas (410 to 440) segmented from each of a predetermined number of time-series accumulated inspection images (400), the equipment anomaly value (d) can be acquired.

[0133] For example, a plurality of abnormality scores corresponding to a plurality of image areas (410 to 440) each segmented from a predetermined number of inspection images (400) accumulated in time series by an encoder (352) can be compressed. The compressed plurality of abnormality scores can be restored by a decoder (353) to obtain a plurality of restored values. As shown in mathematical expression 1, the difference between the accumulated value (S) of the plurality of abnormality scores and the accumulated value (S') of the plurality of restored values ​​can be obtained as the equipment abnormality value (d).

[0134] The processor (180, 260) can compare the equipment abnormality (d) with the set value (TH) (S570) and notify of an abnormality in the inspection equipment (300) (S580). If the equipment abnormality (d) is greater than the set value (TH), it is determined that there is an abnormality in the inspection equipment (300), and notification information regarding the abnormality in the inspection equipment (300) can be sent.

[0135]

[0136] Meanwhile, the embodiment may also be applied to a non-transitory computer-readable recording medium storing commands. That is, the non-transitory computer-readable recording medium may include a step of obtaining a plurality of abnormality scores corresponding to a plurality of image areas segmented from an inspection image of a target object using a plurality of banks, a step of performing auto-encoding on a plurality of abnormality scores for each of a predetermined number of inspection images accumulated in time series to obtain an equipment abnormality, and a step of detecting an abnormality of the inspection equipment based on the obtained equipment abnormality.

[0137]

[0138] The above detailed description should not be construed as limiting in any respect and should be considered illustrative only. The scope of the embodiments should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalency range of the embodiments are intended to be included within the scope of the embodiments.

Claims

1. A method by a device including a memory and a processor including a plurality of banks, A step in which the above processor obtains a plurality of abnormality scores corresponding to a plurality of image areas segmented from an inspection image of the target object using the plurality of banks; A step of performing auto-encoding on multiple abnormality scores for each of a predetermined number of inspection images accumulated in time series by the above processor to obtain equipment abnormality values; and The step of detecting an abnormality in the inspection equipment based on the acquired equipment abnormality value by the above processor; method.

2. In paragraph 1, Each of the above multiple banks, Contains multiple feature data, The above multiple feature data are, When the above inspection equipment is operating normally, each of the multiple image areas for each of the multiple inspection images acquired from the target object is extracted using deep learning. method.

3. In paragraph 1, The steps for obtaining the above multiple ideal scores are: A step of the processor performing wavelet transform to divide the inspection image of the object into a plurality of image regions; method.

4. In paragraph 1, The steps for obtaining the above multiple ideal scores are: A step in which the processor extracts multiple feature data for each of the multiple divided image regions using deep learning; and A step of obtaining an ideal score for each of the plurality of divided image regions by comparing the plurality of feature data extracted for each of the plurality of divided image regions with the plurality of feature data of each of the plurality of banks, wherein the processor comprises; method.

5. In paragraph 4, The step of obtaining an ideal score for each of the above divided plurality of image regions is: A step in which the processor obtains an average value of distances between the plurality of feature data extracted for each of the plurality of divided image regions and the plurality of feature data of each of the plurality of banks as an ideal score for each of the plurality of divided image regions; method.

6. In paragraph 1, The steps for obtaining the above multiple ideal scores are: A step in which the processor assigns different weights to the ideal scores for each of the plurality of divided image regions; method.

7. In paragraph 6, The above divided plurality of image areas contain different frequency components, The above divided plurality of image areas include a first image area including a first frequency component and a second image area including a second frequency component higher than the first frequency component, The steps for obtaining the above multiple ideal scores are: The step of the processor including: assigning a higher weight to the abnormality score obtained corresponding to the first image image area than to the abnormality score obtained corresponding to the second image area; method.

8. In paragraph 1, The steps to obtain the above equipment ideals are: A step of the above processor compressing and restoring a plurality of abnormality scores for each of a predetermined number of time-series accumulated inspection images, and obtaining a plurality of restored values ​​obtained by restoring them; and A step of obtaining the difference between the cumulative value of the plurality of abnormal scores and the cumulative value of the plurality of restoration values ​​as the equipment abnormal value; including; method.

9. In paragraph 8, The steps to obtain the above equipment ideals are: The step of the processor periodically obtaining the equipment abnormality by filling in the inspection images after the last priority as much as the first priority inspection images are missing from the predetermined number of inspection images accumulated in the time series; including; method.

10. Memory comprising multiple banks; and a processor; including; The above processor, Obtaining multiple abnormality scores corresponding to multiple image areas segmented from an inspection image of a target object using multiple banks, Auto-encoding is performed on multiple abnormal scores for each of a predetermined number of sequentially accumulated inspection images to obtain equipment abnormalities. Detecting abnormalities in inspection equipment based on the above-mentioned acquired equipment abnormalities. device.

11. In paragraph 10, Each of the above multiple banks, Contains multiple feature data, The above multiple feature data are, When the above inspection equipment is operating normally, each of the multiple image areas for each of the multiple inspection images acquired from the target object is extracted using deep learning. device.

12. In paragraph 10, The above processor, Splitting the inspection image of the object into multiple image regions by performing wavelet transform, device.

13. In paragraph 10, The above processor, For each of the above divided image regions, multiple feature data are extracted using deep learning. By comparing the extracted plurality of feature data for each of the divided plurality of image regions with the plurality of feature data of each of the divided plurality of banks, an ideal score is obtained for each of the divided plurality of image regions. device.

14. In paragraph 13, The above processor, The average value of the distances between the plurality of feature data extracted for each of the plurality of divided image regions and the plurality of feature data of each of the plurality of banks is obtained as an ideal score for each of the plurality of divided image regions. device.

15. In paragraph 10, The above processor, Assigning different weights to the ideal scores for each of the above-mentioned multiple image regions. device.

16. In paragraph 15, The above divided plurality of image areas contain different frequency components, The above divided plurality of image areas include a first image area including a first frequency component and a second image area including a second frequency component higher than the first frequency component, The above processor, The ideal score obtained corresponding to the first image image area is given a higher weight than the ideal score obtained corresponding to the second image area. device.

17. In paragraph 10, The above processor, Obtain multiple restored values ​​by compressing and restoring the accumulated values ​​of multiple abnormal scores for each of the predetermined number of inspection images accumulated in the above time series, The difference between the cumulative value of the above multiple abnormal scores and the cumulative value of the above multiple restoration values ​​is obtained as the above equipment abnormal value. device.

18. In paragraph 17, The above processor, The above equipment abnormality is periodically obtained by filling in the inspection images after the last priority as the first priority inspection images are omitted from the above time-series accumulated predetermined number of inspection images. device.

19. In paragraph 10, The above inspection equipment includes a camera for obtaining the inspection image of the target object, The above processor, Detecting abnormalities in the above camera, device.

20. In a non-transitory computer-readable recording medium storing commands, A step of obtaining multiple abnormality scores corresponding to multiple image areas segmented from an inspection image of a target object using multiple banks; A step of performing auto-encoding on multiple abnormality scores for each of a predetermined number of sequentially accumulated inspection images to obtain equipment abnormality values; and A step of detecting an abnormality in the inspection equipment based on the above-mentioned acquired equipment abnormality; including; A non-transitory computer-readable recording medium.

Images