Apparatus and method for determining class of data, and recording medium having command stored therein

Abstract

A technique for determining a class of data is disclosed. A system according to one aspect of the present invention may comprise: an optical inspection device for receiving reflected light of light emitted to a component disposed on a substrate to acquire first data on the component, inspecting whether the first data corresponds to a first class, and transmitting an inspection result to a review station; a review station for transmitting a re-inspection request for the inspection result to a re-classification device in response to receiving the inspection result from the optical inspection device; and the re-classification device which acquires the first data, determines whether a model for the first class is stored in one or more memories, determines whether the first data corresponds to the first class by using the model according to the determination that the model for the first class is stored, determines whether the first data corresponds to the first class by using second data corresponding to the first class according to the determination that the model for the first class is not stored, and transmits information indicating whether the first data corresponds to the first class to the review station.

Description

Recording medium recording a device, method and command for determining a class of data

[0001] The present disclosure relates to a technique for determining a class of data.

[0002] Surface Mount Technology (SMT) is used to place one or more components on a board. Numerous components are connected to pads on the board, and for each component to properly connect to the pads, it may be necessary to place it in the correct location on the board. An automated optical inspection (AOI) device uses optical measurement technology to inspect whether each component is placed in the correct location on the board. If a component is placed in the correct location on the board, other components can be placed on the board.

[0003] However, if a component is not positioned correctly on the board, the board may be sent to a review station for re-inspection based on the results of the automated optical inspection (AOI) device. At the review station, an operator can re-inspect components determined to be incorrectly bonded to the board by the AOI device, making a final determination as to whether an error has occurred.

[0004] At least one embodiment of the present disclosure provides a technique for more precisely determining the class of data.

[0005] At least one embodiment of the present disclosure can reduce false calls in which a non-defective substrate is determined to be defective.

[0006] At least one embodiment of the present disclosure can reduce the number of substrates to be re-inspected at a review station.

[0007] At least one embodiment of the present disclosure can generate a model for each class and determine the class of input data using the generated model.

[0008] At least one embodiment of the present disclosure can determine the class of input data even if a model for the class corresponding to the input data has not been created.

[0009] A system according to one aspect of the present disclosure may include an optical inspection device that receives reflected light of light irradiated on a component placed on a substrate to obtain first data about the component, examines whether the first data corresponds to a first class, and transmits the examination result to a review station; a review station that, in response to receiving the examination result from the optical inspection device, transmits a request for re-examination of the examination result to a reclassification device; and a reclassification device that obtains the first data, determines whether a model for the first class is stored in one or more memories, determines whether the first data corresponds to the first class using the model based on a determination that the model for the first class is stored, determines whether the first data corresponds to the first class using second data corresponding to the first class based on a determination that the model for the first class is not stored, and transmits information indicating whether the first data corresponds to the first class to the review station.

[0010] In one embodiment, the optical inspection device can determine whether the first data corresponds to the first class, and, based on a determination that the first data corresponds to the first class, transmit the first data and information indicating that the first data corresponds to the first class to the reclassification device, and, based on a determination that the first data does not correspond to the first class, transmit the first data and information indicating that the first data does not correspond to the first class to the review station.

[0011] In one embodiment, in determining whether the first data corresponds to the first class using the model, the reclassification device may include a device that calculates a probability that the first data corresponds to the first class using the model, determines whether the probability is greater than or equal to a predetermined reference probability, and determines that the first data corresponds to the first class if the probability is greater than or equal to the predetermined reference probability, and determines that the first data does not correspond to the first class if the probability is less than the predetermined reference probability.

[0012] In one embodiment, the predetermined reference probability may be determined using an average of a minimum value of a probability of one or more data determined to correspond to the first class and a maximum value of a probability of one or more data determined not to correspond to the first class.

[0013] In one embodiment, the reclassification device may calculate a similarity between the first data and the second data using a machine learning algorithm based on a determination that the model for the first class is not stored, and determine whether the first data corresponds to the first class based on the similarity between the first data and the second data.

[0014] In one embodiment, the machine learning algorithm may include ArcFace (Additive Angular Margin Loss).

[0015] In one embodiment, the machine learning algorithm can be trained using a plurality of randomly generated virtual data and the second data.

[0016] In one embodiment, the plurality of virtual data may be generated by adjusting the size, font, and thickness of letters in one or more real data stored in one or more memories, and adding blur and defects.

[0017] In one embodiment, the plurality of virtual data may be generated using at least one algorithm among Generative Adversarial Networks (GAN) and stable diffusion.

[0018] In one embodiment, the reclassification device obtains a learning request for the first data, and upon obtaining the learning request, determines whether the model for the first class is stored in the one or more memories, and upon determining that the model for the first class is not stored, adds the first data to a learning queue for generating or learning the model, and upon determining that the model for the first class is stored, adds the first data to the learning queue if the model for the first class satisfies a predetermined condition.

[0019] In one embodiment, the reclassification device may add the first data to the learning queue when it is determined that the model for the first class is not stored, and if the number of one or more pieces of data corresponding to the first class stored on the one or more memories is greater than a predetermined value and the first data does not exist in the learning queue, the first data may be added to the learning queue.

[0020] In one embodiment, the predetermined condition may be a condition that is satisfied when the number of times the model determines that data that does not correspond to the first class corresponds to the first class is greater than a first threshold, or when the number of times the model determines that data that corresponds to the first class corresponds to not correspond to the first class is greater than a second threshold.

[0021] In one embodiment, the model can be trained using a data set including data corresponding to one or more classes similar to the first class and randomly selected data.

[0022] In one embodiment, the proportion of data corresponding to one or more classes similar to the first class among the data set may be less than or equal to the proportion of the randomly selected data among the data set.

[0023] In one embodiment, the review station further includes a display, and the reclassification device transmits information to the review station indicating that it is not possible to determine whether the first data corresponds to the first class based on a determination that the model for the first class and the second data corresponding to the class are not stored, and the review station can output a phrase indicating that it is not possible to determine whether the first data corresponds to the first class on the display.

[0024] In one embodiment, the model may include a feature extractor and a classifier.

[0025] In one embodiment, the feature extractor is commonly applied to models stored in the one or more memories, and the classifier can be learned based on different information for each model stored in the one or more memories.

[0026] In one embodiment, the feature extractor can be trained using a plurality of randomly generated virtual data.

[0027] In one embodiment, the feature extractor and classifier operate on a GPU (Graphics Processing Unit), and the reclassification device loads a classifier included in a model of a class used with a frequency greater than a predetermined value from the memory to a cache on the GPU, and determines whether data corresponds to a class using the feature extractor and the classifier loaded into the cache.

[0028] In one embodiment, the reclassification device can select a least-used classifier from among one or more classifiers loaded into the cache and remove it from the cache.

[0029] A method according to one aspect of the present disclosure is performed in a reclassification device including one or more processors and one or more memories storing instructions to be executed by the one or more processors, the method comprising: obtaining first data indicating a component to be placed on a substrate; determining whether a model for a first class corresponding to the first data is stored in one or more memories; determining whether the first data corresponds to the first class using the model based on a determination that the model for the first class is stored; determining whether the first data corresponds to the first class using second data corresponding to the first class based on a determination that the model for the first class is not stored; and transmitting information indicating whether the first data corresponds to the first class to a review station.

[0030] According to at least one embodiment of the present disclosure, a technique may be provided that can more precisely determine the class of data.

[0031] According to at least one embodiment of the present disclosure, the misjudgment of determining a defect-free substrate as defective can be reduced.

[0032] According to at least one embodiment of the present disclosure, the number of substrates to be re-inspected at a review station can be reduced.

[0033] According to at least one embodiment of the present disclosure, a model for each class can be generated and the class of input data can be determined using the generated model.

[0034] According to at least one embodiment of the present disclosure, the class of input data can be determined even if a model for the class corresponding to the input data is not created.

[0035] The effects according to the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description of the specification.

[0036] FIG. 1 is a block diagram of an electronic device according to one embodiment of the present disclosure.

[0037] FIG. 2 is a block diagram of a model according to one embodiment of the present disclosure.

[0038] FIG. 3a illustrates a process of learning a machine learning algorithm according to one embodiment of the present disclosure.

[0039] FIG. 3b illustrates a process for determining whether input data corresponds to a class according to one embodiment of the present disclosure.

[0040] FIG. 4 illustrates virtual data according to one embodiment of the present disclosure.

[0041] FIG. 5 illustrates a comparison data set generated to learn input data according to one embodiment of the present disclosure.

[0042] FIG. 6 illustrates a re-inspection process at a review station according to one embodiment of the present disclosure.

[0043] FIG. 7 is a block diagram of a model operating on a GPU according to one embodiment of the present disclosure.

[0044] FIG. 8 illustrates an embodiment of determining a reference probability in an electronic device according to one embodiment of the present disclosure.

[0045] FIG. 9 is a flowchart of a method for determining whether first data corresponds to a first class in an electronic device according to one embodiment of the present disclosure.

[0046] FIG. 10 is a flowchart of a method for generating or retraining a model for a first class in an electronic device according to one embodiment of the present disclosure.

[0047] FIG. 11 is a flowchart of a method for an electronic device to generate a model for a first class according to an embodiment of the present disclosure.

[0048] FIG. 12 is a flowchart of a method for an electronic device to relearn a first model for a first class according to an embodiment of the present disclosure.

[0049] FIGS. 13 and 14 illustrate a system for determining a class of data according to one embodiment of the present disclosure.

[0050] The various embodiments described in this document are exemplified for the purpose of clearly explaining the technical concept of the present disclosure and are not intended to limit it to a specific embodiment. The technical concept of the present disclosure includes various modifications, equivalents, alternatives, and embodiments selectively combining all or part of each embodiment described in this document. Furthermore, the scope of the technical concept of the present disclosure is not limited to the various embodiments presented below or the specific descriptions thereof.

[0051] Terms used in this document, including technical or scientific terms, unless otherwise defined, may have the meaning commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0052] The expressions "includes," "may include," "comprises," "may have," "have," and "may have" used in this document imply the presence of a target feature (e.g., a function, operation, or component), but do not exclude the presence of other additional features. In other words, such expressions should be understood as open-ended terms that imply the possibility of including other embodiments.

[0053] The singular forms used in this document may include the plural form unless the context clearly indicates otherwise, and this also applies to the singular forms set forth in the claims.

[0054] The expressions "first," "second," or "first", "second", etc. used in this document, unless the context indicates otherwise, are used to refer to multiple similar objects and to distinguish one object from another, and do not limit the order or importance among the objects.

[0055] As used herein, the expressions "A, B, and C", "A, B, or C", "A, B, and / or C", or "at least one of A, B, and C", "at least one of A, B, or C", "at least one of A, B, and / or C", etc., may refer to each of the listed items or all possible combinations of the listed items. For example, "at least one of A or B" may refer to (1) at least one A, (2) at least one B, (3) at least one A and at least one B.

[0056] The expression "based on" as used in this document is used to describe one or more factors that influence the decision, act of judgment, or action described in the phrase or sentence containing the expression, and this expression does not exclude additional factors that influence the decision, act of judgment, or action.

[0057] As used herein, the expression that a component (e.g., a first component) is “connected” or “connected” to another component (e.g., a second component) may mean that the component is directly connected or connected to the other component, as well as connected or connected via a new other component (e.g., a third component).

[0058] The expression "configured to" as used in this document can mean "set to do", "having the ability to do", "modified to do", "made to do", "capable of doing", etc., depending on the context. The expression is not limited to the meaning of "specifically designed in hardware", and for example, a processor configured to perform a specific operation can mean a generic-purpose processor that can perform the specific operation by executing software.

[0059] Hereinafter, various embodiments of the present disclosure will be described with reference to the attached drawings. In the attached drawings and the description of the drawings, identical or substantially equivalent components may be assigned the same reference numerals. Furthermore, in the description of the various embodiments below, duplicate descriptions of identical or corresponding components may be omitted, but this does not mean that the corresponding components are not included in the embodiments.

[0060] FIG. 1 is a block diagram of an electronic device according to one embodiment of the present disclosure. Hereinafter, the electronic device may be used in the same sense as a re-classification engine (RCE) in the present specification. According to one embodiment, the electronic device (100) may include a memory (110), a processor (120), and / or a communication interface (130). In some embodiments, at least one of these components of the electronic device (100) may be omitted, or another component may be added to the electronic device (100). In some embodiments, additionally or alternatively, some of the components may be implemented in an integrated manner, or may be implemented as a single or multiple entities.

[0061] At least some of the components inside and outside the electronic device (100) are connected to each other through a bus, GPIO (general purpose input / output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface), and can exchange data and / or signals.

[0062] The processor (120) may control at least one component of the electronic device (100) connected to the processor (120) by running software. In addition, the processor (120) may perform various operations related to the present disclosure, such as calculations, processing, data generation, and processing. In addition, the processor (120) may load data from or store data in the memory (110). In one embodiment, the processor (120) may receive an image of each component placed on the substrate as input data and determine whether the component is placed in the correct position on the substrate. A component may refer to one or more parts placed on the substrate.

[0063] The memory (110) can store various data. The data stored in the memory (110) is data acquired, processed, or used by at least one component of the electronic device (100), and may include software. The memory (110) may include volatile and / or non-volatile memory.

[0064] The communication interface (130) can perform wireless or wired communication between the electronic device (100) and a server or between the electronic device (100) and another external electronic device. For example, the communication interface (130) can perform wireless communication according to a method such as LTE (long-term evolution), LTE-A (LTE Advance), CDMA (code division multiple access), WCDMA (wideband CDMA), WiBro (Wireless Broadband), WiFi (wireless fidelity), Bluetooth (Bluetooth), NFC (near field communication), GPS (Global Positioning System), or GNSS (global navigation satellite system). For example, the communication interface (130) can perform wired communication according to a method such as USB (universal serial bus), HDMI (high definition multimedia interface), RS-232 (recommended standard 232), or POTS (plain old telephone service). The communication interface (130) may be omitted depending on the embodiment.

[0065] In one embodiment, the electronic device (100) can communicate with an external device (101) through a server or directly. The external device (101) can be the same type of device as or different from the electronic device (100). All or part of the operations performed in the electronic device (100) can also be performed in the external device (101). In one embodiment, the electronic device (100) can transmit the result of the performed operation to the external device (101). The external device (101) can perform subsequent operations using the transmitted data, etc. According to one embodiment, the external device (101) can be a server in a review station. According to one embodiment, the automatic optical inspection device can obtain an image of the inspection target by irradiating light or patterned light formed by an illumination unit in the automatic optical inspection device to the inspection target and receiving light reflected from the inspection target (e.g., a PCB board or component) by an imaging unit. Afterwards, the automatic optical inspection device can perform a first inspection on the image of the inspection target according to the preset reference information, and determine whether the inspection target is good or bad, that is, whether the inspection target is good (GOOD) or bad (NG). At this time, if the inspection target is determined to be good, the inspection target is transferred to the next line in the SMT line, and if the inspection target is determined to be bad, the image of the inspection target can be re-inspected within the automatic optical inspection device or transmitted to the server of the review station to perform re-inspection through the server.

[0066] For example, during the first inspection, optical character recognition (OCR) or optical character verification can be used to determine whether the characters to be inspected are recognized in the inspection image and judged as good or bad, and during the re-inspection, the similarity of the images can be judged.

[0067] That is, the review station server can receive the inspection results for data from the electronic device (100) and display them on a display so that an operator within the review station can re-inspect the inspection results. Based on the re-inspection results for the data, the review station server can ultimately determine whether the data corresponds to a class. The re-inspection of the data performed at the review station can be performed manually by the operator.

[0068] The processor (120) can obtain input data regarding components placed on a substrate. A component may refer to one or more parts placed on the substrate. Each component may have a predetermined position on the substrate. For example, a first component should be placed at a first position on the substrate, and a second component should be placed at a second position on the substrate. The data may refer to an image including information about the shape and size of the component, letters, numbers, or symbols written on an inspection target such as a component or a PCB, or the like. Alternatively, the data may refer to an image including a crack shape of the component, a foreign substance, a solder fillet, a solder bridge, a lifted lead, a component or a lead offset, etc.

[0069] According to one embodiment, the processor (120) may perform an operation of determining a class of acquired data. Each data may correspond to one class. A class is a data classification corresponding to a specific component, may be created for each component, and may include one or more data corresponding to one component. For example, a first class corresponding to a first component may include first data and second data for the first component, and a second class corresponding to a second component may include third data and fourth data for the second component. According to one embodiment, the data may be an image for the component. For example, the first data and the second data may be images for the first component, and the third data and the fourth data may be images for the second component. Hereinafter, for convenience of explanation, the input data will be described as data corresponding to the first class.

[0070] The processor (120) can check whether a model for a class corresponding to the acquired input data is stored in the memory (110). In response to acquiring the input data, the processor (120) can check whether a model for a first class is stored in the memory (110). The model may refer to an algorithm (e.g., OC-CNN (One Class Convolutional Neural Network)) for determining whether the input data corresponds to a specific class, and different models may be generated and stored for each class. For example, if a first model for a first class is stored in the memory (110), the processor (120) can use the first model to determine that the first data corresponds to the first class and determine that the third data does not correspond to the first class.

[0071] According to one embodiment, the model for each class can calculate the probability that input data corresponds to the class. For example, the probability that the first input data corresponds to the first class can be calculated as 0.6, and the probability that the second input data corresponds to the first class can be calculated as 0.9. The processor (120) can determine a reference probability for determining whether the data corresponds to the class, and compare the probability calculated by the model with the reference probability to determine whether the input data corresponds to the corresponding class. If the probability calculated by the model is greater than or equal to the reference probability, the processor (120) can determine that the input data corresponds to the corresponding class, and if the probability calculated by the model is less than the reference probability, the processor (120) can determine that the input data does not correspond to the corresponding class. For example, if the processor (120) determines the reference probability to be 0.8, the first input data can be determined not to correspond to the first class, and the second input data can be determined to correspond to the first class. The processor (120) can accumulate and store the judgment results for the plurality of data input as described above. That is, the model can accumulate and store the probability produced for each input data and the results of determining whether it corresponds to the corresponding class.

[0072] According to one embodiment, the processor (120) may determine a reference probability of each class based on the accumulated judgment results. The processor (120) may calculate an average of the minimum value among the probabilities calculated for data corresponding to the corresponding class and the maximum value among the probabilities calculated for data not corresponding to the corresponding class. The processor may determine a higher value between the average value and a predetermined threshold probability as the reference probability. For example, when determining whether input data corresponds to the first class, the probability of the first input data determined not to correspond to the first class may be 0.4, the probability of the second input data may be 0.6, the probability of the third input data determined to correspond to the first class may be 0.9, the probability of the fourth input data may be 0.98, and the threshold probability may be determined to be 0.7. At this time, the highest probability calculated for the input data determined not to correspond to the first class is 0.6 for the second input data, and the lowest probability calculated for the input data determined to correspond to the first class is 0.9 for the third input data. Since the average of the two values ​​is 0.75, which is higher than the critical probability of 0.7, the processor (120) can determine the reference probability as 0.75. On the other hand, if the average of the two determined values ​​is lower than 0.7, the processor (120) can determine the reference probability as 0.7.

[0073] If the first model is not stored in the memory (110), the processor (120) can check whether one or more data corresponding to the first class is stored in the memory (110). If one or more data corresponding to the first class is stored, the processor (120) can use the stored data to determine whether the input data corresponds to the first class. In other words, even if the first model is not generated, the processor (120) can use the stored data to determine whether the input data corresponds to the first class. According to one embodiment, the processor (120) can use a machine learning algorithm (e.g., Metric Learning, ArcFace (Additive Angular Margin Loss)) to determine whether the input data corresponds to the first class. The Metric Learning algorithm is a machine learning algorithm that calculates the similarity between two input data to determine whether the two input data are similar. Even if the first model is not generated, the processor (120) can use the Metric Learning algorithm to determine whether the input data corresponds to the first class. For example, the processor (120) may calculate the similarity between first data or second data corresponding to the first class and input data, and if the calculated similarity is greater than a predetermined value, determine that the input data corresponds to the first class.

[0074] On the other hand, if data corresponding to the first class is not stored, the processor (120) may output information indicating that the class of the first data cannot be determined. For example, the processor (120) may output a phrase indicating that the class of the first data cannot be determined (e.g., "Model not found") to an external device, and the external device may be a display.

[0075] Hereinafter, a method for generating a model for each class will be described. For convenience of explanation, the description will assume that data corresponding to the first class has been received. The processor (120) can obtain a learning request for input data from the learning server. The learning server is a component introduced in the electronic device (100) for learning input data, and upon receiving the input data, can transmit a request for initial learning or a request for fine tuning to the processor.

[0076] Upon receiving a learning request, the processor (120) may determine whether a first model is stored in the memory (110). If the first model is not stored, the processor (120) may add input data to a learning queue to generate a model. In one embodiment, the processor (120) may sequentially learn data included in the learning queue to generate a model for a class. This model generation process corresponds to an initial learning process, and if not many data are input and thus not many models are generated, most learning requests from the learning server may be requests for the initial learning process. In one embodiment, if a predetermined number (e.g., 50 or 100) or more of data corresponding to the first class is stored and no data corresponding to the first class already exists in the learning queue, the processor (120) may add the input data to the learning queue for generating the first model. This is because a predetermined number of data or more is required to generate a model with high accuracy, and if input data for generating the first model already exists in the learning queue, there is no need for redundant learning. According to one embodiment, if a predetermined number or more of data corresponding to the first class is stored, but data corresponding to the first class already exists in the learning queue, or if there is no data corresponding to the first class in the learning queue but data corresponding to the first class is less than the predetermined number, the processor may not add input data to the learning queue even if it receives a learning request.

[0077] If the first model is stored, the processor (120) may add the input data to the training queue in order to retrain the currently generated model using the input data. This process may correspond to fine tuning. According to one embodiment, the processor (120) may determine to retrain the current model based on the number of errors generated in the results of determining whether each data corresponds to the first class using the first model. For example, the processor (120) may determine to retrain the first model based on the number of first errors that determine data that does not correspond to the first class as corresponding to the first class and the number of second errors that determine data that corresponds to the first class as not corresponding to the first class, among the results of determining whether the input data corresponds to the first class using the first model. For example, the processor (120) may determine to retrain the first model if the number of first errors exceeds a predetermined value (e.g., 20) or the number of second errors exceeds a predetermined value (e.g., 100).

[0078] The processor (120) may generate a comparative data set to learn input data. In order to increase the accuracy of the generated model, the processor (120) may generate a comparative data set including data having similar characteristics to the input data (hereinafter, “similar data”). Similar data refers to data in which the difference between the input data and the overall size and the size of the letters included in each data is less than a predetermined value, or in which the number of strokes of the letters included in each data is less than a predetermined number. For example, if the input data is “188,” the comparative data set may be generated with similar data having similar visual characteristics, such as “100,” “168,” “186,” and “180.” The processor (120) may use the comparative data set to learn the input data, so that the generated model can more accurately determine whether the input data corresponds to a class.

[0079] In one embodiment, the processor (120) may determine the proportion of similar data in the comparison data set. For example, the processor (120) may determine the proportion of similar data in the comparison data set as a predetermined value (e.g., 25%), and the remainder may be determined as random data. The processor (120) may extract features of the input data using the randomly determined data, and may further extract features of the input data by comparing the similar data with the input data. Through this, a model capable of more accurately determining data corresponding to a class can be created.

[0080] The electronic device (100) according to various embodiments of the present disclosure may be a device of various forms. For example, the electronic device (100) may be a portable communication device, a computer device, a portable multimedia device, a wearable device, a home appliance device, an AOI positioned at a mounter or a post-reflow stage in an SMT line, a review station linked to an AOI, a smart factory monitoring system linked to an SMT line, or a device according to a combination of one or more of the above-described devices. The electronic device (100) of the present disclosure is not limited to the above-described devices.

[0081] FIG. 2 is a block diagram of a model according to one embodiment of the present disclosure. Referring to FIG. 2, the model may include a feature extractor (210), a classifier (220), and a softmax (230). Different models may be generated for each class, and the models may be generated based on input data (200) as described above in FIG. 1. According to one embodiment, multiple models may share the same feature extractor (210) but include different classifiers (220). For example, a first model for a first class may include a common feature extractor (210) and a first classifier, and a second model for a second class may include a common feature extractor (210) and a second classifier.

[0082] The feature extractor (210) can extract features of the data (200) to determine whether the input data (200) corresponds to a first class. The feature extractor (210) can be trained using a plurality of input data (200) or virtual data. The feature extractor (210) is configured to extract features of the input data (200) and needs to be trained using a large amount of data. The feature extractor (210) can be trained using a plurality of virtual data generated based on the input data. The virtual data may not be actual data, but may be data generated for training the feature extractor (210). According to one embodiment, the virtual data used to train the feature extractor (210) may be generated by a processor.

[0083] The classifier (220) can output a value calculated to indicate whether the input data (200) corresponds to the first class based on the features of the data (200) extracted from the feature extractor (210). The classifier (220) can be trained using a plurality of data corresponding to the first class. For example, the classifier (220) can output a value indicating whether the features of the input data (200) and the features of the first class are similar.

[0084] Softmax (230) is an activation function that normalizes the values ​​input from the classifier (220) to values ​​between 0 and 1 and outputs them. Softmax (230) processes the values ​​obtained from the classifier (220) and can output the probability that the input data (200) corresponds to a specific class.

[0085] FIG. 3A illustrates a process of learning a machine learning algorithm according to one embodiment of the present disclosure. FIG. 3B illustrates a process of determining whether input data corresponds to a class according to one embodiment of the present disclosure. FIGS. 3A and 3B illustrate a process of determining a class of input data (320) using a machine learning algorithm when a model for a first class has not been created. Referring to FIG. 3A, the machine learning algorithm may be learned based on data (300) corresponding to the first class. For example, the machine learning algorithm may learn about the features of the first class by learning the first data (300) corresponding to the first class. Referring to FIG. 3B, the machine learning algorithm may generate a representative image (310) based on the learned features of the first class, and compare the input image (320) with the representative image (310) to determine whether the input image (320) corresponds to the first class. For example, a machine learning algorithm can determine whether a representative image (310) of a first class and an input image (320) share similar features. For example, a machine learning algorithm (312, 322) can receive an input image (320) and a representative image (310), extract features of the representative image (310), and extract features of the input image (320). A softmax (330) can obtain features of the representative image (310) and the input image (320) from the machine learning algorithm (312, 322), and determine whether the input image (320) corresponds to the first class.

[0086] FIG. 4 illustrates virtual data according to one embodiment of the present disclosure. The virtual data may be used to train a feature extractor commonly included in each model and a machine learning algorithm for determining whether data corresponds to a class before a model is generated. According to one embodiment, the virtual data may be generated by adjusting the size, font, and thickness of letters in real data, and adding blur and defects. The virtual data may be generated using at least one algorithm among Generative Adversarial Networks (GAN) and stable diffusion. According to one embodiment, the virtual data may be generated by a processor of the electronic device (100). Referring to FIG. 4, a plurality of virtual data (400a, 400b, 400c, and 400d) may be generated by changing the size and font of letters in images actually stored in memory, or by adding blur and defects.

[0087] FIG. 5 illustrates a comparative data set generated to learn input data according to one embodiment of the present disclosure. The processor may generate a comparative data set (510, 520) for learning input data (500). The processor may generate the comparative data set (510, 520) including similar data (510) having similar characteristics to the input data (500) and randomly selected data (520). For example, referring to FIG. 5, the processor may generate the comparative data set (510, 520) including similar data (510) '9670SG' having similar visual characteristics to the input data (500) '96700G'. Examples of similar data (510) included in the comparative data set (510, 520) are not limited to those illustrated in FIG. 5. In the comparison data set (510, 520), data (520) excluding similar data (510) can be randomly selected from data stored in memory. By training the model using the comparison data set (510, 520), it is possible to more accurately determine whether the input data (500) corresponds to a class.

[0088] FIG. 6 illustrates a re-inspection process at a review station according to one embodiment of the present disclosure. The processor determines whether input data corresponds to the first class according to the contents described above in FIGS. 1 to 5, and if the input data does not correspond to the first class, the inspection result for the input data can be transmitted to the review station (610). The review station (610) can re-inspect the inspection result (600) of the processor for the input data based on a sample (612) for data corresponding to the first class. According to one embodiment, the review station (610) can perform a task of re-inspecting the inspection result (600) of the processor by having an operator directly compare the sample (612) with the input data.

[0089] The review station (610) may receive an inspection result (600) for input data from the processor. For example, the review station (610) may receive an inspection result (600) that determines that the first input data (602) and the second input data (604) do not correspond to the first class, and that the third input data (606) corresponds to the first class. The review station (610) may re-examine whether the determination was made correctly based on the data sample (612). For example, the re-examined result (620) may determine that the first input data (622) corresponds to the first class because it is defective but matches the sample (612), and that the second input data (624) also corresponds to the first class because it matches the sample (612). Additionally, the third data (626) may be determined not to correspond to the first class because it does not match the sample (612), unlike the inspection result (600) of the processor that determined that the third data corresponds to the first class.

[0090] FIG. 7 is a block diagram of a model operating on a GPU according to one embodiment of the present disclosure. A model for determining whether input data corresponds to a class may operate on the GPU (700). That is, the model may output a judgment result (740) indicating whether the input data corresponds to a class by using a feature extractor (710) and a classifier (720) operating on the GPU (700). However, since the capacity of the GPU (700) is limited, it may be difficult to load and use the classifiers of all models on the GPU (700) at once. Therefore, in order to save the capacity of the GPU (700), the classifiers of models used more frequently than a predetermined frequency may be loaded from the memory (730) to the cache (722) on the GPU (700). For example, if 100 classifiers can be stored in the cache (722) on the GPU (700), the processor can load the 100 classifiers from the memory (730) to the cache (722) in order of frequency of use, starting with the most frequently used model among the multiple models.

[0091] In one embodiment, the processor may remove the classifier of the least used model from the cache (722) based on the usage frequency of the classifier. For example, if all 100 classifiers are loaded into the cache (722), and a classifier not loaded into the cache (722) is needed, the processor may remove the least used classifier among the classifiers loaded into the cache (722) and load the needed classifier from the memory (730) into the cache (722).

[0092] FIG. 8 illustrates an example of determining a reference probability in an electronic device according to an embodiment of the present disclosure. In the graph of FIG. 8, the horizontal axis represents a class, and the vertical axis represents a probability calculated when a plurality of input data are input to a model of each class. The vertical lines in the graph represent probability values ​​calculated by the model of each class for a plurality of input data, and the bars located at the bottom and top of the vertical lines may represent sections where probability values ​​are widely distributed (e.g., sections where the top 25% to 75% of data are distributed). In addition, the bubble indicated for each class represents the reference probability for the model of the corresponding class. That is, the reference probability for the model of the first class (800) is the value (P) indicated by the first bubble (806). x ), the reference probability for the model of the second class (810) may be the value (0.9) indicated by the second bubble (816).

[0093] The processor can determine the reference probability of each model based on the output values ​​of multiple input data for each model. The probability values ​​output by inputting multiple input data into the model can be accumulated and stored in the memory. The processor can calculate the average value of the minimum value among the probability values ​​of the input data determined to correspond to the first class (800) and the maximum value among the probability values ​​of the input data determined not to correspond to the first class (800), and the determined critical probability (P x) can be determined as the reference probability. For example, referring to FIG. 8, among the data input to the first model, the maximum value (802) of the probability values ​​of the input data determined not to correspond to the first class (800) is 0.1, and the minimum value (804) of the probability values ​​of the input data determined to correspond to the first class (800) is 0.99, so the average value of the two values ​​is approximately 0.5. Among the data input to the second model, the maximum value (812) of the probability values ​​of the input data determined not to correspond to the second class (810) is 0.86, and the minimum value (814) of the probability values ​​of the input data determined to correspond to the second class (810) is 0.98, so the average value (816) of the two values ​​is approximately 0.9. The critical probability (P x ) is approximately 0.7, the processor can determine the reference probability for the model of the first class (800) to be 0.7 and the reference probability for the model of the second class (810) to be 0.9. According to one embodiment, the processor can reset the reference probability each time a model is generated or retrained. According to another embodiment, the processor can reset the reference probability at a set time interval.

[0094] FIG. 9 is a flowchart illustrating a method for determining whether first data corresponds to a first class in an electronic device according to an embodiment of the present disclosure. In operation 900, the electronic device may acquire input data. For example, the input data may be an image of a component placed on a substrate.

[0095] The electronic device (100) can, at operation 910, check whether a model for a first class corresponding to the first data is stored in the memory (110). If the first model is stored, the electronic device (100) can, at operation 912, determine whether the first data corresponds to the first class using the first model. If the first model is not stored, the electronic device (100) can, at operation 914, determine whether the input data corresponds to the first class based on the first data corresponding to the first class.

[0096] The electronic device (100) may, in operation 920, transmit information indicating whether the first data corresponds to the first class to the external device (101). The electronic device (100) may determine whether the first data corresponds to the first class and transmit the inspection result to the review station (610).

[0097] FIG. 10 is a flowchart illustrating a method for generating or retraining a model for a first class in an electronic device according to an embodiment of the present disclosure. In operation 1000, the electronic device (100) may obtain a learning request for input data. Upon obtaining the learning request, in operation 1010, the electronic device (100) may verify whether a model for the first class corresponding to the input data is stored in the memory (110).

[0098] If the first model is stored in the memory (110), the electronic device (100) may determine whether a predetermined condition is satisfied in operation 1020. According to one embodiment, the electronic device (100) may make a decision to retrain the current model based on the number of errors generated in the result of determining whether each data corresponds to the first class using the first model. In operation 1022, the electronic device (100) may determine to retrain the first model based on the number of first errors in determining data that does not correspond to the first class as corresponding to the first class and the number of second errors in determining data that corresponds to the first class as not corresponding to the first class, among the results of determining whether the input data corresponds to the first class using the first model. If the predetermined condition is not satisfied, the electronic device (100) may not retrain the first model.

[0099] If the first model is not stored in the memory (110), the electronic device (100) may, at operation 1030, add the input data to a learning queue for generating the first model. According to one embodiment, the electronic device (100) may add the input data to the learning queue if a predetermined number (e.g., 100) or more of data corresponding to the first class is stored and the data corresponding to the first class does not already exist in the learning queue. The electronic device (100) may, at operation 1032, generate a model for the first class.

[0100] FIG. 11 is a flowchart illustrating a method for an electronic device to generate a model for a first class according to an embodiment of the present disclosure. In operation 1110, the electronic device may receive a learning request for first data. In one embodiment, the electronic device may receive a learning request for input data from a learning server. An automatic optical inspection device may inspect whether a component is placed in a correct position on a substrate, and if it determines that the component is placed in an incorrect position, the device may transmit data (e.g., an image) for the component to the electronic device for re-inspection. In response to receiving data from the automatic optical inspection device, the learning server may check whether a model for a class corresponding to the received data is stored in the electronic device. If the model is not stored in the electronic device, the learning server may transmit a learning request for the received data to the processor.

[0101] In operation 1120, the electronic device may check whether a predetermined number or more of data corresponding to the first class is stored. According to one embodiment, the electronic device may store one or more pieces of received data on the memory (110) by class. For example, the electronic device may store one or more pieces of data in the memory (110) while also storing information about the class corresponding to each piece of data. In response to receiving a learning request for input data, the electronic device may check whether a predetermined number or more of data corresponding to the same class as the corresponding data is stored in the memory (110). For example, in response to receiving the first data, the electronic device may check whether a predetermined number or more of data corresponding to the first class is stored in the memory (110). If a predetermined number or more of data corresponding to the same class as the corresponding data is stored in the memory (110), the electronic device may add the first data to the learning queue in operation 1130. Conversely, if less than a predetermined number of pieces of data corresponding to the same class as the corresponding data is stored in the memory (110), the electronic device does not add the first data to the learning queue. By securing a predetermined number of samples as described above, the electronic device can generate a highly accurate model for the first class.

[0102] In operation 1140, the electronic device may generate a comparison data set for generating a first model. The comparison data set may include similar data having similar characteristics to the input first data, in order to increase the accuracy of the generated first model. According to one embodiment, the electronic device may determine the proportion of similar data in the comparison data set as a predetermined value (e.g., 25%), and the remainder may be determined as random data. The electronic device may extract characteristics of the input data using the randomly determined data, and may compare the similar data with the input data to more precisely extract characteristics of the input data, thereby generating a model that can more accurately determine data corresponding to a class. According to one embodiment, the electronic device may calculate the probability that the input data corresponds to a class using the generated model.

[0103] The electronic device may determine a reference probability in operation 1150. The reference probability may refer to a probability that serves as a criterion for determining whether data corresponds to a specific class. For example, the electronic device may calculate a probability that the first data corresponds to the first class using the first model, and compare the calculated probability with the reference probability to determine whether the first data corresponds to the first class. If the calculated probability is greater than or equal to the reference probability, the electronic device may determine that the first data corresponds to the first class, and if the calculated probability is less than the reference probability, the electronic device may determine that the first data does not correspond to the first class. In operation 1160, the electronic device may generate a model of a class corresponding to the first data as a result of operations 1110 to 1150.

[0104] FIG. 12 is a flowchart illustrating a method for an electronic device to retrain a first model for a first class according to an embodiment of the present disclosure. Descriptions of content overlapping with those described in FIG. 11 will be omitted. In operation 1210, the electronic device may determine whether the first data corresponds to a class using an existing model. For example, the electronic device may determine whether the first data corresponds to a first class using the first model generated according to the method described in FIG. 11.

[0105] In operation 1220, the electronic device can determine whether the number of errors occurring during the judgment process is greater than a predetermined value. When the electronic device determines whether input data corresponds to the first class using the first model, an error may occur in which data that does not correspond to the first class is incorrectly determined to correspond to the first class, or data that corresponds to the first class is incorrectly determined to correspond to not the first class. The electronic device can store information on all errors that occurred in the memory (110) and determine whether the number of errors that occurred is greater than a predetermined value. In operation 1230, if the number of errors that occurred is greater than the predetermined value, the electronic device can add the first data to the learning queue. Conversely, if the number of errors that occurred is less than the predetermined value, the input data is not added to the learning queue.

[0106] The electronic device may generate a comparison data set for retraining the first model at operation 1240. According to one embodiment, the comparison data set for retraining the first model may be generated by the method described in operation 1140 of FIG. 11. The electronic device may determine a reference probability at operation 1250. According to one embodiment, the reference probability for determining whether the input data corresponds to the corresponding class may be determined by the method described in operation 1150 of FIG. 11. The electronic device may retrain a model of the class corresponding to the first data at operation 1260 based on the results of operations 1210 to 1250.

[0107] Although the steps of the method or algorithm according to the present disclosure are described in a sequential order in the flowcharts illustrated in FIGS. 9 to 12, the steps may be performed in any order that can be arbitrarily combined according to the present disclosure, in addition to being performed sequentially. The description according to this flowchart does not exclude changes or modifications to the method or algorithm, and does not imply that any step is essential or desirable. In one embodiment, at least some of the steps may be performed in parallel, iteratively, or heuristically. In one embodiment, at least some of the steps may be omitted, or other steps may be added.

[0108] FIGS. 13 and 14 illustrate a system for determining a class of data according to one embodiment of the present disclosure. Referring to FIG. 13 , the system may include an optical inspection device (1310), a review station (1320), and an electronic device (1330). The optical inspection device (1310) may irradiate light formed from an illumination unit within the device onto an inspection target (e.g., a substrate) and receive reflected light to obtain an image of the inspection target. The optical inspection device (1310) may perform a primary inspection on the inspection target to determine whether the inspection target is passable or fail, i.e., good (GOOD) or bad (NG) for the inspection target. The optical inspection device (1310) may transmit good data to the electronic device (1330) and bad data to the review station (1320).

[0109] The review station (1320) can receive defective data from the optical inspection device (1310) and perform a re-inspection on the defective data. For example, the review station (1320) can re-judge whether the input data is passable or fail. According to one embodiment, the re-inspection performed at the review station (1320) may be a process in which a person directly checks the input data and determines whether it is passable or fail. The review station (1320) can re-inspect the input data to determine whether each data is passable data (PASS) or fail data (NG), and transmit the re-inspection result to the electronic device (1330). According to another embodiment, the review station (1320) can transmit an inference request to the electronic device (1330). The inference request may mean, for example, a request to perform a re-inspection on the defective data received by the review station (1320). The review station (1320) can transmit the defective data received from the optical inspection device (1310) along with the inference request to the electronic device (1330). The review station (1320) can receive an inference response corresponding to the inference request from the electronic device (1330) and determine the re-inspection result based on the inference response.

[0110] The electronic device (1330) may determine whether the received data is passable or not in response to an inference request received from the review station (1320). According to one embodiment, the electronic device (1330) may generate and store a model for each class. If a model for a class corresponding to the received data has been generated, the electronic device (1330) may determine whether the data is passable or not based on the generated model. If a model for a class corresponding to the received data has not been generated, the electronic device (1330) may determine whether the data is passable or not based on one or more pieces of data stored in the memory (110). The electronic device (1330) may determine whether the data is passable or not using the methods described in FIGS. 1 to 12 .

[0111] Referring to FIG. 14, the system may further include a data agent (Data agent, 1340). The data agent (1340) may refer to an interface for transmitting and receiving data between the optical inspection device (1310), the review station (1320), and the electronic device (1330). For example, the data agent (1340) may receive good (GOOD) data from the optical inspection device (1310), and good (PASS), bad (NG) data, and an inference request from the review station (1320) and transmit them to the electronic device (1330). The data agent (1340) may receive an inference response from the electronic device (1330) and transmit it to the review station (1320). The system may smoothly and quickly transmit and receive data between the optical inspection device (1310), the review station (1320), and the electronic device (1330) through the data agent (1340).

[0112] A system for determining the class of data according to one embodiment of the present disclosure can be used throughout the entire semiconductor manufacturing process. In one embodiment, the system according to the present disclosure can be used in all pre-processes, such as wafer manufacturing processes, oxidation processes, photo processes, etching processes, deposition processes, ion implantation processes, metal wiring processes, and EDS (Electrical Die Sorting) processes, as well as post-processes, such as packaging. For example, the system of the present disclosure can be used to determine whether photoresist has been properly applied to a wafer according to a predetermined pattern in a photo process, or to determine whether etching has been performed properly in an etching process. The embodiments in which the system of the present disclosure can be used are not limited by what has been described above.

[0113] Various embodiments of the present disclosure may be implemented as software on a machine-readable storage medium. The software may be software for implementing various embodiments of the present disclosure. The software may be inferred from various embodiments of the present disclosure by programmers skilled in the art to which the present disclosure pertains. For example, the software may be a program including machine-readable instructions (e.g., code or code segments). The device may be a device capable of operating according to instructions called from a storage medium, such as a computer. In one embodiment, the device may be an electronic device (100) according to embodiments of the present disclosure. In one embodiment, the processor of the device may execute the called instructions, causing components of the device to perform functions corresponding to the instructions. In one embodiment, the processor may be a processor (120) according to embodiments of the present disclosure. The storage medium may refer to any type of recording medium that stores data and can be read by the device. The storage medium may include, for example, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, etc. In one embodiment, the storage medium may be memory (110). In one embodiment, the storage medium may be implemented in a distributed form, such as in a network-connected computer system. The software may be distributed and stored and executed in a computer system, etc. The storage medium may be a non-transitory storage medium. A non-transitory storage medium means a tangible medium regardless of whether data is stored semi-permanently or temporarily, and does not include a signal that is propagated transitorily.

[0114] While the technical concept of the present disclosure has been described through various embodiments, it should be understood that the technical concept of the present disclosure encompasses various substitutions, modifications, and variations that can be made within the scope of those skilled in the art to which the present disclosure pertains. Furthermore, it should be understood that such substitutions, modifications, and variations are encompassed within the scope of the appended claims.

Claims

1. An optical inspection device that receives reflected light from a component placed on a substrate to obtain first data about the component, checks whether the first data corresponds to a first class, and transmits the inspection result to a review station; A review station that transmits a re-inspection request for the inspection result to a reclassification device in response to receiving the inspection result from the optical inspection device; and Obtain the above first data, Determine whether a model for the first class is stored in one or more memories, Based on a determination that the model for the first class is stored, determining whether the first data corresponds to the first class using the model, Based on a determination that the model for the first class is not stored, using second data corresponding to the first class, it is determined whether the first data corresponds to the first class, A system comprising a reclassification device that transmits information indicating whether the first data corresponds to the first class to the review station.

2. In paragraph 1, The above optical inspection device, Determine whether the above first data corresponds to the above first class, and A system that transmits the first data and information indicating that the first data corresponds to the first class to the reclassification device upon a determination that the first data corresponds to the first class, and transmits the first data and information indicating that the first data does not correspond to the first class to the review station upon a determination that the first data does not correspond to the first class.

3. In paragraph 1, In determining whether the first data corresponds to the first class using the above model, Using the above model, the probability that the first data corresponds to the first class is calculated, Determine whether the above probability is greater than or equal to a predetermined reference probability, If the above probability is greater than or equal to a predetermined reference probability, the first data is determined to correspond to the first class, A system comprising a reclassification device that determines that the first data does not correspond to the first class if the probability is less than a predetermined reference probability.

4. In paragraph 3, A system in which the above-determined reference probability is determined using the average value of the minimum value of the probability of one or more data determined to correspond to the first class and the maximum value of the probability of one or more data determined not to correspond to the first class.

5. In paragraph 1, The above reclassification device is, Upon determining that the model for the first class is not stored, the similarity between the first data and the second data is calculated using a machine learning algorithm, A system that determines whether the first data corresponds to the first class based on the similarity between the first data and the second data.

6. In paragraph 5, The above machine learning algorithm is a system including ArcFace (Additive Angular Margin Loss).

7. In paragraph 5, The above machine learning algorithm, A system that learns using a plurality of randomly generated virtual data and the second data.

8. In paragraph 7, The above multiple virtual data are, A system that generates text by adjusting the size, font, and thickness of letters and adding blur and defects to one or more actual data stored in one or more memories.

9. In paragraph 7, The above multiple virtual data are, A system generated using at least one algorithm among Generative Adversarial Networks (GAN) and stable diffusion.

10. In paragraph 1, The above reclassification device is, Obtain a learning request for the above first data, Upon obtaining the above learning request, determining whether the model for the first class is stored in one or more memories, Upon determining that the model for the first class is not stored, adding the first data to a training queue for generating or training the model, A system that adds the first data to the learning queue when the model for the first class satisfies a predetermined condition, based on a determination that a model for the first class is stored.

11. In paragraph 10, The reclassification device adds the first data to the learning queue upon determining that the model for the first class is not stored. A system for adding the first data to the learning queue when the number of one or more data corresponding to the first class stored on the one or more memories is greater than a predetermined value and the first data does not exist in the learning queue.

12. In paragraph 10, The above predetermined conditions are, A system in which the condition is satisfied when the number of times the model judges data that does not correspond to the first class as corresponding to the first class is greater than a first threshold, or when the number of times the model judges data that corresponds to the first class as not corresponding to the first class is greater than a second threshold.

13. In paragraph 10, A system in which the above model is trained using a data set including data corresponding to one or more classes similar to the first class and randomly selected data.

14. In paragraph 13, A system wherein the proportion of data corresponding to one or more classes similar to the first class among the data set is less than or equal to the proportion of the randomly selected data among the data set.

15. In paragraph 1, The above review station further includes a display, The reclassification device transmits information indicating that it cannot determine whether the first data corresponds to the first class to the review station based on a determination that the model for the first class and the second data corresponding to the class are not stored, A system in which the review station outputs a phrase on the display indicating that it is unable to determine whether the first data corresponds to the first class.

16. In paragraph 1, The above model is a system including a feature extractor and a classifier.

17. In paragraph 16, A system wherein the feature extractor is commonly applied to models stored in the one or more memories, and the classifier is learned based on different information for each model stored in the one or more memories.

18. In paragraph 16, The above feature extractor is a system that learns using a plurality of randomly generated virtual data.

19. In paragraph 16, The above feature extractor and classifier operate on a GPU (Graphic Processing Unit), The above reclassification device is, Loading a classifier included in a model of a class used with a frequency greater than a predetermined value from the memory to a cache on the GPU, A system that determines whether data corresponds to a class using the above feature extractor and the classifier loaded into the cache.

20. In paragraph 19, The above reclassification device is, A system that selects and removes from the cache the least used classifier among one or more classifiers loaded in the cache.

21. A method performed in a reclassification device including one or more processors and one or more memories storing instructions to be executed by the one or more processors, A step of obtaining first data indicating a component to be placed on a substrate; A step of determining whether a model for a first class corresponding to the first data is stored in one or more memories; A step of determining whether the first data corresponds to the first class using the model, based on a determination that the model for the first class is stored; A step of determining whether the first data corresponds to the first class by using second data corresponding to the first class, based on a determination that a model for the first class is not stored; and A method comprising the step of transmitting information indicating whether the first data corresponds to the first class to a review station.

Images