Self-supervised and unsupervised learning-based image clustering electronic device for automated labeling and noise sampling of PCB surface defect image, and operation method thereof

Abstract

According to various embodiments, a self-supervised and unsupervised learning-based image clustering electronic device for automated labeling and noise sampling of a PCB surface defect image may comprise a PCB manufacturing facility and a processor, wherein the processor is configured to: detect PCB surface defect image data through a quality inspection process system of the PCB manufacturing facility; and input the PCB surface defect image data to a clustering classification model to classify the PCB surface defect image data on the basis of a pre-configured structural feature. According to various embodiments, an operation method of a self-supervised and unsupervised learning-based image clustering electronic device for automated labeling and noise sampling of a PCB surface defect image may comprise the steps of: detecting PCB surface defect image data through a quality inspection process system of a PCB manufacturing facility of the electronic device; and inputting the PCB surface defect image data to a clustering classification model to classify the PCB surface defect image data on the basis of a pre-configured structural feature. Various other embodiments are also possible.

Description

Image clustering electronic device and its operating method based on self-supervised and unsupervised learning for automated labeling and noise sampling of PCB surface defect images

[0001] The present invention relates to an electronic device and its operating method based on self-supervised and unsupervised learning for automated labeling and noise sampling of PCB surface defect images, and more specifically, to a system for performing pre-classification and pseudo-labeling on an image dataset of PCB surface defects detected by traditional image processing-based inspection in AFVI (Auto Final Visual Insepction; automatic final inspection equipment) and VTS (Verify Totaling System; real-time quality information collection and quality management system), which are part of a quality inspection process in a PCB manufacturing facility, based on structural features (pads, patterns, grounds) of the PCB surface, thereby enabling sampling of a noise dataset, and for supplementing and automating existing human-based visual labeling through semi-automatic labeling.

[0002]

[0003] In general, a printed circuit board (PCB) is a structure in which conductors and insulators are laminated in the form of a substrate, and various components such as semiconductors, capacitors, and resistors can be mounted on it, and it is a component that makes electrical connections between components.

[0004] As industrial development progresses, PCBs with various shapes and functions are manufactured, and as electronic components become smaller, thinner, denser, and more packaged, demand for them increases, requiring extensive research to improve precision, productivity, and work efficiency.

[0005] Inspection results recognized as defects in inspections using AFVI (Auto Final Visual Inspection; automatic final inspection equipment) / VTS (Verify Totaling System; real-time quality information collection and quality management system) include both false and genuine defects. Therefore, it is necessary to conduct additional inspections on the inspection results recognized as defects to distinguish between false and genuine defects.

[0006]

[0007] Accordingly, the technical problem to be solved by the present invention is to provide an image clustering electronic device and an operating method thereof based on self-supervised and unsupervised learning for automatic labeling and noise sampling of PCB surface defect images, which were created to solve the aforementioned problems.

[0008] The technical problems to be solved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0009]

[0010] According to the present invention, an image clustering electronic device based on self-supervised and unsupervised learning for automated labeling and noise sampling of PCB surface defect images comprises a PCB manufacturing facility and a processor, wherein the processor is configured to detect PCB surface defect image data through a quality inspection process system of the PCB manufacturing facility, input the PCB surface defect image data into a clustering classification model, and classify the PCB surface defect image data based on preset structural features.

[0011] According to the present invention, a method for operating an image clustering electronic device based on self-supervised and unsupervised learning for automated labeling and noise sampling of PCB surface defect images may include the steps of: detecting PCB surface defect image data through a quality inspection process system of a PCB manufacturing facility of the electronic device; and inputting the PCB surface defect image data into a clustering classification model to classify the PCB surface defect image data based on preset structural features.

[0012]

[0013] The present invention can improve the over-inspection rate of AFVI and similar image-based automated inspection equipment, and reduce classification errors by optimizing the reference image (master image or Gerber data) of the AFVI equipment and the parameters of the inspection system.

[0014] In addition, the problem of conventional visual inspection requiring the inspector's knowledge of the PCB and skill for accurate inspection can be replaced with semi-automated labeling, so that consistent results can be expected for each inspection. In addition, by applying an explicit classification system based on the clustering results or deriving improvements to the master image, more accurate defect inspection and optimization of pass / fail judgment can be enabled.

[0015] The effects of the present invention are not limited to the effects described above, and the tentative effects expected by the technical features of the present invention can be clearly understood from the description below.

[0016]

[0017] FIG. 1 illustrates a block diagram of an electronic device according to various embodiments of the present invention.

[0018] FIG. 2 is a flowchart illustrating a method of operating an electronic device according to various embodiments.

[0019] FIG. 3 is a flowchart illustrating a process of clustering PCB surface defect image data according to various embodiments.

[0020] FIG. 4 is an exemplary drawing specifically illustrating a process of clustering PCB surface defect image data according to various embodiments.

[0021]

[0022] Hereinafter, various embodiments of the present document will be described with reference to the attached drawings. It should be understood that the embodiments and the terms used therein are not intended to limit the technology described in the present document to a specific embodiment, but rather include various modifications, equivalents, and / or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar components. The singular expression may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B" or "at least one of A and / or B" may include all possible combinations of the items listed together. Expressions such as "first," "second," "first," or "second," may modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When it is said that a component (e.g., a first component) is “(functionally or communicatively) connected” or “connected” to another component (e.g., a second component), said component may be directly connected to said other component, or may be connected via another component (e.g., a third component).

[0023] In this document, "configured to" may be used interchangeably with, for example, "suitable for," "capable of," "modified to," "made to," "capable of," or "designed to," either in hardware or software. In some contexts, the phrase "a device configured to" may mean that the device is "capable of" doing something together with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" may mean a dedicated processor (e.g., an embedded processor) for performing the operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform the operations by executing one or more software programs stored in a memory device.

[0024] An electronic device according to various embodiments of the present document may include, for example, at least one of a smartphone, a tablet PC, a desktop PC, a laptop PC, a netbook computer, a workstation, and a server.

[0025] Referring to FIG. 1, an electronic device (101) within a network environment (100) according to various embodiments is described. The electronic device (101) may include a bus (110), a processor (120), a memory (130), an input / output interface (150), a display (160), and a communication interface (170). In some embodiments, the electronic device (101) may omit at least one of the components or additionally include other components. The bus (110) may include a circuit that connects the components (110-170) to each other and transmits communication (e.g., control messages or data) between the components. The processor (120) may include one or more of a central processing unit, an application processor, or a communication processor (CP). The processor (120) may, for example, execute operations or data processing related to control and / or communication of at least one other component of the electronic device (101).

[0026] The memory (130) may include volatile and / or non-volatile memory. The memory (130) may store, for example, commands or data related to at least one other component of the electronic device (101). According to one embodiment, the memory (130) may store software and / or programs (140). The programs (140) may include, for example, a kernel (141), middleware (143), an application programming interface (API) (145), and / or an application program (or “application”) (147). At least a portion of the kernel (141), middleware (143), or API (145) may be referred to as an operating system. The kernel (141) may control or manage system resources (e.g., a bus (110), a processor (120), or a memory (130)) used to execute operations or functions implemented in other programs (e.g., middleware (143), API (145), or application programs (147)). In addition, the kernel (141) may provide an interface that allows the middleware (143), API (145), or application programs (147) to control or manage system resources by accessing individual components of the electronic device (101).

[0027] The middleware (143) may, for example, act as an intermediary to enable the API (145) or the application program (147) to communicate with the kernel (141) to exchange data. In addition, the middleware (143) may process one or more task requests received from the application program (147) according to priority. For example, the middleware (143) may give at least one of the application programs (147) a priority to use the system resources (e.g., bus (110), processor (120), memory (130), etc.) of the electronic device (101) and process the one or more task requests. The API (145) is an interface for the application (147) to control functions provided by the kernel (141) or the middleware (143), and may include, for example, at least one interface or function (e.g., command) for file control, window control, image processing, or character control. The input / output interface (150) can, for example, transmit commands or data input from a user or another external device to other component(s) of the electronic device (101), or output commands or data received from other component(s) of the electronic device (101) to the user or another external device.

[0028] The display (160) may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a micro electro mechanical systems (MEMS) display, or an electronic paper display. The display (160) may, for example, display various contents (e.g., text, images, videos, icons, and / or symbols) to the user. The display (160) may include a touch screen and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a part of the user's body. The communication interface (170) may, for example, establish communication between the electronic device (101) and an external device (e.g., a first external electronic device (102), a second external electronic device (104), or a server (106)). For example, the communication interface (170) can be connected to a network (162) via wireless communication or wired communication to communicate with an external device (e.g., a second external electronic device (104) or a server (106)).

[0029] The wireless communication may include, for example, cellular communication using at least one of LTE, LTE-A (LTE Advance), CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications system), WiBro (Wireless Broadband), or GSM (Global System for Mobile Communications). In one embodiment, the wireless communication may include, for example, at least one of WiFi (wireless fidelity), Bluetooth, Bluetooth low energy (BLE), Zigbee, near field communication (NFC), Magnetic Secure Transmission, radio frequency (RF), or body area network (BAN). In one embodiment, the wireless communication may include GNSS. The GNSS may be, for example, GPS (Global Positioning System), Glonass (Global Navigation Satellite System), Beidou Navigation Satellite System (hereinafter "Beidou"), or Galileo, the European global satellite-based navigation system. Hereinafter, in this document, "GPS" may be used interchangeably with "GNSS." Wired communication may include at least one of, for example, USB (universal serial bus), HDMI (high definition multimedia interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).The network (162) may include at least one of a telecommunications network, for example, a computer network (e.g., a LAN or WAN), the Internet, or a telephone network.

[0030] Each of the first and second external electronic devices (102, 104) may be the same or a different type of device as the electronic device (101). According to various embodiments, all or part of the operations executed in the electronic device (101) may be executed in another one or more electronic devices (e.g., electronic devices (102, 104), or server (106). According to one embodiment, when the electronic device (101) is to perform a certain function or service automatically or upon request, the electronic device (101) may request at least some functions related thereto from another device (e.g., electronic device (102, 104), or server (106)) instead of executing the function or service by itself or in addition. The other electronic device (e.g., electronic device (102, 104), or server (106)) may execute the requested function or additional function and transmit the result to the electronic device (101). The electronic device (101) may process the received result as is or additionally to provide the requested function or service. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology may be used.

[0031]

[0032] FIG. 2 is a flowchart illustrating a method of operating an electronic device according to various embodiments.

[0033] FIG. 3 is a flowchart illustrating a process of clustering PCB surface defect image data according to various embodiments.

[0034] FIG. 4 is an exemplary drawing specifically illustrating a process of clustering PCB surface defect image data according to various embodiments.

[0035]

[0036] Before a detailed description of the present invention, the electronic device (100) may include a tool wear sensor module and a processor (120). In the present invention, the electronic device (100) may typically refer to a portable device such as a tablet or a smartphone, but of course, it may refer to any device that has the function of measuring tool wear data, storing the data, and transmitting the data to an external device, including a tool wear sensor module, which will be described later. The tool wear sensor module may refer to any type of sensor that is installed inside the electronic device (100) and can measure the wear status of a tool in real time. The processor (120) may be a subject that performs operations 201 to 205, which will be described later. The components presented above are merely examples, and it is of course possible to add or remove the components as needed.

[0037]

[0038] In operation 201, according to various embodiments, the processor (120) of the electronic device (100) can detect PCB surface defect image data through a quality inspection process system of a PCB manufacturing facility. Specifically, the quality inspection process system of the PCB manufacturing facility can include AFVI (Auto Final Visual Insepction; automatic final inspection equipment) and VTS (Verify Totaling System; real-time quality information collection and quality management system). Here, the PCB manufacturing facility can use all types of manufacturing facilities currently in use on the market or under development in the future, and the quality inspection process system as an example is only for efficient implementation of the present invention, and of course, other types of quality inspection process systems can be used depending on the situation.

[0039]

[0040] In operation 203, according to various embodiments, the processor (120) of the electronic device (100) may input PCB surface defect image data into a clustering classification model to classify the PCB surface defect image data based on preset structural features. Here, the preset structural feature may mean one of a pad, a pattern, and a ground in the shape of the PCB surface, and it is to be understood that the types thereof may be added or removed to implement the present invention.

[0041] Specifically, with reference to FIGS. 3 and 4, a process of classifying PCB surface defect image data according to structural features by the clustering classification model described in operation 203 can be illustrated.

[0042]

[0043] In operation 301, according to various embodiments, the processor (120) of the electronic device (100) may input a plurality of PCB surface defect image data detected through a quality inspection process system of a PCB manufacturing facility into a backbone model to extract image representation data. Specifically, in order to learn structural features of a PCB surface, the backbone model of RegNet v2 (backbone based on RegNet-AnyNetX) may be used to extract an image representation that can well explain the structural features of the image, characteristic patterns of the image, and spatial relations.

[0044]

[0045] In operation 303, according to various embodiments, the processor (120) of the electronic device (100) uses semi-supervised learning knowledge distillation using image expression data as a Teacher Model and the clustering classification model as a Student Model, and then uses the clustering classification model as a loss function ( ) can be trained based on. Specifically, using the image expression data extracted from 301 movements, self-supervised learning based on SimCLR v2 is performed to train a Teacher model for generating pseudo-labels in the learning stage, and a Student model, which is a more compact model, can be trained using the pseudo-labels of the Teacher model through knowledge distillation, which is a semi-supervised learning, to fit the PCB surface defect image dataset. Here, the distillation loss can be calculated by the following [Mathematical Formula 1].

[0046] [Mathematical Formula 1]

[0047]

[0048]

[0049] Here, stands for distillation loss, represents the input data sample, D represents the entire dataset, y represents the class label, is entered by Teacher Network The probability of class y predicted for temperature ) means using, is entered by Student Network The probability of class y predicted for temperature ) means using, Temperature can mean a parameter that plays a role in smoothing the probability distribution.

[0050]

[0051] In operation 305, according to various embodiments, the processor (120) of the electronic device (100) may use the PCB surface defect image data as input data to the clustering classification model, and perform clustering based on one of the preset structural features corresponding to the PCB surface defect image data. Specifically, unsupervised image clustering based on Reciprocal Agglomerative Clustering (RAC) is performed based on the image embedding of each data extracted from the Student model learned through the PCB surface defect image dataset, so that stable and fast operation can be expected even for large datasets of 100,000 or 1 million or more, and the images clustered in this way may select a cluster representative label through a voting method within each cluster, utilize a cluster label that has been partially manually labeled in advance for each cluster, or assign a pseudo label, so that semi-auto labeling of the background structure of the PCB surface defect image dataset can be enabled.

[0052]

[0053] According to various embodiments, an image clustering electronic device based on self-supervised and unsupervised learning for automated labeling and noise sampling of PCB surface defect images comprises a PCB manufacturing facility and a processor, wherein the processor is configured to detect PCB surface defect image data through a quality inspection process system of the PCB manufacturing facility, and input the PCB surface defect image data into a clustering classification model to classify the PCB surface defect image data based on preset structural features.

[0054] According to various embodiments, the quality inspection process system of the PCB manufacturing facility may include an AFVI (Auto Final Visual Insepction; automatic final inspection equipment) and a VTS (Verify Totaling System; real-time quality information collection and quality management system).

[0055] According to various embodiments, the predetermined structural feature may mean that the shape of the PCB surface is one of a pad, a pattern, and a ground.

[0056] According to various embodiments, the processor may be configured to input a plurality of PCB surface defect image data detected through a quality inspection process system of the PCB manufacturing facility into a backbone model to extract image representation data, train the clustering classification model based on a loss function through semi-supervised learning knowledge distillation using the image representation data as a Teacher Model and the clustering classification model as a Student Model, and perform clustering based on one of the preset structural features corresponding to the PCB surface defect image data by using the PCB surface defect image data as input data to the clustering classification model.

[0057] According to various embodiments, a method for operating an electronic device for image clustering based on self-supervised and unsupervised learning for automated labeling and noise sampling of PCB surface defect images may include: detecting PCB surface defect image data through a quality inspection process system of a PCB manufacturing facility of the electronic device; and inputting the PCB surface defect image data into a clustering classification model to classify the PCB surface defect image data based on preset structural features.

[0058]

[0059] The term "module" or "part" used in this document includes a unit composed of hardware, software, or firmware, and can be used interchangeably with terms such as logic, logic block, component, or circuit, for example. The "module" or "part" can be an integrally configured component or a minimum unit or a part thereof that performs one or more functions. The "module" or "part" can be implemented mechanically or electronically, and can include, for example, an ASIC (application-specific integrated circuit) chip, FPGAs (field-programmable gate arrays), or a programmable logic device, known or to be developed in the future, that performs certain operations, and can be executed by the processor (120). At least a part of the device (e.g., modules or functions thereof) or method (e.g., operations) according to various embodiments can be implemented as instructions stored in a computer-readable storage medium (e.g., memory (130)) in the form of a program module. When the above command is executed by a processor (e.g., processor (120)), the processor can perform a function corresponding to the command. The computer-readable recording medium may include a hard disk, a floppy disk, a magnetic medium (e.g., a magnetic tape), an optical recording medium (e.g., a CD-ROM, a DVD, a magneto-optical medium (e.g., a floptical disk), a built-in memory, etc. The command may include a code generated by a compiler or a code executable by an interpreter. A module or program module according to various embodiments may include at least one or more of the above-described components, some of which may be omitted, or other components may be further included. Operations performed by a module, a program module, or other components according to various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0060] The embodiments disclosed in this document are presented for the purpose of explaining and understanding the disclosed technical content, and do not limit the scope of the present disclosure. Therefore, the scope of the present disclosure should be interpreted to include all modifications or various other embodiments based on the technical concepts of the present disclosure.

Claims

1. Image clustering based on self-supervised and unsupervised learning for automated labeling and noise sampling of PCB surface defect images in electronic devices. PCB manufacturing equipment and Contains a processor, The above processor, Detect PCB surface defect image data through the quality inspection process system of the above PCB manufacturing facility, The above PCB surface defect image data is input into a clustering classification model, and the PCB surface defect image data is set to be classified based on preset structural features. Electronic devices.

2. In paragraph 1, The quality inspection process system of the above PCB manufacturing facility is: Including AFVI (Auto Final Visual Insepction; automatic final inspection equipment) and VTS (Verify Totaling System; real-time quality information collection and quality management system). Electronic devices.

3. In paragraph 1, The above-mentioned structural features are, The shape of the PCB surface means one of pad, pattern, and ground. Electronic devices.

4. In paragraph 1, The above processor, The backbone model inputs multiple PCB surface defect image data detected through the quality inspection process system of the above PCB manufacturing facility to extract image representation data. Through semi-supervised learning knowledge distillation using the image expression data as the Teacher Model and the clustering classification model as the Student Model, the clustering classification model is trained based on the loss function. The above PCB surface defect image data is used as input data to the above clustering classification model, and clustering is performed based on one of the above preset structural features corresponding to the above PCB surface defect image data. Electronic devices.

5. In the method of operating an electronic device for image clustering based on self-supervised and unsupervised learning for automated labeling and noise sampling of PCB surface defect images, A step of detecting PCB surface defect image data through a quality inspection process system of the PCB manufacturing facility of the above electronic device; and A step of inputting the PCB surface defect image data into a clustering classification model and classifying the PCB surface defect image data based on preset structural features; Including, How an electronic device operates.

Images