Method and apparatus for detecting surface defects in a metal-ceramic composite substrate

Abstract

To improve accuracy of the technique for identifying mechanical scratches on a metal-ceramic composite substrate using image processing and machine learning.SOLUTION: A method of detecting scratches on a metal-ceramic composite substrate is provided, comprising capturing an image of the metal-ceramic composite substrate, extracting a scratch candidate area from the captured image using a linear filter, and determining the presence or absence of scratches in the scratch candidate area by applying a convolutional neural network to the scratch candidate area.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a technique for detecting surface defects occurring in a metal-ceramics composite substrate.

Background Art

[0002] In fields related to motor drive control of electric vehicles and electric railways, and renewable energy such as solar power generation and wind power generation, power modules for performing power conversion and control with high efficiency are widespread. A power module includes a substrate and semiconductor chips fixed to the substrate. In a power module, high electrical insulation is required for the substrate in order to extract a large current. Further, in order to control a large current, the temperature of the semiconductor chip during actual operation rises due to heat generation, so high heat dissipation is also required for the substrate. In addition, for miniaturization and high output of the power module, the substrate is required to have reliability (heat cycle tolerance), flexural strength (bending strength), and assembly properties (solder wettability, wire bonding property). Therefore, a metal-ceramics composite substrate in which a metal layer is laminated on a ceramics base material is widely used as the substrate.

[0003] A metal-ceramics composite substrate (hereinafter sometimes simply referred to as "substrate") is a polycrystal composed of a large number of fine crystal grains. And the grain boundaries forming the interfaces of adjacent crystal grains may exhibit various different properties because the atomic arrangement is different from that inside the crystal, and affect many material properties such as mechanical, electrical, and thermal properties of the entire substrate.

[0004] On the other hand, in the manufacturing process of the substrate, defects such as scratches, cracks, stains, and peripheral chipping may occur on the substrate. Therefore, it is necessary to determine the presence or absence of defects from the appearance of the substrate, and the determination is roughly classified into manual determination and machine determination. In manual determination, the substrate is visually inspected from various angles to detect defects. In contrast, in machine determination, the substrate is photographed and defects are detected by performing image processing based on binarization.

[0005] For example, Patent Document 1 discloses a technology for detecting scratches. Specifically, it acquires an image including a light-transmitting substrate and electrodes attached to the substrate, extracts detection targets from this image, and detects targets whose complexity exceeds a certain threshold as scratches. Furthermore, Patent Document 2 discloses an image inspection device. This image inspection device is characterized by having an inspection execution means that applies a normal inspection process to a newly acquired image to be inspected, determines whether the image has features that allow for good or bad product determination, and applies a deep learning process to determine whether the image has features that do not allow for good or bad product determination. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2019-138783 [Patent Document 2] Japanese Patent Publication No. 2020-187656 [Patent Document 3] Japanese Patent Application Publication No. 9-318334 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] Conventionally, in mechanical visual inspection methods, when performing edge detection processing such as differentiation on a 2D image and then binarizing the resulting image at a predetermined threshold to extract only the areas with high brightness levels, the threshold was calculated by performing a predetermined calculation based on a physical quantity related to the noise generated in the edge detection processed image (for example, Patent Document 3). However, actual metals and ceramics have a structure in which many regions with different atomic arrangements are gathered, and multiple metal or ceramic crystals are linked together to form an interface (grain boundary). Figure 12 shows an example of a grain boundary. When performing visual inspection using conventional visual inspection methods, these grain boundaries are often misidentified as defects.

[0008] Therefore, the present invention aims to provide a method and apparatus that can improve the accuracy of mechanical scratch detection in metal-ceramic composite substrates using image processing and machine learning techniques. [Means for solving the problem]

[0009] This application discloses a method for detecting scratches on a metal-ceramic composite substrate, comprising imaging the metal-ceramic composite substrate, extracting candidate scratch regions from the image obtained by imaging using a linear filter, and determining whether or not a scratch is present in the candidate scratch region using a convolutional neural network.

[0010] Furthermore, the present invention discloses a device for detecting scratches on a metal-ceramic composite substrate, comprising an imaging device and a scratch discrimination device for processing images obtained by the imaging device, wherein the scratch discrimination device processes the image using a linear filter to extract candidate scratch regions and performs calculations on the candidate scratch regions using a convolutional neural network to determine whether or not a scratch is present in the candidate scratch region. [Effects of the Invention]

[0011] According to the present invention, the accuracy of scratch detection in mechanical scratch detection technology for metal-ceramic composite substrates can be improved by using image processing and machine learning techniques. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 is a conceptual diagram showing the configuration of the inspection device 1. [Figure 2] Figure 2 shows an example of an image to be examined. [Figure 3] Figure 3 illustrates an example of the configuration of the damage detection device 20. [Figure 4] Figure 4 shows the flow of the damage identification method S1. [Figure 5]FIG. 5 is a diagram showing the flow of the process S10 for extracting a defect candidate region. [Figure 6] FIG. 6 is a diagram for explaining the process S11 of grayscale conversion processing. [Figure 7] FIG. 7 is a diagram for explaining the process S12 of binarization processing. [Figure 8] FIG. 8 is a diagram for explaining the filter in the process S13 of extracting a defect candidate region. [Figure 9] FIG. 9 is a diagram for explaining the process S14 of determining the position of the original image. [Figure 10] FIG. 10 is a diagram conceptually showing a convolutional neural network. [Figure 11] FIG. 11 shows examples of a defect region image (a) and a non-defect region image (b). [Figure 12] FIG. 12 is a diagram showing an example of a crystal grain boundary.

MODE FOR CARRYING OUT THE INVENTION

[0013] 1. Defect Detection Device A defect detection device 1 according to one example will be described while referring to the drawings. FIG. 1 is a diagram conceptually showing the configuration of the defect detection device 1. The defect detection device 1 is a device for detecting defects on a metal-ceramic composite substrate used in a power module, and is arranged, for example, on a manufacturing line of a metal-ceramic composite substrate.

[0014] The defect detection device 1 includes an imaging device 10 that captures an image of a substrate to be inspected, and a defect discrimination device 20 that discriminates defects from the image (digital image data) obtained by the imaging device 10.

[0015] 1.1. Imaging Device The imaging device 10 generates digital image data of the substrate for inspection by irradiating the substrate to be inspected with illumination light and taking a picture with a digital camera. Figure 2 is an example of an image taken with pseudo-coaxial incident illumination. The pseudo-coaxial incident illumination method is a known illumination method that can be used in this disclosure. The pseudo-coaxial incident illumination method is an illumination method that directly illuminates the object, and without passing through an optical lens, the illumination light path is made the same as the optical axis of the lens using a half mirror or beam splitter, and the object is illuminated perpendicularly.

[0016] 1.2.Flaw identification device The scratch detection device 20 performs image processing on the digital image obtained by the imaging device 10 to detect scratches. As conceptually shown in Figure 3, the scratch detection device 20 includes a CPU (Central Processing Unit) 21, which is a processor and performs image processing calculations; RAM (Random Access Memory) 22, which functions as a work area; ROM (Read-Only Memory) 23, which functions as a recording medium; a receiving unit 24, which is an interface that accepts information into the scratch detection device 20 whether wired or wireless; and a transmitting unit 25, which is an interface that sends information from the scratch detection device 20 to the outside whether wired or wireless. Therefore, the scratch detection device 20 is connected to the imaging device 10 and other devices via the receiving unit 24 and the transmitting unit 25, and is configured to transmit and receive signals.

[0017] The scratch detection device 20 stores a program that performs image processing to detect scratches. In the scratch detection device 1, the CPU 21, RAM 22, and ROM 23, which serve as hardware resources, work in cooperation with the program. Specifically, the CPU 21 executes the computer program recorded in ROM 23 in RAM 22, which functions as a work area, thereby realizing various functions, including scratch detection. Information acquired or generated by the CPU 21 is stored in RAM 22. In addition, the scratch detection device 20 may be equipped with recording media located inside or outside it, on which the program and various data may be recorded.

[0018] In this configuration, the scratch detection device 20 acquires a digital image of the substrate to be inspected from the imaging device 10 via the receiving unit 24. Based on the data of the acquired digital image, the scratch detection device 20 executes a computer program recorded in the ROM 23 or other recording media, using a database recorded in the ROM 23 or other recording media, to perform image processing, identify scratches, and record the results in the RAM 22 or recording media. The specific details of the scratch detection process will be explained later. The results of the scratch detection are output from the transmitting unit 25 to the monitor 27 and other devices, along with other necessary information.

[0019] Such a scratch detection device 20 can typically be configured using a computer. Therefore, the scratch detection device 20 has an input means (such as a keyboard or mouse) 26 connected to the receiving unit 24 for operating the scratch detection device 20.

[0020] 2. Scratch detection Next, we will explain the procedure for detecting damage. In this configuration, scratch detection is performed by processing the digital image data obtained by the imaging device 10 with the scratch detection device 20, as described above. Therefore, the method of scratch detection will be described here. Specifically, to execute this method, a computer program having each step corresponding to each process of the method is created, stored in the ROM 23 or recording medium of the scratch detection device 20, and executed to perform scratch detection. Below, a scratch detection method according to one example will be described, but as described above, the computer program based on this method functions as one component of the scratch detection device 1 by being recorded in the ROM 23 or recording medium of the scratch detection device 20.

[0021] Figure 4 shows the flow of the scratch detection method S1 for one form. As can be seen from Figure 4, the scratch detection method S1 of this disclosure has a process S10 for extracting candidate scratch regions and a scratch discrimination process S20. Each process will be described below.

[0022] 2.1. Process of extracting candidate injury regions In the process of extracting potential damage areas S10, a filter is applied to extract potential damage areas. Figure 5 shows the flow of the process of extracting potential damage areas S10. As can be seen from Figure 5, the process of extracting potential damage areas S10 includes a grayscale conversion process S11, a binarization process S12, a process of extracting potential damage areas S13, and a process of determining the position of the original image S14.

[0023] 2.1.1. Grayscale Conversion Process In the grayscale conversion process S11 (sometimes referred to as "process S11"), the original image (digital image, e.g., Figure 2) obtained by the imaging device 10 is converted to grayscale. The conversion to grayscale is well known. Figure 6 shows the image obtained by converting the image of Figure 2 to grayscale. In this embodiment, the RGB values ​​were converted to grayscale values ​​by calculating a weighted sum of the R, G, and B components for each pixel of the image as shown in the following formula. Grayscale value = 0.2989·R + 0.5870·G + 0.1140·B

[0024] 2.1.2. The process of binarization In the binarization process S12 (sometimes referred to as "process S12"), the grayscale data obtained in process S11 is binarized. The binarization method is well known, but one example is Otsu's binarization method (discriminant analysis method). Figure 7 shows the image obtained by binarizing Figure 6.

[0025] 2.1.3. Extraction of potential injury regions In the process of extracting candidate defect regions S13 (sometimes referred to as "process S13"), features are extracted from the binarized image obtained in process S12 to identify regions that are candidates for defects. In this disclosure, linear filters are used to obtain candidate defect regions for feature extraction. More specifically, in this embodiment, four types of linear filters are used as shown in Figure 8: vertical ((a)), horizontal ((b)), right diagonal ((c)), and left diagonal ((d)). In this embodiment, since the linear filters are linear filters, the size of the filter (the length of the filter in this embodiment) is expressed as 2k+1. The specific value of k is determined through testing to find a value that is effective for defect detection.

[0026] The four types of linear filters used and the binarized image obtained in process S12 are convolved using the exclusive sum. Below are the feature extraction calculation formulas (1) to (4) by the convolution operation. Formula (1) is for the vertical linear filter (Figure 8(a)), formula (2) is for the horizontal (Figure 8(b)), formula (3) is for the right diagonal (Figure 8(c)), and formula (4) is for the left diagonal (Figure 8(d)). In these formulas, x is the x-coordinate, y is the y-coordinate, N(N1~N4) is the extracted feature quantity, f(f1~f4) is the linear filter, and P is the value of the binarized image at the corresponding coordinate (0 or 1, i.e., black or white).

[0027]

number

[0028] As can be seen from the right-hand side of equations (1) to (4), N is obtained as the solution to the exclusive sum of the linear filter (f) and the binarized image (P). That is, features of potential defects are extracted by comparing the linear filter (f) and the binarized image (P). When f=0 (black) and P=1 (white), and when f=1 (white) and P=0 (black), they are judged as potential defects, and when f=P=0 (white) and f=P=1 (black), they are judged as not being potential defects.

[0029] Note that the term "candidate injury region" means that there is a high probability that one of the obtained candidate injury regions contains an injury, and does not mean that all of the candidate injury regions are injuries at this stage of the process. Injuries are identified from these candidate injury regions using the injury identification process S20 described later.

[0030] 2.1.4. Determining the position in the original image The process of determining the position in the original image, S14 (sometimes referred to as "process S14"), involves applying the candidate scratch area obtained in process S13 to the original image (generated digital image data, Figure 2) to determine the position of the region on the metal-ceramic composite substrate that has become the candidate scratch area. Figure 9 shows an image representing the candidate scratch area. The square area in Figure 9 is the candidate scratch area. In the example shown in Figure 9, the location of scratches on the metal-ceramic composite substrate from which the original image was derived was investigated beforehand. As a result, it was found that two of the candidate scratch areas shown in Figure 9 were indeed scratches. Furthermore, there were no scratches in the areas that were not identified as candidate scratch areas.

[0031] 2.2. The process of identifying injuries 2.2.1. Identifying the wound In the defect detection process S20, for each candidate defect region obtained in process S14, it is determined whether or not it is a defect in the metal-ceramic composite substrate that is the subject of inspection. This disclosure uses deep neural networks, specifically convolutional neural networks (CNNs), to perform discrimination. Figure 10 shows an overview of a CNN. As is well known, a CNN comprises an input layer, multiple hidden layers, and an output layer, and at least a portion of these layers are composed of convolutional and pooling layers. When an image of a candidate scratch region extracted in the process of extracting candidate scratch regions S10 is input to the input layer, calculations are performed sequentially in multiple hidden layers, and finally the output layer outputs a result indicating whether or not a scratch is present in this candidate scratch region. This allows for the detection of scratches in the metal-ceramic composite substrate being inspected.

[0032] 2.2.2. Learning To determine the CNN used for scratch detection, it is necessary to specifically define the weights and biases in the hidden layers, which are done through learning. In this model, learning was performed as follows, and the results were also verified.

[0033] [Correct labeling] The candidate scratch regions obtained using the same method as in the process of extracting candidate scratch regions S10 are compared with the actual scratches on the metal-ceramic composite substrate that were the target of the scratch candidate region detection. For each candidate scratch region, scratch regions with actual scratches are designated as "scratch region images," and scratch candidate regions without scratches are designated as "unscratched region images," and these are then labeled as correct. Figure 11 shows examples of scratch region images (a) and unscratched region images (b), respectively. In this embodiment, the condition is that a candidate region containing part of a defect is treated as a "defect region image," while small defects that are indistinguishable from grain boundaries are treated as "undefect region images."

[0034] [Number of data points] As described above, multiple "damaged region images" and "undamaged region images" with correct labels were prepared and used as training data (labeled data) and test data. Specifically, the following was done: Dataset A consisted of 242 images of "damaged areas" and 1100 images of "undamaged areas," for a total of 1342 images. 80% of these were used as training data and 20% as test data. In dataset B, dataset A was expanded to include 1649 "damaged area images" and 1649 "undamaged area images," for a total of 3388 images. Of these, 80% were used as training data and 20% as test data. In this data set B, the expansion involves adding the following images to the "injured area images": for each of the 242 "injured area images" in data set A, an image shifted 5 pixels to the left, an image shifted 5 pixels to the right, an image shifted 5 pixels up, an image shifted 5 pixels down, an image flipped vertically, and an image flipped horizontally (242 × 7 = 1694 images). For the "non-injured area images," 1100 to 1694 images were added randomly.

[0035] [Learning methods] In this example, we trained the CNN for scratch detection using the following four learning methods (Learning Example 1 to Learning Example 4). In training example 1, the CNN used for scratch detection was determined by training it using dataset B under conditions without prior training. In training example 2, we used dataset A to retrain the pre-trained AlexNet to determine the CNN to be used for scratch detection. In training example 3, we used dataset B to retrain the pre-trained AlexNet to determine the CNN to be used for scratch detection. In training example 4, the CNN used for scratch detection was determined by retraining the pre-trained GoogLeNet using dataset B. The scratch detection accuracy of the CNN determined for each training example was calculated. The results are shown in Table 1. Here, the scratch detection accuracy is expressed as a percentage of the percentage of all images used for scratch detection that were correctly classified as having a scratch or not having one.

[0036] [Table 1]

[0037] As can be seen from the above, the scratch detection method described herein was able to distinguish scratches from other elements such as grain boundaries with a high accuracy of over 90%. Among these methods, applying a CNN trained using a pre-training tool further improved accuracy. Furthermore, comparing training example 2 and training example 3, it can be seen that the accuracy has improved due to data augmentation.

[0038] As an example, in training example 1 (no pre-training), when we used the constructed CNN classifier to detect defects in a metal-ceramic composite substrate that was not used in training, following the procedure in process S1 above, 62 candidate defect regions were extracted in process S10, and in process S20, we determined whether each candidate defect region was a defect or not. Of the 62 determinations, 55 were correctly detected as either a defect or not. Furthermore, the accurate detection of scratches was also achieved from the perspective that there were no scratches on the actual metal-ceramic composite substrate in locations other than the extracted scratch candidate areas.

[0039] As another example, when we used the CNN constructed using training example 4 to perform scratch detection on a metal-ceramic composite substrate in one example, following the procedure in process S1 above, 49 candidate scratch regions were extracted in process S10, and in process S20, we determined whether each candidate scratch region was a scratch or not. Of the 49 regions, we were able to correctly detect whether it was a scratch or not in 47 of them. Furthermore, the accurate detection of scratches was also achieved from the perspective that there were no scratches on the actual metal-ceramic composite substrate in locations other than the extracted scratch candidate areas. [Explanation of symbols]

[0040] 1. Scratch detection device 10 Imaging device 20 Scratch identification device

Claims

1. A method for detecting defects in a metal-ceramic composite substrate, The aforementioned metal-ceramic composite substrate is imaged, From the image obtained by the aforementioned imaging, a filter is used to extract vertical, horizontal, right-diagonal, and left-diagonal lines to identify potential damage areas. A convolutional neural network is used to determine whether or not a defect is present in the candidate defect region. A method for detecting surface defects in a metal-ceramic composite substrate.

2. A device for detecting scratches in a metal-ceramic composite substrate, Imaging device and The system includes a scratch detection device that processes images obtained by the aforementioned imaging device, The aforementioned scratch detection device processes the image using filters that extract vertical, horizontal, right-diagonal, and left-diagonal lines to extract candidate scratch regions. A convolutional neural network is used to perform calculations on the candidate region of a potential injury to determine whether or not the region contains an injury. A device for detecting defects in metal-ceramic composite substrates.

Images