Void and crack detection device for joints connecting electronic components

Abstract

To provide a detection device capable of automatically detecting voids with different shades and a wide variety of cracks.SOLUTION: An automatic detection device 1 includes a control unit 11 that controls the entire device, a main memory unit 12, a communication unit 13, an operation unit 14, a display panel 15, and an auxiliary memory unit 16. Images captured by apparatuses of an ultrasonic microscope 2 and an X-ray CT device 3 are uploaded to the auxiliary memory unit 16 by a captured image upload program 103 and saved in a captured image (1) DB 104. Using a machine learning model trained using a machine learning approach including deep learning, the captured images are binarized and void / crack areas on the images are visualized. The automatic detection performance of the automatic detection device 1 is improved by collecting transmitted X-ray images and ultrasonic microscope images, and training again.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004] ,

[0005] , , , , , , ,

[0001] The present invention relates to a visualization device, a visualization method, and a computer program, etc., which use transmission X-ray images and ultrasonic microscope images to non-destructively visualize spaces, voids, cracks, etc. generated inside an object or at a joint of an electronic component, a semiconductor element, a thin film device, a thick film device, an electronic device, etc.

Background Art

[0002] The terminals of an electronic component and the electrodes of a substrate on which the electronic component is mounted are soldered. Voids 36 are generated in solder joints and the like due to preheating, reflow conditions, etc. Also, the electronic component and the substrate repeatedly expand and contract due to thermal stress, and cracks 39 are generated in joints and the like due to the difference in their expansion rates. In order to improve the problem of the substrate yield of an electronic circuit, the detection and quality inspection of voids 36 and cracks 39 in joints and the like are important. Conventionally, as a method for non-destructively inspecting joints such as solder, a method of visually inspecting transmission X-ray images and ultrasonic microscope images has been applied.

[0003] Visual inspection can capture slight image changes, but there is a large variation in the pass / fail judgment by inspectors, and the inspection locations are also limited due to time constraints. In visual inspection, it is difficult to accurately detect and measure voids 36 and cracks 39, and the process takes a long time.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Attempts have been made to automate and visualize the detection of voids 36 and cracks 39 in order to achieve high-quality, rapid, and non-destructive inspection. However, while image processing methods such as binarization of transmission X-ray images and ultrasound microscope images are common for automation and visualization, the binarization threshold must be set appropriately, making it impossible to automatically detect voids 36 with different shading and a wide variety of cracks 39 at once.

[0006] In particular, with transmission X-ray images, it was difficult to achieve robustness and automatically detect voids 36 and cracks 39 due to challenges such as reflections from vias, plating or vias, inconsistent illumination, and noise.

[0007] In other words, conventional image processing methods require humans to define each image feature, such as brightness values, individually in order to detect voids 36 and cracks 39, making it impossible to detect the wide variety of voids 36 and cracks 39. [Means for solving the problem]

[0008] To address these challenges, the present invention takes a machine learning approach, including deep learning, considering the need for high-quality and rapid inspection of the inside of objects and joints. The AI ​​learns image features that humans cannot define, enabling high-precision detection of a wide variety of voids 36 and cracks 39, significantly reducing conventional inspection time.

[0009] The present invention relates to a method for visualizing the inside of an object, characterized in that the method comprises a first operation of classifying at least one of a transmitted X-ray image acquired by an X-ray device and an ultrasonic image acquired by an ultrasonic device into a plurality of image patterns using a first machine learning model, and a second operation of detecting a predetermined region from the image patterns using a second machine learning model.

[0010] In the first operation described above, the performance of the newly trained machine learning model is periodically evaluated by the training program for image pattern classification, and if degradation is detected, the machine learning model is rebuilt and retrained.

[0011] The system further includes a training program for region detection, and is characterized by periodically evaluating the performance of newly trained machine learning models, and rebuilding and retraining the model if degradation is detected.

[0012] The method is characterized by performing at least one of the following processes—noise reduction, contrast processing, brightness processing, smoothing, enlargement, reduction, rotation, translation, and masking—on at least one of the images obtained from a transmission X-ray device and an ultrasound device: noise reduction, contrast processing, brightness processing, smoothing, enlargement, reduction, rotation, translation, and masking, in order to generate an image pattern for the second machine learning model.

[0013] The present invention relates to a computer program for visualizing the interior of an object, characterized in that the program visualizes the interior of an object, comprising: a first process of classifying at least one of a transmitted X-ray image acquired by an X-ray device and an ultrasonic image acquired by an ultrasonic device into a plurality of image patterns using a first machine learning model; and a second process of visualizing the interior of the object by detecting a predetermined region from the image patterns using a second machine learning model.

[0014] In the first process described above, the performance of the newly trained machine learning model is periodically evaluated by the training program for pattern classification, and if degradation is detected, the model is rebuilt and retrained to ensure pattern classification performance.

[0015] The system further includes a training program for region detection, and is characterized by the fact that the performance of newly trained machine learning models is periodically evaluated, and if degradation is detected, the model is automatically rebuilt and retrained, thereby ensuring region detection performance.

[0016] The method is characterized by performing at least one of the following processes—noise reduction, contrast processing, brightness processing, smoothing, enlargement, reduction, rotation, translation, and masking—on at least one of the images obtained from a transmission X-ray device and an ultrasound device: noise reduction, contrast processing, brightness processing, smoothing, enlargement, reduction, rotation, translation, and masking, in order to generate an image pattern for the second machine learning model.

[0017] The present invention relates to a method for visualizing the inside of an object, characterized by performing a first process by predicting and dividing the region into multiple regions during region detection, performing a second process to integrate the processing results from the first process, and calculating the average of the processing results for regions that were detected overlapping in the first process.

[0018] The present invention relates to a computer program for visualizing the interior of an object, characterized in that it performs a first process of dividing the object into multiple regions using a prediction program for region detection, performs a second process of integrating the processing results from the first process, and calculates the average of the processing results for regions that were detected overlapping in the first process. This invention provides an automated void and crack detection system for solder joints, etc., implemented on a cloud service, and offers a void ratio and crack ratio measurement service.

[0019] The automatic detection device 1 comprises a control unit 11 for controlling the entire device, a main memory unit 12, a communication unit 13, an operation unit 14, a display panel 15, and an auxiliary memory unit 16. Images captured by the ultrasonic microscope 2 and the X-ray CT scanner 3 are uploaded to the auxiliary memory unit 16 by the captured image upload program 103 and stored in the captured image (1) DB. Image processing, including deep learning, is performed by binarizing transmitted X-ray images. By being used by users, transmitted X-ray images and ultrasonic microscope images are collected, and the automatic detection performance is improved through learning. [Effects of the Invention]

[0020] According to the present invention, in the conventional void inspection and crack inspection of solder joints by image analysis, it was impossible to detect voids with different shadows, various cracks, etc. at once.

[0021] By taking an approach of machine learning including deep learning, the present invention enables automatic detection of voids 36 and cracks 39 inside substances, joints, etc. with high precision, and can significantly shorten the conventional inspection time, detection time, etc.

Brief Description of the Drawings

[0022] [Figure 1] It is an explanatory diagram of a void and crack visualization device for the inside of an object and joints of the present invention. [Figure 2] It is an explanatory diagram of a method for visualizing voids and cracks in the inside of an object and joints of the present invention. [Figure 3] It is an explanatory diagram of a method for pattern classification of a photographed image in the method for visualizing the inside of an object and joints of the present invention. [Figure 4] It is an explanatory diagram of the method for visualizing the inside of an object and joints of the present invention. [Figure 5] It is a schematic diagram schematically showing a transmission X-ray image of a void generated in a solder part. [Figure 6] It is an explanatory diagram explaining a method for machine learning of voids and cracks generated in the inside of an object and joints of the present invention. [Figure 7] It is an explanatory diagram explaining a method for machine learning of voids and cracks generated in the inside of an object and joints of the present invention. [Figure 8] It is an explanatory diagram explaining a method for dividing an image of voids and cracks generated in the inside of an object and joints into patch images. [Figure 9] It is an explanatory diagram explaining a method for updating machine learning data for reconstructing and retraining a machine learning model for automatically detecting voids and cracks generated in the inside of an object and joints of the present invention. [Figure 10] It is an explanatory diagram of the method for visualizing the inside of an object and joints of the present invention and the computer program. [Figure 11] This is an explanatory diagram of the present invention regarding a method for visualizing the interior and joints of an object and the computer program said therefor. [Figure 12] This is an explanatory diagram of a method for visualizing the interior and joints of an object, and the computer program for which it was developed. [Figure 13] This invention relates to a method for visualizing the interior and joints of an object, and an explanatory diagram of the computer program said therefor. [Modes for carrying out the invention]

[0023] The present invention will be described below with reference to drawings illustrating the implementation.

[0024] In the embodiments described in the specification, a solder joint of electronic component 4 is used as an example for ease of understanding, but the invention is not limited thereto. The technical concept of the present invention can be applied to many objects and various types of structures, such as buildings, electrical equipment, and internal diagnostic equipment for the human body.

[0025] In each drawing illustrating embodiments for carrying out the invention, elements having the same function are denoted by the same reference numeral, and their descriptions may be omitted. Furthermore, embodiments of the present invention can be combined from one embodiment to another. In addition, elements may be enlarged, reduced, or omitted for purposes such as facilitating understanding or simplifying the illustration.

[0026] Figure 1 is a block diagram showing the configuration of a void / crack automatic detection (visualization) device 1, an ultrasonic microscope 2, an X-ray CT (Computed Tomography) device 3, and a joint between electronic components 4 and solder 5. In the examples described herein, the pattern 40 is described as being acquired using an X-ray CT scanner 3, but this may be replaced with an X-ray fluoroscopy device or an ultrasonic microscope 2.

[0027] An X-ray microscope (X-ray CT) is a microscope that non-destructively observes the interior of an object. Unlike an X-ray fluoroscopy device, it converts X-rays that have passed through the sample into light and magnifies it with an optical lens. X-rays have the property of penetrating matter. When X-rays pass through a sample, some are absorbed. The rate of absorption increases with higher material density (higher atomic number) and thickness, resulting in a lower intensity of transmitted X-rays.

[0028] If voids 36 and cracks 39 occur within the object, the X-ray transmittance of the voids 36 and cracks 39 increases, and therefore the voids 36 and cracks 39 are displayed in pattern 40.

[0029] The X-ray CT scanner 3 uses 360° (DEG.) X-ray transmission information of the sample under observation to perform computer calculations and construct 3D data of the sample. By narrowing the rotation pitch and obtaining more information, highly accurate 3D data can be obtained.

[0030] X-ray images are basically two-dimensional images. Numerous voids 36 and cracks 39 occur, and these voids 36 and cracks 39 are distributed in three dimensions. Therefore, the acquired X-ray image will have a pattern 40 in which voids 36 and cracks 39 overlap.

[0031] Ultrasonic microscope 2 is a microscope that non-destructively observes defects such as delamination, voids, cracks, and foreign matter inside an object. When ultrasound propagates through different materials, some of it is reflected and some is transmitted. If there are delamination areas (voids) in the object being observed, for example, strong reflected waves will be detected. By performing a planar scan of the object being observed and marking the positions where these reflected waves are detected, it is possible to obtain a two-dimensional distribution of delamination areas (voids). Therefore, the resulting ultrasonic microscope image will be similar to an X-ray image, resembling a pattern 40 with overlapping voids 36 and cracks 39.

[0032] The void crack automatic detection (visualization) device 1 of the present invention has the function of automatically observing, inspecting, judging, and determining the location and occurrence state of void cracks through predetermined operations, and automatically visualizing and detecting void cracks. Hereinafter, the void crack automatic detection (visualization) device will be referred to as an automatic detection device or a visualization device.

[0033] The X-ray CT apparatus 3 of the present invention can obtain the internal structure of an object by performing a reconstruction process that utilizes the difference in "ease of penetration" and "ease of absorption" when X-rays pass through an object.

[0034] The ultrasonic microscope 2 of the present invention can obtain the internal structure of an object by performing a reconstruction process that utilizes the difference in "ease of transmission" and "ease of reflection" when ultrasonic waves propagate through the object. The automatic detection device 1 comprises a control unit 11 that controls the entire device, a main memory unit 12, a communication unit 13, an operation unit 14, a display panel 15, and an auxiliary memory unit 16.

[0035] The automatic detection device 1 can be composed of, for example, a desktop computer, a notebook personal computer, a tablet, a smartphone, or a computer in a cloud environment that provides computer resources as a service via a computer network such as the internet.

[0036] The control unit 11 can be composed of a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc. The control unit 11 may also include a GPU (Graphics Processing Unit).

[0037] The main memory unit 12 is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory, and temporarily stores the data necessary for the control unit 11 to perform calculation processing. The communication unit 13 has the function of communicating with the automatic detection device 1 via the network 17 and can send and receive necessary information. The control unit 14 is composed of, for example, a hardware keyboard, mouse, touch panel, etc.

[0038] The display panel 15 can be made of a liquid crystal panel or an organic EL (Electro-Luminescence) display panel, etc. The control unit 11 performs control to display the required information on the display panel 15.

[0039] The auxiliary storage unit 16 is a large-capacity memory, hard disk, etc., and stores the programs necessary for the control unit 11 to execute processing, as well as the user authentication program 101, the case management program 102, the captured image upload program 103, the machine learning program 110, the automatic detection program 111, the analysis result editing and viewing program 112, the automatic analysis report creation program 113, the analysis report notification program 114, and the analysis report editing and viewing program 115.

[0040] The user authentication program 101, case management program 102, captured image upload program 103, machine learning program 110, automatic detection program 111, analysis result editing and viewing program 112, automatic analysis report creation program 113, analysis report notification program 114, and analysis report editing and viewing program 115, which are stored in the auxiliary storage unit 16, may be provided by a recording medium 18 on which each program is recorded in a readable manner.

[0041] The recording medium 18 is, for example, a portable memory such as a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, a microSD card, or a CompactFlash (registered trademark).

[0042] Each program recorded on the recording medium 18 is provided via communication through the communication unit 13. Alternatively, it may be read from the recording medium 18 using a reading device (not shown in the figure) and installed in the auxiliary storage unit 16. Next, we will explain how images taken by the ultrasonic microscope 2 and the X-ray CT scanner 3 are stored in the auxiliary storage unit 16.

[0043] If the ultrasonic microscope 2 and X-ray CT scanner 3 are connected to the internet, user authentication is automatically performed by the user authentication program 101 based on the configuration information of the ultrasonic microscope 2 and X-ray CT scanner 3, and case registration is automatically performed by the case management program 102.

[0044] Images captured by the ultrasonic microscope 2 and the X-ray CT scanner 3 are uploaded to the auxiliary storage unit 16 by the image upload program 103 and stored in the image database (1).

[0045] On the other hand, if the ultrasound microscope 2 or X-ray CT scanner 3 is not connected to the internet, the images taken from the equipment are retrieved and copied to a computer or other device connected to the internet.

[0046] On the computer, the user authentication program 101 performs the login operation, and the case management program 102 registers the case. Next, the captured image upload program 103 uploads the image to the auxiliary storage unit 16, where it can be saved as captured image (1) DB 104.

[0047] In the auxiliary storage unit 16, uploaded image data is stored in Captured Images (1) DB104, the results of automatic detection of voids and cracks are stored in Analysis Results (1) DB105, the analysis report containing the void rate and crack rate automatically measured from the automatic detection results is stored in Analysis Report (1) DB106, and the dataset created using Captured Images (1) DB104 and Analysis Results (1) DB105 is stored in Machine Learning Data (1) DB107, each for each user, group, and project. Only users authenticated by the user authentication program 101 can access each of these data sets.

[0048] Furthermore, the auxiliary storage unit 16 stores an automatic detection program 111. This automatic detection program 111 uses a machine learning model (pattern classification) DB108 and a machine learning model (region detection) DB109 to perform automatic detection of voids and cracks in the captured images (1) DB104. In addition, the captured images (1) DB104 can be processed in batches at regular intervals. Details of this automatic detection program 111 will be described later.

[0049] The results automatically detected by the automatic detection program 111 are stored in the analysis results (1) DB 105. The stored analysis result data is read by the analysis result editing and viewing program 112, and users can edit and view the analysis results. In this editing process, users authorized by the user authentication program 101 can collaboratively edit the data.

[0050] The automated analysis report generation program 113 reads the images stored in the captured images (1) DB104 and the analysis results (1) DB105, and automatically generates numerical data such as void ratio and crack ratio, as well as text data such as the cause of defects, according to the format specified by the user. It then compiles these into an analysis report and saves it in the analysis report (1) DB106.

[0051] If a new or updated analysis report is available, the analysis report notification program 114 will notify the user of its contents via email, message, chat service, etc. The user can then access the analysis report editing and viewing program 115 to edit and view the report. This editing process can be collaboratively performed by users authorized by the user authentication program.

[0052] The auxiliary storage unit 16 stores a machine learning program 110. This machine learning program 110 uses captured images (1) DB104 and analysis results (1) DB105 to create machine learning data (1) DB, performs machine learning using this dataset, and creates a machine learning model (pattern classification) DB108 and a machine learning model (region detection) DB109.

[0053] Regarding the datasets used for machine learning, sometimes individual databases of captured images and analysis results, stored for each user, group, or project, are used, while other times they are shared and used together.

[0054] The created machine learning models (pattern classification) DB108 and (region detection) DB109 are used in the aforementioned automatic detection program 111. Therefore, the machine learning program 110 is executed at a different time than when the automatic detection program 111 is executed (for example, during times when the user is not using the system).

[0055] Figure 2 illustrates the details of the automatic detection program 111. First, the newly uploaded image data 21 from the ultrasound microscope 2 and X-ray CT scanner 3 are loaded into the pattern classification prediction program 201.

[0056] The pattern classification and prediction program 201 classifies patterns in images. Examples of patterns to be classified include those with different mounted electronic components such as chip capacitors, resistors, and coils; those with different pin configurations of IC packages, such as leaded and leadless types; those with different substrate types, such as surface mount and through-hole; those with different materials of electronic components, such as copper, brass, and iron; and those with different imaging methods, such as optical microscopes and electron microscopes. The captured image group 21 is classified by the pattern classification prediction program 201 into, for example, pattern A, pattern B, and pattern C.

[0057] The image set for Pattern A is loaded into the Pattern A region detection and prediction program 202. The image set for Pattern B is loaded into the Pattern B region detection and prediction program 203. The image set for Pattern C is loaded into the Pattern C region detection and prediction program 204, after which each program automatically detects the void 36 regions and crack 39 regions. The resulting analysis images are saved as the analysis result (mask image) group 22.

[0058] This invention focuses on the fact that there is an optimal region detection method for each pattern, and therefore provides a region detection program for each pattern. This improves the accuracy of region detection compared to not providing a separate program for each pattern. For this reason, a separate machine learning model must be prepared for each pattern. Figure 3 illustrates the processing flow of the training program and prediction program in the pattern classification described above.

[0059] The training program for pattern classification aims to create a machine learning model (pattern classification) DB108. For example, if a set of captured images contains two types of patterns, A and B, a machine learning model is created that can classify whether a given image is pattern A or pattern B, using the captured image (AB) DB301 and the pattern (AB) DB302, which allows referencing whether each captured image is pattern A or pattern B.

[0060] The data augmentation preprocessing program 303 reads the captured images (AB)DB301 and patterns (AB)DB302 and creates a dataset for machine learning. The created dataset is saved in the machine learning data (AB)DB304.

[0061] Next, the machine learning (training) program 305 reads the machine learning data (AB) DB304, performs training, and saves the trained machine learning model to the machine learning model (pattern classification) DB108. When performing training, we may also use existing machine learning models (pattern classification) from DB108.

[0062] When training a machine learning model, it's sometimes possible to differentiate the importance of data by assigning specific weights to machine learning data from certain users, groups, or projects, or to machine learning data with recent upload dates.

[0063] In performance evaluations of newly trained machine learning models, if they are determined to have higher accuracy than existing machine learning models, they may be automatically replaced with existing models in the Machine Learning Models (Pattern Classification) DB108.

[0064] In performance evaluation of newly trained machine learning models, if they are determined to have lower accuracy than existing machine learning models, the trained models may be automatically discarded.

[0065] As described above, a key feature of the present invention is that, through a training program for pattern classification, the performance of newly trained machine learning models is periodically evaluated, and if degradation is detected, the model is automatically rebuilt and retrained.

[0066] In Figure 3, in the pattern classification prediction program, the preprocessing program 307 reads the captured image group 306 and performs image preprocessing. The preprocessed images are read by the machine learning (prediction) program 308, which reads the machine learning model trained by the training program from the machine learning model (pattern classification) DB 108 and performs pattern classification. The patterns classified for the captured image group are stored in the pattern classification result group 309. Figure 4 illustrates the processing flow of the training program and prediction program in the region detection described above.

[0067] The training program for region detection aims to create a machine learning model (region detection) DB109. Using the captured image (A) DB401 and the mask image (A) DB402, a machine learning model is created that can detect which regions of the captured image correspond to voids 36 and cracks 39.

[0068] The data augmentation preprocessing program 403 reads the captured image (A)DB401 and the mask image (A)DB402 and creates a dataset for machine learning.

[0069] The created dataset is saved in the machine learning data (A) DB404. Subsequently, the machine learning (training) program 405 reads the machine learning data (A) DB404, performs training, and saves the trained machine learning model in the machine learning model (region detection) DB109. When training, we may use existing machine learning models in the machine learning model (region detection) DB109.

[0070] When training a machine learning model, it's sometimes possible to differentiate the importance of data by assigning specific weights to machine learning data from certain users, groups, or projects, or to machine learning data with recent upload dates.

[0071] In performance evaluations of newly trained machine learning models, if they are determined to have higher accuracy than existing machine learning models, they may be automatically replaced with existing models in the Machine Learning Models (Pattern Classification) DB108.

[0072] In performance evaluation of newly trained machine learning models, if they are determined to have lower accuracy than existing machine learning models, the trained models may be automatically discarded.

[0073] As described above, a key feature of the present invention is that the performance of a newly trained machine learning model is periodically evaluated by the training program in region detection, and if degradation is detected, the model is automatically rebuilt and retrained.

[0074] Specific training examples will be explained using Figures 5, 6, and 7. While the acquired image pattern 40 is described as a void 36 pattern, it is not limited to this; for example, void 36 could be replaced with cracks 39, etc. Furthermore, even in patterns containing both voids 36 and cracks 39, training and learning will enable the system to distinguish and detect them.

[0075] Figure 5 schematically illustrates the occurrence of voids 36 in the solder joint 5 as captured by a transmission X-ray image (pattern 40). Normally, voids 36 do not form a circle as shown in Figure 5, but rather multiple circles are connected, resulting in shadows and reduced contrast.

[0076] In the following embodiments, pattern 40 will be described as pattern 40 acquired by a transmission X-ray apparatus, but it goes without saying that pattern 40 can also be acquired by an X-ray CT apparatus. Thousands of transmission X-ray images and images in which the void regions 36 are masked (data obtained through manual inspection) are prepared.

[0077] These are randomly cut out as 256x256 (pixel) patch images (section 37), and then one or more processing steps, or combinations of multiple processing steps, are applied to augment the data, such as increasing noise, adjusting contrast, adjusting brightness, inverting brightness, smoothing, scaling, rotating, shifting horizontally or vertically, or partially masking (filling with black), to generate a dataset of hundreds of thousands of images.

[0078] As an example, in Figure 5, the soldered portion 38 is represented by four patch images (sections 37a, 37b, 37c, and 37d). It is preferable that each section 37 has the same shape and area.

[0079] Each patch image (section 37) contains a void 36. As shown in Figure 6(a), section 37a is inverted in black and white and used as training data. Also, as shown in Figure 6(b), section 37a is enlarged and used as training data. Furthermore, as shown in Figure 6(c), section 37a is reduced and used as training data.

[0080] Section 37a, as shown in Figure 7(a), is rotated to the left by a predetermined angle (10° in the figure) as shown in Figure 6(b) to create training data. Then, as shown in Figure 6(c), it is rotated to the right by a predetermined angle (10° in the figure) to create training data. Alternatively, section 37 may be inverted by line symmetry.

[0081] A key feature of this invention is that, even when there is limited machine learning data available, the amount of data can be expanded by methods such as black and white inversion, rotation, reduction, and scaling, as described above, thereby generating a large dataset independently.

[0082] Next, 70% of the patch images (section 37) are used as training data, and the remaining 30% as test data. A deep learning model of U-Net is then performed using an encoder-decoder model for segmentation, specifically for medical segmentation (particularly cell segmentation). In other words, artificial intelligence (AI) functions are used. It is preferable to use 60% or more of the patch images (section 37) as training data. It is preferable to use 40% or less as test data.

[0083] The loss function during training is set to the Dice coefficient × (-1), where Dice coefficient is the ratio between the mask image used as training data and the mask image used as network output. The Dice coefficient uses the average number of elements in the two sets instead of the number of elements in the union of the two sets. During training, the machine learning model that minimizes the loss on the aforementioned test data is saved.

[0084] Meanwhile, in the prediction program for region detection, the preprocessing program 407 reads the captured image set 406 and performs image preprocessing. The preprocessed images are read by the machine learning (prediction) program 408, which reads the machine learning model trained by the training program from the machine learning model (region detection) DB 109 and performs region prediction. The predicted regions for the captured images are stored in the mask image set 409.

[0085] A concrete example of prediction is explained using Figure 8. The captured transmission X-ray image is divided into 256 x 256 (pixel) patch images at equal intervals. Alternatively, the patch image can be divided into multiple parts. In Figure 8, the image is divided vertically and horizontally starting from section 37e in the upper left corner. However, depending on the width and height of the captured transmission X-ray image, the patch images can be divided into sections 37f, 37g, 37h, etc., at equal intervals. Next, each patch image is input into a trained machine learning model.

[0086] The trained machine learning model obtains a probability map (values ​​from 0 to 1) indicating the presence of voids at each pixel location. To obtain a binary map (mask image), the obtained probability map is binarized using a set threshold of 0.5 (1 indicates a void, and 0 indicates the background).

[0087] The obtained 256×256 (pixel) binary maps are combined (as shown in Figure 8, divided into sections 37e, 37f, 37g, and 37h) to generate an overall binary map (mask image) corresponding to the original transmitted X-ray image.

[0088] Note that 256 is 2 to the power of 8. It is preferable to use 2 to the power of n (where n is an integer greater than or equal to 2) for each pixel. Finally, as a post-processing step, the binary map (mask image) is optimized using the CRF (Conditional Random Field) of the transmitted X-ray image. As described above, the characteristic of the present invention is that it divides the process into multiple fine regions (sections) 37, performs processing on each region, and integrates the processing results of each region to obtain a single processing result.

[0089] For example, even if a void 36 must be contained within the boundary of section 37, it may still be clearly detectable in other sections 37. For overlapping detected areas, the average of the processing results from each batch image is taken, and pixels exceeding a threshold of 0.5 are considered voids, enabling robust automatic void detection.

[0090] For example, even if void 36 cannot be detected in an image of section 37, it may be possible to detect it by rotating the image by 90 degrees. By taking the average of the processing results for images of section 37 rotated multiple times, and defining pixels exceeding the threshold of 0.5 as void 36, robust automatic detection of void 36 can be achieved.

[0091] Figure 9 illustrates how to update the machine learning data in order to rebuild and retrain the machine learning model. The automatic detection results of void 36 mentioned above are not always correct. Users can check and edit the automatic detection results of void 36 using the system's analysis result editing and viewing program 112. To improve the automatic detection performance, it is necessary to incorporate user confirmations and edits into the machine learning data of this system and update it accordingly.

[0092] Figure 9(a) shows the case where the captured image group 901 is input to the region detection and prediction program 902, and the analysis result (mask image) group 903 is output, but the user has not reviewed these analysis results. In this case, the review status and editing result (mask image) DB 904 are registered as "unreviewed," and the data is not incorporated into the machine learning data and is not updated.

[0093] Figure 9(b) shows the case where, after the captured image group 901 is input to the region detection and prediction program 902 and the analysis result (mask image) group 903 is output, the user views the analysis result for a certain period of time or presses the confirmation button or the analysis result output button. In this case, the confirmation status and editing result (mask image) DB 904 are registered as "confirmed" and incorporated into the machine learning data to update it.

[0094] Figure 9(c) shows the case where the captured image group 901 is input to the region detection prediction program 902, and after the analysis result (mask image) group 903 is output, the user edits the analysis result. In this case, the confirmation status and the edited result (mask image) DB 904 are registered as "edited," and it is determined that the analysis result was incorrect, so it is weighted and incorporated into the machine learning data to update it. Furthermore, the larger the difference between the analysis result (mask image) group 903 and the result edited by the user, the weighted it is incorporated into the machine learning data to update it. One way to apply weights is to expand the amount of data by using the aforementioned methods of inverting black and white, rotating, shrinking, and enlarging the captured image group 901 and the edited result (mask image) DB 904 in Figure 9(c), and then incorporate them into the machine learning data. As described above, the image set 901 is classified and registered as unconfirmed, confirmed, edited, etc., and then incorporated into the machine learning data to update it. By updating the machine learning data as described above, the machine learning model is automatically rebuilt and retrained, improving its automatic detection performance.

[0095] This invention employs a machine learning approach, including deep learning. By using this approach, voids 36 and cracks 39 can be detected effectively and accurately.

[0096] The embodiments of the present invention described above relate to a device and method for detecting and visualizing voids 36 inside a material. Cracks 39 can also be identified by acquiring a pattern 40 using an X-ray CT device 3, etc. Although the images of voids 36 and cracks 39 are different, by training the device to recognize them as cracks 39, it becomes possible to detect them. Therefore, it goes without saying that the items described as voids 36 in the embodiments of this specification can be replaced with items described as cracks 39.

[0097] Furthermore, pattern 40 can be acquired not only by obtaining a transmitted X-ray image using an X-ray CT scanner 3, but also, for example, by using an ultrasonic microscope 2 to obtain a pattern 40 corresponding to a transmitted X-ray image. Therefore, it goes without saying that in the apparatus of the present invention, the X-ray device such as an X-ray CT scanner may be replaced with an ultrasonic device 2 such as an ultrasonic microscope.

[0098] Furthermore, by combining an X-ray device such as an X-ray CT scanner 3 with an ultrasonic microscope 2 to acquire a pattern 40, the detection accuracy of voids 36, cracks 39, etc., is improved.

[0099] The above embodiments describe methods or apparatus for detecting voids 36, cracks 39, etc., in transmitted X-ray images, but the present invention is not limited thereto. Needless to say, it is possible to non-destructively analyze a wide variety of things, such as air bubbles in adhesive resins, spaces in concrete blocks, inclusions of other metals in iron material parts, and fat particles in the internal organs of living organisms. Figures 10, 11, 12, and 13 are explanatory diagrams of a business model using the automatic detection method and computer program of the present invention.

[0100] Figure 10 is an explanatory diagram of the business model in the first embodiment. To customer A, who possesses images taken with an ultrasonic microscope 2, an X-ray CT scanner 3, etc., company A provides or sells a program (hereinafter referred to as the "void / crack automatic detection program") that includes a user authentication program 101, a case management program 102, an image upload program 103, a machine learning program 110, an automatic detection program 111, an analysis result editing and viewing program 112, an automatic analysis report creation program 113, an analysis report notification program 114, an analysis report editing and viewing program 115, etc., as shown in Figure 1.

[0101] Customer A will be able to install and use the void crack automatic detection program on their desktop computer, laptop computer, tablet, and smartphone. Company A will receive a license fee for the void crack automatic detection program from Customer A.

[0102] Figure 11 is an explanatory diagram of the business model in the second embodiment. To customer A, who does not possess images taken with an ultrasonic microscope 2, an X-ray CT scanner 3, etc., company A provides or sells a collection of images taken with company A's own ultrasonic microscope 2, an X-ray CT scanner 3, etc., and an automated void / crack detection program.

[0103] Customer A will be able to install and use the void crack automatic detection program on their desktop computer, laptop computer, tablet, and smartphone. Company A will receive image acquisition fees and license fees for the void crack automatic detection program from Customer A.

[0104] Figure 12 is an explanatory diagram of the business model in the third embodiment. For customer A, who possesses images taken with an ultrasonic microscope 2, an X-ray CT scanner 3, etc., company A operates an automated void / crack detection program on a cloud server.

[0105] Customer A can use the void crack automated detection program using a web browser on a desktop computer, laptop computer, tablet, or smartphone. Company A receives a fee from Customer A for using the void crack automated detection program.

[0106] Figure 13 is an explanatory diagram of the business model in the fourth embodiment. For customer A, who does not possess images taken with an ultrasonic microscope 2, X-ray CT scanner 3, etc., company A prepares a set of images taken with its own ultrasonic microscope 2, X-ray CT scanner 3, etc., and runs an automated void / crack detection program on a cloud server.

[0107] Customer A can use the void crack automatic detection program using a web browser on a desktop computer, laptop computer, tablet, or smartphone. Company A receives image acquisition fees and usage fees for the void crack automatic detection program from Customer A.

[0108] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims, not in the sense described above, and all modifications are intended to be in the sense and scope equivalent to the claims. Needless to say, the matters or contents described herein and in the drawings can be combined with each other. [Explanation of symbols]

[0109] 1. Automatic Void and Crack Detection System 2. Ultrasonic Microscope 3 X-ray CT device 4. Electronic components (measurement samples) 5 solder 11 Control Unit 12 Main memory 13 Communications Department 14 Control section 15 Display Panel 16 Auxiliary storage 17 Network 18 Recording media 21 Images 22 Analysis Results (Mask Images) Group 31 Pattern Classification Training Program 32 Pattern Classification Prediction Program 36 Void 37 categories (processing units) 38 Soldering section 39 Crack 40 patterns 41. Area detection training program 42 Region Detection and Prediction Program 101 User Authentication Program 102 Project Management Program 103 Image Upload Program 104 Captured Image (1) DB 105 Analysis results (1) DB 106 Analysis Report (1) DB 107 Data for Machine Learning (1) DB 108 Machine Learning Model (Pattern Classification) Database 109 Machine Learning Models (Region Detection) Database 110 Machine Learning Programs 111 Automatic detection program 112 Analysis Result Editing and Viewing Program 113 Automatic Analysis Report Generation Program 114 Analysis Report Notification Program 115 Analysis Report Editing and Viewing Program 201 Pattern Classification Prediction Program 202 Area detection and prediction program for Pattern A 203 Area detection and prediction program for Pattern B 204 Area detection and prediction program for Pattern C 301 Captured Images (AB) DB 302 Pattern (AB) DB 303 Data augmentation preprocessing program 304 Machine Learning Data (AB) DB 305 Machine Learning (Training) Programs 306 Images 307 Preprocessing Program 308 Machine Learning (Prediction) Programs 309 Pattern Classification Results 401 Captured Image (A) DB 402 Mask Image (A) DB 403 Data augmentation preprocessing program 404 Machine Learning Data (A) DB 405 Machine Learning (Training) Programs 406 Images 407 Preprocessing Program 408 Machine Learning (Prediction) Programs 409 Analysis Results (Mask Images) Group 901 Image series 902 Region Detection and Prediction Program 903 Analysis Results (Mask Images) Group 904 Confirmation Status and Editing Results (Mask Image) DB

Claims

1. A device for detecting voids and cracks in joints connecting electronic components, A database of images in which at least one of the following images is stored: an X-ray image of the joint taken with an X-ray device and an ultrasound image of the joint taken with an ultrasound device. A classification program that classifies image patterns based on at least one of the following: the type of electronic component, the shape of the connection part of the electronic component, the type of substrate on which the electronic component is mounted, and the type of imaging device for the joint; A detection program for detecting void and crack regions at joints in the classified image pattern, The system includes an analysis results database that stores the aforementioned void and crack regions as analysis result images. A void crack detection device for joints connecting electronic components, characterized by confirming or editing the void crack region and updating the analysis results database.

2. A device for detecting voids and cracks in joints connecting electronic components, A database of images in which at least one of the following images is stored: an X-ray image of the joint taken with an X-ray device and an ultrasound image of the joint taken with an ultrasound device. An image processing program that performs at least one of the following processes on the captured image: noise reduction, contrast processing, brightness processing, smoothing, enlargement, reduction, rotation, translation, and masking. A classification program that classifies at least one of the X-ray image of the joint and the ultrasound image of the joint into image patterns based on at least one of the following: the type of electronic component, the shape of the connection part of the electronic component, the type of substrate on which the electronic component is mounted, and the type of imaging device for the joint. The system includes an analysis results database that detects void and crack regions at joints in the classified image patterns and saves these void and crack regions as analysis result images. A void crack detection device for joints connecting electronic components, characterized by confirming or editing the void crack region and updating the analysis results database.

3. A device for detecting voids and cracks in joints connecting electronic components, A database of images in which at least one of the following images is stored: an X-ray image of the joint taken with an X-ray device and an ultrasound image of the joint taken with an ultrasound device. A classification program that classifies image patterns based on at least one of the following: the type of electronic component, the shape of the connection part of the electronic component, the type of substrate on which the electronic component is mounted, and the type of imaging device for the joint; A detection program for detecting void and crack regions at joints in the classified image pattern, A void crack detection device for joints connecting electronic components, characterized by comprising an analysis result database that stores the aforementioned void crack regions as analysis result images.

4. A device for detecting voids and cracks in joints connecting electronic components, A database of images in which at least one of the following images is stored: an X-ray image of the joint taken with an X-ray device and an ultrasound image of the joint taken with an ultrasound device. An image processing program that performs at least one of the following processes on the captured image: noise reduction, contrast processing, brightness processing, smoothing, enlargement, reduction, rotation, translation, and masking. A classification program that classifies at least one of the X-ray image of the joint and the ultrasound image of the joint into image patterns based on at least one of the following: the type of electronic component, the shape of the connection part of the electronic component, the type of substrate on which the electronic component is mounted, and the type of imaging device for the joint. A void crack detection device for joints connecting electronic components, characterized by comprising an analysis result database that detects void crack regions in the classified image patterns and saves the void crack regions as analysis result images.

5. The void crack detection device for a joint connecting electronic components according to claim 1, 2, 3, or 4, characterized in that the joint is made of solder.

6. A void crack detection device for a joint connecting electronic components according to claim 1, 2, 3, or 4, characterized in that it generates images of multiple regions from the captured image, detects void crack regions in the images, and saves the combined image as the analysis result image.

7. The void and crack detection device for a joint connecting electronic components according to claim 1, 2, 3, or 4, characterized in that the captured image is an image combining an X-ray image of the joint and an ultrasonic image of the joint.

8. A void and crack detection device for joints connecting electronic components according to claim 1, 2, 3, or 4, characterized by applying weights to the aforementioned analysis result image.

Images