Detection device for voids and cracks in joint portion connecting electronic components

Abstract

To provide a detection device capable of automatically detecting voids with different shades and a wide variety of cracks.SOLUTION: An automatic detection device 1 includes a control unit 11 that controls the entire device, a main memory unit 12, a communication unit 13, an operation unit 14, a display panel 15, and an auxiliary memory unit 16. Images captured by apparatuses of an ultrasonic microscope 2 and an X-ray CT device 3 are uploaded to the auxiliary memory unit 16 by a captured image upload program 103 and saved in a captured image (1) DB 104. Using a machine learning model trained using a machine learning approach including deep learning, the captured images are binarized and void / crack areas on the images are visualized. The automatic detection performance of the automatic detection device 1 is improved by collecting transmitted X-ray images and ultrasonic microscope images, and training again.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a visualization device, a visualization method, and a computer program for non-destructively visualizing spaces, voids, cracks, etc. generated inside an object or at a joint such as an electronic component, a semiconductor element, a thin film device, a thick film device, an electronic device, etc. using a transmission X-ray image or an ultrasonic microscope image.

Background Art

[0002] A terminal of an electronic component and an electrode of a substrate on which the electronic component is mounted are soldered. Voids are generated in the solder joint or the like due to preheating, reflow conditions, etc. In addition, the electronic component and the substrate repeatedly expand and contract due to thermal stress, and cracks are generated in the joint or the like due to the difference in their expansion rates. In order to improve the problem of the substrate yield of an electronic circuit, detection and quality inspection of voids and cracks in joints and the like are important. Conventionally, as a method for non-destructively inspecting joints such as solder, a method of visually inspecting a transmission X-ray image or an ultrasonic microscope image has been applied.

[0003] Visual inspection can capture slight image changes, but there is a large variation in the pass / fail judgment by inspectors, and the inspection locations are also limited due to time constraints. In visual inspection, it is difficult to accurately detect and measure voids and cracks, and the process takes time.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In order to achieve high-quality and rapid non-destructive inspections, attempts have been made to automate and visualize the detection of voids 36 and cracks 39. However, in automation and visualization, image processing techniques based on the binarization of transmission X-ray images and ultrasonic microscope images are common, but the binarization threshold must be set appropriately, and it has not been possible to automatically detect voids 36 with different shadows and various cracks 39 at once.

[0006] In particular, in transmission X-ray images, due to problems such as vias, plating or reflections from vias, inconsistent illumination, and noise, it has been difficult to automatically detect voids 36 and cracks 39 with high robustness.

[0007] That is, with conventional image processing techniques, in order to detect voids 36 and cracks 39, it is necessary for humans to define image features such as luminance values one by one, and it has not been possible to detect various voids 36 and cracks 39.

Means for Solving the Problems

[0008] In response to these problems, the present invention takes an approach of machine learning including deep learning in consideration of the need for high-quality and rapid inspections inside objects and at joints. An AI learns image features that cannot be defined by humans, detects various voids 36 and cracks 39 with high accuracy, and can significantly shorten the conventional inspection time.

[0009] The present invention is a method for visualizing the inside of an object, characterized in that in the object, a first operation of classifying at least one of a transmission X-ray image acquired by an X-ray device and an ultrasonic image acquired by an ultrasonic device into a plurality of image patterns using a first machine learning model, and a second operation of detecting a predetermined region from the image patterns using a second machine learning model.

[0010] In the first operation, the performance of the newly trained machine learning model is periodically evaluated by a training program in image pattern classification, and when deterioration is recognized, reconstruction and retraining of the machine learning model are performed.

[0011] It further has a training program in region detection, the performance of the newly trained machine learning model is periodically evaluated, and when deterioration is recognized, reconstruction and retraining of the model are performed.

[0012] At least one of the processes of noise processing, contrast processing, brightness processing, smoothing processing, enlargement processing, reduction processing, rotation processing, translation processing, and masking processing is performed on at least one of the transmission X-ray image acquired by the X-ray apparatus and the ultrasonic image acquired by the ultrasonic apparatus, and an image pattern of the second machine learning model is generated.

[0013] The present invention is a computer program for visualizing the inside of an object, and in the object, a first process of classifying at least one of the transmission X-ray image acquired by the X-ray apparatus and the ultrasonic image acquired by the ultrasonic apparatus into a plurality of image patterns using a first machine learning model, and a second process of visualizing the inside of the object by detecting a predetermined region from the image pattern using a second machine learning model.

[0014] In the first process, the performance of the newly trained machine learning model is periodically evaluated by a training program in pattern classification, and when deterioration is recognized, reconstruction and retraining of the model are performed to ensure the pattern classification performance.

[0015] It further has a training program in region detection, the performance of the newly trained machine learning model is periodically evaluated, and when deterioration is recognized, reconstruction and retraining of the model are automatically performed to ensure the region detection performance.

[0016] At least one of the transmission X-ray image obtained by the X-ray device and the ultrasonic image obtained by the ultrasonic device is subjected to at least one or more of noise processing, contrast processing, brightness processing, smoothing processing, enlargement processing, reduction processing, rotation processing, translation processing, and masking processing to generate an image pattern of the second machine learning model.

[0017] The present invention is a method for visualizing the inside of an object, which includes predicting in region detection, dividing into a plurality of regions and performing a first process, and performing a second process of integrating the processing results in the first process. In the first process, for regions detected repeatedly, the average of the respective processing results is obtained.

[0018] The present invention is a computer program for visualizing the inside of an object, which includes performing a first process of dividing into a plurality of regions by a prediction program in region detection, and performing a second process of integrating the processing results in the first process. In the first process, for regions detected repeatedly, the average of the respective processing results is obtained. The present invention constructs an automatic detection system for voids and cracks in joints such as solder mounted on a cloud service and provides a void ratio and crack ratio measurement service.

[0019] The automatic detection device 1 includes a control unit 11 for controlling the entire device, a main storage unit 12, a communication unit 13, an operation unit 14, a display panel 15, and an auxiliary storage unit 16. Images taken by devices such as the ultrasonic microscope 2 and the X-ray CT device 3 are uploaded to the auxiliary storage unit 16 by the captured image upload program 103 and stored in the captured image (1) DB. In the image processing by binarization of the transmission X-ray image, a machine learning approach including deep learning is taken. By being used by the user, transmission X-ray images and ultrasonic microscope images are collected, and the automatic detection performance is improved by learning.

Advantages of the Invention

[0020] According to the present invention, in the conventional void inspection and crack inspection of solder joints by image analysis, voids with different shadows, various cracks, etc. could not be detected at once.

[0021] By taking an approach of machine learning including deep learning, the present invention enables automatic detection of voids 36 and cracks 39 inside a substance, at joints, etc. with high precision, and can significantly shorten the conventional inspection time, detection time, etc.

Brief Description of Drawings

[0022]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0023] Hereinafter, the present invention will be described based on the drawings showing the implementation of the present invention.

[0024] In the embodiments described in the specification, for ease of understanding, the solder joint of the electronic component 4 will be exemplified and described, but it is not limited thereto. The technical idea of the present invention can be applied to many objects such as buildings, electrical equipment, and internal diagnosis of the human body, and various types of structures.

[0025] In each of the drawings for explaining the embodiments for carrying out the invention, elements having the same function are denoted by the same reference numerals, and the description may be omitted. In addition, the embodiments of the present invention can be combined with each other. Also, for the purpose of facilitating understanding, facilitating illustration, etc., there may be cases of magnification, reduction, and omission.

[0026] FIG. 1 is a block diagram showing the configuration of a void crack automatic detection (visualization) device 1, an ultrasonic microscope 2, an X-ray CT (Computed Tomography) device 3, and joints such as an electronic component 4 and solder 5. In the embodiments of this specification, it will be described that the pattern 40 is acquired by the X-ray CT device 3, but it may be replaced with an X-ray fluoroscopy observation device or the ultrasonic microscope 2.

[0027] The X-ray microscope (X-ray CT) 3 is a microscope that non-destructively observes the interior of an object. Different from an X-ray fluoroscopic observation device, it converts the X-rays transmitted through the sample into light and magnifies it with an optical lens. X-rays have the property of passing through substances. When X-rays pass through a sample, a part of them is absorbed. The absorption rate increases as the density of the material is high (the atomic number is large) and the thickness is thick, so the intensity of the transmitted X-rays decreases.

[0028] If voids 36 and cracks 39 occur inside the object, the X-ray transmittance of the void 36 and crack 39 parts increases, so the voids 36 and cracks 39 are displayed in the pattern 40.

[0029] The X-ray CT device 3 performs computational processing with a computer based on the X-ray transmission information of 360° (DEG.) for the sample to be observed and constructs three-dimensional data of the sample. By narrowing the rotation pitch and obtaining more information, high-precision three-dimensional data can be obtained.

[0030] An X-ray image is basically a two-dimensional image. Many voids 36 and cracks 39 occur, and the generated voids 36 and cracks 39 are three-dimensionally distributed. Therefore, the obtained X-ray image has a pattern 40 in which the voids 36 and cracks 39 overlap.

[0031] The ultrasonic microscope 2 is a microscope that non-destructively observes defects such as delamination, voids, cracks, and foreign substances inside an object. When ultrasonic waves propagate through different substances, part of them is reflected and part of them is transmitted. If there is, for example, a delamination location (void) in the observation object, a strong reflected wave is detected. By performing a planar scan on the observation object and marking the positions where this reflected wave is detected, the two-dimensional distribution of the delamination location (void) can be obtained. Therefore, the ultrasonic microscope image obtained by imaging it also becomes an image such as the pattern 40 in which the voids 36 and cracks 39 overlap, similar to the X-ray image.

[0032] The void crack automatic detection (visualization) device 1 of the present invention has the function of automatically observing, inspecting, judging, determining, etc. the position and occurrence state of void cracks, and automatically visualizing and detecting void cracks. Hereinafter, the void crack automatic detection (visualization) device is referred to as an automatic detection device or a visualization device.

[0033] The X-ray CT device 3 of the present invention can obtain the internal structure of an object by performing reconstruction processing by utilizing the differences in "ease of transmission" and "ease of absorption" when X-rays penetrate the object.

[0034] The ultrasonic microscope 2 of the present invention can obtain the internal structure of an object by performing reconstruction processing by utilizing the differences in "ease of transmission" and "ease of reflection" when ultrasonic waves propagate through the object. The automatic detection device 1 includes a control unit 11 for controlling the entire device, a main memory unit 12, a communication unit 13, an operation unit 14, a display panel 15, and an auxiliary storage unit 16.

[0035] The automatic detection device 1 can be composed of, for example, a desktop computer, a notebook personal computer, a tablet, a smartphone, a computer, etc. in a cloud environment that provides computer resources in the form of services via a computer network such as the Internet.

[0036] The control unit 11 can be composed of a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The control unit 11 may include a GPU (Graphics Processing Unit).

[0037] The main memory unit 12 is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory, and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 has a function of communicating with the automatic detection device 1 via the network 17 and can transmit and receive required information. The operation unit 14 is composed of, for example, a hardware keyboard, a mouse, a touch panel, etc.

[0038] The display panel 15 can be composed of a liquid crystal panel or an organic EL (Electro Luminescence) display panel, etc. The control unit 11 performs control for displaying required information on the display panel 15.

[0039] The auxiliary storage unit 16 is a large-capacity memory, a hard disk, etc., and stores programs necessary for the control unit 11 to execute processing, as well as a user authentication program 101, a case management program 102, a photographed image upload program 103, a machine learning program 110, an automatic detection program 111, an analysis result editing and viewing program 112, an analysis report automatic creation program 113, an analysis report notification program 114, and an analysis report editing and viewing program 115.

[0040] The user authentication program 101, the case management program 102, the photographed image upload program 103, the machine learning program 110, the automatic detection program 111, the analysis result editing and viewing program 112, the analysis report automatic creation program 113, the analysis report notification program 114, and the analysis report editing and viewing program 115 stored in the auxiliary storage unit 16 may be provided by a recording medium 18 that records each program in a readable manner.

[0041] The recording medium 18 is, for example, a portable memory such as a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, a micro SD card, a CompactFlash (registered trademark), etc.

[0042] Each program recorded on the recording medium 18 is provided by communication via the communication unit 13. Also, it may be read from the recording medium 18 using a reading device (not shown in the figure) and installed in the auxiliary storage unit 16. Next, an explanation will be given of how the images taken by the equipment of the ultrasonic microscope 2 and the X-ray CT apparatus 3 are stored in the auxiliary storage unit 16.

[0043] When the equipment of the ultrasonic microscope 2 and the X-ray CT apparatus 3 are connected to the Internet, based on the setting information of the ultrasonic microscope 2 and the X-ray CT apparatus 3, user authentication is automatically performed by the user authentication program 101, and case registration is automatically performed by the case management program 102.

[0044] The images taken by the equipment of the ultrasonic microscope 2 and the X-ray CT apparatus 3 are uploaded to the auxiliary storage unit 16 by the captured image upload program 103 and stored in the captured image (1) DB.

[0045] On the other hand, when the equipment of the ultrasonic microscope 2 and the X-ray CT apparatus 3 are not connected to the Internet, the images taken by the equipment are taken out and copied to a computer or the like connected to the Internet.

[0046] On the computer, a login operation is performed by the user authentication program 101, and case registration is performed by the case management program 102. Next, when the image is uploaded to the auxiliary storage unit 16 by the captured image upload program 103, it can be stored in the captured image (1) DB 104.

[0047] In the auxiliary storage unit 16, the uploaded image data is stored in the photographed image (1) DB104, the results of automatically detecting void cracks are stored in the analysis result (1) DB105, the analysis report recording the void ratio and crack ratio automatically measured from the automatic detection results is stored in the analysis report (1) DB106, and the dataset created using the photographed image (1) DB104 and the analysis result (1) DB105 is stored in the machine learning data (1) DB107, each stored for each user, group, or project. The users who can access each data are only the users authenticated by the user authentication program 101.

[0048] Also, the auxiliary storage unit 16 stores the automatic detection program 111. This automatic detection program 111 uses the machine learning model (pattern classification) DB108 and the machine learning model (area detection) DB109 to perform automatic detection of void cracks on the images stored in the photographed image (1) DB104. Also, the images stored in the photographed image (1) DB104 can be processed in batches at regular intervals. Details of this automatic detection program 111 will be described later.

[0049] The results automatically detected by the automatic detection program 111 are stored in the analysis result (1) DB105. The stored analysis result data is read out by the analysis result editing and viewing program 112, and the user can edit and view the analysis results. In this editing operation, users permitted by the user authentication program 101 can edit jointly.

[0050] The analysis report automatic creation program 113 reads out the images stored in the photographed image (1) DB104 and the analysis results in the analysis result (1) DB105, and automatically generates numerical data such as void ratio and crack ratio and text data such as defect causes according to the format specified by the user, summarizes them as an analysis report, and stores them in the analysis report (1) DB106.

[0051] When there is a newly created or updated analysis report, the content is notified to the user through mail, messages, chat services, etc. by the analysis report notification program 114. The user can read out the analysis report editing and viewing program 115 and edit and view the analysis report. In this editing work, users permitted by the user authentication program can jointly edit it.

[0052] The auxiliary storage unit 16 stores the machine learning program 110. Using the photographed image (1) DB 104 and the analysis result (1) DB 105, the machine learning program 110 creates the machine learning data (1) DB, performs machine learning using the data set, and creates the machine learning model (pattern classification) DB 108 and the machine learning model (area detection) DB 109.

[0053] Regarding the data sets used for machine learning, there are cases where the photographed image DB and the analysis result DB stored for each user, group, or project are used individually, and there are also cases where they are shared and used.

[0054] The created machine learning model (pattern classification) DB 108 and the machine learning model (area detection) DB 109 are used in the aforementioned automatic detection program 111. Therefore, the machine learning program 110 is executed at a timing different from the timing when the automatic detection program 111 is executed (for example, during a time period when the user is not using this system).

[0055] Figure 2 illustrates the details of the automatic detection program 111. First, the group of photographed images 21 newly uploaded from devices such as the ultrasonic microscope 2 and the X-ray CT device 3 is read into the pattern classification prediction program 201.

[0056] The pattern classification prediction program 201 classifies the patterns of images. Examples of patterns to be classified include cases where the electronic components mounted, such as chip capacitors, resistors, and coils, are different. Also, examples include cases where the pin shapes of IC packages are different, such as leaded and leadless types. Further, examples include cases where the types of substrates are different, such as surface mount and through-hole. Also, examples include cases where the materials of electronic components are different, such as copper, brass, and iron. Also, examples include cases where the imaging methods are different, such as optical microscopes and electron microscopes. The captured image group 2l is classified by the pattern classification prediction program 201, for example, into pattern A, pattern B, and pattern C.

[0057] The image group of pattern A is read into the region detection prediction program 202 for pattern A. The image group of pattern B is read into the region detection prediction program 203 for pattern B. After the image group of pattern C is read into the region detection prediction program 204 for pattern C, each program automatically detects the void 36 region and the crack 39 region. Those analysis result images are saved as the analysis result (mask image) group 22.

[0058] The present invention focuses on the fact that there is an optimal region detection method for each pattern, and prepares a region detection program for each pattern. This is because the accuracy of region detection can be improved compared to the case where it is not prepared for each pattern. Therefore, the machine learning model must be prepared for each pattern. FIG. 3 illustrates the processing flow of the training program and the prediction program in the aforementioned pattern classification.

[0059] In the training program for pattern classification, the purpose is to create a machine learning model (pattern classification) DB108. For example, when there are two types, pattern A and pattern B, in a group of captured images, a machine learning model that can classify whether a given captured image is pattern A or pattern B is created using a captured image (AB) DB301 and a pattern (AB) DB302 that can refer to whether each captured image is pattern A or pattern B.

[0060] The data augmentation preprocessing program 303 reads the captured image (AB) DB301 and the pattern (AB) DB302, and creates a data set for machine learning. The created data set is stored in the machine learning data (AB) DB304.

[0061] Next, the machine learning (training) program 305 reads the machine learning data (AB) DB304, performs training, and stores the trained machine learning model in the machine learning model (pattern classification) DB108. When performing training, there are cases where training is performed using an existing machine learning model in the machine learning model (pattern classification) DB108.

[0062] When performing training, there are also cases where the machine learning model is trained by setting specific weights for machine learning data held by a specific user, group, or project, or for machine learning data with a newer upload date and time, thereby differentiating the importance of the data.

[0063] In the performance evaluation of a newly trained machine learning model, if it is determined that the performance is of higher accuracy than an existing machine learning model, it may automatically replace the existing machine learning model in the machine learning model (pattern classification) DB108.

[0064] In the performance evaluation of a newly trained machine learning model, if it is determined that the performance is of lower accuracy than an existing machine learning model, the trained machine learning model may be automatically discarded.

[0065] As described above, the feature of the present invention is that the performance evaluation of a newly trained machine learning model is periodically performed by a training program in pattern classification, and when deterioration is recognized, the model is automatically reconstructed and retrained.

[0066] In FIG. 3, in the prediction program in pattern classification, the preprocessing program 307 reads the captured image group 306 and executes image preprocessing. The preprocessed image is read by the machine learning (prediction) program 308, and a machine learning model trained by the training program is read from the machine learning model (pattern classification) DB 108, and pattern classification is performed. The patterns classified for the captured image group are stored in the pattern classification result group 309. FIG. 4 illustrates the processing flow of the training program and the prediction program in the above-described region detection.

[0067] The purpose of the training program in region detection is to create a machine learning model (region detection) DB 109. Using the captured image (A) DB 401 and the mask image (A) DB 402, a machine learning model capable of detecting which region of the captured image corresponds to the void 36 or the crack 39 is created.

[0068] The data augmentation preprocessing program 403 reads the captured image (A) DB 401 and the mask image (A) DB 402 and creates a data set for machine learning.

[0069] The created data set is stored in the machine learning data (A) DB 404. Thereafter, the machine learning (training) program 405 reads the machine learning data (A) DB 404, performs training, and stores the trained machine learning model in the machine learning model (region detection) DB 109. When performing training, training may be performed using an existing machine learning model in the machine learning model (region detection) DB 109.

[0070] When performing training, in some cases, a machine learning model is trained by setting specific weights for machine learning data owned by a certain specific user, group, or project, or machine learning data with a newer upload date and time, and differentiating the importance of the data.

[0071] In the performance evaluation of a newly trained machine learning model, if it is determined that the performance has higher accuracy than an existing machine learning model, it may automatically replace the existing machine learning model in the machine learning model (pattern classification) DB108.

[0072] In the performance evaluation of a newly trained machine learning model, if it is determined that the performance has lower accuracy than an existing machine learning model, the trained machine learning model may be automatically discarded.

[0073] As described above, the feature of the present invention is that the performance evaluation of a newly trained machine learning model by a training program in region detection is periodically performed, and when deterioration is recognized, the model is automatically reconstructed and retrained.

[0074] A specific training example will be described with reference to FIGS. 5, 6, and 7. Although the acquired image pattern 40 is described as the pattern 40 of the void 36, it is not limited thereto. Needless to say, for example, the void 36 may be replaced with a crack 39 or the like. Also, even if it is a pattern in which the void 36 and the crack 39 are mixed, by training and learning, the void 36 and the crack 39 can be distinguished and detected.

[0075] FIG. 5 schematically shows (pattern 40) the occurrence status of the solder part 5 and the void 36 photographed by a transmission X-ray image. Normally, the void 36 does not become circular as shown in FIG. 5, but a plurality of circles are connected, with a shadow and deteriorated contrast.

[0076] In the following examples, Pattern 40 will be described as the pattern obtained by a transmission X-ray device. Needless to say, Pattern 40 can also be obtained by an X-ray CT device. Prepare thousands of transmission X-ray images and images in which the void region 36 is masked from the images (manually inspected data).

[0077] Randomly cut these into 256×256 (pixel) patch images (section 37), and perform one or more processes such as adding noise, adjusting contrast, adjusting brightness, inverting brightness, smoothing, enlarging / reducing, rotating, shifting in the horizontal or vertical direction, partially masking (painting black), or a combination of multiple processes to expand the data and generate hundreds of thousands of data sets.

[0078] As an example, in FIG. 5, the solder portion 38 is divided into four patch images (section 37a, section 37b, section 37c, section 37d). Each section 37 preferably has the same shape and area.

[0079] Each patch image (section 37) contains a void 36. As shown in FIG. 6(a), section 37a is inverted in black and white to obtain training data. Also, as shown in FIG. 6(b), section 37a is enlarged to obtain training data. Further, as shown in FIG. 6(c), section 37a is reduced to obtain training data.

[0080] The section 37a shown in FIG. 7(a) is rotated by a predetermined angle (10° in the figure) to the left as shown in FIG. 6(b) to obtain training data. As shown in FIG. 6(c), it is rotated by a predetermined angle (10° in the figure) to the right to obtain training data. Also, section 37 may be inverted by a line object.

[0081] The feature of the present invention is that even when there is little machine learning data available, the number of data is expanded by the method of inverting black and white, rotating, reducing, and enlarging the sections as described above, and a large amount of data sets are self-generated.

[0082] Next, using 70% of the patch images (section 37) as training data and the remaining 30% as test data, perform deep learning of U-Net by an encoder-decoder model for segmentation, specifically for medical segmentation (such as cell segmentation). That is, use the AI (artificial intelligence) function. It is preferable to use 60% or more of the patch images (section 37) as training data. The test data is preferably 40% or less.

[0083] As the Dice coefficient between the mask image as the teacher data and the mask image as the network output, set the loss function during learning to the Dice coefficient × (-1). The Dice coefficient uses the average of the number of elements in two sets instead of the number of elements in the union of the two sets. During training, save the machine learning model with the smallest loss of the aforementioned test data.

[0084] On the other hand, in the prediction program for region detection, the preprocessing program 407 reads the photographed image group 406 and executes image preprocessing. The preprocessed image is read by the machine learning (prediction) program 408, and the machine learning model trained by the training program is read from the machine learning model (region detection) DB109, and region prediction is executed. The region predicted for the photographed image is saved in the mask image group 409.

[0085] A specific prediction example will be described with reference to FIG. 8. The photographed transmission X-ray image is divided at equal intervals into patch images of 256×256 (pixels). Alternatively, the patch image is divided into a plurality of parts. In FIG. 8, it is divided vertically and horizontally from the upper left section 37e of the image, but in accordance with the width and height of the photographed transmission X-ray image, the patch images are divided by overlapping them at equal intervals like sections 37f, 37g, and 37h. Next, each patch image is input into the trained machine learning model.

[0086] From the trained machine learning model, a probability map (values from 0 to 1) indicating the presence of voids at each pixel position is obtained. To obtain a binary map (mask image), the obtained probability map is binarized at a set threshold of 0.5 (indicating voids for 1 and the background for 0).

[0087] The obtained binary map group of 256×256 (pixels) is combined (as divided into sections 37e, 37f, 37g, 37h in FIG. 8) to generate an overall binary map (mask image) corresponding to the original transmission X-ray image.

[0088] Note that 256 is 2 to the 8th power. It is preferable that the pixels be 2 to the nth power (n is an integer of 2 or more). Finally, as post-processing, the binary map (mask image) is optimized using the CRF (conditional random field) of the transmission X-ray image. As described above, the feature of the present invention is to perform processing by dividing into a plurality of fine regions (sections) 37, integrating the respective processing results, and obtaining one processing result.

[0089] For example, even for a void 36 where an object must fit within the boundary of section 37, it may be clearly detected in other sections 37. For the regions detected repeatedly, by taking the average of the processing results in the respective batch images and setting pixels exceeding the threshold of 0.5 as voids, voids can be automatically detected with high robustness.

[0090] For example, even if a void 36 cannot be detected in the image of section 37, it may be detectable if the image is rotated by 90 degrees. By taking the average of the processing results for the images obtained by rotating section 37 multiple times and setting pixels exceeding the threshold of 0.5 as void 36, void 36 can be automatically detected with high robustness.

[0091] A method for updating machine learning data for reconstructing and retraining a machine learning model will be described with reference to FIG. 9. The above-described automatic detection result of the void 36 is not always correct. The user can check and edit the automatic detection result of the void 36 by the analysis result editing and browsing program 112 of the present system. In order to improve the automatic detection performance, it is necessary to incorporate the user's check and edit content into the machine learning data of the present system and update it.

[0092] FIG. 9(a) shows a case where the captured image group 901 is input to the region detection prediction program 902 and the analysis result (mask image) group 903 is output, but the user has not checked the analysis results. In this case, it is registered as "unconfirmed" in the confirmation status and edit result (mask image) DB 904 and is not incorporated into the machine learning data and updated.

[0093] FIG. 9(b) shows a case where, after the captured image group 901 is input to the region detection prediction program 902 and the analysis result (mask image) group 903 is output, the user views the analysis results for a specific period of time or presses a confirmation button or an analysis result output button. In this case, it is registered as "confirmed" in the confirmation status and edit result (mask image) DB 904 and is incorporated into the machine learning data and updated.

[0094] FIG. 9(c) shows the case where the captured image group 901 is input to the region detection prediction program 902 and the analysis result (mask image) group 903 is output, and then the user edits the analysis result. In this case, it is registered as "edited" in the confirmation status and edited result (mask image) DB 904. If it is determined that the analysis result is incorrect, it is weighted and incorporated into the machine learning data for update. Also, the greater the difference between the analysis result (mask image) group 903 and the result edited by the user, the greater the weight and the more it is incorporated into the machine learning data for update. Examples of how to assign weights include expanding the number of data by the methods of black-and-white inversion, rotation, reduction, and enlargement of the captured image group 901 and the edited result (mask image) DB 904 in FIG. 9(c) and incorporating them into the machine learning data. As described above, the captured image group 901 is classified and registered as unconfirmed, confirmed, edited, etc., incorporated into the machine learning data, and updated. By the method of updating the above machine learning data, the machine learning model is automatically reconstructed and retrained, and the automatic detection performance is improved.

[0095] The present invention uses an approach of machine learning including deep learning. By using this approach, voids 36 and cracks 39 can be detected well and accurately.

[0096] The above embodiments of the present invention have described the detection, visualization device, and visualization method of voids 36 inside a substance. Cracks 39 can also obtain a pattern 40 with an X-ray CT device 3 or the like. Although the image of voids 36 and the image of cracks 39 are different, by learning as cracks 39, cracks 39 can be detected. Therefore, it goes without saying that the matters described as voids 36 in the embodiments of this specification can be replaced with cracks 39.

[0097] In addition, the pattern 40 can be obtained not only by acquiring it from a transmission X-ray image obtained by an X-ray CT apparatus 3 or the like, but also, for example, by using an ultrasonic microscope 2. Therefore, in the apparatus of the present invention, it goes without saying that an X-ray apparatus such as an X-ray CT apparatus may be replaced with an ultrasonic apparatus 2 such as an ultrasonic microscope.

[0098] Further, by combining an X-ray apparatus such as an X-ray CT apparatus 3 and an ultrasonic microscope 2 to obtain the pattern 40, the detection accuracy of voids 36, cracks 39, etc. is improved.

[0099] The above embodiments are methods or apparatuses for detecting voids 36, cracks 39, etc. in a transmission X-ray image, but the present invention is not limited thereto. Needless to say, various things such as bubbles in an adhesive resin, spaces in a concrete block, mixtures of other metals in an iron material part, and fat particles in a living body can be nondestructively analyzed and examined. FIGS. 10, 11, 12, and 13 are explanatory diagrams of an automatic detection method of the present invention and a business model using a computer program.

[0100] FIG. 10 is an explanatory diagram of a business model in the first embodiment. For customer A who owns images taken by an ultrasonic microscope 2, an X-ray CT apparatus 3, etc., company A provides or sells a program (hereinafter referred to as "void / crack automatic detection program") that includes a user authentication program 101, a project management program 102, a captured image upload program 103, a machine learning program 110, an automatic detection program 111, an analysis result editing and viewing program 112, an analysis report automatic creation program 113, an analysis report notification program 114, an analysis report editing and viewing program 115, etc. shown in FIG. 1.

[0101] Customer A can install and use the void crack automatic detection program on desktop computers, notebook personal computers, tablets, and smartphones. Company A receives a license fee for the void crack automatic detection program from Customer A.

[0102] Figure 11 is an explanatory diagram of the business model in the second embodiment. For Customer A who does not own images taken by the ultrasonic microscope 2, X-ray CT device 3, etc., Company A provides or sells a group of images taken by the ultrasonic microscope 2, X-ray CT device 3, etc. owned by Company A and the void crack automatic detection program.

[0103] Customer A can install and use the void crack automatic detection program on desktop computers, notebook personal computers, tablets, and smartphones. Company A receives an image shooting fee and a license fee for the void crack automatic detection program from Customer A.

[0104] Figure 12 is an explanatory diagram of the business model in the third embodiment. For Customer A who owns images taken by the ultrasonic microscope 2, X-ray CT device 3, etc., Company A operates the void crack automatic detection program on a cloud server.

[0105] Customer A can use the void crack automatic detection program by using the web browser of desktop computers, notebook personal computers, tablets, and smartphones. Company A receives a usage fee for the void crack automatic detection program from Customer A.

[0106] Figure 13 is an explanatory diagram of the business model in the fourth embodiment. For Customer A who does not own images taken by the ultrasonic microscope 2, X-ray CT device 3, etc., Company A prepares a group of images taken by the ultrasonic microscope 2, X-ray CT device 3, etc. owned by Company A and operates the void crack automatic detection program on a cloud server.

[0107] Customer A can use the web browser of a desktop computer, a notebook personal computer, a tablet, or a smartphone to utilize the void crack automatic detection program. Company A receives an image shooting fee and a usage fee for the void crack automatic detection program from Customer A.

[0108] The embodiments disclosed this time should be considered as illustrative in all respects and not restrictive. The scope of the present invention is shown not by the above meaning but by the claims, and it is intended that all modifications within the meaning and scope equivalent to the claims are included. Needless to say, the matters or contents described in this specification and the drawings can be combined with each other.

Explanation of Signs

[0109] 1 Void crack automatic detection device 2 Ultrasonic microscope 3 X-ray CT device 4 Electronic component (measurement sample) 5 Solder 11 Control unit 12 Main memory unit 13 Communication unit 14 Operation unit 15 Display panel 16 Auxiliary storage unit 17 Network 18 Recording medium 21 Group of photographed images 22 Group of analysis results (mask images) 31 Pattern classification training program 32 Pattern classification prediction program 36 Void 37 Division (processing unit) 38 Solder part 39 Crack 40 Pattern 41 Region detection training program 42 Region detection prediction program 101 User authentication program 102 Case Management Program 103 Photographed Image Upload Program 104 Photographed Image (1) DB 105 Analysis Result (1) DB 106 Analysis Report (1) DB 107 Machine Learning Data (1) DB 108 Machine Learning Model (Pattern Classification) DB 109 Machine Learning Model (Region Detection) DB 110 Machine Learning Program 111 Automatic Detection Program 112 Analysis Result Editing and Browsing Program 113 Analysis Report Automatic Generation Program 114 Analysis Report Notification Program 115 Analysis Report Editing and Browsing Program 201 Pattern Classification Prediction Program 202 Region Detection Prediction Program for Pattern A 203 Region Detection Prediction Program for Pattern B 204 Region Detection Prediction Program for Pattern C 301 Photographed Image (AB) DB 302 Pattern (AB) DB 303 Data Augmentation Preprocessing Program 304 Machine Learning Data (AB) DB 305 Machine Learning (Training) Program 306 Group of Photographed Images 307 Preprocessing Program 308 Machine Learning (Prediction) Program 309 Group of Pattern Classification Results 401 Photographed Image (A) DB 402 Mask Image (A) DB 403 Data Augmentation Preprocessing Program 404 Machine Learning Data (A) DB 405 Machine Learning (Training) Program 406 Group of Photographed Images 407 Preprocessing Program 408 Machine Learning (Prediction) Program Group of 409 analysis results (mask images) Group of 901 captured images 902 Region detection prediction program Group of 903 analysis results (mask images) 904 Confirmation status and editing results (mask image) DB

Claims

1. A method for detecting voids and cracks in a joint for connecting electronic components, comprising: a first process of classifying at least one of an X-ray image of the joint and an ultrasonic image of the joint into an image pattern based on at least one of the type of the electronic component, the shape of the connection portion of the electronic component, the type of substrate on which the electronic component is mounted, and the imaging method; a second process of detecting a void and crack region of the joint in the classified image pattern and storing the void and crack region as an analysis result image. The method for detecting voids and cracks in a joint for connecting electronic components is characterized by having the second process.

2. A method for detecting voids and cracks in a joint for connecting electronic components, comprising: a first process of classifying at least one of an X-ray image of the joint and an ultrasonic image of the joint into an image pattern based on at least one of the type of the electronic component, the shape of the connection portion of the electronic component, the type of substrate on which the electronic component is mounted, and the type of imaging method; a second process of detecting a void and crack region of the joint in the classified image pattern and storing the void and crack region as an analysis result image; a third process of checking or editing the void and crack region with respect to the analysis result image and updating the analysis result image. The method for detecting voids and cracks in a joint for connecting electronic components is characterized by having the third process.

3. A method for detecting voids and cracks in a joint for connecting electronic components, comprising: a first process of classifying at least one of an X-ray image of the joint and an ultrasonic image of the joint into an image pattern based on at least one of the type of the electronic component, the shape of the connection portion of the electronic component, the type of substrate on which the electronic component is mounted, and the type of imaging method; a second process of detecting a void and crack region of the joint in the classified image pattern and storing the void and crack region as an analysis result image; performing at least one or more processes of noise processing, contrast processing, brightness processing, smoothing processing, enlargement processing, reduction processing, rotation processing, parallel movement processing, and mask processing on the captured image to generate a first dataset, and performing the first process based on the first dataset. At least one or more of noise processing, contrast processing, brightness processing, smoothing processing, enlargement processing, reduction processing, rotation processing, translation processing, and masking processing are performed on the captured image to generate a second dataset, and a second process is performed based on the second dataset. A method for detecting void cracks in a joint for connecting an electronic component, characterized in that.

4. At least one or more of noise processing, contrast processing, brightness processing, smoothing processing, enlargement processing, reduction processing, rotation processing, translation processing, and masking processing are performed on the captured image to generate a first dataset, and the first process is performed based on the first dataset. At least one or more of noise processing, contrast processing, brightness processing, smoothing processing, enlargement processing, reduction processing, rotation processing, translation processing, and masking processing are performed on the captured image to generate a second dataset, and a second process is performed based on the second dataset. Performance evaluation of image pattern classification of the first process is performed based on the first dataset. A method for detecting void cracks in a joint for connecting an electronic component according to claim 1 or claim 2, characterized in that performance evaluation of void crack region detection of the second process is performed based on the second dataset.

5. A method for detecting void cracks in a joint for connecting an electronic component according to claim 1 or claim 2 or claim 3, characterized in that images of a plurality of regions are generated from the captured image, and an image obtained by detecting and combining void crack regions of the images of the plurality of regions is stored as the analysis result image.

6. A method for detecting void cracks in a joint for connecting an electronic component according to claim 1 or claim 2 or claim 3, characterized in that the captured image is an image combining an X-ray image of the joint and an ultrasonic image of the joint.

7. A device for detecting void cracks in a joint for connecting an electronic component, A captured image database in which at least one of the X-ray image of the joint captured by an X-ray device and the ultrasonic image of the joint captured by an ultrasonic device is stored, It comprises an analysis result database that detects the void crack region of the joint in the captured image and stores the void crack region as an analysis result image. A void crack detection device for a joint connecting electronic components, characterized by checking or editing the void crack region with respect to the analysis result image and updating the analysis result image. **Claim 8** A void crack detection device for a joint connecting electronic components, a captured image database storing at least one of an X-ray image of the joint captured by an X-ray device and an ultrasonic image of the joint captured by an ultrasonic device; an analysis result database that performs at least one or more of noise processing, contrast processing, brightness processing, smoothing processing, enlargement processing, reduction processing, rotation processing, translation processing, and masking processing on the captured image, detects a void crack region of the joint of the captured image, and stores the void crack region as an analysis result image. A void crack detection device for a joint connecting electronic components, characterized by comprising. **Claim 9** The void crack detection device for a joint connecting electronic components according to claim 7 or claim 8, characterized in that a plurality of region images are generated from the captured image, the void crack region of the image is detected and combined, and the combined image is stored as the analysis result image. **Claim 10** The void crack detection device for a joint connecting electronic components according to claim 7 or claim 8, characterized in that the captured image is an image combining an X-ray image of the joint and an ultrasonic image of the joint.

Images