Apparatus for visualizing voids and cracks in joints and method for visualizing voids and cracks in joints

Abstract

To solve a problem that it is easy to determine a predictive model that outputs the most likely prediction result for a known inspection target, but it is difficult for an unknown inspection target, when performing image inspection using a predictive model based on semantic segmentation, etc.SOLUTION: A pattern classification program 133 and an image scale adjustment program 134 classify and adjust an image of an unknown inspection target, thereby narrowing down an optimal machine learning model from among a plurality of machine learning models. In a process of selecting the best machine learning model from a limited candidate group, a model selection program 504 determines a prediction model that outputs the most likely prediction result by using a display method that allows a user who does not have special knowledge of a machine learning model to easily select the optimal machine learning model and by checking an editing status of a visualization object.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a visualization device, a visualization method, a computer program, an operation method of the computer program, a display method, etc., which use transmission X-ray images and ultrasonic microscope images to nondestructively visualize spaces, voids, and cracks generated inside objects or joints such as electronic components, semiconductor elements, thin film devices, thick film devices, and electronic devices. Furthermore, the present invention relates to a method for selecting and applying a machine learning model, a method for creating a prediction model, and a method for using a prediction model.

Background Art

[0002] The terminals of an electronic component and the electrodes of a substrate on which the electronic component is mounted are soldered. Voids 36 are generated in joints such as solder 5 due to preheating, reflow conditions, etc. Also, the electronic component and the substrate repeatedly expand and contract due to thermal stress, and cracks 39 are generated in joints etc. due to the difference in their expansion rates. In order to improve the problem of the substrate yield of an electronic circuit, nondestructive detection of voids 36 and cracks 39 in joints etc. and quality inspection are important. Conventionally, as a method for nondestructively inspecting joints such as solder 5, a method of visually inspecting transmission X-ray images and ultrasonic microscope images has been applied.

[0003] Visual inspection can capture slight image changes, but there is a large variation in the quality judgment by inspectors, and the inspection locations are also limited due to time constraints. In visual inspection, it is difficult to accurately detect and measure voids 36 and cracks 39, and the process takes time.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] Conventional image inspection methods using machine learning models required multiple machine learning models tailored to the target object, rather than a single model, in order to robustly detect voids 36 and cracks 39, which have various characteristics. Therefore, when an image of an unknown subject was input, it was difficult to automatically determine which machine learning model to apply to make a prediction. [Means for solving the problem]

[0006] To address these challenges, the present invention takes an approach that allows the user to appropriately select the optimal machine learning model from among multiple machine learning models using a dedicated display method. By selecting the machine learning model that yields the best prediction results, a wide variety of voids 36 and cracks 39 can be detected with high accuracy, significantly reducing conventional inspection time.

[0007] The present invention relates to an object interior visualization device, an object interior visualization method, and a computer program for an object interior visualization device, characterized by selecting the optimal machine learning model from among multiple machine learning models used for visualizing the inside of an object.

[0008] The present invention is characterized by classifying and adjusting images of unknown subjects using a pattern classification program 133 and an image scale adjustment program 134, thereby narrowing down the optimal machine learning model from among multiple machine learning models.

[0009] The present invention is characterized in that, in the process of selecting the optimal machine learning model from the limited group of candidate machine learning models described above, the model selection program 504 of the present invention provides a display method that allows users without special knowledge of machine learning models to easily select the optimal machine learning model.

[0010] The present invention is characterized by taking an approach that indirectly obtains the user's evaluation of the prediction results using a user editing status confirmation program 135, thereby maintaining a record of which machine learning model should be applied when the same type of image input is provided, and improving the convenience of image inspection for the user. Furthermore, the automatic detection device 1 includes a control unit 11 that controls the entire device, a main memory unit 12, a communication unit 13, an operation unit 14, a display panel 15, and an auxiliary memory unit 16.

[0011] Images captured by the ultrasonic microscope 2 and the X-ray CT scanner 3 are uploaded to the auxiliary storage unit 16 by the image upload program 103 and stored in the image database (1).

[0012] The present invention involves a first operation in which a machine learning model is used to classify the type of image data from at least one of the following images in an object: a transmission X-ray image acquired by an X-ray device, a cross-sectional image acquired by an X-ray CT device, and an ultrasound image acquired by an ultrasound device; and a second operation in which the position, angle, and scale of the object in the image data are adjusted to narrow down the optimal machine learning model from among multiple machine learning models used to visualize the inside of an object.

[0013] Furthermore, the present invention provides a display method that allows a user without special knowledge of machine learning models to easily select the optimal machine learning model in the process of selecting the optimal machine learning model from a limited group of candidate machine learning models by performing a first operation to classify the type of image data using a machine learning model from at least one of the images obtained from a transmission X-ray image obtained from an X-ray device, a cross-sectional image obtained from an X-ray CT device, and an ultrasound image obtained from an ultrasound device, and a second operation to adjust the position, angle, and scale of the object in the image data.

[0014] Furthermore, the present invention improves the convenience of image inspection for the user by performing a first operation in which, in an object, a machine learning model is used to classify the type of image data from at least one of the following images: a transmitted X-ray image acquired by an X-ray device, a cross-sectional image acquired by an X-ray CT device, and an ultrasonic image acquired by an ultrasonic device; a second operation in which the position, angle, and scale of the object in the image data are adjusted; and a third operation in which the inside of the object in the image data is visualized using a machine learning model selected by the user. After this, the user's evaluation of the visualization result is indirectly obtained, and the system stores whether the selected machine learning model should be applied next time. [Effects of the Invention]

[0015] By using multiple machine learning models instead of a single one, and applying the most appropriate model from among them, it becomes possible to automatically detect a wide variety of voids 36 and cracks 39 in electronic components with high accuracy, significantly reducing conventional inspection or detection times. [Brief explanation of the drawing]

[0016] [Figure 1] This is an explanatory diagram of the visualization device for the interior and joints of an object according to the present invention. [Figure 2] This is an explanatory diagram of a visualization image of solder joints and other similar areas. [Figure 3] This is an explanatory diagram of a method for detecting void and crack regions from captured images in the visualization method for the interior and joints of an object according to the present invention. [Figure 4] This is an explanatory diagram of a method for detecting void and crack regions from captured images in the visualization method for the interior and joints of an object according to the present invention. [Figure 5] This is an explanatory diagram illustrating the method for selecting a machine learning model (region detection) for automatically detecting voids and cracks occurring inside objects and at joints according to the present invention. [Figure 6] This is an explanatory diagram illustrating the method for selecting a machine learning model (region detection) for automatically detecting voids and cracks occurring inside objects and at joints according to the present invention. [Figure 7] It is an explanatory diagram for explaining a method of selecting a machine learning model (region detection) for automatically detecting voids and cracks generated inside an object and at joints of the present invention. [Figure 8] It is an explanatory diagram for explaining a method of selecting a machine learning model (region detection) for automatically detecting voids and cracks generated inside an object and at joints of the present invention. [Figure 9] It is an explanatory diagram for explaining a method of selecting a machine learning model (region detection) for automatically detecting voids and cracks generated inside an object and at joints of the present invention. [Figure 10] It is an explanatory diagram for explaining a method of selecting a machine learning model (region detection) for automatically detecting voids and cracks generated inside an object and at joints of the present invention. [Figure 11] It is an explanatory diagram for explaining a method of selecting a machine learning model (region detection) for automatically detecting voids and cracks generated inside an object and at joints of the present invention. [Figure 12] It is an explanatory diagram for explaining a method of selecting a machine learning model (region detection) for automatically detecting voids and cracks generated inside an object and at joints of the present invention. [Figure 13] It is an explanatory diagram for explaining a method of selecting a machine learning model (region detection) for automatically detecting voids and cracks generated inside an object and at joints of the present invention. [Figure 14] It is an explanatory diagram for explaining a method of selecting a machine learning model (region detection) for automatically detecting voids and cracks generated inside an object and at joints of the present invention. [Figure 15] It is an explanatory diagram for explaining a method of selecting a machine learning model (region detection) for automatically detecting voids and cracks generated inside an object and at joints of the present invention. [Figure 16] It is an explanatory diagram for explaining a method of selecting a machine learning model (region detection) for automatically detecting voids and cracks generated inside an object and at joints of the present invention. [Figure 17] It is an explanatory diagram for explaining a method of visualizing the inside of an object and joints of the present invention. [Figure 18]This is an explanatory diagram of the present invention regarding a method for visualizing the interior and joints of an object and the computer program said therefor. [Figure 19] This is an explanatory diagram of the present invention regarding a method for visualizing the interior and joints of an object and the computer program said therefor. [Figure 20] This invention relates to a method for visualizing the interior and joints of an object, and an explanatory diagram of the computer program said therefor. [Figure 21] This invention relates to a method for visualizing the interior and joints of an object, and an explanatory diagram of the computer program said therefor. [Modes for carrying out the invention]

[0017] The present invention will be described below with reference to the drawings illustrating the embodiments. In the embodiments described in the specification, the solder joint of the electronic component 4 will be used as an example for ease of understanding, but the invention is not limited thereto.

[0018] The technical concept of this invention can be applied to many objects and various structures and shapes, such as buildings, electrical equipment, electronic equipment, mechanical parts, mechanical machinery, electronic components, electronic component mounting boards, mechanical parts, mechanical jigs, and internal human diagnostic equipment. In the drawings illustrating embodiments for carrying out the invention, elements having the same function are denoted by the same reference numeral, and their descriptions may be omitted. Furthermore, the embodiments of the present invention can be combined, and parts of them can be combined with other embodiments. Please note that for the purpose of facilitating understanding, simplifying illustrations, etc., images may be enlarged, reduced, or omitted.

[0019] Figure 1 is a block diagram showing the configuration of a void / crack automatic detection (visualization) device 1, an ultrasonic microscope 2, an X-ray CT (Computed Tomography) device 3, and joints between electronic components 4 and solder 5.

[0020] In the embodiments described herein, patterns are acquired using an X-ray CT apparatus 3, but this can be replaced with an X-ray fluoroscopy apparatus, an ultrasonic microscope 2, or other non-destructive observation devices.

[0021] Furthermore, the devices used to acquire patterns are not limited to ultrasonic microscopes 2 and X-ray CT 3. Examples include infrared thermometers that measure patterns consisting of heat distributions, and gamma-ray measuring devices that measure or observe the internal state of objects using gamma rays.

[0022] X-ray microscopes (such as X-ray CT scanners) are microscopes that observe the inside of an object non-destructively. Unlike X-ray fluoroscopy devices, they convert the X-rays that pass through the sample into light and magnify it using optical lenses.

[0023] X-rays have the property of penetrating matter. Some X-rays are absorbed as they pass through a sample. The rate of absorption increases with higher material density (higher atomic number) and thickness, resulting in lower transmitted X-ray intensity (see, for example, Figure 17 and its explanation).

[0024] If voids 36 and cracks 39 occur within the object, the X-ray transmittance of the voids 36 and cracks 39 increases, and therefore the voids 36 and cracks 39 are displayed in the acquired pattern.

[0025] The X-ray CT scanner 3 uses 360° (DEG.) X-ray transmission information of the sample under observation to perform computer calculations and construct 3D data of the sample. By narrowing the rotation pitch and obtaining more information, highly accurate 3D data can be obtained.

[0026] X-ray images are basically two-dimensional images. Numerous voids 36 and cracks 39 occur, and these voids 36 and cracks 39 are distributed in three dimensions. Therefore, the acquired X-ray image will have a pattern of overlapping voids 36 and cracks 39.

[0027] Ultrasonic microscope 2 is a microscope that non-destructively observes defects such as delamination, voids, cracks, and foreign matter inside an object. When ultrasound propagates through different materials, some of it is reflected and some is transmitted. If there are delamination areas (voids) in the object being observed, for example, strong reflected waves will be detected. By performing a planar scan of the object being observed and marking the positions where these reflected waves are detected, it is possible to obtain a two-dimensional distribution of delamination areas (voids). The resulting ultrasound microscope image, like the X-ray image, shows a pattern of overlapping voids 36 and cracks 39.

[0028] The void crack automatic detection (visualization) device 1 of the present invention has the function of automatically or through predetermined operations observing, inspecting, judging, and determining the location and occurrence state of void cracks, and also automatically visualizing and detecting void cracks. Hereinafter, the void crack automatic detection (visualization) device will be referred to as the automatic detection device.

[0029] The X-ray CT apparatus 3 of the present invention can obtain the internal structure of an object by performing a reconstruction process that utilizes the difference in "ease of penetration" and "ease of absorption" when X-rays pass through an object.

[0030] The ultrasonic microscope 2 of the present invention can obtain the internal structure of an object by performing a reconstruction process that utilizes the difference in "ease of transmission" and "ease of reflection" when ultrasonic waves propagate through the object. The automatic detection device 1 comprises a control unit 11 that controls the entire device, a main memory unit 12, a communication unit 13, an operation unit 14, a display panel 15, and an auxiliary memory unit 16.

[0031] The automatic detection device 1 can be composed of, for example, a desktop computer, a notebook personal computer, a tablet, a smartphone, or a computer in a cloud environment that provides computer resources as a service via a computer network such as the Internet.

[0032] The control unit 11 can be composed of a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc. The control unit 11 may also include a GPU (Graphics Processing Unit).

[0033] The main memory unit 12 is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory, and temporarily stores the data necessary for the control unit 11 to perform arithmetic processing. The communication unit 13 has the function of communicating with the automatic detection device 1 via the network 17 and can send and receive necessary information. The control unit 14 is composed of, for example, a hardware keyboard, mouse, touch panel, etc.

[0034] The display panel 15 can be made of a liquid crystal panel or an organic EL (Electro-Luminescence) display panel, etc. The control unit 11 performs control to display the required information on the display panel 15.

[0035] The auxiliary storage unit 16 is a large-capacity memory, hard disk, etc., and stores the programs necessary for the control unit 11 to execute processing, as well as the user authentication program 101, case management program 102, captured image upload program 103, machine learning program 110, automatic detection program 111, analysis result editing and viewing program 112, automatic analysis report creation program 113, analysis report notification program 114, analysis report editing and viewing program 115, pattern classification program 133, image scale adjustment program 134, and user editing status confirmation program 135.

[0036] Figure 17 is an explanatory diagram of the method for visualizing the inside of an object and its joints according to the present invention. It is an explanatory diagram for determining or adjusting the acquisition range when the object to be measured or observed to be visualized is an electronic component 4.

[0037] The distance between the X-ray focus 162 and the X-ray detector is the distance from the X-ray focus to the X-ray detector, which is 161. The distance between the X-ray focus 162 and the electronic component 4 is the distance from the X-ray focus to the sample, which is 160.

[0038] X-rays are emitted from the X-ray focus 162. By adjusting or setting the distance 160 from the X-ray focus (X-ray focus 162) to the sample, the size of the magnified projection image 164 projected onto the X-ray detector 163 changes. Additionally, the distance 161 from the X-ray focus to the X-ray detector is adjusted or set.

[0039] When acquiring data with an X-ray CT scanner, the magnification is the ratio of the size of the magnified projection image 164 on the detector input surface of the X-ray CT scanner to the size of the sample. It can be calculated as the distance from the X-ray focus 162 to the X-ray detector 163 (161) divided by the distance from the X-ray focus to the sample (160). As the distance from the X-ray focus 163 to the sample (160) increases, the magnification decreases and the acquisition range widens.

[0040] By adjusting or changing the size of the magnified projection image 164 on the X-ray detector, many images, machine learning data, etc. can be obtained. In addition, by rotating, tilting, moving the electronic component 4 as a sample, many images, machine learning data, etc. can be obtained.

[0041] The user authentication program 101, case management program 102, captured image upload program 103, machine learning program 110, automatic detection program 111, analysis result editing and viewing program 112, automatic analysis report creation program 113, analysis report notification program 114, analysis report editing and viewing program 115, pattern classification program 133, image scale adjustment program 134, and user editing status confirmation program 135, which are stored in the auxiliary storage unit 16, may be provided by a recording medium 18 on which each program is recorded in a readable format.

[0042] The recording medium 18 is, for example, a portable memory such as a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, a microSD card, or a CompactFlash (registered trademark). Alternatively, data may be read from the recording medium 18 using a reading device not shown in the diagram and installed in the auxiliary storage unit 16. Next, we will explain how images taken by the ultrasonic microscope 2 and the X-ray CT scanner 3 are stored in the auxiliary storage unit 16.

[0043] If the ultrasonic microscope 2 and X-ray CT scanner 3 are connected to the internet, user authentication is automatically performed by the user authentication program 101 based on the configuration information of the ultrasonic microscope 2 and X-ray CT scanner 3, and case registration is automatically performed by the case management program 102.

[0044] Therefore, images captured by the ultrasonic microscope 2 and the X-ray CT scanner 3 are uploaded to the auxiliary storage unit 16 by the image upload program 103 and stored in the image database (1).

[0045] On the other hand, if the ultrasound microscope 2 or X-ray CT scanner 3 is not connected to the internet, the images taken from the equipment are retrieved and copied to a computer or other device connected to the internet.

[0046] On the computer, the user authentication program 101 performs the login operation, and the case management program 102 registers the case. Next, the captured image upload program 103 uploads the image to the auxiliary storage unit 16, where it can be saved as captured image (1) DB 104.

[0047] In the auxiliary storage unit 16, uploaded image data is saved in the captured image (1) DB104, and the results of the automatic detection of voids and cracks are saved in the analysis results (1) DB105, each for each user, group, and case.

[0048] Analysis reports containing void ratios and crack ratios automatically measured from the automated detection results are saved in Analysis Report (1) DB106, separately for each user, group, and project.

[0049] The datasets created using the captured images (1) DB104 and the analysis results (1) DB105 are stored in the machine learning data (1) DB107, separately for each user, group, and project. Only users authenticated by user authentication program 101 can access each piece of data.

[0050] The auxiliary storage unit 16 stores an automatic detection program 111. This automatic detection program 111 uses a machine learning model (region detection) DB109 to perform automatic crack detection and automatic measurement of the crack rate on the captured image (1) DB104.

[0051] Furthermore, the captured images (1) stored in DB104 can be processed in batches at regular intervals. Details of this automatic detection program 111 will be described later.

[0052] The results automatically detected and measured by the automatic detection program 111 are stored in the analysis results (1) DB 105. The stored analysis result data is read by the analysis result editing and viewing program 112, and the user can edit and view the analysis results. In this editing operation, users authorized by the user authentication program 101 can collaboratively edit the data.

[0053] The automated analysis report generation program 113 reads the images stored in the captured images (1) DB104 and the analysis results (1) DB105, and automatically generates numerical data such as void ratio and crack ratio, as well as text data such as the cause of defects, in accordance with the format specified by the user. It then compiles these into an analysis report and saves it in the analysis report (1) DB106.

[0054] If a new or updated analysis report is available, the analysis report notification program 114 will notify the user of its contents via email, message, chat service, etc. The user can then access the analysis report editing and viewing program 115 to edit and view the report. This editing process can be collaboratively performed by users authorized by the user authentication program.

[0055] The auxiliary storage unit 16 stores a machine learning program 110. This machine learning program 110 uses captured images (1) DB104 and analysis results (1) DB105 to create machine learning data (1) DB, performs machine learning using this dataset, and creates a machine learning model (pattern classification) DB108 and a machine learning model (region detection) DB109.

[0056] Regarding the datasets used for machine learning, sometimes individual databases of captured images and analysis results, stored for each user, group, or project, are used, while other times they are shared and used together.

[0057] The created machine learning models (pattern classification) DB108 and (region detection) DB109 are used in the pattern classification program 133 and the automatic detection program 111. Therefore, the machine learning program 110 is executed at a different time than when the pattern classification program 133 and the automatic detection program 111 are executed (for example, during times when the user is not using the system).

[0058] The pattern classification program 133 uses the machine learning model (pattern classification) DB108 to classify electronic components by pattern, particularly by shape, form, structure, etc., as shown on the vertical axis of Figure 2.

[0059] The image scale adjustment program 134 is a program that adjusts the differences in scale, particularly enlargement, rotation, and position, as shown on the horizontal axis of Figure 2. The adjustment method may use the scale value 501 indicated in the image (for example, how many micrometers one pixel corresponds to), the BGA ball diameter 502, or the component length 503, and a specific adjustment method is defined according to the pattern classified by the pattern classification program 133. This image scale adjustment program 134 reduces the variation in the input image to the subsequent automatic detection program 111, contributing to improved accuracy of the automatic detection results.

[0060] The user editing status confirmation program 135 is a program that checks the user's editing status in the analysis result editing and viewing program 112, etc., for example, how much editing has been done on the output results of the automatic detection program 111, the viewing time, etc., and decides whether to retain the selection result of the model selection program 504, which will be explained later.

[0061] Figure 3 is an explanatory diagram illustrating the flowchart for automatically detecting voids 36 and cracks 39 in a group of captured images using machine learning models (pattern classification) DB108 and (region detection) DB109 created by the machine learning program 110.

[0062] First, a set of images is input and uploaded using at least one of the ultrasound microscope 2 or the X-ray CT scanner 3. Uploading is exemplified by saving the images to the case management program 102. The upload destination is not limited to a cloud server; for example, it could be a smartphone with the functionality of the automatic detection device 1.

[0063] Next, the pattern classification program 133 uses the machine learning model (pattern classification) DB108 to classify the captured images based on the shape, form, structure, etc. of the electronic components.

[0064] The captured images, categorized into each pattern, are then subjected to image adjustments such as scaling, rotation, and repositioning by the image scaling adjustment program 134, using specific adjustment methods based on each pattern.

[0065] Next, the automated detection program 111 automatically detects the regions of voids 36 and cracks 39 in the images of the image-adjusted captured image set using an appropriate machine learning model (region detection) from the machine learning model (region detection) DB109. Details of the automated detection program 111 will be described later.

[0066] The automatically detected results can be edited and viewed by the user using the analysis result editing and viewing program 112, and the editing and viewing status is monitored by the user editing status confirmation program. Details of the user editing status confirmation program will be described later. Figure 4 is an explanatory diagram showing the details of the automatic detection program 111 and the user editing status confirmation program 134.

[0067] First, the image scale adjustment program 134 checks whether a machine learning model (region detection) exists for the project to which the captured image belongs. As mentioned above, each captured image is saved for each user, group, or project, and some of these images may have a machine learning model (region detection) that does not.

[0068] This is because, as mentioned above, the machine learning program 110 that creates the machine learning model (region detection) is executed at a different time than when the pattern classification program 133 is executed (for example, during times when the user is not using the system).

[0069] If the answer to "Does the case to which this image belongs have a machine learning model (region detection)?" in Figure 4 is "Yes," that is, if a machine learning model (region detection) exists for the case to which the captured image belongs (let's call this case (1) for explanation purposes), the region detection prediction program 505 uses the machine learning model (region detection) (1) to automatically detect voids 36 and cracks 39 in the image, and the results are saved in the analysis results (mask image) group 22.

[0070] On the other hand, if the answer to "Does the case to which this image belongs have a machine learning model (region detection)?" in Figure 4 is "No," that is, if there is no machine learning model (region detection) for the case to which the captured image belongs (for the sake of explanation, this will be referred to as case (3)), the model selection program 504 selects a machine learning model (region detection) from the machine learning model (region detection) DB 109 to automatically detect voids 36 and cracks 39 in the image. The selection method will be described later.

[0071] Using the machine learning model (region detection) selected by the user, the region detection prediction program 505 automatically detects voids 36 and cracks 39 in the image, as in the previous case, and saves the results to the analysis results (mask images) group 22.

[0072] Next, the analysis results editing and viewing program 112 reads the results, making them editable and viewable by the user. At this time, the user editing status confirmation program 134 checks to what extent the results have been edited and viewed.

[0073] If the answer to "Has the analysis result been edited to some extent?" in Figure 4 is "No," meaning the user editing status confirmation program 134 determines that the analysis result was edited minimally or that the viewing time was short, then the model selection program 504 determines that the user's selection of the machine learning model (region detection) was appropriate, and the machine learning model copy program 506 copies the reference of the machine learning model (region detection) selected by the user for its own case (case (3)).

[0074] This copy allows the user to use the same machine learning model previously selected when handling images for their own projects (Project (3)) in the future, without having to select a model again. This improves the user's convenience in image inspection.

[0075] On the other hand, if the answer to "Have the analysis results been edited to a certain extent?" in Figure 4 is "Yes," that is, if the user editing status confirmation program 134 determines that the analysis results have been edited extensively or that the viewing time has been long, then the aforementioned model selection program 504 determines that the user's selection of a machine learning model (region detection) was inappropriate, and the aforementioned copying is not performed. Therefore, in the future, when handling images for my own projects (Project (3)), I will again select an appropriate machine learning model (region detection).

[0076] Figure 5 is an explanatory diagram of the operation screen of the model selection program 133. Figure 5 shows an embodiment in which a region detection result area 137 and four region detection result areas 137 (region detection result area 137a, region detection result area 137b, region detection result area 137c, and region detection result area 137d) are provided.

[0077] The area detection result area 137 may be one area in the display area 40, or it may be four or more areas in the display area 40. One or more area detection result areas 137 are generated in the display area 40 in accordance with the operation state, usage status, or processing status. The display area 40 is displayed on the display panel 15. It can also be displayed online or offline on a user-controlled display device.

[0078] In the following explanation, void 36 is used as an example of the pattern displayed in the region detection result area 137, but it goes without saying that it is not limited to void 36, and other evaluation patterns such as crack 39 may also be used.

[0079] The present invention is characterized in that, in the process of selecting the optimal machine learning model from the limited group of candidate machine learning models described above, the model selection program 504 of the present invention provides a display method that allows users without special knowledge of machine learning models to easily select the optimal machine learning model.

[0080] The present invention is characterized by taking an approach that indirectly obtains the user's evaluation of the prediction results using a user editing status confirmation program 135, thereby maintaining a record of which machine learning model should be applied when the same type of image input is provided, and improving the convenience of image inspection for the user.

[0081] In the display area 40, the results of region detection by multiple machine learning models (region detection) DB109 are displayed in the region detection result area 137, divided into four screens. Furthermore, to facilitate understanding of operations using input devices (finger touch input, mouse input, input pad input, etc.), these are illustrated as input device operations 140.

[0082] The model selection program 133 displays the detection results of candidate machine learning models (region detection) in the display area 40. The display area 40 is divided into multiple sections, and the detection results are displayed in each of the divided sections.

[0083] By performing an operation 140 on the input device, it is possible to switch to the display of the captured image 138b by pressing the left corner of the region detection result area 137b or swiping to the right (arrow direction) on the region detection result area 137b. The animation state during the switching is illustrated as the switching animation 139.

[0084] Figure 6 is an explanatory diagram of the operation screen of the model selection program 133. As shown in Figure 6, during the transition from the region detection result area 137b to the display of the captured image 138b, the display is changed by switching animations 139 such as front / back display and page turning display to highlight the difference between the two.

[0085] Figure 7 is an explanatory diagram of the operation screen of the model selection program 133. Switching from the display of the captured image 138b to the display of the region detection result area 137b is possible by pressing the left corner of the display of the captured image 138b or by swiping left on the display of the captured image 138b.

[0086] Figure 8 is an explanatory diagram of the operation screen of the model selection program 133. As shown in Figure 8, it is also possible to add switching animation effects, similar to Figure 6. As shown in Figure 8, while the display transitions from the display of the captured image 138b to the region detection result area 137b, the display changes with switching animations 139 such as front / back display and page turning display, which highlights the difference between the two.

[0087] Figure 9 is an explanatory diagram of the operation screen of the model selection program 133. To display the selected region detection result area 137b across the entire display area 40, press near the center of the region detection result area 137b.

[0088] For example, press and hold the area near the center of the region detection result area 137b for more than one second. By pressing the area near the center of the region detection result area 137b, the region detection result area 137b will be displayed across the entire display area 40.

[0089] Figure 10 is an explanatory diagram of the operation screen of the model selection program 133. In the embodiment shown in Figure 9, it is also effective to add a switching animation effect, similar to that in Figure 6. By pressing near the center of the region detection result area 137b, the region detection result area 137b expands as shown by the arrow. The region detection result area 137b expands to cover the entire display area 40, and the operation is designed so that the expansion of the region detection result area 137b is visually recognizable.

[0090] Figure 11 is an explanatory diagram of the operation screen of the model selection program 133. The size of each region of the divided region detection result area 137 (region detection result area 137a, region detection result area 137b, region detection result area 137c, region detection result area 137b) can also be changed by dragging the dividing lines.

[0091] As shown in Figure 12, hold down the intersection of region detection result area 137a, region detection result area 137b, region detection result area 137c, and region detection result area 137b, and move the operation 140 on the input device as shown in Figure 12.

[0092] Figure 12 is an explanatory diagram of the operation screen of the model selection program 133. As shown in Figure 12, the display area of ​​the region detection result area 137a, region detection result area 137c, etc. can be varied by moving the position of the operation 140 on the input device.

[0093] Figure 13 is an explanatory diagram of the operation screen of the model selection program 133. Similar to the operation in Figure 9, even for the single-display region detection result area 137, it is possible to switch to the display of the captured image 138 by pressing the left corner of the region detection result area 137 or swiping right on the region detection result area 137. The selection of the machine learning model (region detection) can be completed by pressing the Adopt button (OK) 141.

[0094] Furthermore, the "Adopt" button may display multiple selection buttons, such as "Back" and "Update," in addition to "OK," allowing the user to click or select any one or more of these buttons.

[0095] Figure 14 is an explanatory diagram of the operation screen of the model selection program 133. Similar to Figure 6, it is also possible to add switching animation effects. As shown in Figure 14, while the display transitions from the region detection result area 137 to the display of the captured image 138, the difference between the two is highlighted by switching animations 139 such as front / back display and page turning display.

[0096] Figure 15 is an explanatory diagram of the operation screen of the model selection program 133. Switching to the captured image display 138 is possible by pressing the left corner of the captured image display 138 or by swiping left on the captured image display 138. Also, similar to Figure 13, the selection of the machine learning model (region detection) can be completed by pressing the adopt button 141. Figure 16 is an explanatory diagram of the operation screen of the model selection program 133. Similar to Figure 6, it is also possible to add switching animation effects.

[0097] The embodiments of the present invention described above relate to a device and method for detecting and visualizing voids 36 inside a material. Cracks 39 can also be patterned using an X-ray CT device 3, etc. Although the images of voids 36 and cracks 39 are different, by training the device to recognize them as cracks 39, it becomes possible to detect them. Therefore, it goes without saying that the items described as voids 36 in the embodiments of this specification can be replaced with items described as cracks 39.

[0098] Furthermore, patterns can be acquired not only by obtaining transmitted X-ray images using an X-ray CT scanner 3, but also, for example, by using an ultrasonic microscope 2 to obtain patterns corresponding to transmitted X-ray images. A gamma-ray device can also be used. Therefore, it goes without saying that in the apparatus of the present invention, the X-ray device such as an X-ray CT scanner may be replaced with an ultrasonic device 2 such as an ultrasonic microscope. Furthermore, by combining an X-ray device such as an X-ray CT scanner 3 with an ultrasonic microscope 2 to acquire patterns, the detection accuracy of voids 36, cracks 39, etc., is improved.

[0099] The above embodiments describe methods or apparatus for detecting voids 36, cracks 39, etc., in transmitted X-ray images, but the present invention is not limited thereto. Needless to say, it is possible to non-destructively analyze a wide variety of things, such as air bubbles in adhesive resins, spaces in concrete blocks, inclusions of other metals in iron material parts, and fat particles in the internal organs of living organisms. Figures 18, 19, 20, and 21 are explanatory diagrams of a business model using the automatic detection method and computer program of the present invention.

[0100] Figure 18 is an explanatory diagram of the business model in the first embodiment. To customer A, who possesses images taken with an ultrasonic microscope 2, an X-ray CT scanner 3, etc., company A provides or sells a program (hereinafter referred to as the "void / crack automatic detection program") that includes a user authentication program 101, a case management program 102, an image upload program 103, a machine learning program 110, an automatic detection program 111, an analysis result editing and viewing program 112, an automatic analysis report creation program 113, an analysis report notification program 114, an analysis report editing and viewing program 115, etc., as shown in Figure 1.

[0101] Customer A will be able to install and use the void crack automatic detection program on their desktop computer, laptop computer, tablet, and smartphone. Company A will receive a license fee for the void crack automatic detection program from Customer A.

[0102] Figure 19 is an explanatory diagram of the business model in the second embodiment. To customer A, who does not possess images taken with an ultrasonic microscope 2, an X-ray CT scanner 3, etc., company A provides or sells a collection of images taken with company A's own ultrasonic microscope 2, X-ray CT scanner 3, etc., and an automated void / crack detection program.

[0103] Customer A will be able to install and use the void crack automatic detection program on their desktop computer, laptop computer, tablet, and smartphone. Company A will receive image acquisition fees and license fees for the void crack automatic detection program from Customer A.

[0104] Figure 20 is an explanatory diagram of the business model in the third embodiment. For customer A, who possesses images taken with an ultrasonic microscope 2, an X-ray CT scanner 3, etc., company A operates an automated void / crack detection program on a cloud server.

[0105] Customer A can use the void crack automated detection program using a web browser on a desktop computer, laptop computer, tablet, or smartphone. Company A receives a fee from Customer A for using the void crack automated detection program.

[0106] Figure 21 is an explanatory diagram of the business model in the fourth embodiment. For customer A, who does not possess images taken with an ultrasonic microscope 2, X-ray CT scanner 3, etc., company A prepares a set of images taken with its own ultrasonic microscope 2, X-ray CT scanner 3, etc., and runs an automated void / crack detection program on a cloud server.

[0107] Customer A can use the void crack automatic detection program using a web browser on a desktop computer, laptop computer, tablet, or smartphone. Company A receives image acquisition fees and usage fees for the void crack automatic detection program from Customer A.

[0108] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims, not in the sense described above, and all modifications are intended to be in the sense and scope equivalent to the claims. Needless to say, the matters or contents described herein and in the drawings can be combined with each other. [Explanation of Symbols]

[0109] 1. Automatic Void and Crack Detection System 2. Ultrasonic Microscope 3 X-ray CT device 4. Electronic components (measurement samples) 5 solder 11 Control Unit 12 Main memory 13 Communications Department 14 Control section 15 Display Panel 16 Auxiliary storage 17 Network 18 Recording media 21 Images 22 Analysis Results (Mask Images) Group 31 Pattern Classification Training Program 32 Pattern Classification Prediction Program 36 Void 37 categories (processing units) 38 Soldering section 39 Crack 40 display area 41. Area detection training program 42 Region Detection and Prediction Program 101 User Authentication Program 102 Project Management Program 103 Image Upload Program 104 Captured Image (1) DB 105 Analysis results (1) DB 106 Analysis Report (1) DB 107 Data for Machine Learning (1) DB 108 Machine Learning Model (Pattern Classification) Database 109 Machine Learning Models (Region Detection) Database 110 Machine Learning Programs 111 Automatic detection program 112 Analysis Result Editing and Viewing Program 113 Automatic Analysis Report Generation Program 114 Analysis Report Notification Program 115 Analysis Report Editing and Viewing Program 133 Model Compatibility Program 134 Image Scale Adjustment Program 135 User Editing Status Confirmation Program 137 Area detection results by candidate models 138 Display of captured images 139 Front / Back Display Switching Animation 140 Operation with input devices 141 Accept button 160 Distance from X-ray focus to sample 161 Distance from X-ray focus to X-ray detector 162 X-ray focus 163 X-ray detector 164 Enlarged projection image 201 Pattern Classification Prediction Program 202 Area detection and prediction program for Pattern A 203 Area detection and prediction program for Pattern B 204 Area detection and prediction program for Pattern C 301 Captured Images (AB) DB 302 Pattern (AB) DB 303 Data augmentation preprocessing program 304 Machine Learning Data (AB) DB 305 Machine Learning (Training) Programs 306 Images 307 Preprocessing Program 308 Machine Learning (Prediction) Programs 309 Pattern Classification Results 401 Captured Image (A) DB 402 Mask Image (A) DB 403 Data augmentation preprocessing program 404 Machine Learning Data (A) DB 405 Machine Learning (Training) Programs 406 Images 407 Preprocessing Program 408 Machine Learning (Prediction) Programs 409 Analysis Results (Mask Images) Group 501 Scale values ​​indicated in the image 502 BGA ball diameter 503 Part length 504 Model Selection Program 505 Region Detection and Prediction Program 506 Machine Learning Model Copy Program

Claims

1. An image acquisition device that acquires at least one image from among an X-ray image of the joint, a cross-sectional image, and an ultrasound image, A pattern classification program that classifies the acquired images into patterns using a machine learning model, An image adjustment program that adjusts at least one of the position, rotation, and scaling of the object to be visualized in the aforementioned image, A detection program for detecting objects to be visualized from the aforementioned image, A display device that displays images of multiple objects to be visualized, A visualization device for voids and cracks in a joint, characterized by comprising an editing status confirmation program for confirming the editing status of the object to be visualized.

2. An image acquisition device that acquires at least one image from among an X-ray image of the joint, a cross-sectional image, and an ultrasound image, A pattern classification program that classifies the acquired images into patterns using a machine learning model, An image adjustment program that adjusts at least one of the position, rotation, and scaling of the object to be visualized in the aforementioned image, A detection program for detecting objects to be visualized from the aforementioned image, A display device that displays images of multiple objects to be visualized, A selection program that selects an image of a predetermined object to be visualized from the images of multiple objects to be visualized that are displayed, A visualization device for voids and cracks in a joint, characterized by comprising an editing status confirmation program for confirming the editing status of the object to be visualized.

3. An image acquisition device that acquires at least one image from among an X-ray image of the joint, a cross-sectional image, and an ultrasound image, A pattern classification program that classifies the acquired images into patterns using a machine learning model, An image adjustment program that adjusts at least one of the position, rotation, and scaling of the object to be visualized in the aforementioned image, A detection program for detecting objects to be visualized from the aforementioned image, A selection program in which the user views images of the objects to be visualized and selects an image of the object to be visualized, A visualization device for voids and cracks in a joint, characterized by comprising an editing status confirmation program for confirming the editing status of the object to be visualized.

4. An image acquisition device that acquires at least one image from among an X-ray image of the joint, a cross-sectional image, and an ultrasound image, A pattern classification program that classifies the acquired images into patterns using a machine learning model, A detection program for detecting a first object to be visualized from the objects to be visualized in the aforementioned image, A display device that displays an image of the first object to be visualized, A selection program that selects a predetermined second visualization target image from the first visualization target image displayed on the display device, A visualization device for voids and cracks in a joint, characterized by comprising an editing status confirmation program for confirming the editing status of the object to be visualized.

5. The joint is a soldered portion, and the object to be visualized is at least one of a void and a crack. The apparatus for visualizing voids and cracks in a joint according to claim 1, 2, 3, or 4, characterized in that the pattern is based on the shape, form, or structure of the joint.

6. A first operation of classifying at least one image from among the X-ray image, cross-sectional image, and ultrasound image of the joint into a pattern using a machine learning model, A second operation to adjust at least one of the position, rotation, and scaling of the object to be visualized in the aforementioned image, A fourth operation involves selecting a predetermined machine learning model from a plurality of machine learning models based on an image of the object to be visualized selected by the user, A method for visualizing voids and cracks in a joint, characterized by having a sixth operation for checking the editing status of the object to be visualized.

7. A first operation of classifying at least one image from among the X-ray image, cross-sectional image, and ultrasound image of the joint into a pattern using a machine learning model, A second operation to adjust at least one of the position, rotation, and scaling of the object to be visualized in the aforementioned image, A third operation involves displaying an image of the object to be visualized on a display device, and the user selecting an image of a predetermined object to be visualized from the images of the object to be visualized. A fourth operation involves selecting a predetermined machine learning model from a plurality of machine learning models based on an image of the object to be visualized selected by the user, A method for visualizing voids and cracks in a joint, characterized by having a sixth operation for checking the editing status of the object to be visualized.

8. A first operation of classifying at least one image from among the X-ray image, cross-sectional image, and ultrasound image of the joint into a pattern using a machine learning model, A second operation to adjust at least one of the position, rotation, and scaling of the object to be visualized in the aforementioned image, A third operation involves displaying an image of the object to be visualized on a display device, and the user selecting an image of a predetermined object to be visualized from the images of the object to be visualized. A fifth operation involves determining whether the machine learning model is appropriate based on the time the user viewed the image of the object to be visualized, and then modifying or maintaining the machine learning model. A method for visualizing voids and cracks in a joint, characterized by having a sixth operation for checking the editing status of the object to be visualized.

9. The method for visualizing voids and cracks in a joint according to claim 6, claim 7, or claim 8, characterized in that the pattern is based on the shape, form, or structure of the joint.

10. The method for visualizing voids and cracks in a joint according to claim 6, claim 7, or claim 8, characterized in that the joint is a soldered portion and the object to be visualized is at least one of a void or a crack.

Images