Visualization device for void and crack in joint portion and method for visualizing void and crack in joint portion

Abstract

To solve a problem that it is easy to determine a predictive model that outputs the most likely prediction result for a known inspection target, but it is difficult for an unknown inspection target, when performing image inspection using a predictive model based on semantic segmentation, etc.SOLUTION: A pattern classification program 133 and an image scale adjustment program 134 classify and adjust an image of an unknown inspection target, thereby narrowing down an optimal machine learning model from among a plurality of machine learning models. In a process of selecting the best machine learning model from a limited candidate group, a model selection program 504 determines a prediction model that outputs the most likely prediction result by using a display method that allows a user who does not have special knowledge of a machine learning model to easily select the optimal machine learning model and by checking an editing status of a visualization object.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a visualization device and a visualization method that non-destructively visualizes spaces, voids, and cracks that occur inside or at the joints of objects such as electronic components, semiconductor elements, thin film devices, thick film devices, and electronic equipment using transmission X-ray images and ultrasonic microscope images, as well as a computer program, a method for operating the computer program, a display method, etc. The present invention also relates to a method for selecting and applying a machine learning model, a method for creating a predictive model, and a method for using a predictive model. [Background technology]

[0002] The terminals of the electronic components are soldered to the electrodes of the substrate on which the electronic components are mounted. Voids 36 occur in the joints of the solder 5 due to preheating and reflow conditions. Furthermore, the electronic components and substrate repeatedly expand and contract due to thermal stress, and the difference in their expansion rates causes cracks 39 to occur in the joints. In order to improve the yield of electronic circuit boards, it is important to non-destructively detect voids 36 and cracks 39 in joints and inspect their quality. Conventionally, methods of visually inspecting transmitted X-ray images or ultrasonic microscope images have been used as non-destructive methods of inspecting joints such as solder 5 .

[0003] Visual inspection can detect slight changes in the image, but there is a large variability in the pass / fail judgments made by inspectors, and the inspection points are limited due to time constraints. It is difficult to accurately detect and measure voids36 and cracks39 with visual inspection, and the process is time-consuming. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2012-68209 Summary of the Invention [Problem to be solved by the invention]

[0005] In conventional image inspection methods using machine learning models, in order to robustly detect voids36 and cracks39, which have various characteristics, it was necessary to prepare multiple machine learning models tailored to the object being photographed, rather than using a single machine learning model. Therefore, when an image of an unknown subject is input, it is difficult to automatically determine which machine learning model should be applied to make a prediction. [Means for solving the problem]

[0006] To address these issues, the present invention takes an approach that allows users to appropriately select the optimal machine learning model from multiple machine learning models using a dedicated display method. By selecting the machine learning model that provides the best prediction results, the user can detect a wide variety of voids 36 and cracks 39 with high accuracy, significantly reducing the inspection time compared to conventional methods.

[0007] The present invention relates to a device for visualizing the inside of an object, a method for visualizing the inside of an object, and a computer program for the device for visualizing the inside of an object, which are characterized by limiting an optimal machine learning model from among multiple machine learning models used to visualize the inside of an object.

[0008] The present invention is characterized by classifying and adjusting images of unknown subjects using a pattern classification program 133 and an image scale adjustment program 134, thereby narrowing down the optimal machine learning model from among multiple machine learning models.

[0009] The present invention is characterized by having a display method that allows a user who does not have special knowledge about machine learning models to easily select the optimal machine learning model using the model selection program 504 of the present invention in the process of selecting the optimal model from the group of candidate machine learning models limited as described above.

[0010] The present invention is characterized by taking an approach that uses a user editing status confirmation program 135 to indirectly obtain the user's evaluation of the prediction results, thereby retaining which machine learning model should be applied when the same type of image is input, thereby improving the convenience of image inspection for the user. The automatic detection device 1 also includes a control unit 11 that controls the entire device, a main memory unit 12, a communication unit 13, an operation unit 14, a display panel 15, and an auxiliary memory unit 16.

[0011] Images captured by the ultrasonic microscope 2 and the X-ray CT device 3 are uploaded to the auxiliary storage unit 16 by the captured image upload program 103 and stored in the captured image (1) DB.

[0012] The present invention narrows down the optimal machine learning model from among multiple machine learning models used to visualize the interior of an object by performing a first operation of classifying the type of image data using a machine learning model from at least one of a transmission X-ray image acquired by an X-ray device, a cross-sectional image acquired by an X-ray CT device, and an ultrasound image acquired by an ultrasound device, and a second operation of adjusting the position, angle, and scale of the object in the image data.

[0013] The present invention also provides a display method that allows a user with no special knowledge of machine learning models to easily select the most appropriate machine learning model from a limited group of candidate models by performing a first operation of classifying the type of image data from at least one of a transmission X-ray image acquired by an X-ray device, a cross-sectional image acquired by an X-ray CT device, and an ultrasound image acquired by an ultrasound device, and a second operation of adjusting the position, angle, and scale of the object in the image data.

[0014] In addition, the present invention improves the convenience of image inspections for users by performing a first operation of classifying the type of image data using a machine learning model from at least one of a transmission X-ray image acquired by an X-ray device, a cross-sectional image acquired by an X-ray CT device, and an ultrasound image acquired by an ultrasound device, a second operation of adjusting the position, angle, and scale of the object in the image data, and a third operation of visualizing the interior of the object in the image data using a machine learning model selected by the user, and then indirectly obtaining the user's evaluation of the visualization results and retaining information on whether the selected machine learning model should be applied next time. [Effects of the Invention]

[0015] By using multiple machine learning models rather than a single model and then selecting the most appropriate one, it becomes possible to automatically detect a wide variety of voids 36 and cracks 39 in electronic components with high accuracy, thereby significantly reducing the inspection time or detection time required in the past. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is an explanatory diagram of a visualization device for the inside and joints of an object according to the present invention. [Figure 2] 10A and 10B are explanatory diagrams of visualized observation images of solder joints and the like. [Figure 3] 1 is an explanatory diagram of a method for detecting void and crack regions from a captured image in a method for visualizing the inside of an object and a joint of the present invention. [Figure 4] 1 is an explanatory diagram of a method for detecting void and crack regions from a captured image in a method for visualizing the inside of an object and a joint of the present invention. [Figure 5] FIG. 1 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint according to the present invention. [Figure 6] FIG. 1 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint according to the present invention. [Figure 7] FIG. 1 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint according to the present invention. [Figure 8] FIG. 1 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint according to the present invention. [Figure 9] FIG. 1 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint according to the present invention. [Figure 10] FIG. 1 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint according to the present invention. [Figure 11] FIG. 1 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint according to the present invention. [Figure 12] FIG. 1 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint according to the present invention. [Figure 13] FIG. 1 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint according to the present invention. [Figure 14] FIG. 1 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint according to the present invention. [Figure 15] FIG. 1 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint according to the present invention. [Figure 16] FIG. 1 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint according to the present invention. [Figure 17] 1A to 1C are explanatory diagrams illustrating a method for visualizing the inside of an object and a joint according to the present invention. [Figure 18]1A to 1C are explanatory diagrams of a method for visualizing the inside of an object and a joint of the object according to the present invention and a computer program therefor. [Figure 19] 1A to 1C are explanatory diagrams of a method for visualizing the inside of an object and a joint of the object according to the present invention and a computer program therefor. [Figure 20] 1 is an explanatory diagram of a method for visualizing the inside of an object and a joint of the object according to the present invention and a computer program therefor. [Figure 21] 1 is an explanatory diagram of a method for visualizing the inside of an object and a joint of the object according to the present invention and a computer program therefor. DETAILED DESCRIPTION OF THE INVENTION

[0017] The present invention will be described below with reference to the drawings showing embodiments thereof. In the embodiments described in the specification, for ease of understanding, the solder joints of the electronic component 4 will be described as an example, but the present invention is not limited to this.

[0018] The technical concept of the present invention can be applied to many objects and various structures and shapes, such as buildings, electrical equipment, electronic equipment, mechanical parts, mechanical equipment, electronic parts, electronic part mounting boards, mechanical parts, mechanical jigs, and internal human body diagnosis. In the drawings for explaining the embodiments of the invention, elements having the same functions are given the same reference numerals, and descriptions thereof may be omitted. In addition, the embodiments of the present invention can be combined with each other, and some of the embodiments can be combined with other embodiments. In addition, for the purpose of facilitating understanding and facilitating illustration, the drawings may be enlarged, reduced, or omitted.

[0019] FIG. 1 is a block diagram showing the configuration of an automatic void / crack detection (visualization) device 1, an ultrasonic microscope 2, an X-ray CT (Computed Tomography) device 3, and a joint between an electronic component 4 and solder 5, etc.

[0020] In the examples of this specification, the pattern is described as being acquired by the X-ray CT device 3, but this may be replaced by an X-ray fluoroscopic observation device, an ultrasonic microscope 2, or other non-destructive observation device.

[0021] Furthermore, the device for acquiring the pattern is not limited to the ultrasonic microscope 2 or the X-ray CT 3, but may be, for example, an infrared thermometer that measures a pattern consisting of heat distribution, or a gamma ray measuring device that measures or observes the internal state of an object using gamma rays.

[0022] The X-ray microscope (X-ray CT, etc.) 3 is a microscope that observes the inside of an object non-destructively. Unlike an X-ray fluoroscopic observation device, it converts X-rays that pass through the sample into light, etc., and magnifies them using optical lenses, etc.

[0023] X-rays have the property of penetrating matter. As X-rays pass through a sample, some of them are absorbed. The absorption rate increases as the density of the material (high atomic number) and thickness increase, resulting in a decrease in the intensity of the penetrating X-rays (see, for example, Figure 17 and its explanation).

[0024] If voids 36 and cracks 39 occur in the object, the X-ray transmittance of the voids 36 and cracks 39 increases, and the voids 36 and cracks 39 appear in the acquired pattern.

[0025] The X-ray CT device 3 uses a computer to process the X-ray transmission information from 360° (DEG.) of the sample being observed, and constructs 3D data of the sample. By narrowing the rotation pitch and obtaining more information, it is possible to obtain highly accurate 3D data.

[0026] X-ray images are basically two-dimensional images. Many voids 36 and cracks 39 occur, and the voids 36 and cracks 39 are distributed three-dimensionally. Therefore, the acquired X-ray image shows a pattern in which the voids 36 and cracks 39 overlap.

[0027] The ultrasonic microscope 2 is a microscope that non-destructively observes defects such as delamination, voids, cracks, and foreign bodies inside an object. When ultrasonic waves propagate through different materials, some of the waves are reflected and some are transmitted. If the object being observed has a delamination (void), for example, a strong reflected wave will be detected. By scanning the object in a plane and marking the positions where this reflected wave is detected, it is possible to obtain a two-dimensional distribution of delaminations (voids). The ultrasonic microscope image, like the X-ray image, also shows a pattern in which voids 36 and cracks 39 overlap.

[0028] The automatic void / crack detection (visualization) device 1 of the present invention observes, inspects, judges, and determines the location and occurrence state of voids and cracks automatically or through predetermined operations, and also has the function of automatically visualizing and detecting voids and cracks. Hereinafter, the automatic void / crack detection (visualization) device will be referred to as the automatic detection device.

[0029] The X-ray CT device 3 of the present invention can obtain the internal structure of an object by performing reconstruction processing using the differences in the "ease of penetration" and "ease of absorption" of X-rays when they pass through the object.

[0030] The ultrasonic microscope 2 of the present invention can obtain the internal structure of an object by performing reconstruction processing using the differences in the "ease of transmission" and "ease of reflection" that occur when ultrasonic waves propagate within the object. The automatic detection device 1 includes a control unit 11 that controls the entire device, a main memory unit 12, a communication unit 13, an operation unit 14, a display panel 15, and an auxiliary memory unit 16.

[0031] The automatic detection device 1 can be configured, for example, as a desktop computer, a notebook personal computer, a tablet, a smartphone, or a computer in a cloud environment that provides computer resources as a service via a computer network such as the Internet.

[0032] The control unit 11 can be configured with a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The control unit 11 may be configured to include a GPU (Graphics Processing Unit).

[0033] The main memory unit 12 is a temporary storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, and temporarily stores data required for the control unit 11 to execute arithmetic processing. The communication unit 13 has a function of communicating with the automatic detection device 1 via the network 17, and can send and receive required information. The operation unit 14 is configured with, for example, a hardware keyboard, a mouse, a touch panel, and the like.

[0034] The display panel 15 can be configured with a liquid crystal panel, an organic EL (Electro Luminescence) display panel, etc. The control unit 11 controls the display panel 15 to display required information.

[0035] The auxiliary memory unit 16 is a large-capacity memory, a hard disk, etc., and stores programs necessary for the control unit 11 to execute processing, as well as a user authentication program 101, a case management program 102, a captured image upload program 103, a machine learning program 110, an automatic detection program 111, an analysis result editing and viewing program 112, an automatic analysis report creation program 113, an analysis report notification program 114, an analysis report editing and viewing program 115, a pattern classification program 133, an image scale adjustment program 134, and a user editing status confirmation program 135.

[0036] 17 is an explanatory diagram of a method for visualizing the inside of an object and a joint according to the present invention, which is an explanatory diagram for determining or adjusting the acquisition range when the object to be measured or observed is an electronic component 4.

[0037] The distance between the X-ray focus 162 and the X-ray detector is defined as the distance 161 from the X-ray focus to the X-ray detector. The distance between the X-ray focus 162 and the electronic component 4 is defined as the distance 160 from the X-ray focus to the sample.

[0038] X-rays are emitted from an X-ray focal point 162. The size of an enlarged projection image 164 projected onto an X-ray detector 163 changes by adjusting or setting a distance 160 from the X-ray focal point, which is the electronic component 4 and the X-ray focal point 162, to the sample. Also, a distance 161 from the X-ray focal point to the X-ray detector is adjusted or set.

[0039] When data is acquired by an X-ray CT 3 or the like, the magnification is the ratio of the size of the enlarged projection image 164 on the detector input surface of the X-ray CT 3 to the size of the sample, and can be calculated by dividing the distance 161 from the X-ray focus 162 to the X-ray detector 163 by the distance 160 from the X-ray focus to the sample. As the distance 160 from the X-ray focus 163 to the sample increases, the magnification decreases and the acquisition range becomes wider.

[0040] By adjusting or changing the size of the enlarged projection image 164 on the X-ray detector, it is possible to obtain many captured images, machine learning data, etc. Furthermore, by rotating, adjusting the tilt, moving the electronic component 4 as a sample left and right, it is possible to obtain many captured images, machine learning data, etc.

[0041] The user authentication program 101, case management program 102, captured image upload program 103, machine learning program 110, automatic detection program 111, analysis result editing and viewing program 112, automatic analysis report creation program 113, analysis report notification program 114, analysis report editing and viewing program 115, pattern classification program 133, image scale adjustment program 134, and user editing status confirmation program 135 stored in the auxiliary memory unit 16 may be provided by a recording medium 18 on which each program is readably recorded.

[0042] The recording medium 18 is, for example, a portable memory such as a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, a micro SD card, or a CompactFlash (registered trademark). Alternatively, the program may be read from the recording medium 18 using a reading device (not shown) and installed in the auxiliary storage unit 16. Next, how images captured by the ultrasonic microscope 2 and the X-ray CT device 3 are stored in the auxiliary storage unit 16 will be described.

[0043] When the ultrasonic microscope 2 or the X-ray CT scanner 3 is connected to the Internet, the user authentication program 101 automatically performs user authentication based on the setting information of the ultrasonic microscope 2 or the X-ray CT scanner 3, and the case management program 102 automatically performs case registration.

[0044] Therefore, images captured by the ultrasonic microscope 2 and the X-ray CT device 3 are uploaded to the auxiliary storage unit 16 by the captured image upload program 103 and stored in the captured image (1) DB.

[0045] On the other hand, if the ultrasonic microscope 2 or the X-ray CT scanner 3 is not connected to the Internet, the images captured by the device are extracted and copied to a computer or the like that is connected to the Internet.

[0046] In the computer, a login operation is performed using a user authentication program 101, and a case is registered using a case management program 102. Next, when an image is uploaded to the auxiliary storage unit 16 using a photographed image upload program 103, the image can be stored in a photographed image (1) DB 104.

[0047] In the auxiliary storage unit 16, the uploaded image data is stored in the photographed image (1) DB 104, and the results of automatically detected voids and cracks are stored in the analysis result (1) DB 105, each for each user, group, and case.

[0048] Analysis reports that record the void rate and crack rate that are automatically measured from the results of automatic detection are saved in the analysis report (1) DB 106 for each user, group, and case.

[0049] Data sets created using the captured image (1) DB 104 and the analysis result (1) DB 105 are stored in the machine learning data (1) DB 107 for each user, group, and case. Only users who have been authenticated by the user authentication program 101 can access each piece of data.

[0050] The auxiliary storage unit 16 stores an automatic detection program 111. This automatic detection program 111 uses a machine learning model (area detection) DB 109 to automatically detect cracks and automatically measure the crack rate for images stored in the captured image (1) DB 104.

[0051] Furthermore, the images stored in the photographed image (1) DB 104 can be processed in batches of a certain amount. Details of this automatic detection program 111 will be described later.

[0052] The results of automatic detection and automatic measurement by the automatic detection program 111 are stored in the analysis result (1) DB 105. The stored analysis result data is read by the analysis result editing and viewing program 112, and users can edit and view the analysis results. In this editing work, users authorized by the user authentication program 101 can edit together.

[0053] The automatic analysis report creation program 113 reads the images stored in the captured image (1) DB 104 and the analysis results stored in the analysis result (1) DB 105, automatically generates numerical data such as void rate and crack rate and text data such as defect causes in accordance with the format specified by the user, compiles it into an analysis report, and saves it in the analysis report (1) DB 106.

[0054] When a newly created or updated analysis report is available, the analysis report notification program 114 notifies the user of the contents via email, message, chat service, etc. The user can then read the analysis report editing and viewing program 115 to edit and view the analysis report. This editing work can be done collaboratively by users authorized by the user authentication program.

[0055] The auxiliary storage unit 16 stores a machine learning program 110. This machine learning program 110 uses a captured image (1) DB 104 and an analysis result (1) DB 105 to create a machine learning data (1) DB, performs machine learning using the data set, and creates a machine learning model (pattern classification) DB 108 and a machine learning model (area detection) DB 109.

[0056] The datasets used for machine learning may be individual databases of captured images and analysis results stored for each user, group, or project, or they may be shared and used.

[0057] The created machine learning model (pattern classification) DB 108 and machine learning model (area detection) DB 109 are used by the pattern classification program 133 and the automatic detection program 111. Therefore, the machine learning program 110 is executed at a different timing (for example, during a time period when the user is not using the system) from the timing at which the pattern classification program 133 and the automatic detection program 111 are executed.

[0058] The pattern classification program 133 is a program that uses the machine learning model (pattern classification) DB 108 to classify electronic components by patterns shown on the vertical axis of FIG. 2, particularly by shape, form, structure, etc. of the electronic components.

[0059] The image scale adjustment program 134 is a program that adjusts the differences in enlargement / reduction, rotation, and position according to the scale shown on the horizontal axis of Fig. 2. The adjustment method may use a scale value 501 written in the image (for example, how many micrometers one pixel corresponds to), a BGA ball diameter 502, or a component length 503, and a specific adjustment method is defined according to the pattern classified by the pattern classification program 133. This image scale adjustment program 134 reduces the variation in the image input to the automatic detection program 111 at the subsequent stage, contributing to improving the accuracy of the automatic detection results.

[0060] The user editing status confirmation program 135 is a program that checks the user's editing status in the analysis result editing and viewing program 112, etc., such as the amount of editing that has been done on the output results of the automatic detection program 111, the viewing time, etc., and determines whether to retain the selection results of the model selection program 504, which will be described later.

[0061] Figure 3 is an explanatory diagram showing a flowchart for automatically detecting voids 36 and cracks 39 in a group of captured images using a machine learning model (pattern classification) DB108 and a machine learning model (area detection) DB109 created by a machine learning program 110.

[0062] First, a group of captured images is input and uploaded using at least one of the ultrasonic microscope 2 and the X-ray CT 3. Here, uploading means, for example, saving the images in the case management program 102. Here, the upload destination is not limited to a cloud server, and may be, for example, a smartphone having the functions of the automatic detection device 1.

[0063] Next, the pattern classification program 133 uses the machine learning model (pattern classification) DB 108 to classify the captured images into patterns according to the shape, form, structure, etc. of the electronic components.

[0064] The captured images classified into each pattern are subjected to image adjustments such as enlargement / reduction, rotation, position, etc. by an image scale adjustment program 134 using a specific adjustment method depending on each pattern.

[0065] Next, for the adjusted captured images, an appropriate machine learning model (area detection) is selected from the machine learning model (area detection) DB 109, and an automatic detection program 111 automatically detects areas of voids 36 and cracks 39 in the images. Details of the automatic detection program 111 will be described later.

[0066] The automatically detected results can be edited and viewed by the user using the analysis result editing and viewing program 112, and the status of editing and viewing is monitored by a user editing status checking program, which will be described in detail later. FIG. 4 is an explanatory diagram showing details of the automatic detection program 111 and the user editing status confirmation program 134. As shown in FIG.

[0067] First, it is checked whether a machine learning model (area detection) exists for the case to which the captured image belongs, for the captured image output by the image scale adjustment program 134. As mentioned above, each captured image is saved for each user, group, or case, and some of these have a machine learning model (area detection) and some do not.

[0068] This is because, as mentioned above, the machine learning program 110 that creates the machine learning model (area detection) is executed at a different time (for example, during a time period when the user is not using the system) from the time when the pattern classification program 133 is executed.

[0069] If the answer to the question in Figure 4, "Does the case to which this image belongs have a machine learning model (area detection)?" is "Yes," that is, if the case to which the captured image belongs has a machine learning model (area detection) (for the sake of explanation, let us call this case case (1)), the area detection prediction program 505 uses the machine learning model (area detection) (1) to automatically detect voids 36 and cracks 39 in the image, and saves the results in the analysis result (mask image) group 22.

[0070] 4, if the answer is "No" to the question "Does the case to which this image belongs have a machine learning model (area detection)?", that is, if the case to which the captured image belongs does not have a machine learning model (area detection) (for the sake of explanation, this will be case (3)), the model selection program 504 selects a machine learning model (area detection) for automatically detecting voids 36 and cracks 39 in the image from the machine learning model (area detection) DB 109. The selection method will be described later.

[0071] As in the previous case, the area detection prediction program 505 automatically detects voids 36 and cracks 39 in the image using the machine learning model (area detection) selected by the user, and saves the results in the analysis result (mask image) group 22.

[0072] Next, the analysis result editing and viewing program 112 reads the results, allowing the user to edit and view them. At this time, the user editing status confirmation program 134 confirms to what extent the results have been edited and viewed.

[0073] If the answer to the question "Have the analysis results been edited to a certain extent?" in Figure 4 is "No," that is, if the user editing status confirmation program 134 determines that the analysis results have not been edited much or that the viewing time has been short, it determines that the user's selection of the machine learning model (area detection) by the aforementioned model selection program 504 was appropriate, and the machine learning model copy program 506 copies a reference to the machine learning model (area detection) selected by the user for its own case (case (3)).

[0074] This copy allows users to use the same machine learning model they previously selected when handling images from their own case (case (3)) in the future, without having to select a model again, improving the convenience of image inspection for users.

[0075] On the other hand, if the answer to the question "Have the analysis results been edited to a certain extent?" in Figure 4 is "Yes," that is, if the user editing status confirmation program 134 determines that the analysis results have been edited frequently or that the viewing time has been long, it determines that the user's selection of the machine learning model (area detection) by the model selection program 504 described above was inappropriate, and does not perform the above-mentioned copying. Therefore, in the future, when handling images captured for one's own project (project (3)), the appropriate machine learning model (area detection) will be selected again.

[0076] Fig. 5 is an explanatory diagram of the operation screen of the model selection program 133. Fig. 5 etc. shows an embodiment in which a region detection result area 137, four region detection result areas 137 (region detection result area 137a, region detection result area 137b, region detection result area 137c, and region detection result area 137d) are provided.

[0077] The region detection result area 137 may be one region in the display region 40, or may be four or more regions in the display region 40. One or more region detection result areas 137 are generated in the display region 40 in accordance with the operation state, the usage status, or the processing state. The display area 40 is displayed on the display panel 15. It is also displayed online or offline on a display device used by the user.

[0078] In the following explanation, void 36 will be used as an example of a pattern to be displayed in the region detection result area 137, but it goes without saying that this is not limited to void 36 and other evaluation patterns such as crack 39 may also be used.

[0079] The present invention is characterized by having a display method that allows a user who does not have special knowledge about machine learning models to easily select the optimal machine learning model using the model selection program 504 of the present invention in the process of selecting the optimal model from the group of candidate machine learning models limited as described above.

[0080] The present invention is characterized by taking an approach that uses a user editing status confirmation program 135 to indirectly obtain the user's evaluation of the prediction results, thereby retaining which machine learning model should be applied when the same type of image is input, thereby improving the convenience of image inspection for the user.

[0081] In the display area 40, the results of area detection by a plurality of machine learning model (area detection) DBs 109 are displayed in an area detection result area 137, divided into four sections. Note that to make it easier to understand the operations on the input device (finger touch input, mouse input, input pad input, etc.), these are illustrated as operations on the input device 140.

[0082] The model selection program 133 displays the detection results of the candidate machine learning models (area detection) in the display area 40. The display area 40 is divided into a plurality of areas, and the detection results are displayed in each divided area.

[0083] It is possible to switch to the captured image display 138b by pressing the left corner of the area detection result area 137b or swiping rightward (in the direction of the arrow) on the area detection result area 137b using an input device operation 140. The animation state at the time of switching is illustrated as switching animation 139.

[0084] Fig. 6 is an explanatory diagram of the operation screen of the model selection program 133. As shown in Fig. 6, while the display is transitioning from the area detection result area 137b to the photographed image display 138b, the difference between the two is emphasized by changing the display using switching animation 139 such as front / back display or page turning display.

[0085] 7 is an explanatory diagram of the operation screen of the model selection program 133. The display can be switched from the captured image display 138b to the region detection result area 137b by pressing the left corner of the captured image display 138b or by swiping leftward on the captured image display 138b.

[0086] Fig. 8 is an explanatory diagram of the operation screen of the model selection program 133. As shown in Fig. 8, it is also possible to add a switching animation effect, as in Fig. 6. As shown in Fig. 8, while the display is transitioning from the captured image display 138b to the region detection result area 137b, by changing it with switching animation 139 such as front / back display or page turning display, an effect of highlighting the difference between the two is exhibited.

[0087] 9 is an explanatory diagram of the operation screen of the model selection program 133. The selected region detection result area 137b can be displayed in the entire display area 40 by pressing near the center of the region detection result area 137b.

[0088] For example, by pressing and holding the center of the region detection result area 137b for one second or more, the region detection result area 137b is displayed in the entire display area 40 by pressing and holding the center of the region detection result area 137b.

[0089] Figure 10 is an explanatory diagram of the operation screen of the model selection program 133. In the embodiment of Figure 9, it is also effective to add a switching animation effect, as in Figure 6. By pressing near the center of the region detection result area 137b, the region detection result area 137b expands as shown by the arrow. The region detection result area 137b expands to fill the entire display area 40, and the expansion of the region detection result area 137b is visually recognized.

[0090] 11 is an explanatory diagram of the operation screen of the model selection program 133. The size of each area in the divided area detection result area 137 (area detection result area 137a, area detection result area 137b, area detection result area 137c, area detection result area 137b) can also be changed by dragging the dividing line.

[0091] As shown in FIG. 12, the intersection of region detection result area 137a, region detection result area 137b, region detection result area 137c, and region detection result area 137b is pressed, and operation 140 on the input device is moved as shown in FIG.

[0092] Fig. 12 is an explanatory diagram of the operation screen of the model selection program 133. As shown in Fig. 12, by moving the position where an operation 140 is performed on the input device, the display areas of the region detection result area 137a, region detection result area 137c, etc. can be changed.

[0093] 13 is an explanatory diagram of the operation screen of the model selection program 133. As with the operation in FIG. 9, the region detection result area 137, which is now displayed singly, can also be switched to a captured image display 138 by pressing the left corner of the region detection result area 137 or swiping right on the region detection result area 137. Selection of the machine learning model (region detection) can be completed by pressing an adopt button (OK) 141.

[0094] The adoption button may be configured to display multiple selection buttons such as "Back" and "Update" in addition to "OK" so that any one or more buttons can be clicked or selected.

[0095] Fig. 14 is an explanatory diagram of the operation screen of the model selection program 133. It is also possible to add a switching animation effect, as in Fig. 6. As shown in Fig. 14, while the display is transitioning from the area detection result area 137 to the captured image display 138, a switching animation 139 such as a front / back display or a page-turning display is used to highlight the difference between the two.

[0096] 15 is an explanatory diagram of the operation screen of the model selection program 133. It is possible to switch to the photographed image display 138 by pressing the left corner of the photographed image display 138 or swiping left on the photographed image display 138. Also, as in FIG. 13, it is possible to complete the selection of the machine learning model (area detection) by pressing the adopt button 141. 16 is an explanatory diagram of the operation screen of the model selection program 133. As in FIG. 6, it is also possible to add a switching animation effect.

[0097] The above embodiments of the present invention have described the detection and visualization device and visualization method for voids 36 inside a material. Patterns of cracks 39 can also be acquired using an X-ray CT scanner 3 or the like. Although images of voids 36 and cracks 39 are different, by learning them as cracks 39, it becomes possible to detect cracks 39. Therefore, it goes without saying that matters described as voids 36 in the embodiments of this specification can be replaced with cracks 39.

[0098] Furthermore, the pattern can be obtained not only by a transmission X-ray image using an X-ray CT device 3 or the like, but also by using, for example, an ultrasonic microscope 2. A gamma ray device can also be used. Therefore, it goes without saying that in the device of the present invention, an X-ray device such as an X-ray CT device may be replaced with an ultrasonic device 2 such as an ultrasonic microscope. Furthermore, by combining an X-ray device such as the X-ray CT device 3 with the ultrasonic microscope 2 to obtain a pattern, the accuracy of detecting voids 36, cracks 39, etc. can be improved.

[0099] The above embodiments are methods and devices for detecting voids 36, cracks 39, etc. in transmitted X-ray images, but the present invention is not limited to these. It goes without saying that the present invention can be used to non-destructively analyze a wide variety of objects, such as air bubbles in adhesive resin, spaces in concrete blocks, metal inclusions in iron parts, and fat particles in the internal organs of living organisms. 18, 19, 20 and 21 are explanatory diagrams of business models that use the automatic detection method and computer program of the present invention.

[0100] 18 is an explanatory diagram of a business model in the first embodiment. Company A provides or sells to customer A, who owns images captured by an ultrasonic microscope 2, an X-ray CT scanner 3, or the like, a program (hereinafter referred to as an "automatic void / crack detection program") that includes a user authentication program 101, a case management program 102, a captured image upload program 103, a machine learning program 110, an automatic detection program 111, an analysis result editing / viewing program 112, an automatic analysis report creation program 113, an analysis report notification program 114, an analysis report editing / viewing program 115, and the like, shown in FIG. 1.

[0101] Customer A will be able to install and use the Boyd Crack Auto-Detection Program on desktop computers, laptops, tablets, and smartphones. Company A will receive license fees for the Boyd Crack Auto-Detection Program from Customer A.

[0102] 19 is an explanatory diagram of a business model in the second embodiment. Company A provides or sells a group of images taken by the ultrasonic microscope 2, X-ray CT scanner 3, etc. owned by Company A, and an automatic void / crack detection program to Customer A, who does not own images taken by the ultrasonic microscope 2, X-ray CT scanner 3, etc.

[0103] Customer A will be able to install and use the void / crack automatic detection program on desktop computers, laptops, tablets, and smartphones. Company A will receive image capture fees and license fees for the void / crack automatic detection program from Customer A.

[0104] 20 is an explanatory diagram of a business model in the third embodiment. Company A runs an automatic void / crack detection program on a cloud server for customer A, who owns images taken with an ultrasonic microscope 2, an X-ray CT scanner 3, etc.

[0105] Customer A can use the Boyd Crack Auto-Detection Program via a web browser on a desktop computer, laptop computer, tablet, or smartphone. Company A receives a usage fee for the Boyd Crack Auto-Detection Program from Customer A.

[0106] 21 is an explanatory diagram of a business model in the fourth embodiment. For customer A who does not own images taken with an ultrasonic microscope 2, an X-ray CT scanner 3, etc., company A prepares a group of images taken with the ultrasonic microscope 2, the X-ray CT scanner 3, etc. owned by company A, and runs an automatic void / crack detection program on a cloud server.

[0107] Customer A can use the void and crack automatic detection program using a web browser on a desktop computer, laptop computer, tablet, or smartphone. Company A receives image capture fees and usage fees for the void and crack automatic detection program from Customer A.

[0108] The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. It goes without saying that the matters or contents described in this specification and drawings can be mutually combined. [Explanation of symbols]

[0109] 1. Automatic void and crack detection device 2. Ultrasonic microscope 3 X-ray CT device 4. Electronic components (measurement samples) 5. Solder 11 Control section 12 Main memory 13 Communications Department 14 Control section 15 Display panel 16 Auxiliary storage 17 Network 18 Recording Media 21 Photographed images 22 Analysis results (mask images) 31 Pattern Classification Training Program 32 Pattern Classification and Prediction Program 36 Void 37 Classification (processing unit) 38 Soldering part 39 Crack 40 display area 41 Area Detection Training Program 42 Area detection and prediction program 101 User Authentication Program 102 Case Management Program 103 Photo Upload Program 104 Photographed Images (1) DB 105 Analysis results (1) DB 106 Analysis Report (1) DB 107 Machine Learning Data (1) DB 108 Machine Learning Model (Pattern Classification) DB 109 Machine Learning Model (Area Detection) DB 110 Machine Learning Programs 111 Auto-detection program 112 Analysis result editing and viewing program 113 Automatic analysis report creation program 114 Analysis Report Notification Program 115 Analysis report editing and viewing program 133 Model Adaptation Program 134 Image Scale Adjustment Program 135 User Edit Status Check Program 137 Area detection results by candidate model 138 Displaying captured images 139 Front and back display switching animation 140 Input Device Operations 141 Recruitment button 160 Distance from X-ray focus to sample 161 Distance from X-ray focus to X-ray detector 162 X-ray focus 163 X-ray detector 164 Enlarged Projection Image 201 Pattern Classification and Prediction Program 202 Area detection and prediction program for pattern A 203 Area detection and prediction program for pattern B 204 Area detection and prediction program for pattern C 301 Photographic Image (AB) DB 302 Pattern (AB) DB 303 Data augmentation preprocessing program 304 Machine Learning Data (AB) DB 305 Machine Learning (Training) Program 306 images 307 Preprocessing Program 308 Machine Learning (Prediction) Program 309 Pattern Classification Results 401 Photographic Image (A) DB 402 Mask Image (A) DB 403 Data augmentation preprocessing program 404 Machine Learning Data (A)DB 405 Machine Learning (Training) Program 406 images 407 Preprocessing Program 408 Machine Learning (Prediction) Program 409 Analysis results (mask images) 501 Scale value written in the image 502 BGA ball diameter 503 Part length 504 Model Selection Program 505 Area detection and prediction program 506 Machine Learning Model Copy Program

Claims

1. an image acquisition device that acquires at least one of a transmission X-ray image, a cross-sectional image, and an ultrasonic image of the joint; a pattern classification program that classifies the acquired image into patterns using a machine learning model; an image adjustment program for adjusting at least one of the position, rotation, and scaling of a visualized object of the image; a detection program for detecting a visualized object from the image; a display device that displays a plurality of images of the visualization object; A visualization device for voids and cracks in a joint, comprising a confirmation program for confirming the image viewing time of the visualization object.

2. an image acquisition device that acquires at least one of a transmission X-ray image, a cross-sectional image, and an ultrasonic image of the joint; a pattern classification program that classifies the acquired image into patterns using a machine learning model; an image adjustment program for adjusting at least one of the position, rotation, and scaling of a visualized object of the image; a detection program for detecting a visualized object from the image; a display device that displays a plurality of images of the visualization object; a selection program for selecting an image of a predetermined visualization object from the displayed images of the plurality of visualization objects; A visualization device for voids and cracks in a joint, comprising a confirmation program for confirming the image viewing time of the visualization object.

3. an image acquisition device that acquires at least one of a transmission X-ray image, a cross-sectional image, and an ultrasonic image of the joint; a pattern classification program that classifies the acquired image into patterns using a machine learning model; an image adjustment program for adjusting at least one of the position, rotation, and scaling of a visualized object of the image; a detection program for detecting a visualized object from the image; a selection program that allows a user to view images of the visualization object and select an image of the visualization object; A visualization device for voids and cracks in a joint, comprising a confirmation program for confirming the image viewing time of the visualization object.

4. an image acquisition device that acquires at least one of a transmission X-ray image, a cross-sectional image, and an ultrasonic image of the joint; a pattern classification program that classifies the acquired image into patterns using a machine learning model; a detection program that detects a first visualization object from the visualization objects in the image; a display device that displays an image of the first visualization object; a selection program that selects a predetermined second visualized object image from the image of the first visualized object displayed on the display device; A visualization device for voids and cracks in a joint, comprising a confirmation program for confirming the image viewing time of the first visualization object and the image viewing time of the second visualization object.

5. the joint is a solder joint, and the visualization object is at least one of a void and a crack, 5. The apparatus for visualizing voids and cracks in a joint according to claim 1, 2, 3 or 4, wherein the pattern is based on the shape, form or structure of the joint.

6. a first operation of classifying at least one of a transmission X-ray image, a cross-sectional image, and an ultrasound image of the joint into a pattern using a machine learning model; a second operation of adjusting at least one of a position, a rotation, and a scale of a visualization object of the image; a fourth operation of selecting a predetermined machine learning model from the plurality of machine learning models based on an image of the visualization object selected by a user; A method for visualizing voids and cracks in a joint, comprising a sixth operation of confirming the image viewing time of the visualization object.

7. a first operation of classifying at least one of a transmission X-ray image, a cross-sectional image, and an ultrasound image of the joint into a pattern using a machine learning model; a second operation of adjusting at least one of a position, a rotation, and a scale of a visualization object of the image; a third operation of displaying images of the visualization objects on a display device and allowing a user to select an image of a predetermined visualization object from the images of the visualization objects; a fourth operation of selecting a predetermined machine learning model from the plurality of machine learning models based on an image of the visualization object selected by a user; A method for visualizing voids and cracks in a joint, comprising a sixth operation of confirming the image viewing time of the visualization object.

8. a first operation of classifying at least one of a transmission X-ray image, a cross-sectional image, and an ultrasound image of the joint into a pattern using a machine learning model; a second operation of adjusting at least one of a position, a rotation, and a scale of a visualization object of the image; a third operation of displaying images of the visualization objects on a display device and allowing a user to select an image of a predetermined visualization object from the images of the visualization objects; a fifth operation of determining whether the machine learning model is appropriate based on the time the user viewed the image of the visualization object, and changing or maintaining the machine learning model; A method for visualizing voids and cracks in a joint, comprising a sixth operation of confirming the image viewing time of the visualization object.

9. 9. The method for visualizing voids and cracks in a joint according to claim 6, 7 or 8, wherein the pattern is based on the shape, form or structure of the joint.

10. 9. The method for visualizing voids and cracks in a joint according to claim 6, claim 7 or claim 8, wherein the joint is a solder joint, and the visualization object is at least one of a void and a crack.

Images