Visualization device for voids and cracks in joints and method for visualizing voids and cracks in joints

Abstract

To solve the problem that a crack cannot be automatically detected in an image processing method by a transmission X-ray image.SOLUTION: An X-ray CT device 53 acquires a first X-ray image of a printed board 89 and an electronic component 54. With a camera 60, positions of a resist 84 and a terminal electrode 58 of the printed board 89 are acquired. After the electronic component 54 is attached to the terminal electrode 58 by a solder 86, a height-range measurement device 62 acquires area and height information data of the solder 86, and the X-ray CT device 53 acquires a second X-ray image of the printed board 89 after the electronic component 54 is mounted. Information data of the solder 86 is extracted from the second X-ray image, the first X-ray image, and the area and height information data of the solder 86, and the information data is subjected to processing by AI learning, and thus, a crack is detected.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an apparatus and method for non-destructively visualizing and analyzing voids, cracks, non-bonded parts, crystal orientation, etc. that occur inside or at joints of objects such as printed circuit boards, electronic components, semiconductor elements, thin film devices, thick film devices, and electronic equipment, using transmitted X-ray images, X-ray CT cross-sectional images, ultrasonic microscope images, etc.

[0002] In particular, the present invention relates to an apparatus for detecting, analyzing and analyzing cracks and voids in soldered parts of mounted parts mounted on ceramic boards, flexible boards, printed boards and the like, and a method for detecting, analyzing and analyzing defective parts. [Background technology]

[0003] Terminals of mounting parts such as electronic components are soldered to terminal electrodes of a substrate (ceramic substrate, flexible substrate, printed substrate, etc.) on which the mounting parts are mounted, and to the mounting electrodes of the mounting parts. Cracks 39, voids 36, etc. occur in solder joints due to preheating and reflow conditions. In addition, electronic components and substrates repeatedly expand and contract due to thermal stress, and the difference in their expansion rates causes cracks 39, voids 36, etc. to occur in joints. In order to improve the yield of electronic circuit boards, it is important to detect cracks 39 and voids 36 in joints and perform quality inspections (quantitative evaluations). Conventionally, methods for non-destructively inspecting solder joints and the like have involved visual inspection of transmitted X-ray images, X-ray CT cross-sectional images, and ultrasonic microscope images.

[0004] Visual inspection can detect slight changes in images, but there is a large variation in pass / fail judgments between inspectors, and inspection locations are limited due to time constraints. It is difficult to accurately detect and measure cracks39 and other defects with visual inspection, and the process takes time.

[0005] In addition, because cracks occur three-dimensionally in solder joints, etc., it is difficult to grasp the cracks in the entire joint by only observing a projection surface or reflection surface in one direction or a few cross sections. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication 2011-169791 Summary of the Invention [Problem to be solved by the invention]

[0007] Cracks 39 and voids 36 that occur in solder joints can be detected by polishing the cross section of the solder joints. However, because polishing the cross section is a destructive test, it cannot be used for mounting boards manufactured on a production line, although it can be used to verify cracks.

[0008] As shown in FIG. 36, in conventional non-destructive testing, an X-ray irradiator 88 irradiates a printed circuit board 89 on which electronic components 54 are mounted with X-rays 61, and the image on an X-ray receiver 83 is analyzed.

[0009] 36, X-rays 61a irradiated from an X-ray irradiator 88a located above an electronic component 54a pass through a lead pin 57 and a solder 86, forming an image of an X-ray receiving section 83. A printed circuit board 89 has many wiring patterns (not shown) formed thereon.

[0010] Therefore, the wiring pattern of the printed circuit board 89 is also reflected as an image in the image formed on the X-ray receiving unit 83. Also, the image on the X-ray receiving unit 83 is generally processed by a binarization image processing method. Therefore, an image is formed on the X-ray receiving unit 83 by irradiating X-rays 61b from an oblique direction as in the case of the X-ray irradiator 88b. However, the X-rays 61b are blocked by the electronic component 54b adjacent to the electronic component 54a, and the X-rays 61b do not pass through the lead pins 57 and the solder 86 of the electronic component 54a. Therefore, the X-ray image of the X-ray receiving unit 83 has problems such as inconsistent images and noise, making it impossible to automatically detect the cracks 39 and the like with high accuracy. [Means for solving the problem]

[0011] The structural analysis method of the present invention for analyzing defective parts inside an object uses an X-ray CT device (X-ray irradiation device) 53 to measure (obtain) and store X-ray CT images (X-ray images) of a printed circuit board 89 (bare board) on which electronic components 54 are not mounted and of the electronic components (mounted components) 54. The camera 60 photographs the terminal electrodes 58 and resist 84 of the printed circuit board 89, and the coordinate positions and areas of the terminal electrodes 58 and resist 84 are obtained and registered in a database.

[0012] After mounting electronic components (mounted components) 54 on a printed circuit board 89 and soldering electrodes 87, lead pins 57, etc. of the electronic components (mounted components) 54 to terminal electrodes 85 with solder 86, an X-ray CT image (X-ray image) of the printed circuit board 89 after mounting the electronic components 54 is measured (obtained) and stored by an X-ray CT device (X-ray irradiation device) 53. In addition, unevenness data and thickness / area data of the soldering 86 are obtained by a height / area measuring device 62.

[0013] In the solder joint 86 obtained by an X-ray CT device (X-ray irradiation device) 53 or the like, a start point 71 and an end point 72 are set or predicted using the finite element method or geometrically, and AI processing is performed by AI learning taking into account the positions of the start point 71 and the end point 72.

[0014] The structural analysis device for analyzing defects inside an object of the present invention obtains a first X-ray image of a printed circuit board 89 and an electronic component 54 by an X-ray CT device 53. The positions of terminal electrodes 58 and resist 84 of the printed circuit board 89 are obtained by a camera 60.

[0015] The camera 60 , the X-ray CT device (X-ray irradiation device) 53 , the laser device 64 , the height / range measuring device 62 , and the rotation device 63 are controlled by the control unit 11 of the automatic crack detection device 51 .

[0016] After electronic component 54 is soldered to terminal electrode 58 with solder 86, height / range measuring device 62 obtains area and height information data of solder 86, and X-ray CT device 53 obtains a second X-ray image of printed circuit board 89 after electronic component 54 is mounted.

[0017] Information data of the solder 86 is extracted from the second X-ray image, the first X-ray image, and the area and height information data of the solder 86 portion, and the information data is processed using AI learning to detect cracks 39, voids 36, etc. Effect of the Invention

[0018] It is possible to automatically detect the cracks 39 and the like with high accuracy. Also, it is possible to introduce the method into a manufacturing line in which electronic components 54 are mounted on a printed circuit board 89. Also, it is possible to detect crack defects at an early stage. [Brief description of the drawings]

[0019] [Figure 1] 1 is a block diagram and an explanatory diagram of a structural analysis device according to the present invention; [Diagram 2] 1 is an explanatory diagram of a structural analysis method according to the present invention; [Diagram 3] FIG. 2 is an explanatory diagram of a printed circuit board and mounted components. [Figure 4] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Diagram 5] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 6] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 7] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 8] 1 is an explanatory diagram of a mounted component and a structural analysis method according to the present invention; [Figure 9] FIG. 1 is an explanatory diagram of a graph showing a cycle test and connection strength. [Figure 10] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 11] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 12] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 13] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 14] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 15] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 16] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 17] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 18] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 19] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 20] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 21] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 22] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Diagram 23] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 24] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Diagram 25] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 26] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 27] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 28] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Figure 29] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Diagram 30] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Diagram 31] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Diagram 32] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Diagram 33] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Diagram 34] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Diagram 35] 1 is an explanatory diagram of a structural analysis device and a structural analysis method according to the present invention; [Diagram 36] FIG. 1 is an explanatory diagram of a conventional structural analysis method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The present invention will be described below with reference to the drawings showing the embodiments. In the embodiments described in the specification, for ease of understanding, as shown in FIG. 5 and the like, a solder joint between a lead pin 57 of an electronic component (mounted component) 54 and a terminal electrode 58 of a printed circuit board 89 will be described as an example, but the present invention is not limited to this. The mounting electrode 87 of the mounted component (electronic component) 54 and the terminal electrode of the printed circuit board 89 may be soldered 86. Furthermore, the present invention is not limited to solder 86, and may be connected, for example, with a conductive paste or the like.

[0021] The joints, connections, or adhesives are not limited to solder 86, but may be conductive paste, conductive adhesive, organic adhesive, ACF (anisotropic conductive film), conductive rubber, thermocompression bonding, or the like.

[0022] In this embodiment, a printed circuit board 89 is taken as an example for explanation, but the base material is not limited to the printed circuit board 89, and may be a flexible board, a flexible film, a ceramic board, an inorganic material board, a plastic board, a glass board, a liquid crystal film, a resin film, a conductive board, etc. Also, the base material is not limited to a board, and may be a sheet or a film.

[0023] Although the description will be given assuming that electronic components 54 are mounted on the printed circuit board 89, the mounting members are not limited to electronic components. For example, any member to be mounted may be used, such as a metal processed chip, a resin chip, a metal component, or a ceramic component.

[0024] The connection with the electronic components is exemplified by solder 86, but is not limited to this. Conductive paste, conductive adhesive, organic adhesive, adhesive, fusing agent, rubber, resin, varnish, paint, etc. may also be used. The technical concept of the present invention can be applied to many object joints, contacts, object interiors, and object structures, such as electronic components (mounted components), buildings, electrical equipment, and the inside of the human body. Although the present invention uses cracks as an example, it can also be applied to the observation, understanding, detection, etc. of voids, material densities, mixed states of different materials, disjointed parts, crystal orientation, etc.

[0025] The present invention will be described using an X-ray CT (Computed Tomography) device as an example, but the present invention is not limited to this, and it goes without saying that a two-dimensional X-ray device, an X-ray fluoroscopy observation device, an ultrasonic microscope device, an X-ray irradiator, etc. may also be used.

[0026] In each drawing for explaining the embodiment of the invention, elements having the same function are given the same reference numerals, and the description may be omitted. In addition, the embodiments of the present invention may be combined with each other. In addition, in order to facilitate understanding and to facilitate illustration, the drawings may be enlarged, reduced, or omitted.

[0027] Fig. 1 is a block diagram and an explanatory diagram of a structural analysis device of the present invention. As shown in Fig. 1, the present invention is equipped with a height / range measuring device 62 such as a 3D macroscope or a laser microscope that measures unevenness, height, and range, a camera 60, and an X-ray CT device 53. In addition, a rotation device 63 that rotates a printed circuit board 89 is also equipped. Images taken or acquired by the height / range measuring device 62, camera 60, and X-ray CT device 53 are used together or in combination to carry out AI processing, etc.

[0028] The present invention uses images or data taken or acquired by a height / range measuring device 62, a camera 60, and an X-ray CT scanner 53 for photographing the surface condition, but for the outside of the object, an optical image may be obtained by photographing the observation point from the outside using an optical microscope or the like, and the optical image may be combined with the image or data acquired by the X-ray CT scanner 53. Examples of the height / range measuring device 62 include a device capable of measuring, measuring and acquiring unevenness and height, such as a 3D macroscope device or a laser microscope.

[0029] The present invention is not limited to X-ray CT devices. The technical scope of the present invention also includes the use and control of radiation devices that use radiation such as gamma rays, α (alpha) rays, β (beta) rays, γ (gamma) rays, microwaves, infrared rays, visible light, and ultraviolet rays.

[0030] It goes without saying that the device and method of the present invention are not limited to X-rays, but can also be applied to images obtained by irradiating or receiving gamma rays, α (alpha) rays, β (beta) rays, γ (gamma) rays, microwaves, infrared rays, visible light, or ultraviolet rays. The camera 60 is not limited to measuring images or image data in the visible range, but may also measure images or image data in the ultraviolet range or infrared range.

[0031] Examples of the height / range measuring device 62 include a 3D macroscope and a laser microscope. A 3D macroscope is a device that irradiates a sample with fringe projection light and measures the distortion of the fringe projection image by photographing it. It can measure a wide range in a short time.

[0032] The 3D macroscope of the present invention is equipped with one camera and two light sources. The camera photographs the object from directly above, while shining striped light on it from diagonally above. The striped pattern becomes distorted where the height changes, and the height is measured by triangulation based on this distortion. Height data is added to the image captured by the camera, and an image of the 3D shape is output. The 3D image can be obtained in a short time.

[0033] In the present invention, by combining the image or image data of the 3D macroscope 62 with the X-ray CT device 53 and by implementing the AI ​​processing of the present invention, it is possible to obtain highly accurate elevation and range data.

[0034] By using the image data from the camera 60 in combination, the 3D macroscope 62 can obtain the range and area of ​​the solder 86 with high accuracy. It can also obtain the surface condition of the solder 86. The 3D macroscope 62 mainly obtains information on the height and unevenness of the solder 86, and the camera 60 obtains the area and range of the solder 86. From the information from the 3D macroscope 62 and the camera 60, 3D shape information of the solder 86 can be obtained.

[0035] It is also effective to use infrared light as the light source of the 3D macroscope 62. Infrared light is absorbed by the solder 86, but is less absorbed by the electrode 58. By detecting or measuring the difference in absorption rate, the area, height, and range of the solder 86 can be obtained with high precision.

[0036] When a portion of solder 86 to be observed or analyzed is subjected to structural analysis or analysis using X-rays, the solder portion is irradiated with ultraviolet rays or laser light 65 and heated or warmed. Heating or warming makes it easier to observe and analyze cracks 39 and voids 36. When laser light 65 is used, the laser light 65 of a laser microscope can be used, and when infrared light is used, light from a 3D macroscope 62 can be used. It goes without saying that a separate laser device 64 for heating or an infrared device can also be used.

[0037] The 3D macroscope 62 of the present invention has a depth stacking function that focuses on the screen of the entire substrate 89. In addition, telecentric lenses are installed on both the light projecting side and the light receiving side to improve measurement accuracy. The working distance is 75 mm.

[0038] The structural analysis apparatus and method of the present invention can be used not only to inspect joints such as solder joints, but also to inspect the surfaces of syringes and catheters for irregularities (scratches), and to check medicine packaging for tears or foreign objects.

[0039] If the height / range measuring device 62 is a laser microscope, it irradiates the solder 86 and the lead pins 57 with ultraviolet or infrared light, and measures the unevenness of the sample from the reflected light. Small areas can be measured with high accuracy. The height / range measuring device 62 may be configured using both a 3D microscope and a laser microscope.

[0040] The camera 60 is exemplified by a camera that has a magnifying function and captures images using visible light. However, it is not limited to visible light. When detecting the position of the resist 84, a filter is provided to match the color of the resist. For example, if the resist is colored green, a filter that transmits green light is provided. When capturing an image with the camera, a separate illuminator (not shown) is used to irradiate the object with telecentric light, capture the object, and obtain the image and image data.

[0041] The height / range measuring device 62 may be configured with a camera and an LCD display. When measuring surface roughness, if the surface of the target part is a mirror, the light is reflected specularly, and if it is rough, the light is scattered. Therefore, the light source is moved, and light is applied from various angles to measure the scattered light. The mechanism for moving this light source is configured with an LCD display. Rather than illuminating the LCD display uniformly, an image is presented as if the light source were moved.

[0042] The light source position is moved so that the mean square error is minimized, and the image is captured repeatedly. Measurements can be taken quickly because there is no need to mechanically move the light-emitting (display) position of the LCD. In addition, because the movement of the light source can be changed each time to minimize the error, there is no need to align the measurement target with the light source. The automatic detection device 51 includes a control unit 11 that controls the entire device, a main memory unit 12, a communication unit 13, an operation unit 14, a display panel 15, and an auxiliary memory unit 16.

[0043] The automatic detection device 51 can be configured, for example, as a desktop computer, a notebook personal computer, a tablet, a smartphone, or a computer in a cloud environment that provides computer resources in the form of a service via a computer network such as the Internet.

[0044] The control unit 11 controls the automatic crack detection device 51 and the rotation device 63. It also controls the main memory unit 12, the auxiliary memory unit 16, the communication unit 13, the operation unit 14, the display panel 15, and the like.

[0045] The control unit 11 can be configured with a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The control unit 11 may be configured to include a GPU (Graphics Processing Unit).

[0046] The main memory unit 12 is a temporary storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 has a function of communicating with the automatic detection device 51 via the network 17, and can transmit and receive required information. The operation unit 14 is composed of, for example, a hardware keyboard, a mouse, a touch panel, and the like.

[0047] The display panel 15 can be configured with a liquid crystal panel, an organic EL (Electro Luminescence) display panel, etc. The control unit 11 performs control for displaying required information on the display panel 15.

[0048] The auxiliary memory unit 16 is a large-capacity memory, a hard disk, etc., and stores programs necessary for the control unit 11 to execute processing, as well as a user authentication program 101, a case management program 102, a captured image upload program 103, a machine learning program 110, an automatic detection program 111, an analysis result editing and viewing program 112, an automatic analysis report creation program 113, an analysis report notification program 114, and an analysis report editing and viewing program 115.

[0049] The user authentication program 101, case management program 102, captured image upload program 103, machine learning program 110, automatic detection program 111, analysis result editing and viewing program 112, automatic analysis report creation program 113, analysis report notification program 114, and analysis report editing and viewing program 115 stored in the auxiliary memory unit 16 may be provided by a recording medium 18 on which each program is recorded in a readable manner.

[0050] The recording medium 18 is, for example, a portable memory such as a Universal Serial Bus (USB) memory, a Secure Digital (SD) card, a micro SD card, or a Compact Flash (registered trademark).

[0051] Each program recorded on the recording medium 18 is provided by communication via the communication unit 13. Alternatively, the program may be read from the recording medium 18 using a reading device (not shown) and installed in the auxiliary storage unit 16. Next, how images captured or acquired by the camera 60 or the X-ray CT scanner 53 are stored in the auxiliary storage unit 16 will be described. The data format for saving the crack 39 and void 36 detection results is exemplified by image data formats (JPEG, PNG, GIF, TIFF, etc.).

[0052] The format for storing information such as the measured crack length, position, and crack width is JSON, which is a data description method, or a labeled image PNG format in which each pixel of the image is labeled (0, 1, 2, ...).

[0053] When the camera 60 and the X-ray CT scanner 53 are connected to the Internet, user authentication is performed by a user authentication program 101 based on the setting information of the camera 60 and the X-ray CT scanner 53, and case registration is performed by a case management program 102. It uses a cloud service for user authentication, allowing quick and easy user sign-up, sign-in and access control.

[0054] Security measures include multi-factor authentication and encryption of stored and transmitted data, and it complies with PCI DSS, SOC, ISO / IEC 27001, ISO / IEC 27017, ISO / IEC 27018, and ISO 9001.

[0055] Images captured by the camera 60 and the X-ray CT scanner 53 are uploaded to the auxiliary storage unit 16 by the captured image upload program 103 and stored in a captured image (1) DB (database).

[0056] On the other hand, if the camera 60 and the X-ray CT scanner 53 are not connected to the Internet, the images captured by the devices are extracted and copied to a computer or the like connected to the Internet.

[0057] C-scan data from an ultrasonic microscope and slice data from an X-ray CT scanner are stored as image data and used. Image formats include JPG, PNG, GIF, TIFF, WebP, and SVG.

[0058] In the computer, a login operation is performed by a user authentication program 101, and a case is registered by a case management program 102. Next, an image is uploaded to the auxiliary storage unit 16 by a captured image upload program 103, and can be stored in a captured image (1) DB (database) 104.

[0059] In the auxiliary storage unit 16, the uploaded image data is stored in the photographed image (1) DB 104, and the results of automatically detected cracks are stored in the analysis result (1) DB 105, respectively, for each component, user, group, or case.

[0060] Analysis reports indicating the void rate and crack rate automatically measured from the automatic detection results are stored in the analysis report (1) DB 106 for each component, user, group, and case. The analysis report containing the automatically measured void rate and crack rate is stored in Excel, CSV, and PDF formats.

[0061] A data set created using the captured image (1) DB104 and the analysis result (1) DB105 is stored in the machine learning data (1) DB107 for each component, user, group, or case.

[0062] Only users who have been authenticated by the user authentication program 101 can access each data. The authentication method is an account and a password.

[0063] The auxiliary storage unit 16 stores an automatic detection program 111. This automatic detection program 111 uses a machine learning model (pattern classification) DB 108, a machine learning model (area detection) DB 109, and a machine learning model (object detection) DB 116. In addition, automatic crack detection and automatic crack rate measurement are performed on images stored in the captured image (1) DB 104. The images stored in the photographed image (1) DB 104 can be processed in batches of a certain amount. The details of this automatic detection program 111 will be described later.

[0064] The results of automatic detection and automatic measurement by the automatic detection program 111 are stored in the analysis result (1) DB 105. The stored analysis result data is read by the analysis result editing and viewing program 112, and the user can edit and view the analysis results. In this editing work, users authorized by the user authentication program 101 can edit together.

[0065] The automatic analysis report creation program 113 reads images stored in the captured image (1) DB 104 and analysis results in the analysis result (1) DB 105, and automatically generates numerical data such as void rate and crack rate, and text data such as the cause of defects, based on a format specified by the user.

[0066] The results are compiled as an analysis report and stored in the analysis report (1) DB 106. The software of the present application automatically measures the void rate and the crack rate and automatically generates a report.

[0067] When there is a newly created or updated analysis report, the analysis report notification program 114 notifies the user of the contents via email, message, chat service, or the like.

[0068] A user can edit and view the analysis report by reading the analysis report editing and viewing program 115. In this editing operation, users authorized by the user authentication program can edit together.

[0069] The auxiliary storage unit 16 stores a machine learning program 110. The machine learning program 110 uses the captured image (1) DB 104 and the analysis result (1) DB 105 to create a machine learning data (1) DB.

[0070] Machine learning is performed using the created dataset to create a machine learning model (pattern classification) DB108, a machine learning model (area detection) DB109, and a machine learning model (object detection) DB116.

[0071] The data sets used for machine learning may be the captured image DB and analysis result DB stored for each user, group, or case, which may be used individually, or they may be shared and used.

[0072] By sharing a workspace, for example, user A can edit the measurement results and user B can check them. It also enables division of labor. It also makes it possible to work on a single project at multiple locations.

[0073] The created machine learning model (pattern classification) DB 108, machine learning model (area detection) DB 109, and machine learning model (object detection) DB 116 are used in the automatic detection program 111 described above.

[0074] The machine learning program 110 is executed at a timing different from that at which the automatic detection program 111 is executed (for example, during a time period when the user is not using the system). For each database (DB), the timing for updating the AI ​​model used during automatic detection and the timing for using the DB are mutually exclusive.

[0075] 19 illustrates details of the automatic detection program 111. First, the captured image group 21 uploaded from the camera 60, the height / range measuring device 62, and the X-ray CT scanner 53 is read (saved) into the pattern classification prediction program 201.

[0076] The pattern classification prediction program 201 classifies the patterns of an image. Examples of patterns to be classified include mounted electronic components 54 such as chip capacitors, resistors, coils, etc. Examples include cases where the pin shapes of IC packages are different, such as lead pin types (FIG. 11, etc.) and leadless types (FIGS. 14, 18, etc.). Examples include cases where the lead pin shapes and the shapes of the terminal electrodes to be joined are different. Other examples include cases where the types of substrates are different, such as surface mount and through hole, and cases where the materials of electronic components are different, such as copper, brass, iron, etc. Also, examples include cases where different imaging methods are used, such as an optical microscope, an electron microscope, etc. In order to improve the accuracy of automatic crack detection in the subsequent stage, the parts are classified.

[0077] The captured image group 21 is classified, for example, into crack images such as pattern A, pattern B, and pattern C by the pattern classification prediction program 201. Also, the starting point 71, the end point 72, etc. are classified.

[0078] Different processing methods are applied to the crack images for Pattern A, Pattern B, and Pattern C. For example, different processing methods are applied depending on the pin shape of the IC package and the mounting pattern of the lead part.

[0079] The image group of pattern A is read into a pattern A area detection and prediction program 202. The image group of pattern B is read into a pattern B area detection and prediction program 203.

[0080] The image group of pattern C is read into the area detection and prediction program 204 for pattern C, and then each program automatically detects areas such as cracks 39. These analysis result images are stored as an analysis result (mask image) group 22.

[0081] The present invention focuses on the fact that there is an optimal area detection method for each pattern, and prepares an area detection program for each pattern. It is also used for the shape and pattern of electronic components 54. Compared to when it is not prepared for each pattern, it is possible to improve the accuracy of area detection. A machine learning model (area detection) must be prepared for each pattern. FIG. 20 illustrates the process flow of the training program and the prediction program in the above-mentioned pattern classification.

[0082] The purpose of the training program for pattern classification is to create a machine learning model (pattern classification) DB 108. For example, if there are two types of patterns, pattern A and pattern B, in a group of captured images, a machine learning model capable of classifying a given captured image as pattern A or pattern B is created using a captured image (A) DB 301 and a pattern (A) DB 302 that can refer to whether each captured image is pattern A or pattern B.

[0083] The detection performance is improved by classifying the input image as an image of a lead mounting pattern (pattern A) or an image of a BGA mounting pattern (pattern B) and then performing area detection using a machine learning model (area detection) suitable for each pattern in the latter stage. Also, the image of the camera 60, image data, 3D macroscope image, and image data are used.

[0084] In one embodiment, the X-ray CT device 53 acquires sample images by the following method or the like. The sample temperature is set to a predetermined temperature, and an X-ray image of the sample is taken. The predetermined temperature is set to multiple temperatures, and multiple images are taken. The AI ​​is trained using the multiple obtained X-ray images. By using various images under various conditions for AI training, for example, data on strength versus number of cycles as explained in FIG. 9 is trained. The AI ​​becomes able to find crack characteristics that are independent of various conditions.

[0085] In one embodiment, a heat shock test is performed on a sample to produce a sample with cracks. The temperature conditions of the sample, the position conditions of the soldered terminals, the solder thickness conditions, and the solder type are changed. An X-ray image is taken, and then the temperature is raised to take an X-ray image after a heat shock (after the cracks have progressed further), and the X-ray images before and after the temperature application are combined and used for AI learning. For AI learning, it is desirable to prepare image data of the crack progress under various conditions (crack rate 0% to 100%).

[0086] FIG. 9 is a graph showing the heat shock test (Number of cycles N) and the adhesive (bonding) strength (Strength / N) of the solder part. The adhesive (bonding) strength (Strength / N) decreases as the number of heat cycles N increases. At 1500 cycles, the adhesive (bonding) strength (Strength / N) is about 13, and at 2500 cycles, the adhesive (bonding) strength (Strength / N) is about 12. The adhesive (bonding) strength (Strength / N) decreases as the number of heat cycles increases. The decrease in adhesive (bonding) strength (Strength / N) is due to the occurrence of cracks 39 and the like.

[0087] AI learning is performed taking into account the number of cycles in the heat shock test (Number of cycles N). The number of heat cycles affects the crack propagation (crack rate 0% to 100%) and causes differences in X-ray image data, so this data is used for learning. By utilizing the information data in Figure 9, it is possible to predict, for example, the crack occurrence position, crack line length, and crack area at 1500 cycles and 3000 cycles.

[0088] In one embodiment, a sample that is prone to cracking is prepared, ultrasonic images, X-ray images, etc. are taken of the sample, and the crack position and range (crack line length, crack area, etc.) are detected by AI. After that, it is effective to confirm the crack position, range, etc. by polishing the cross section, etc., and have the AI ​​relearn. In addition, if the cracks are not continuous, it is also effective to manually specify them as if they were continuous using a pen tool, etc., and have the AI ​​learn that they are continuous. The AI ​​will be trained using these multiple ultrasound images and X-ray images to build a database. The above is implemented in each embodiment of the present invention.

[0089] In an X-ray CT, an ultrasonic microscope, or the like, images are acquired by changing the temperature, ultrasonic frequency, etc., and the present invention is implemented for each image, such as AI (Artificial Intelligence) processing.

[0090] The tilted CT scanning method irradiates the X-rays from an oblique direction while rotating the object to be observed horizontally, so it is possible to obtain clear, high-magnification transmitted images even for plate-shaped objects. From the perspective of AI learning and AI crack detection performance, it is preferable to obtain clearer images.

[0091] 2, in the present invention, the mounting member 54 (FIG. 2(a)) is rotated using a rotation device 63. Also, the rotation device 63 is used to rotate a printed circuit board 89 (FIG. 2(b)).

[0092] The observation object is fixed to a θ table so that the angle with respect to the light direction can be changed, and the θ table is controlled to change the irradiation direction of radiation such as X-rays or ultrasound. Alternatively, the irradiation unit of radiation such as X-rays or the generation unit of ultrasound is mounted on the θ table or the like and rotated.

[0093] It is configured so that image data of the observation point can be acquired in real time in the θ direction. The image (image data) during rotational movement in the θ direction is configured so that the number of pixels is reduced and visualization can be performed quickly. When observing with the θ direction fixed, the number of pixels of the image (image data) is increased. The system is trained using X-ray images and a database is constructed. The above-mentioned items are implemented in each embodiment of the present invention.

[0094] The X-ray microscope (X-ray CT device) 53 is a microscope for non-destructively observing the inside of an object. X-rays transmitted through a sample are converted into light or the like, and are magnified or reduced by optical lenses or the like.

[0095] X-rays have the property of penetrating materials, and some of them are absorbed as they pass through a sample. The rate of absorption increases as the density of the material increases (the atomic number increases) and the thickness increases, resulting in a decrease in the intensity of the penetrating X-rays. If a crack 39 or the like occurs in the object, the X-ray transmittance of the crack 39 or the like increases, so that the crack 39 or the like is displayed in the pattern 40 .

[0096] 2(a), 3D data of a mounting member 54 to be mounted on a printed circuit board 89 or the like is measured by an X-ray CT device 53. The mounting member (electronic component) 54 is a component to be loaded (mounted) on an electronic circuit board (printed circuit board 89) to be manufactured. In the present invention, the mounting member (electronic component) 54 is rotated by a rotation device 63.

[0097] The X-ray 3D data of the mounting member (electronic component) 54 is stored in a captured image (1) database (DB) 104 together with data on the mounting direction on the printed circuit board 89, mounting height, and other data.

[0098] By subtracting the X-ray 3D data of the mounting member (electronic component) 54 from the X-ray image of the printed circuit board 89 on which the electronic component 54 is mounted, information data on the printed circuit board 89 and the solder 86 can be obtained. By subtracting the X-ray 3D data of the printed circuit board 89, information data on the solder 86 can be obtained. Since the X-ray image of the printed circuit board 89 on which the electronic component 54 is mounted contains X-ray image data of the electronic component 54 and the printed circuit board 89, it is difficult for human vision to judge, but by using information obtained through learning, it is possible to subtract and remove the electronic component 54 and the printed circuit board 89 from the X-ray image of the printed circuit board 89 on which the electronic component 54 is mounted.

[0099] An electronic board is manufactured by mounting mounting members (electronic components) 54 on a printed circuit board 89. If cracks 39 occur in the soldered portion 86 during the manufacture of the electronic board, a product defect occurs. Therefore, it is effective to detect cracks 39 during manufacture and detect electronic boards with poor reliability.

[0100] The 3D data of electronic component 54 to be mounted on printed circuit board 89 stored in the database is extracted by subtracting the data of electronic component 54 from data photographed by an X-ray CT device while mounted on printed circuit board 89, and also by subtracting the X-ray data of printed circuit board 89, to extract images and image data of solder 86 of terminal electrode 58. Therefore, the wiring pattern of printed circuit board 89 and the like are deleted from the image data.

[0101] In reality, wiring patterns and electronic component data cannot be clearly differentiated. Image data can be trained to become clear through AI learning. This is an area or processing where AI learning processing excels.

[0102] 2(b) is an explanatory diagram of a method for acquiring X-ray CT data of a printed circuit board 89. The printed circuit board 89 is a bare board with no mounted components 54. The terminal electrodes 58 are exposed on the surface of the printed circuit board 89, and a resist 84 is printed on the surface.

[0103] The mounted components 54 may be not only components or components mounted on the printed circuit board 89 or the like, but also components, members, components, etc. configured by a 3D printer or the like. Therefore, it is not limited to mounting, but includes things that are formed, bonded, or placed.

[0104] The printed circuit board 89 is usually a multi-layer wiring board. In the multi-layer wiring board, wiring patterns are formed on the front surface, back surface, and inner layers. When photographed with X-rays, the wiring patterns on the front surface, back surface, and inner layers are photographed as images.

[0105] Cracks 39 occurring in solder 86 of electronic component 54 occur in the joint (solder 86) between electronic component 54 and terminal electrode 58 mounted on the surface of printed circuit board 89. Therefore, the internal wiring pattern of printed circuit board 89 is not necessary for the occurrence and analysis of cracks 39.

[0106] Although the state of the joints (solder 86) differs for each printed circuit board to be manufactured, the wiring pattern of printed circuit board 89 is basically the same. Therefore, the data required for the wiring pattern of printed circuit board 89 is the data for terminal electrodes 58 that connect with lead pins 57 of electronic component 54, and no internal wiring pattern is required.

[0107] Images and image data of the position and state of the terminal electrode 58 and resist 84 can be acquired by a camera 60. Images and image data of the height and range of the solder 86 on the terminal electrode 58 can be acquired by a height / range measuring device 62. Images and image data of the internal state of the solder 86 can be acquired by an X-ray CT device (X-ray imaging device) 53. Analysis of the joints from each image and image data can be performed by AI processing, and cracks 39, voids 36, etc. can be extracted and detected.

[0108] As shown in Fig. 2(b), the printed circuit board 89 is attached to the rotating device 63 and rotates while an X-ray image of the printed circuit board 89 is taken (observed) by the X-ray CT device 53. Based on the X-ray transmission information of 360° (DEG.) for the observed printed circuit board 89, calculation processing is performed by a computer to construct three-dimensional data of the sample. By narrowing the rotation pitch and obtaining more information, it is possible to obtain highly accurate three-dimensional data (3D image data).

[0109] The image data of the printed circuit board 89 is recorded in the photographed image (1) DB 104. The image data of the printed circuit board 89 includes data of the terminal electrodes 58, the internal wiring pattern, and the back electrodes of the printed circuit board 89.

[0110] If the design data (CAD data, Gerber data) of the printed circuit board 89 is available, it is not necessary to obtain the X-ray data described in Fig. 2(b). Alternatively, X-ray data can be obtained as a supplement, and the design data (CAD data, Gerber data) can be used as the main data.

[0111] Fig. 3 is an explanatory diagram of a printed circuit board 89. As shown in Fig. 3(b), electronic components 54 are mounted on the printed circuit board 89. As shown in Fig. 3(a), X-ray data is acquired in a state where no electronic components 54 are mounted. In addition, the camera 60 acquires the positions and sizes of the terminal electrodes 58. In addition, the position and size of the resist 84 are acquired.

[0112] 3(b) is a schematic diagram showing a state in which electronic component 54 is mounted on printed circuit board 89. Although not shown in the figure, solder 86 is placed or applied to terminal electrode 58, and mounting electrode 87, lead pin 57 and terminal electrode 58 are electrically connected by solder 86.

[0113] Fig. 4 is a schematic diagram showing a state in which an electronic component 54 is mounted on a printed circuit board 89. When an electronic component 54 is mounted, as shown in Fig. 36, an X-ray 61 is blocked by the adjacently mounted electronic component 54, and a complete X-ray image of the electronic component 54 cannot be obtained.

[0114] In the present invention, the X-ray CT images of the electronic components and the X-ray CT images of the printed circuit board 89 are stored in a database. Therefore, the image of the electronic components 54 can be subtracted from the image of the printed circuit board on which the electronic components 54 are mounted by image processing using AI processing. 4 is an explanatory diagram of the structural analysis device and the structural analysis method of the present invention. Printed circuit boards 89 on which electronic components 54 are mounted are manufactured in sequence on a manufacturing line.

[0115] For ease of explanation, it is assumed that the printed circuit board 89 on which the electronic components 54 are mounted is manufactured on a manufacturing line, but this is not limited to this. It goes without saying that the printed circuit board 89 may be manufactured, formed, or produced offline.

[0116] An X-ray image of a printed circuit board 89 on which electronic components 54 are mounted is obtained. In the embodiment of Fig. 4, the printed circuit board 89 is fixed, and the X-ray CT device is changed at angles from 0 to θ and rotated to capture the X-ray CT image. The X-ray image is captured at an angle of θ. The X-ray CT image 53 device irradiates X-rays 61 , and an X-ray image is formed or acquired on the X-ray receiving section 83 .

[0117] When the printed circuit board 89 is small (small area), it is possible to capture a 360-degree X-ray CT image, but when the area of ​​the printed circuit board 89 is large, it is not possible to capture a 360-degree X-ray CT image. Also, it is difficult to capture a 360-degree X-ray CT image of the printed circuit board 89 on which electronic components 54 are mounted on the production line.

[0118] The present invention acquires X-ray images (X-ray image data) at tilt angles ranging from 0 degrees to θ on the printed circuit board 89. A tilt of 0 degrees (°) is when X-rays are irradiated from a direction perpendicular to the printed circuit board 89. A tilt of θ degrees (°) is when X-rays are irradiated from an angle of θ with respect to the vertical axis on the printed circuit board 89.

[0119] The smaller the angle θ, the smaller the shadow of the adjacent electronic component 54. The larger the angle θ (approaching 180°), the more image data of the states of the mounting electrodes 87, lead pins 57, and solder 86 of the electronic component 54 can be acquired.

[0120] In the present invention, oblique X-ray data is acquired mainly in the range of angles from 0° to θ°. From the oblique X-ray data, X-ray images and X-ray image data of the solder 86, such as the mounting electrode 87 and the lead pin 57, are acquired. Although the number of pieces of data in these X-ray images and X-ray image data is smaller than that of a 360° X-ray CT image, by carrying out AI learning, it is possible to reproduce the states of the mounting electrode 87, the lead pin 57, and the solder 86, and it is possible to detect and evaluate cracks 39 occurring in the solder 86.

[0121] 4, the X-ray CT device (X-ray irradiation device) 53 is moved at angles from 0 to θ to acquire an X-ray image of the printed circuit board 89, but the present invention is not limited to this. As shown in FIG. 6, the printed circuit board 89 may be attached to a rotation device 63, the X-ray CT device (X-ray irradiation device) 53 may be fixed in the vertical direction (θ=0°), and the printed circuit board 89 may be rotated at angles from 0 to θ.

[0122] Fig. 5 is an explanatory diagram of a method of obtaining X-ray image data from the rear surface of a printed circuit board 89 in the same manner as the operation described in Fig. 4. As shown in Fig. 5, an X-ray image and X-ray image data are obtained from the rear surface of a printed circuit board 89.

[0123] By using the X-ray image and X-ray image data from the front side of the printed circuit board 89 in Figure 4 and the X-ray image and X-ray image data from the back side of the printed circuit board 89 in Figure 5 and by having AI learn, it is possible to accurately reproduce the conditions of the mounting electrode 87, lead pin 57, and solder 86, and to detect and evaluate cracks 39 that have occurred in the solder 86.

[0124] Fig. 7(a) is an explanatory diagram for explaining observation and data acquisition of a printed circuit board 89 on which no electronic components 54 are mounted. Fig. 7(b) is an explanatory diagram for explaining observation and data acquisition of a printed circuit board 89 on which electronic components 54 are mounted.

[0125] As shown in Fig. 7, X-ray image data and X-ray images are observed and acquired in an inclination angle range of 0° to θ° by an X-ray CT device (X-ray irradiation device) 61. θ is 90° (DEG.) or less, preferably 45° or less, and more preferably 30° or less. θ is preferably set to be 5° (DEG.) or more and 30° or less. The printed circuit board 89 is mounted and positioned on the XY stage 66. Also, the rotation device 63 is configured so that the angle θ can be set and changed. 7A, a printed circuit board 89 is observed and an image is acquired in a state where no electronic component 54 is mounted (unmounted). Image data is also acquired.

[0126] The camera 60 measures the position coordinates and area data of the resist 84 when no electronic components are mounted and no solder 86 is applied, and stores the data in a database. Also, the XY stage 66 is moved as necessary, and the position coordinates and area data of the resist 84 are measured, and the data is stored in a database. Also, the position coordinates and area data of the terminal electrodes 58 are measured, and the data is stored in a database.

[0127] Height / range measuring device 62 measures the position coordinates and area data of resist 84 when no electronic components are mounted and no solder 86 is applied, and stores the data in a database. Also, it moves XY stage 66 as necessary, and in synchronization with the movement of XY stage 66 and in response to the moved position, measures the position coordinates and area data of resist 84 when no electronic components are mounted and no solder 86 is applied, and stores the data in a database. Also, it measures the position coordinates and area data of terminal electrodes 58, and stores the data in a database.

[0128] The camera 60 measures the position coordinates and area data of the resist 84 when no electronic components are mounted and no solder 86 is present, and stores the data in a database. Also, the XY stage 66 is moved as necessary, and the position coordinates and area data of the resist 84 are measured, and the data is stored in a database. Also, the position coordinates and area data of the terminal electrodes 58 are measured, and the data is stored in a database.

[0129] The height / range measuring device 62 measures the position coordinates and area data of the resist 84 when no electronic components are mounted and no solder 86 is applied, and stores the data in a database. Also, the XY stage 66 is moved as necessary, and in synchronization with the movement of the XY stage 66, the position coordinates and area data of the resist 84 when no electronic components are mounted and no solder 86 is applied are measured in accordance with the movement position of the XY stage 66, and stores the data in a database. Also, the position coordinates and area data of the terminal electrodes 58 are measured, and stores the data in a database.

[0130] X-ray CT device 53 measures X-ray images and X-ray data of printed circuit board 89 and terminal electrodes 58 in a state where electronic components are not mounted and solder 86 is not applied, and stores the data in a database. Also, XY stage 66 is moved as necessary, and X-ray images and X-ray data of printed circuit board 89 and terminal electrodes 58 in a state where electronic components are not mounted and solder 86 is not applied are measured in synchronization with the movement of XY stage 66, and stores the data in a database. Control unit 11 controls rotation device 63 to change θ, and measures X-ray data corresponding to θ.

[0131] As shown in FIG. 7, image data is acquired and learned by a height / range measuring device 62, such as a 3D macroscope or laser microscope, which measures unevenness, height, and range, and a camera 60.

[0132] AI processing and the like is performed on the images taken or acquired by the height / range measuring device 62, the camera 60, and the X-ray CT device 53, alternately or in combination. The X-ray image data is subjected to a learning process which will be described with reference to Figs. 24 to 35 etc. The present invention uses images or data taken or acquired by a height / range measuring device 62, a camera 60, and an X-ray CT device 53 for photographing the surface condition.

[0133] The present invention includes a laser device 64 as required. Examples of the laser device 64 include a YAG laser device, a carbon dioxide (CO2) laser device, and a neon-helium laser device. The laser device 64 emits laser light 65. The laser light 65 heats or warms the mounting electrode 87, the lead pin 57, and the solder 86. The heating or warming causes a change in the crack 39 generated in the solder 86, and the change is detected and evaluated by the X-ray CT device 53. The irradiation of the laser light 65 makes it easier to detect the crack 39 and voids 36, and also improves the detection accuracy.

[0134] It is also effective to measure the temperature and temperature distribution of the solder 86 with a thermal viewer while irradiating the laser light 65. From the temperature and temperature distribution, cracks 39 and voids 36 inside the solder 86 can be detected. The surface condition of the solder 86 is observed by a height / range measuring device 62 and a camera 60, and image data is obtained.

[0135] The camera 60 is not limited to one that measures images or image data in the visible range. It may be one that measures images or image data in the ultraviolet range or infrared range. Alternatively, it may be one that measures or acquires light of a specific color wavelength, such as green.

[0136] The height / range measuring device 62 is exemplified by a 3D macroscope and a laser microscope. The 3D macroscope irradiates a sample with fringe projection light. By irradiating the sample with the fringe projection light, the distortion of the fringe projection image is photographed. By measuring the distortion, the height, surface condition, and range of the mounting electrode 87, the lead pin 57, and the solder 86 can be measured or obtained.

[0137] A 3D macroscope is equipped with one camera and two light sources. The camera takes a photo of the object from directly above, while shining striped light on it from diagonally above. The striped pattern becomes distorted in places where the height changes, and the height is measured by triangulation based on this distortion.

[0138] Height data is added to the image captured by the camera to output an image of the 3D shape. By differentiating the wavelengths of the light emitted from the two light sources, interference occurs, making it possible to output an image of the 3D shape with greater accuracy.

[0139] When the height / range measuring device 62 is a laser microscope, a laser beam 65 is irradiated onto the object to measure or acquire data on the height, surface condition, and range of the object. It is also effective to measure the temperature and temperature distribution of the solder 86 with a thermal viewer.

[0140] The present invention can obtain high-precision elevation and range data by combining images or image data from a 3D macroscope 62 or the like with an X-ray CT scanner 53 and implementing the AI ​​processing and AI learning of the present invention.

[0141] The height / range measuring device 62, by using the image data from the camera 60 in combination, can obtain the range and area of ​​the solder 86 with high accuracy. Also, the surface condition of the solder 86 can be obtained.

[0142] It is also effective to use infrared light as the light source of the 3D macroscope serving as the height / range measuring device 62. Infrared light is absorbed by the solder 86, and is less absorbed by the electrode 58 and the resist 84. By detecting or measuring the difference in absorption rate, the area, surface condition, height difference, and formation range of the solder 86 can be obtained with high precision.

[0143] When performing structural analysis or analysis of the solder 86 to be observed or analyzed by X-rays, it is effective to heat or warm the solder part by irradiating it with laser light 65 from a laser device 64. By heating or warming with the laser light 65, it becomes easier to observe and analyze the cracks 39 and voids 36.

[0144] The laser light 65 may be that of a laser microscope, or in the case of using infrared light, may be that from a 3D macroscope 62. As shown in FIG. 7(b), it goes without saying that a separate laser device 64 for heating may also be used.

[0145] 7(a), the position coordinates and area data of resist 84 and the position coordinates and area data of terminal electrode 58 are measured when no electronic components are mounted and no solder 86 is present. Furthermore, height data such as unevenness of terminal electrode 58 and resist 84, and surface condition data are stored in a database. X-ray images and X-ray data of printed circuit board 89 and terminal electrode 58 are measured, and the data is stored in a database.

[0146] 7(a), as shown in FIG. 7(b), electronic component 54 is mounted, and mounting electrodes 87 of electronic component 54 and terminal electrodes 58 are soldered (mounted) with solder 86. Electronic component 54 is mounted, and lead pins 57 and terminal electrodes 58 are soldered with solder 86.

[0147] The solder 86 may be applied using a reflow device, or may be applied by irradiating the applied portion with a laser beam 65 from a laser device 64 to melt the solder 86. During soldering, it is effective to measure X-ray images and X-ray data of the solder 86 at the soldering portion using an X-ray CT device 53. Information on voids and cracks occurring within the solder 86 can be obtained. Information (data) on the crystal orientation (the direction of the arrangement of the crystal planes (perpendicular to the planes, horizontal)) occurring within the solder 86 can also be obtained.

[0148] It is also effective to irradiate the terminal electrodes 58 of the electronic components 54 with laser light 65 from a laser device 64 to sequentially melt the paste solder 86, and obtain X-ray image data before and after melting with an X-ray CT device (X-ray irradiator) 53. Cracks 39 and voids 36 occurring in the solder 86 can be detected with high accuracy. 7(b), the printed circuit board 89 is observed with the electronic components 54 mounted thereon, and image data is also acquired.

[0149] As shown in Fig. 7(b), X-ray image data and X-ray images are observed and acquired in an inclination angle range of θ° by an X-ray CT device (X-ray irradiation device) 61. θ is 90° (DEG.) or less, and preferably 45° or less. More preferably, 30° or less. θ is preferably set to be 5° (DEG.) or more and 30° or less.

[0150] θ is set taking into consideration the state in which X-rays 61 from X-ray CT device 53 are blocked by adjacent electronic components 54. If electronic components 54 are tall, θ is made small. If electronic components 54 such as chip capacitors and chip resistors have a small thickness (back), θ is made large. It is preferable to make θ large and obtain more X-ray CT data and X-ray images, since this allows the internal states of more solder 86 to be obtained.

[0151] The printed circuit board 89 is mounted and positioned on the XY stage 66. Also, the angle θ can be set and changed by the rotation device 63 as necessary. By tilting the printed circuit board 89 (θ), more X-ray CT data and X-ray images of the inside of the solder 86 can be obtained, which is preferable. In FIG. 7B, the printed circuit board 89 is observed with the electronic components 54 mounted and soldered, and an image, or image data, is obtained.

[0152] The camera 60 photographs the solder 86, the lead pins 57, the mounting electrodes 87, and the terminal electrodes 58 after the electronic components are mounted and soldered, and measures their position coordinates and area data. The measured coordinates, data, etc. are stored in a database.

[0153] If necessary, the XY stage 66 is moved to photograph the solder 86, the lead pin 57, the mounting electrode 87, and the terminal electrode 58, and their position coordinates and area data are measured. The measured coordinates, data, etc. are stored in a database.

[0154] The height / range measuring device 62 measures the position coordinates, area data, and unevenness of the solder 86 and resist 84 after the electronic component is mounted and the solder 86 is applied, and stores the data in a database. It also measures the unevenness and height data of the surfaces of the solder 86, lead pins 57, mounting electrodes 87, and terminal electrodes 58, and stores the data in a database 104.

[0155] Furthermore, XY stage 66 is moved as necessary, and in synchronization with the movement, the unevenness and height data of the surfaces of solder 86, lead pins 57, mounting electrodes 87, and terminal electrodes 58 in a state where the electronic components are mounted and solder 86 is applied are measured in accordance with the movement position, and the data is stored in database 104. The data in database 104 is analyzed, and the analysis results (1) are stored in DB 105. The X-ray CT device 53 measures and acquires X-ray data and X-ray CT data of the solder 86 in a state in which the electronic components are mounted and soldered.

[0156] The X-ray data and the X-ray CT data also include information data about the printed circuit board 89, and the data about the printed circuit board 89 is obtained as the X-ray data and the X-ray CT data of the printed circuit board 89, as described with reference to Figures 2(b) and 7(a).

[0157] The acquired data is stored in the photographed image (1) database 104. Therefore, by subtracting the X-ray data, X-ray CT data, etc. of the printed circuit board 89 stored in the photographed image (1) database 104 from the X-ray data, X-ray CT data, etc. acquired in FIG. 7(b), data other than the printed circuit board 89 can be extracted.

[0158] 2(a) is obtained as data on the electronic component 54, such as X-ray data and X-ray CT data. Therefore, by subtracting the X-ray data, X-ray CT data, etc. of the electronic component 54 stored in the captured image (1) database 104 from the X-ray data, X-ray CT data, etc. obtained in FIG. 7(b), data other than the electronic component 54 can be extracted.

[0159] By subtracting the X-ray data and X-ray CT data of the electronic component 54 and the printed circuit board 89 stored in the captured image (1) database 104 from the X-ray data and X-ray CT data of Figure 7 (b), the X-ray data and X-ray CT data of the solder 86 can be extracted.

[0160] Also, in Figures 7(a) and 7(b), it is preferable to irradiate X-rays 61 onto a printed circuit board 89, etc., using an X-ray CT device (X-ray irradiation device) 53, and obtain X-ray data and X-ray CT data of lead pins 57, mounting electrodes 87, and terminal electrodes 58 of electronic components 54, etc.

[0161] By using these data and the results of AI processing and AI learning processing of the X-ray data, X-ray CT data, etc. stored in the captured image (1) database 104, it is possible to extract and analyze the internal state of the solder 86 part of the electronic component 54, etc.

[0162] In FIG. 7(b), it is preferable to irradiate laser light 65 from a laser device 64 and obtain X-ray data or X-ray CT data or obtain a rate of change at each time point during irradiation of the laser light 65.

[0163] The camera 60 photographs the solder 86, lead pins 57, mounting electrodes 87, and terminal electrodes 58 in a state in which the electronic component is mounted and soldered, and measures their position coordinates and area data.

[0164] The measured coordinates, data, etc. are stored in the photographed image (1) database 104. Also, it is measured whether there is any difference between the state (position, area) of the resist 84 in Fig. 7(a) and the state (position, area) of the resist 84 in Fig. 7(b). These data are stored and saved in the database.

[0165] If necessary, the XY stage 66 is moved at an angle θ to measure whether any difference occurs in the states (positions, areas) of the terminal electrodes 58, the mounting electrodes 87, and the resist 84.

[0166] By subtracting the X-ray data and X-ray CT data of the electronic component 54 and the printed circuit board 89 stored in the captured image (1) database 104 from the X-ray data and X-ray CT data of Figure 7 (b), the X-ray data and X-ray CT data of the solder 86 can be extracted.

[0167] The height / range measuring device 62 measures the position coordinates, area and height of the solder 86 portion after the electronic component is mounted and the solder 86 is applied, and stores the data in a database. The measurements are performed at the portion where the solder 86 is formed, such as the terminal of each electronic component 54. The height / range measuring device 62 acquires unevenness (height) data of the solder 86 portion.

[0168] The height / range measuring device 62 mainly measures the unevenness (height) of the solder 86. In addition, position data of the solder 86 (position information with respect to the mounting electrode 87 and position information with respect to the lead pin 57) is obtained. Information on cracks 39 exposed on the surface of the solder 86 is obtained.

[0169] The camera 60 acquires the area of ​​the solder 86 when the electronic component is mounted, the area of ​​the terminal electrode 58 without the solder 86, the solder 86, and position data of the solder 86 (position information with respect to the mounting electrode 87, and position information with respect to the lead pin 57). Also, information on cracks 39 exposed on the surface of the solder 86 is acquired. The X-ray CT device (X-ray irradiation device) 53 mainly obtains data related to the internal state of the solder 86. Information on the cracks 39 and voids 36 inside the solder 86 is obtained.

[0170] Prior to carrying out the process shown in Fig. 7(b), the height / range measuring device 62, the camera 60, and the X-ray CT device (X-ray irradiation device) 53 acquire information data on the printed circuit board 89 and the electronic components 54. This information is processed by the AI ​​functions and AI learning described in Figs. 19 to 35, etc. The above items have been described in Figs. 1, 2, 3, 7(a), etc., so a detailed description will be omitted.

[0171] 7(b), in a printed circuit board 89 on which electronic components 54 are mounted, the height / range measuring device 62, the camera 60, and the X-ray CT device (X-ray irradiation device) 53 acquire information data of the printed circuit board 89 and the electronic components 54. In addition, this information is processed by the AI ​​function and AI learning described in Figs. 19 to 35, etc.

[0172] By subtracting the information data obtained in Figures 1, 2, 3, 7(a), etc. from the information data obtained in Figure 7(b), etc., and by using the AI ​​functions and AI learning described in Figures 19 to 35, etc., information data such as the shape and internal state of solder 86 can be extracted and the information data can be obtained.

[0173] The camera 60 and height / range measuring device 62 obtain three-dimensional shape information of the solder 86, information on the attachment / connection state of the solder 86 and mounting electrode 87, information on the attachment / connection state of the solder 86 and terminal electrode 58, information on the attachment / connection state of the solder 86 and lead pin 57, and information on the attachment / connection state of the solder 86 and terminal electrode 58. The X-ray CT device (X-ray irradiation device) 53 mainly obtains data related to the internal state of the solder 86. Information on the cracks 39 and voids 36 inside the solder 86 is obtained. In the present invention, the solder 86 and the like are subjected to image processing by AI using the AI ​​function and AI learning described in Figs. 19 to 35 and the like.

[0174] By subtracting the X-ray CT data of the electronic component 54 and the X-ray CT data of the printed circuit board 89 from the (inclined) X-ray CT data of the printed circuit board 89 on which the electronic component 54 is mounted, X-ray data of the mounting portion (solder 86 attachment portion) of the electronic component (mounted component) 54 is extracted.

[0175] The surface state of the solder 86 of the electrodes (mounting electrodes 87, terminal electrodes 58, lead pins 57) of the mounted components is shown in FIG. 6 by photographing the positions of the electrodes (mounting electrodes 87, terminal electrodes 58) of the printed circuit board, the lead pins 57, the positions of the resist 84 of the printed circuit board 89 (relative electrode positions), the mounted component positions, The height / range measuring device 62 acquires information data such as unevenness of the surfaces of the solder 86, the lead pins 57, the mounting electrodes 87, and the terminal electrodes 58, height data, and the like. From the X-ray CT data of the mounted component 54 acquired by the X-ray CT, information data on the internal state of the solder 86 is obtained. Using the above information data, AI functions, and AI learning, the cross-section, surface, and internal condition of 86 solder parts can be analyzed.

[0176] Fig. 8 is an explanatory diagram of an electronic component 54. Fig. 8(a) is an explanatory diagram of an electronic component such as a chip capacitor or chip resistor that does not have lead pins 57. Fig. 8(b) is an explanatory diagram of an electronic component such as an IC that has lead pins 57.

[0177] As shown and explained in FIG. 2(a), X-ray CT information of the electronic component 54 is acquired by the X-ray CT device (X-ray irradiation device) 53, and as shown and explained in FIG. 2(b), X-ray CT information of the printed circuit board 89 is acquired and the information is stored in the captured image (1) DB 104.

[0178] Information data such as the position, range, surface condition, height, and unevenness of the resist 84, mounting electrode 87, terminal electrode 58, lead pin 57, and solder 86 can be obtained by measurements using a height / range measuring device 62 and a camera 60. Internal information data of the solder 86 can be obtained by an X-ray CT device (X-ray irradiation device) 53.

[0179] In Fig. 7(b), information data is acquired in the state of Fig. 8 in which electronic components 54 and the like are mounted. Information data of electronic components 54 and printed circuit board 89 is stored in photographed image (1) database (DB) 104. This information data is differentiated from the information data obtained in the state of Fig. 8. The information data obtained in the state of Fig. 8 is complex, but by having AI learn, it is possible to distinguish and differentiate the information data of electronic components 54 and printed circuit board 89.

[0180] The position, area, and range of the resist 84 printed on the printed circuit board 89 define the area and position of the terminal electrode 58 to be soldered. By measuring the position of the resist 84 and the position of the terminal electrode 58 (openings of the resist 84, etc.) with the camera 60, the position and area of ​​the terminal electrode 58 to which solder is to be attached can be determined.

[0181] 8(a), information data relating to the coordinates, positions, and areas of the mounting electrodes 87, solder 86, and terminal electrodes 58 is obtained by the camera 60. In FIG. 8(b), information data relating to the coordinates, positions, and areas of the lead pins 57, solder 86, and terminal electrodes 58 is obtained.

[0182] The height / range measuring device 62 can measure the unevenness and height data of the solder 86. The X-ray CT device (X-ray irradiation device) 53 can measure internal information of the solder 86.

[0183] The above information data and AI learning function allow the analysis of the solder 86. The location and state of crack 39 should preferably be determined by taking into account information from a heat shock test, as described in FIG. 9 and other figures. The location and state of crack 39 change during a heat shock test. By having the AI ​​learn these changes, it is possible to analyze crack occurrence with greater accuracy. In addition, the crack occurrence rate and location for a 3000-cycle heat cycle test can be predicted using a 1500-cycle heat cycle test.

[0184] For example, by having the AI ​​learn the crack area, crack line length, crack occurrence rate, and crack occurrence position after 1,000 cycles of heat cycle tests, 2,000 cycles of heat cycle tests, and 3,000 cycles of heat cycle tests, the number of heat cycles at the time of sample measurement can be estimated from the crack area, crack line length, crack occurrence rate, and crack occurrence position.

[0185] In addition, from the crack area, crack line length, crack occurrence rate, and crack occurrence position measured or detected after a specified number of heat cycles (e.g., 1,000 cycles), the crack area, crack line length, crack occurrence rate, and crack occurrence position after 3,000 heat cycles can be predicted.

[0186] As shown in Figure 9, there is a relationship between the heat shock test (Number of cycles N) and the adhesive (bonding) strength (Strength / N) of the solder part. The greater the number of heat cycles N, the lower the adhesive (bonding) strength (Strength / N). The greater the number of heat cycles, the lower the adhesive (bonding) strength (Strength / N). The decrease in adhesive (bonding) strength (Strength / N) is caused by the occurrence of cracks 39, etc. The occurrence of cracks 39, etc. is detected and evaluated by the crack line length, crack area, and crack occurrence position.

[0187] AI learning is performed taking into account the number of cycles in the heat shock test (Number of cycles N). The number of heat cycles affects the degree of crack progression (crack rate 0% to 100%).

[0188] The crack 39 is detected and evaluated by its crack line length, crack area, and crack occurrence position. The crack line length, crack area, and crack occurrence position are detected and analyzed as differences in the X-ray image data. This information data is used for learning.

[0189] The AI ​​learning method involves creating samples that are prone to cracking, taking and measuring ultrasound images, X-ray images, etc. of the samples, and determining the crack line length, crack area, and crack position.

[0190] After that, it is effective to confirm the crack line length, crack area, crack position, range, etc. by polishing and observing the cross section, and then have the AI ​​re-learn. Also, if the cracks are not continuous, it is effective to manually specify them as if they were continuous with a pen tool, etc., and have the AI ​​learn that they are continuous. The AI ​​will be trained using these multiple ultrasound images and X-ray images to build a database. The above is implemented in each embodiment of the present invention.

[0191] As shown in FIG. 7, image data is acquired and learned by a height / range measuring device 62, such as a 3D macroscope or laser microscope, which measures unevenness, height, and range, and a camera 60.

[0192] Images taken or acquired by the height / range measuring device 62, the camera 60, and the X-ray CT device 53 are mutually, alternately, or in combination to carry out AI processing, etc. The present invention uses images or data taken or acquired by a height / range measuring device 62, a camera 60, and an X-ray CT device 53 for photographing the surface condition.

[0193] The present invention can obtain high-precision elevation and range data by combining images or image data from a 3D macroscope 62 or the like with an X-ray CT scanner 53 and implementing the AI ​​processing and AI learning of the present invention.

[0194] The height / range measuring device 62, by using the image data from the camera 60 in combination, can obtain the range and area of ​​the solder 86 with high accuracy. Also, the surface condition of the solder 86 can be obtained.

[0195] It is also effective to use infrared light as the light source of the 3D macroscope serving as the height / range measuring device 62. Infrared light is absorbed by the solder 86, and is less absorbed by the electrode 58 and the resist 84. By detecting or measuring the difference in absorption rate, the area, surface condition, height difference, and formation range of the solder 86 can be obtained with high precision. In this specification and the drawings, as an example, an internal analysis of solder 86 will be described. FIG. 10 relates to a structure analysis device for solder 86 and a structure analysis method for solder 86.

[0196] The state of the solder 86 surface of the electrodes (mounting electrodes 87, terminal electrodes 58, lead pins 57) of the mounted components 54 is measured and acquired by photographing the positions of the electrodes (mounting electrodes 87, terminal electrodes 58) of the printed circuit board, the lead pins 57, the positions of the resist 84 of the printed circuit board 89 (relative electrode positions), and mounted component position information. The height / range measuring device 62 acquires information data such as the unevenness of the surfaces of the solder 86, lead pins 57, mounting electrodes 87, and terminal electrodes 58, and height data.

[0197] 10 and 11, the physical area, position, and height (thickness) of the solder 86 formed on the terminal electrode 58 are determined (known). In addition, the connection position and connection state between the solder 86 and the mounting electrode 87, and the connection position and connection state between the solder 86 and the terminal electrode 58 are determined (known). In addition, the surface state of the solder 86 (unevenness information, etc.) can also be obtained.

[0198] 10 and other figures, a start point 71 and an end point 72 are determined or estimated using the finite element method and geometric methods for internal analysis of solder 86. Using the start point 71 and end point 72, AI learning and AI processing are performed to analyze and measure the crack line length, crack area, crack position, etc. An X-ray CT device (X-ray irradiation device) 53 acquires the internal state of solder 86.

[0199] When using the finite element method, the solder 86 to be analyzed is regularly subdivided as shown in the schematic diagram of Figure 10. It is specified what kind of force is being applied to the solder 86 and how the solder 86 component is connected, bonded or fixed to the mounting electrode 87, lead pin 57 and terminal electrode 58. In the finite element method, the object or space to be analyzed is divided into elements, and the elements are connected to each other at nodes.

[0200] There are as many equations as there are elements, but because the crystal orientation of the terminal electrode 58 and mounting electrode 87 within the solder 86 is specific, the boundary conditions can often be kept relatively simple. The more divisions there are, the more equations that need to be solved, and the longer the analysis time. Elements are made small for parts that are likely to have a large impact, and large for other elements. The key to the analysis is which parts to make the mesh finer (small elements, more divisions) and which parts to make the mesh coarser (large elements, smaller divisions).

[0201] In the case of Fig. 10, the mesh is divided into a larger number of parts near the black circle that is likely to be the starting point 71. The elements are also made smaller. The part where the solder 86 comes into contact with the outside air and is the shortest distance from the starting point 71 is assumed to be the end point 72, and the mesh is divided into a larger number of parts and the elements are also made smaller.

[0202] By using the finite element method, the positions of the start point 71 and the end point 72 can be estimated with high accuracy. Changes in the shape of the solder 86 are measured by the camera 60 and the height / range measuring device 62. The finite element method is carried out using these measurement data. The positions of the start point 71 and the end point 72 can be estimated with high accuracy by the finite element method. In addition, by performing AI processing using AI learning, the position where the crack 39 occurs can be detected.

[0203] 10 and other examples, the finite element method is used, but the present invention is not limited to this. In many cases, the shortest position from the start point 71 to the surface position of the solder 86 becomes the end point 72. Therefore, the position of the end point 72 can be obtained geometrically.

[0204] The camera 60 measures and acquires the positions of the electrodes (mounting electrodes 87, terminal electrodes 58) of the printed circuit board, the positions of the lead pins 57 and the resist 84 of the printed circuit board 89 (relative electrode positions), and mounted component position information. The height / range measuring device 62 acquires information data such as unevenness and height data on the surfaces of the solder 86, the lead pins 57, the mounting electrodes 87, and the terminal electrodes 58. The X-ray CT device (X-ray irradiation device) 53 measures, analyzes, and acquires the internal state of the solder 86.

[0205] Using these information data and the finite element method, or using these information data and geometric processing, information such as deformation, stress, and strain of the solder 86 is obtained, and the start point 71 and end point 72 are identified. Using the information data, the start point 71, the end point 72, and AI processing through AI learning, the crack line length, crack area, crack position, etc. are analyzed and identified.

[0206] The starting point 71 is often an end of the mounting electrode 76 that contacts the solder 86, or an end of the lead pin 57 that contacts the solder 86. The end point 72 is often an end point 72a or 72b that is located on the surface where the starting point 71 and the solder 86 contact the air and that is located at the shortest distance. It goes without saying that as the AI's learning deepens, the end point 72 may be assumed geometrically.

[0207] FIG. 10 shows one cross section in a schematic manner. For ease of understanding, a start point 71 and an end point 72 in one cross section are shown. However, in actual processing, as will be described in FIG. 22, this is performed in a large number of interfaces. The shape and area of ​​the cross section differ depending on the cross section position. The shape and area of ​​the cross section are known by the camera 60 and the height / range measuring device 62. Therefore, the shape and area of ​​the cross section are also known in each cross section in FIG. 22, and the start point 71 and end point 72 in each cross section can be predicted and set geometrically using the finite element method.

[0208] As shown in Fig. 22, data processing etc. is performed at intervals t parallel to the long side direction of lead pin 57. Data processing etc. may also be performed at intervals s parallel to the short side direction. Data processing etc. may also be performed at both intervals t and intervals s. Data processing may also be performed on a cross section in an oblique direction. Data processing may also be performed on a cross section in a vertical direction. The start point 71 and end point 72 for each cross section are determined or estimated using the finite element method or geometrically, and processing is performed using AI learning, which will be described later.

[0209] Information data such as the positions of the electrodes (mounting electrodes 87, terminal electrodes 58) of the printed circuit board, the positions of the lead pins 57 and the resist 84 of the printed circuit board 89 (relative electrode positions), the unevenness and height data of the surfaces of the solder 86, the lead pins 57, the mounting electrodes 87 and the terminal electrodes 58, are images and information data relating to the shape of the surface of the solder 86.

[0210] The X-ray image is basically a two-dimensional image of the cross section of the solder 86. The cross-sectional shape of the two-dimensional image is known from the surface data information of the solder 86 obtained by the camera 60 and the height / range measuring device 62. A large number of cracks 39 etc. occur, and the generated cracks 39 etc. are distributed three-dimensionally. The acquired X-ray image etc. may have a pattern 40 in which the cracks 39 etc. overlap.

[0211] The camera 60 is a microscope that non-destructively observes defects such as peeling, cracks, foreign bodies, etc. inside the object. When ultrasonic waves propagate through different materials, some of them are reflected and some are transmitted.

[0212] If the object being observed has, for example, a peeled off area (void), a strong reflected wave will be detected. By scanning the object in a plane and marking the position where this reflected wave is detected, it is possible to obtain a two-dimensional distribution of the peeled off area (void).

[0213] For each cross section, a start point 71 and an end point 72 are set, and are also assumed or estimated. In addition, the marked position is stored, and the image of the marked position is used as a reference to acquire images before and after the marked position, and AI processing and feature detection processing for cracks, etc. are performed. It is preferable to acquire images (image data) by changing the distance from the marked position as necessary.

[0214] Image information (measurement information) from the height / range measuring device 62, image information from the camera 60, X-ray CT images 53, etc. are combined to perform AI processing and feature detection processing for cracks, etc. Learning efficiency and detection performance are improved by using two or more pieces of information (multimodal) for the same object, such as camera images and X-ray CT images. In addition, the finite element method is used to geometrically predict and process the start point 71 and end point 72, and the combined results are used to perform AI processing and feature detection processing for cracks, etc.

[0215] The structural analysis device (automatic crack detection (visualization and measurement) device) 51 of the present invention has the function of observing, inspecting, judging, determining, etc. the position and occurrence state of cracks automatically or through specified operations, and automatically detecting, visualizing, measuring, etc. cracks.

[0216] Hereinafter, the automatic crack detection (visualization and measurement) device 51 may be referred to as an automatic detection device or a visualization and measurement device. For ease of explanation, it will be described as an automatic detection device 51, but this does not mean that the entire device is limited to being automatic. It goes without saying that it may include a device in which some or most of the processing is performed manually, or manual work is performed, or separation work is performed, etc.

[0217] In the present invention, cross-sectional and internal images are acquired by an imaging method using an X-ray CT device 53 or the like. Training data is created separately from the automatic processing and used for learning. In addition, the crack rate is automatically measured from one image or multiple images. In addition, processing is applied to all slice images or multiple slice images to visualize the three-dimensional distribution of cracks.

[0218] The X-ray CT device 53 of the present invention can obtain the internal structure of an object by performing reconstruction processing using the differences in "ease of penetration" and "ease of absorption" that occur when X-rays pass through the object.

[0219] It is preferable to change or set the t interval and the s interval by determining or judging "ease of transmission" and "ease of absorption." It is also preferable to change or set θ by the rotation device 63.

[0220] The t interval and the s interval correspond to the slice image pitch and the resolution (spatial resolution) obtained from the X-ray CT device 53, etc. In order to increase the resolution of the obtained image, the focal length during shooting of the X-ray CT device 53, etc., the measurement field of view, the distance to the object, etc. have an effect.

[0221] Fig. 12 is a schematic diagram showing the positional relationship between the lead pin 57, the terminal electrode 58, and the solder 86. Fig. 12(a) is a diagram showing the lead pin 57 as viewed from above, and Fig. 12(b) is a diagram showing the lead pin 57 as viewed from a cross-sectional direction.

[0222] The structural analysis device of the present invention mainly analyzes the solder 86 portion indicated by the hatched area in FIG. 12, and analyzes, analyzes or predicts the positions of cracks and voids in the solder 86.

[0223] In Figures 11(a) and 11(b), black circles indicate locations where cracks are likely to occur depending on the component and mounting method. Solder cracks occur in the solder due to differences in the expansion rates of boards and electronic components that repeatedly expand and contract due to thermal stress.

[0224] Fig. 11(a) shows a case where the alignment between the lead pin 57 and the terminal electrode 58 is good. Fig. 11(b) shows a case where the alignment between the lead pin 57 and the terminal electrode 58 is misaligned. The position of the starting point 71 is not significantly different between Fig. 11(a) and Fig. 11(b), but the position of the end point 72 is different between Fig. 11(a) and Fig. 11(b).

[0225] 11(a) and 11(b), maximum strain occurs at the end of lead pin 57, and cracks are likely to occur at the location marked with a black circle. The black circle is starting point 71 described in FIG. 10. In the case of FIG. 11, it is below the tip of lead pin 57, and is the contact point between lead pin 57 and solder 86.

[0226] The position assumed as end point 72 is the position where solder 86 faces air (outside air). In the case of Fig. 11(a), end point 72b is the position of a perpendicular line from start point 71 to the surface of solder 86. End point 72a is end point 72a, which is a position along lead pin 57. When start point 71 is used as a reference, end point 72a is the shortest distance to the position where solder 86 comes into contact with outside air.

[0227] The end point 72 and the start point 71 can be found or assumed (estimated) by the finite element method. In the case of Fig. 11(a), when the start point 71 is used as the reference point, it is the geometrically shortest position. Therefore, it is possible to find or assume (estimate) geometrically at each cross section.

[0228] 11(b), the terminal electrode 58 and the lead pin 57 are misaligned from their normal positions. The occurrence of the misalignment can be detected by an image captured by the camera 60. In addition, the thickness, position, etc. of the solder 86 can be detected by the height / range detection device 62. 11B, the starting point 71 is also below the tip of the lead pin 57, and is the contact point between the lead pin 57 and the solder 86.

[0229] 11(b), end point 72b is a position of a perpendicular line from start point 71 to the surface of solder 86. End point 72a is a position of solder 86 facing air (outside air), and end point 72b is a position of a perpendicular line from start point 71 to the surface of solder 86. Cracks are likely to occur at the locations marked with white and black circles. The white and black circles are the starting points 71 and the ending points 72 shown in FIG. The structural analysis device of the present invention mainly analyzes the solder 86 portion, and detects, analyzes, predicts, or predicts the positions of cracks and voids in the solder 86.

[0230] 13 shows the case of a chip capacitor, a chip resistor, etc. The chip capacitor (electronic component 54) has a mounting electrode 87. Solder 86 is provided between the mounting electrode 87 and the terminal electrode 58.

[0231] 13 and 14 show schematic diagrams of the positional relationship between mounting electrodes 87, terminal electrodes 58, and solder 86. Fig. 14(a) is a diagram of chip component 54 as viewed from above, and Fig. 14(b) is a diagram of chip component 54 as viewed from a cross-sectional direction.

[0232] In Fig. 14(a), for ease of explanation and for ease of illustration, one start point 71 and one end point 72b are shown. In reality, as shown and explained in Fig. 22, the start point 71 and the end point 72 are found in each cross section, and the start point 71 and the end point 72 are set. Using these start points 71 and end points 72, AI processing is performed to detect cracks and find the crack positions, crack line lengths, and crack areas.

[0233] Fig. 14(b) shows a case where the mounting electrode 87 and the terminal electrode 58 are well aligned. Fig. 14(b) shows a case where the mounting electrode 87 and the terminal electrode 58 are misaligned. The position of the starting point 71 is not significantly different between Fig. 14 and Fig. 15, but the position of the end point 72 is different between Fig. 14(a) and Fig. 15(b).

[0234] 14(a) and 14(b) show locations where cracks are likely to occur with black and white circles. Cracks 39 (not shown) in solder 86 are caused by differences in the expansion rates of the board and electronic components as they repeatedly expand and contract due to thermal stress, and occur in solder 86.

[0235] 14(a) and 14(b), the maximum strain occurs at the end of mounting electrode 76, and cracks are likely to occur at the locations marked with white and black circles. The black circles are starting points 71, which will be described in FIG. 10 and elsewhere. In the case of FIG. 14, this is the lower end of mounting electrode 87, and is the contact point between mounting electrode 87 and solder 86.

[0236] The position assumed as end point 72 is the position where solder 86 faces the air (outside air). In the case of Fig. 14(a), end point 72b is the position of a perpendicular line from start point 71 to the surface of solder 86. End point 72a is end point 72a, which is a position along mounting electrode 87. When start point 71 is used as a reference, end point 72a is the shortest distance to the position where solder 86 comes into contact with the outside air.

[0237] The end point 72 and the start point 71 can be found or estimated by the finite element method. In the case of Fig. 14(b), when the start point 71 is used as the reference point, it is the geometrically shortest position. Therefore, it can be found or estimated geometrically at each cross section in the same way as described in Fig. 22.

[0238] 15, a positional deviation occurs between the mounting electrode 87 and the terminal electrode 58 and their normal positions. The occurrence of the positional deviation can be detected by an image captured by the camera 60. In addition, the thickness, position, etc. of the solder 86 can be detected by the height / range detection device 62. 15(b), the starting point 71 is also located substantially below the tip of the mounting electrode 87, and is the contact point between the mounting electrode 87 and the solder 86.

[0239] The position assumed for end point 72 is the position where solder 86 faces the air (outside air). In the case of Fig. 15(b), end point 72b is the position of a perpendicular line from start point 71 to the surface of solder 86. End point 72a is the position where solder 86 faces the air (outside air), and is the closest pole to the surface of solder 86 that comes into contact with the air. End point 72b is the position of a perpendicular line from start point 71 to the surface of solder 86. Cracks are likely to occur at the locations marked with white circles and black circles. The white circles and black circles are the starting points 71 and the ending points 72.

[0240] 16, a constriction may occur at an end point 72a, etc. The occurrence of the constriction can be detected by the X-ray CT device (X-ray irradiation device) 53 and the rotation device 63. 16, the starting point 71 is also located substantially below the tip of the mounting electrode 87, and is the contact point between the mounting electrode 87 and the solder 86.

[0241] The position assumed for end point 72 is the position where solder 86 faces the air (outside air). In the case of Fig. 16(b), end point 72b is the position of a perpendicular line from start point 71 to the surface of solder 86. End point 72a is the position where solder 86 faces the air (outside air), and is the shortest pole to the surface of solder 86 that comes into contact with the air. End point 72b is the position of a constriction on the surface of solder 86 from start point 71. The position of the constriction is the shortest distance based on start point 71. The start point 71 and the end point 72 are set by using the finite element method, or by geometrically determining or predicting. 17 and 18 show the end of a BGA (solder) part of an electrode terminal of an IC and the end of a BGA (solder) part of an electrode terminal of a substrate.

[0242] The starting points 71a and 71b correspond to the starting point 71. The starting points 71a and 71b are end portions, particularly corner portions, of the mounting electrode 87, where the mounting electrode 87 contacts the solder 86. End point 72 is the surface of solder 86 that comes into contact with air, and end points 72 a and 72 b are located at the shortest distance from start point 71 .

[0243] The locations where cracks are likely to occur (start point 71, end point 72) vary depending on the component and mounting method. Figure 18(a) shows a case where the alignment between mounting electrode 87 and terminal electrode 58 is normal. Figure 18(b) shows a case where the alignment between mounting electrode 87 and terminal electrode 58 is misaligned.

[0244] 18(b), a positional deviation occurs between the mounting electrode 87 and the terminal electrode 58 and their normal positions. The occurrence of the positional deviation can be detected by an image captured by the camera 60. Furthermore, the thickness, position, etc. of the solder 86 can be detected by the height / range detection device 62. The position and state of the solder 86 can be measured by the X-ray CT device 53. 18(b), the starting point 71 is also located substantially below the tip of the mounting electrode 87, and is the contact portion or contact point between the mounting electrode 87 and the solder portion 86.

[0245] The position assumed for end point 72 is the position where solder 86 faces the air (outside air). In the case of Fig. 18(b), end point 72b is the position of a perpendicular line from start point 71 to the surface of solder 86. End point 72a is the position where solder 86 faces the air (outside air), and is the closest pole to the surface of solder 86 that comes into contact with the air. End point 72b is the position of a perpendicular line from start point 71 to the surface of solder 86.

[0246] Cracks are likely to occur at the locations marked with white circles and black circles. The white circles and black circles are the starting point 71 and the end point 72. The starting point 71 and the end point 72 are set using the finite element method, or are geometrically determined or predicted.

[0247] The start point 71 and the end point 72 are obtained or assumed by the finite element method and AI learning. Also, the start point 71 and the end point 72 are geometrically obtained or assumed (estimated) from the shape of the cross section.

[0248] From a structural standpoint, it is possible to predict the locations of the joint boundaries where cracks are likely to occur. Furthermore, the positions of the crack starting point 71 and point 72 change depending on the progress of the crack, the way the crack occurs, and the mounting state. After pattern classification according to each component and mounting method, a predetermined position or location is designated as a start point 71 or an end point 72 to carry out processing. 19, 20, and 21 are explanatory diagrams of AI processing for detecting cracks, etc.

[0249] The input image is subjected to pattern classification for each component and mounting method, and then the locations where cracks are likely to occur, as well as the starting point 71 and the end point 72, are specified, and detection performance is improved by using the finite element method or setting geometrically.

[0250] As explained before, Fig. 19 illustrates details of the automatic detection program 111. First, a group of captured images 21 newly uploaded from the height / range measuring device 62, the camera 60, and the X-ray CT scanner 53 are read into a pattern classification and prediction program 201.

[0251] The pattern classification prediction program 201 classifies the patterns of an image. Examples of patterns to be classified include cases where different mounted electronic components, such as chip capacitors, resistors, coils, etc., are included. Furthermore, information on the start point 71 and end point 72 is classified for the shapes of the mounting electrode 87, the terminal electrode 58, and the lead pin 57.

[0252] Examples include cases where the pin shapes of IC packages are different, such as a lead pin 57 type and a leadless type, and cases where the shapes of the lead pins 57 and the shapes of the terminal electrodes to be joined are different. Other examples include cases where the types of substrates are different, such as surface mount and through hole, and cases where the materials of electronic components are different, such as copper, brass, iron, etc.

[0253] Examples include different photographing methods such as a height / range measuring device 62, a camera 60, an optical microscope, an electron microscope, etc. In order to improve the accuracy of automatic crack detection in the subsequent stage, each part is classified. The captured image group 21 is classified by a pattern classification prediction program 201 into, for example, crack images into pattern A, pattern B, and pattern C, respectively.

[0254] Different processing methods are applied to the crack images for Pattern A, Pattern B, and Pattern C. For example, different processing methods are applied depending on the pin shape of the IC package and the mounting pattern of the lead part.

[0255] The image group of pattern A is read into a pattern A area detection and prediction program 202. The image group of pattern B is read into a pattern B area detection and prediction program 203.

[0256] The image group of pattern C is read into the area detection and prediction program 204 for pattern C, and then each program automatically detects areas such as cracks 39. These analysis result images are stored as an analysis result (mask image) group 22.

[0257] The present invention focuses on the fact that there is an optimal area detection method for each pattern, and prepares an area detection program for each pattern. This is because the accuracy of area detection can be improved compared to when no area detection program is prepared for each pattern. Therefore, a machine learning model (area detection) must be prepared for each pattern.

[0258] Fig. 20 illustrates the process flow of the training program and prediction program in the above-mentioned pattern classification. Information from the camera 60, the height and range measurement device 62, and the X-ray CT device 53 is used for AI learning.

[0259] The purpose of the training program for pattern classification is to create a machine learning model (pattern classification) DB 108. For example, if there are two types of patterns, pattern A and pattern B, in a group of captured images, a machine learning model capable of classifying a given captured image as pattern A or pattern B is created using a captured image (A) DB 301 and a pattern (A) DB 302 that can refer to whether each captured image is pattern A or pattern B.

[0260] Detection performance can be improved by first classifying the input image as either an image of the lead mounting pattern (Pattern A) or an image of the BGA mounting pattern (Pattern B), and then performing area detection using a subsequent machine learning model (area detection) appropriate for each pattern.

[0261] 20, the data augmentation pre-processing program 303 reads the captured image (A) DB 301 and the pattern (A) DB 302, and creates a data set for machine learning. The created data set is stored in the machine learning data (A) DB 304.

[0262] Next, the machine learning (training) program 305 reads the machine learning data (A) DB 304, performs training (for example, in the case of a neural network, adjusts the weights and biases of the network), and stores the trained machine learning model in the machine learning model (pattern classification) DB 108. When training, an existing machine learning model in the machine learning model (pattern classification) DB 108 may be used to perform the training.

[0263] When training, a machine learning model may be trained by setting specific weights for machine learning data owned by a specific user, group, or case, or for machine learning data with a recent upload date and time, and differentiating the importance of the data.

[0264] For example, User A's company handles BGA type data and measures crack rates. User B's company handles lead components and measures crack rates. In this case, when User A uses the system, the detection accuracy will be better if they use an AI that has learned more about BGA types. In this way, it is preferable to change the proportion (weight) of image types used in the training data depending on the user.

[0265] If the performance evaluation of the newly trained machine learning model determines that it has higher accuracy than the existing machine learning model, that is, if the newly trained machine learning model can perform pattern classification more accurately than the existing machine learning model for the evaluation dataset in the machine learning data (A) DB304, it may replace the existing machine learning model in the machine learning model (pattern classification) DB108.

[0266] As is clear when comparing binary classification with multi-value classification, the fewer the number of patterns to be classified, the lower the difficulty (the fewer features to focus on), and therefore the higher the classification performance.

[0267] When verifying the accuracy of machine learning model 1 and machine learning model 2 for a specific test dataset, the one with the better detection accuracy is adopted. The accuracy rate for the test dataset is converted into a score, or quantified and recorded. In other words, the accuracy rate ("Accuracy", an evaluation index for machine learning; the total percentage of correctly predicted positive data for data that is actually positive, and correctly predicted negative data for data that is actually negative) is compared between machine learning model 1 and machine learning model 2, and the one with the higher accuracy rate is adopted.

[0268] If the performance evaluation of the newly trained machine learning model determines that the model has lower accuracy than the existing machine learning model, that is, if the newly trained machine learning model is unable to classify patterns more accurately than the existing machine learning model for the evaluation dataset in the machine learning data (A) DB304, the trained machine learning model may be discarded.

[0269] When verifying the accuracy of machine learning model 1 and machine learning model 2 for a specific test dataset, the one with the lower accuracy is rejected. The accuracy rate for the test dataset is scored, or quantified and recorded. In other words, the accuracy rate ("Accuracy", an evaluation index for machine learning; the total percentage of correctly predicted positive data for data that is actually positive, and correctly predicted negative data for data that is actually negative) is compared between machine learning model 1 and machine learning model 2, and the one with the lower accuracy rate is rejected.

[0270] As described above, a feature of the present invention is that a training program for pattern classification periodically evaluates the performance of a newly trained machine learning model, and if the newly trained machine learning model is able to perform pattern classification more correctly than an existing machine learning model, the existing machine learning model in machine learning model (pattern classification) DB108 is replaced with the newly trained machine learning model.

[0271] A machine learning model trained on a group of image data collected at a certain point in time T0 and a machine learning model trained on a group of image data collected at a different point in time T1 are called a "newly trained machine learning model." The training methods used at T0 and T1 are the same or similar processes. Examples of periodic execution include execution after the processing of one item is completed, monitoring the usage time and predicting the time when the device is not in use and executing the execution during that time period. As time passes, as the amount of processing increases, or as processing progresses, the amount of data increases, so the AI ​​model is updated regularly.

[0272] Using these functions and operations, information data of the solder 86 is extracted from the printed circuit board 89 on which the electronic component 54 is mounted. Also, crack information of the solder 86 is extracted and detected.

[0273] 20, in the prediction program for pattern classification, a preprocessing program 307 reads a captured image group 306 and performs preprocessing of the images. The preprocessed images are read into a machine learning (prediction) program 308, and a machine learning model trained by a training program is read from the machine learning model (pattern classification) DB 108, and pattern classification is performed. The patterns classified for the captured image group are stored in a pattern classification result group 309. FIG. 21 illustrates the process flow of the training program and prediction program in the above-mentioned area detection.

[0274] The training program for area detection aims to create a machine learning model (area detection) DB 109. Using the captured image (B) DB 401 and the mask image (B) DB 402, a machine learning model is created that can detect, in pixel units, which areas of the captured image correspond to voids or cracks 39.

[0275] A data augmentation preprocessing program 403 reads the captured image (B) DB 401 and the mask image (B) DB 402, and creates a data set for machine learning.

[0276] The created data set is stored in the machine learning data (B) DB 404. Thereafter, the machine learning (training) program 405 reads the machine learning data (B) DB 404, performs training (for example, in the case of a neural network, adjusts the weights and biases of the network), and stores the trained machine learning model in the machine learning model (area detection) DB 109. When training, an existing machine learning model in the machine learning model (area detection) DB 109 may be used to perform the training.

[0277] When training, specific weights are assigned to machine learning data held by a specific user, group, or project, or to machine learning data that has been recently uploaded, and the importance of the data is differentiated to train the machine learning model.

[0278] If the performance evaluation of the newly trained machine learning model determines that it has higher accuracy than the existing machine learning model, that is, if the newly trained machine learning model can detect areas more accurately than the existing machine learning model for the evaluation dataset in the machine learning data (B) DB404, it may replace the existing machine learning model in the machine learning model (area detection) DB109.

[0279] Whether the area was detected correctly is determined by looking up each pixel in the image against manually created ground truth data (data in which crack areas have been manually annotated) and checking whether the labeling is correct.

[0280] If the performance evaluation of the newly trained machine learning model determines that it has lower accuracy than the existing machine learning model, that is, if the newly trained machine learning model is unable to detect regions more accurately than the existing machine learning model for the evaluation dataset in the machine learning data (B) DB404, the trained machine learning model may be discarded. As described above, a feature of the present invention is that the performance evaluation of a newly trained machine learning model in area detection is periodically performed by a training program.

[0281] If the newly trained machine learning model detects regions more correctly than the existing machine learning model, the existing machine learning model in the machine learning model (region detection) DB109 is replaced with the newly trained machine learning model. In FIG. 21, in the prediction program for area detection, a pre-processing program 407 reads out a captured image group 406 and executes pre-processing of the images. As an example of the effect of preprocessing, errors, noise, and missing values ​​are removed, improving the prediction performance of area detection.

[0282] The preprocessed image is read into the machine learning (prediction) program 408, and the machine learning model trained by the training program is read from the machine learning model (area detection) DB 109, and area detection is performed. The results of area detection for the captured images are stored in an analysis result (mask image) group 409 .

[0283] Using these functions and operations, information data of the solder 86 is extracted from the printed circuit board 89 on which the electronic component 54 is mounted. Also, crack information of the solder 86 is extracted and detected. 22 and 23 are explanatory diagrams illustrating the joint between the lead pin 57 and the terminal electrode 58 of the electronic component 54. FIG.

[0284] Electrode wiring such as terminal electrodes 58 is formed on the substrate 89. The terminal electrodes 58 and the lead pins 57 of the electronic component 54 are joined by solder parts 86. There is a joint between the terminal electrodes 58 and the lead pins 57, and cracks 39 and the like mainly occur at this joint.

[0285] The X-ray CT device 53 acquires an X-ray image centered on the joint 59. The X-ray CT device 53 changes or moves the observation position of an observation head (not shown) in the X-axis, Y-axis, and Z-axis directions to acquire an X-ray image pattern 40. Alternatively, for example, two-dimensional planar images such as an XZ plane are acquired all at once, and the focal position is changed sequentially in the Y-axis direction to continuously observe or acquire two-dimensional planar images. Also, the rotation device 63 rotates the printed circuit board 89 to acquire the X-ray image. The acquisition of images is not limited to the X-axis, Y-axis, and Z-axis directions. For example, images may be acquired in the directions of axes oblique to the X-axis and Y-axis directions.

[0286] The tilted CT scan method rotates the object to be observed horizontally while irradiating it with X-rays from an oblique direction, so it is possible to obtain clear, high-magnification transmitted images even for plate-shaped objects. Clearer images improve learning efficiency and detection performance.

[0287] An optically photographed image of the surface state of the lead pins 57 and terminal electrodes 58 or the gaps between the lead pins 57 and terminal electrodes 58 may be obtained and used to detect cracks 39 and voids.

[0288] The cracks 39 and voids may be exposed on the solder surface, etc., and by carrying out the present invention while taking into account these exposed areas, the positions of the cracks 39 and voids can be observed and analyzed accurately.

[0289] In addition, by carrying out the present invention by adding element mapping images by energy dispersive X-ray spectroscopy (EDS), the positions of cracks 39 and voids can be observed and analyzed accurately. It goes without saying that a temperature distribution image of the object surface taken by an infrared thermoviewer camera may also be used.

[0290] 24(a), A of bonding portion 59 is the side where lead pin 57 and bonding portion 59 contact each other. B of bonding portion 59 is the side where terminal electrode 58 and bonding portion 59 contact each other. The X-axis is the direction of the long side of lead pin 57, the Z-axis is the direction of the short side of lead pin 57, and the Y-axis is the film thickness direction of bonding portion 59.

[0291] Fig. 24(b) is a schematic diagram showing an example of image processing of an X-ray image acquired by X-ray CT device 53. Fig. 24(b) shows image processing data in the XZ plane of joint 59. As shown in Fig. 24(b), linear crack 39 can be detected in joint 59 in image data by the image processing method described in Fig. 1 etc., but crack 39 is not clearly seen in other locations.

[0292] As for unclear causes, examples include that cracks 39 often occur in the XZ plane where lead pin 57 joins, while the number of cracks 39 appearing in the XY and XZ planes is small, and that the joining material such as solder changes into layers in the XY and XZ planes.

[0293] There are brightness and shade differences in the images, and it is difficult to distinguish cracks in a single two-dimensional plane image of cracks 39. By learning the position and occurrence state of cracks 39 in each cross-sectional image, it becomes possible to detect and understand cracks 39. Also, processing is carried out using information data from the camera 60 and the height / range measuring device 62.

[0294] "Learning" in this specification is a technical category of "training" mentioned above. Therefore, various images with different brightness differences and shading are collected and the AI ​​model is retrained. The image or image data may be subjected to contrast enhancement, brightness enhancement, brightness inversion processing, sharpness processing, etc. In the embodiment, the position and occurrence state of the cracks 39 are learned, but it goes without saying that they may be replaced with voids or the like.

[0295] In the present invention, as shown in FIG. 24, cross-sectional data of an X-ray image is processed at a predetermined distance t in the X-axis direction, where the brightness and shading of the crack portion are emphasized. In other words, the image is cross-sectional data on the XY axes. It is preferable that the distance t is as short as possible. However, the shorter the distance, the longer the image processing time per lead pin 57. However, the distribution and position of cracks 39 become accurate, and the learning effect is also improved. When the possibility of crack 39 occurrence increases, it is also effective to shorten the predetermined distance t and the predetermined distance s.

[0296] The shorter the specified distance t and the specified distance s, the more slice images are acquired, and the longer the image processing time per lead pin 57 becomes, but the higher the accuracy of the distribution and position of the cracks 39 becomes, and the more effective the AI ​​learning becomes. Also, by comparing and processing the characteristic parts (differences) of each image, the image processing time can be reduced.

[0297] When a portion of the acquired image is unclear, it is effective to shorten the specified distance t and the specified distance s. The AI ​​will have higher recognition accuracy if an image in which the image features of a crack are clearer is used for learning. Even if an AI processes a blurred image and has it discriminated, the accuracy will be poor. However, it is effective to perform a filtering process that multiplies a matrix on a blurred image, including sharpening, emphasis, and edge emphasis.

[0298] Distance t and distance s are preferably shorter than the thickness of joint 59. More preferably, distance t and distance s are greater than or equal to ¼ and less than or equal to ½ of the thickness of joint 59. The thickness of joint 59 is the average thickness of the portion where terminal electrode 58 and lead pin 57 face each other.

[0299] Fig. 25 is an explanatory diagram of a method for detecting and learning about a crack 39 in an XY plane. As shown in Fig. 25, the explanation will be given assuming that a crack occurs parallel to the XZ plane of a joint 59. The X-ray data shown in Fig. 25 is an image pattern 40 in the XY plane of a joint 59, and is arbitrary image data at a cross section at intervals t in Fig. 22.

[0300] In the present invention, the image patterns 40 in the XY plane are selected sequentially, in a predetermined order, or randomly, and the image processing described above is performed to obtain the crack 39 area and crack 39 distribution in the joint 59. The recognition rate of the crack 39 can be improved by specifying the crack position point by point. The recognized crack is also used as training data.

[0301] Also, image processing is performed, and the crack 39 area and crack 39 distribution are displayed on a display or the like so that they can be seen optically or visually, and are also converted into digital data so that they can be sent via a network.

[0302] In the embodiment of Fig. 25, cracks 39a, 39b, and 39c are detected on the XY cross section of the joint 59. Each crack 39 is a crack 39 detected on the XY cross section. The XY cross section data is detected at intervals of t. By detecting the cracks 39 on the XY cross section at intervals of t, the distribution of cracks 39 on the joint 59 in Fig. 30 is obtained. By detecting the cracks 39 on the XY cross section at intervals of t, the volume fraction of cracks 39 can also be obtained by integration. Measurement of the three-dimensional crack fraction will be described later.

[0303] The crack area is calculated for each pixel in each slice image. By integrating the areas of each slice image, the crack volume ratio can also be calculated as voxel data. In this specification, the XY cross section is mainly described as an example, but it goes without saying that the present invention may also be practiced with an XZ cross section or a YZ cross section.

[0304] The crack distribution in each XY cross section can be measured by implementing the present invention. By overlapping the measurement results in the Z direction for each △t (delta t), a three-dimensional crack distribution can be obtained. When a part of the acquired image is unclear, it is also effective to implement the present invention by shortening the specified distance t and the specified distance s.

[0305] Fig. 26(a) is an XY cross section at intervals t in Fig. 22. Learning and detection processing is performed on an image pattern 40 consisting of the XY cross section of Fig. 26(a). The present invention is not limited to processing on the XY cross section of Fig. 26(a). For example, processing may be performed in the cross section direction shown by the dotted line in Fig. 26(b).

[0306] Fig. 26(b) is a cross section at intervals s in Fig. 22. That is, this is processing on a ZY cross section. Learning and detection processing are performed on an image pattern 40 consisting of the ZY cross section in Fig. 26(b). It goes without saying that learning and detection processing may be performed on both image patterns 40 on the XY cross section and the ZY cross section.

[0307] Specific training examples will be described with reference to Figures 27, 28, and 29. The acquired image pattern 40 will be described as a pattern 40 of cracks 39, etc., but it goes without saying that the present invention is not limited to this and, for example, the cracks 39, etc. may be replaced with voids, etc. Also, it goes without saying that the present invention is not limited to cracks and voids and can also be applied to non-bonded parts, crystal orientation, etc. Even in a pattern in which voids 36 and cracks 39, etc. are mixed, the voids and cracks 39, etc. can be distinguished and detected by learning. By learning foreign matter, it is possible to detect voids 36, cracks 39, etc., by distinguishing between them.

[0308] In the present invention, the teacher data is prepared as correct answer data classified into void data and crack data. The trained AI distinguishes between voids and cracks based on the learned criteria. It also classifies them. The trained AI analyzes and visualizes the occurrence state of voids 36 and cracks 39 based on the learned criteria.

[0309] Fig. 29 is a schematic diagram (pattern 40) showing the solder part 86 and the occurrence of cracks 39 photographed by X-ray. Cracks 39 are not independent linear shapes, and cracks 39b and 39c in Fig. 29 may branch off. Cracks 39e and 39d may also be connected. Cracks 39 may be multiple cracks that are connected or branch off, and may be shaded, reducing the contrast.

[0310] In the following embodiment, the pattern 40 will be described as a pattern 40 acquired by an X-ray device, but it goes without saying that the pattern 40 can also be acquired by an X-ray CT device and a camera 60. It also goes without saying that an X-ray CT device and an ultrasonic microscope device may be combined.

[0311] Prepare thousands of X-ray images and images with the crack39 areas masked (manually inspected data). Prepare original images such as X-ray images and images with marks indicating where voids and cracks are located in the image. These images and data will be the training data (correct answer data).

[0312] These are randomly cut out as 512 x 256 (pixel) patch images 37, and one or more processes, such as making an image with increased noise, adjusting contrast, adjusting brightness, inverting brightness, smoothing, enlarging / reducing, rotating, converting the number of pixels, performing sharpness processing, shifting horizontally or vertically, converting the number of pixels horizontally or vertically, setting gradation data in a specific range to a constant value, performing gamma conversion on the image data, partially masking (painting black), shrinking, converting the number of pixels, and enlarging again, or a combination of multiple processes are performed. Therefore, many training images can be generated from one image or image data.

[0313] Data augmentation to generate a dataset of hundreds of thousands of images. By learning and training on a dataset of hundreds of thousands of images, the crack39 detection rate can be significantly improved.

[0314] 29, the solder portion 86 is divided into three patch images (section 37a, section 37b, and section 37c). Each section 37 preferably has the same shape and area.

[0315] Each patch image (section 37) includes a crack 39. Also, a continuous crack 39 is included between adjacent sections 37. By performing image processing on adjacent patch images, the crack 39 in the patch image can be detected.

[0316] The accuracy of crack detection can be improved by effectively utilizing the processing results of adjacent patch images. As an example, the crack position of the patch image to be processed can be estimated and corrected based on the positional relationship of the cracks in the adjacent patch images.

[0317] As shown in Fig. 27, the section 37 in Fig. 27(a) is inverted to obtain training data (Fig. 27(b)). Also, as shown in Fig. 27(c), the section 37 in Fig. 27(b) is reduced to obtain training data (Fig. 27(c)).

[0318] Section 37 shown in FIG. 28(a) is rotated a predetermined angle (+3° in the figure) to the left as shown in FIG. 28(b) to obtain training data. Section 37 may also be inverted by line symmetry as shown in FIG. 28(c) to obtain training data. The rotation angle is preferably 1° (degrees) or more and 15° (degrees) or less.

[0319] A feature of the present invention is that even if there is only a small amount of data available for machine learning, the amount of data can be expanded by performing one or more processes, such as inverting, rotating, shrinking, or enlarging the divisions, as described above, or a combination of multiple processes, thereby self-generating a large dataset.

[0320] Next, 70% of the patch images (division 37) are used as training data and the remaining 30% are used as test data to perform deep learning of U-Net for medical segmentation (especially cell segmentation, etc.) using an encoder-decoder model for segmentation.

[0321] Segmentation refers to image segmentation, a type of image processing technique, in which, when an image is given, each pixel is assigned a label to indicate what it corresponds to.

[0322] During AI training, it is preferable to use 60% or more of the patch images (category 37) as training data. It is preferable to use 40% or less as test data. It goes without saying that the ratio may be changed based on the progress and results of AI training.

[0323] The loss function during learning is the Dice coefficient between the mask image as training data and the mask image as the network output, and is the Dice coefficient × (-1).

[0324] The Dice coefficient uses the average number of elements in the two sets instead of the number of elements in the union of the two sets. During training, the machine learning model that has the smallest loss on the aforementioned test data is saved.

[0325] The Dice coefficient may be changed based on the progress and results of AI learning. The value calculated by the loss function is called Loss. The machine learning model with the smallest value of the loss function (how close the predicted value is to the true value) for the test data is saved.

[0326] On the other hand, in the prediction program for area detection, a pre-processing program reads the captured images and performs pre-processing of the images. The pre-processed images are read into a machine learning (prediction) program, and a machine learning model trained by a training program is read from the machine learning model (area detection) DB 109 to perform area prediction.

[0327] The predicted areas for the captured image are stored in a mask image set. The mask image is a binary image of 0 and 1, where 1 indicates the area where the crack is located and 0 indicates the background. It goes without saying that it is also possible to set it so that 0 indicates the area where the crack is located and 1 indicates the background.

[0328] A specific prediction example will be described with reference to Fig. 29. A captured X-ray image is divided into patch images 37 of 512 x 256 (pixels) at equal intervals. Alternatively, the patch image is divided into a plurality of images.

[0329] 28, the patch image is divided from the leftmost section 37a to the right, but the patch image 37 is divided into sections 37a, 37b, and 37c that are equally spaced and overlap each other according to the width and height of the captured X-ray image. Next, each patch image is input to a trained machine learning model.

[0330] From the trained machine learning model, a probability map (values ​​between 0 and 1) indicating the presence of a void at each pixel position is obtained. To obtain a binary map (mask image), the obtained probability map is binarized with a set threshold of 0.5 (1 indicates a crack, 0 is the background). The threshold of 0.5 can be changed. It is also effective to change the threshold based on the learning effect based on AI processing.

[0331] The resulting 512 x 256 (pixel) binary maps are then combined (as divided into sections 37a, 37b, 37c in Figure 29) to generate an overall binary map (mask image) that corresponds to the original X-ray image.

[0332] In this embodiment, the number of pixels in the vertical direction of the image data is 256. 256 is the 8th power of 2. It is preferable that the number of pixels in the horizontal direction and the number of pixels in the vertical direction of the image data are the nth power of 2 (n is an integer equal to or greater than 2). As a post-processing step, we optimize the binary map (mask image) using a conditional random field (CRF).

[0333] The boundaries can be clarified by applying a probability model called a conditional random field (CRF) to the probability map output by the AI ​​model. The binary map (mask image) is optimized using the CRF for the probability map (values ​​between 0 and 1) that indicates the presence of voids mentioned above. Optimizing the binary map (mask image) means that it is possible to remove ambiguity and sharpen object boundaries. As described above, the present invention is characterized in that processing is performed by dividing into a plurality of small regions (sections) 37, and the respective processing results are integrated to obtain a single processing result.

[0334] For example, even if a crack 39, whose object must be contained within the boundary of a section 37, is detected clearly in another section 37, it may be possible to detect the crack 39 clearly. For overlapping detected areas, the processing results for each batch image are averaged, and pixels exceeding a threshold value of 0.5 are regarded as voids, enabling automatic detection of the crack 39 with high robustness.

[0335] In this embodiment, 1 corresponds to the presence of voids and 0 corresponds to the background (no voids). Also, an example is shown in which processing is performed with a threshold value of 0.5. The threshold value may be changed based on the progress and results of AI learning.

[0336] For example, even if crack 39b cannot be detected in the image of section 37a, it may be possible to detect crack 39b in the image of section 37b. Even if crack 39b cannot be detected in the image of section 37b, it may be possible to detect it by flipping the image horizontally.

[0337] If the processing results for the images obtained by rotating the section 37 multiple times are averaged and pixels exceeding a threshold value of 0.5 are determined to be cracks 39, the method is highly robust and allows automatic detection of cracks 39.

[0338] Fig. 30 is a diagram showing the detection results of cracks 39 in each XY cross section at intervals t. Fig. 30(a), Fig. 30(b), Fig. 30(c), Fig. 30(d), and Fig. 30(e) show the state of cracks 39 in each cross section.

[0339] For ease of understanding, Fig. 30 is illustrated in a schematic manner. The XY cross section of Fig. 30(a) taken at interval t is Fig. 30(b), and the XY cross section of Fig. 30(b) taken at interval t is Fig. 30(c). The XY cross section of Fig. 30(c) taken at interval t is Fig. 30(d), and the XY cross section of Fig. 30(d) taken at interval t is Fig. 30(e).

[0340] Information on the XY cross section can be obtained using information data from the camera 60 and the height / range measuring device 62. Also, information data on the solder 86 is extracted from a printed circuit board 89 on which the electronic component 54 is mounted.

[0341] The cracks in adjacent XY cross sections are subjected to a process of determining whether they are continuous or discontinuous through AI learning. The cracks 39 detected by processing the image pattern 40 in Figure 30 are determined to be continuous or discontinuous with the cracks 39 in the adjacent XY cross section pattern, and whether they are partially continuous or partially discontinuous, and the distribution of cracks 39 shown in Figure 31 is created.

[0342] Fig. 31 shows an example of a cross-sectional image at intervals t in Fig. 22 that has been subjected to deep cultivation, but a cross-sectional image at intervals s may also be subjected to deep cultivation. Needless to say, cross-sectional images at intervals t and s may also be subjected to deep cultivation. By studying the images in Figs. 30(a) to 30(e), it is possible to learn whether each crack is continuous or discontinuous. FIG. 32 is an explanatory diagram illustrating a method for updating machine learning data in order to reconstruct and retrain a machine learning model.

[0343] The results of the automatic detection of the cracks 39 described above are not always correct. The user can confirm and edit the results of the automatic detection of the cracks 39 using the analysis result editing and viewing program 112 of this system.

[0344] The automatic detection result of cracks 39 output from this system is output between 0 and 1, and if it exceeds the threshold value of 0.5, it is recognized as a void or crack. Therefore, for example, 0.999 can be said to be clearly a void or crack (high reliability), but 0.55 can be said to be doubtful as to whether it is a void or crack (low reliability).

[0345] The program allows users to manually correct the results of the AI's automatic detection when the results are incomplete. For example, for areas that are voids but were not determined to be voids, the user can manually edit the location and area of ​​the void or crack using a circle tool, just like a paint tool. The edits made by the user will override the AI's automatic detections, meaning that the edits made by the user will be recorded in the system as the correct information.

[0346] The positions and areas of possible voids and the positions and areas of possible cracks are specified or displayed by the information data of the camera 60, height and range measuring device 62, the finite element method, start point 71 and end point 72 processing or detection using geometric processing, and the AI ​​processing of the present invention.

[0347] The positions and areas of voids and cracks can be specified or determined by using the circle tool of the paint tool. By specifying the positions and areas, the AI ​​processing unit of the present invention learns and determines the positions and areas of voids and cracks. In order to improve the automatic detection performance, it is necessary to incorporate the user's confirmations and edits into the machine learning data of this system and update it.

[0348] 32(a) shows a case where a captured image group 901 is input to an area detection and prediction program 902, and an analysis result (mask image) group 903 is output, but the user has not confirmed these analysis results. In this case, the result is registered as "unconfirmed" in the confirmation state and editing result (mask image) DB 904, and is not incorporated into the machine learning data, and is not updated.

[0349] The AI's detection results are checked by looking at the images to see whether they are complete (e.g., a void has been determined to be a void) or incomplete (e.g., a void has not been determined to be a void).

[0350] 32(b) shows a case where, after a captured image group 901 is input into an area detection and prediction program 902 and an analysis result (mask image) group 903 is output, the user views the analysis results for a specific period of time or more, or the user presses a confirmation button or an analysis result output button, etc. In this case, the confirmation status and editing result (mask image) DB 904 is registered as "confirmed", and the result is incorporated into and updated as data for machine learning.

[0351] FIG. 32C shows a case where a captured image group 901 is input to an area detection and prediction program 902, and after an analysis result (mask image) group 903 is output, the user edits the analysis result.

[0352] The confirmation state and editing result (mask image) DB 904 is registered as "edited", the analysis result is determined to be incorrect, and the data is weighted and incorporated into the machine learning data for updating. Also, the greater the difference between the analysis result (mask image) group 903 and the result edited by the user, the more weighted the data is incorporated into the machine learning data for updating.

[0353] The large amount of user editing on the AI's automatic detection results suggests that the target image is an image that the AI ​​has never learned or has not learned well. Therefore, that image should be trained more than other images. For example, in the aforementioned data padding process, the former can be padded by n x 30 and the latter by n x 10 to create a difference in the amount of data used for learning.

[0354] One method of weighting is to expand the amount of data by, for example, inverting the captured images 901 and the edited results (mask images) DB 904 in Figure 32 (c) and applying the aforementioned black and white inversion, rotation, shrinking, enlargement, etc., and then incorporating the data into the machine learning data. As described above, the captured image group 901 is classified as unconfirmed, confirmed, edited, etc., and registered, and is incorporated into the machine learning data for updating. By using the above method of updating machine learning data, the machine learning model is reconstructed and retrained to improve automatic detection performance.

[0355] FIG. 33 is an explanatory diagram for explaining a method for measuring the crack rate of the entire joint by combining the crack rate in each XY cross-sectional image and the detection results of cracks 39 in all the XY cross-sectional images. As a method for quantitatively evaluating solder cracks, the crack rate (%) is measured based on images taken during cross-sectional observation after mechanical polishing or CP.

[0356] For images of solder joints, the start and end of the cracks and how they will progress are shown on the image, the length of each line (path length) is measured, and the ratio is calculated.

[0357] In the present invention, assuming that the length of crack 39 shown in Figure 33(a) is a and the length of predicted line 70 along which crack 39 may enter (grow) in the future is b, the crack rate (%) is calculated using (Equation 1). The length of the crack 39 is called the crack line length. The length along which the crack is predicted to grow is called the predicted crack line length.

[0358]

number

[0359] The above embodiment has been described taking the crack 39 as an example, but the same can be applied to the void 36. As with the crack 39, there are regions (areas) in the solder 86 where voids have occurred, or regions (areas) where voids 36 are predicted to occur in the future. The region (area) in which the void 36 occurs is called the area of ​​the void 36. The area in which the void 36 is predicted to grow in the future is called the predicted void 36 area. The void rate (%) is defined based on the area of ​​voids that have occurred in the solder 86 and the predicted area where voids are predicted to occur. The voids 36 are quantitatively evaluated using the void ratio (%), which is defined as follows: Void ratio (%) = (void area) / (void area + predicted void area) x 100

[0360] For an image showing a solder joint, the area in which voids have occurred and how the voids will progress in the future are shown on the image, the area (region) of each is measured, and the ratio (void area divided by the sum of the void area and void prediction area) is calculated.

[0361] This measurement method is destructive, so there is a risk that the crack cross section to be observed may disappear. Also, since cracks 39 occur three-dimensionally in solder joints, it is difficult to grasp the cracks in the entire joint by only observing the cross sections in a few places. The present invention is characterized in that, in response to this problem, the three-dimensional distribution of cracks 39 can be visualized non-destructively and the cracks 39 can be quantitatively evaluated. Cracks are automatically detected in each slice image, and the results are overlaid to visualize the 3D distribution of cracks.

[0362] First, slice images are extracted one by one from the X-ray CT data. Then, the aforementioned segmentation model (U-Net, etc.) is used to detect solder areas and crack areas in the images. On the other hand, the end points of the solder joints are detected using an object detection model (such as YOLO) described below. Finally, these results are combined to draw a single polyline including the crack line and the crack prediction line. These processes are applied to all slice images of the X-ray CT data, and the results are integrated to create 3D data.

[0363] As shown in Figure 33(b), N XY cross-sectional images are processed using multiple types of AI with deep learning, and the results of each process are combined to visualize the three-dimensional distribution of cracks, making it possible to measure the crack rate of the entire joint (hereinafter referred to as the 3D crack rate) using (Equation 2).

[0364] Furthermore, by performing image processing using AI from N XY cross-sectional images, it is possible to create N or more images. For example, an image between the Nth and N+1th images can be generated by interpolation from the Nth and N+1th images. This is the same method used to generate intermediate images from the Nth field image and the N+1th field image in a video television.

[0365]

number

[0366] On the other hand, even in the case of the deep plowing treatment results shown in FIG. 31, it is possible to measure the crack ratio projected onto the Z plane in the solder joint (hereinafter, Z plane projected crack ratio).

[0367] If the area of ​​the wireless crack region in FIG. 31(a) is a and the area of ​​the dotted crack prediction region in FIG. 31(b) is b, the Z-plane projected crack rate can be calculated using Equation 1. FIG. 34 is a diagram showing an overall flow chart for visualizing the three-dimensional distribution of cracks 39 and measuring the 3D crack rate. First, as shown in FIG. 34(a), it is necessary to detect the areas of the solder parts 86 and the voids and cracks 39.

[0368] Cross-sectional images from conventional X-ray CT scans are often unclear, making it difficult to automatically detect crack 39 areas using general rule-based image processing methods such as binarization.

[0369] To address this issue, the segmentation technique using the aforementioned deep-learned U-Net is used for each XY cross-sectional image in Figure 33(b), in which the brightness and shading of the crack 39 area are emphasized, to automatically detect the crack 39 area in the image with high accuracy.

[0370] Moreover, processing is performed using information data from the camera 60 and the height / range measuring device 62. Information data of the solder 86 is extracted from the printed circuit board 89 on which the electronic components 54 are mounted, and processing is performed. At the same time, the area of ​​the solder portion 86 is automatically detected using a similar method for use in the subsequent polyline generation process.

[0371] 34(b), a start point 71 and an end point 72 where a crack 39 may occur are detected at the joint boundary between the lead pin 57 and the solder part 86. However, there is no distinction between the start point 71 and the end point 72, and there is no problem even if they are interchanged.

[0372] To calculate the crack rate, the crack 39 and a predicted line 70 where a crack may occur in the future are drawn in the XY cross-sectional image. A polyline must be generated using the predicted line 70. The predicted line, start point 71, and end point 72 are obtained using the finite element method. Alternatively, they are estimated using the finite element method. However, if the predicted line is a simple line such as a perpendicular line, it can be obtained geometrically.

[0373] By using object detection technology based on deep learning YOLO (You Look Only Once), the starting point 71 and the end point 72 where the crack 39 may enter at the aforementioned joint boundary are detected using the finite element method or the like.

[0374] YOLO is a real-time object detection algorithm. The YOLO algorithm does not use a mechanism for sliding the detection window, but instead detects objects by passing the image through a CNN (Convolutional Neural Network) once. CNN is composed of layers with unique functions such as "convolutional layers" and "pooling layers". Figure 35 illustrates the processing flow of the training program and prediction program for object detection using YOLO described above.

[0375] The training program for object detection aims to create a machine learning model (object detection) DB 116. A machine learning model capable of detecting the start point 71 and end point 72 of the solder joint boundary in the photographed image is created by using a photographed image (C) DB 801 and a position / size (C) DB 802.

[0376] A data augmentation pre-processing program 803 reads a captured image (C) DB 801 and a position / size (C) DB 802, and creates a data set for machine learning.

[0377] The created data set is stored in the machine learning data (C) DB 804. Thereafter, the machine learning (training) program 805 reads the machine learning data (C) DB 804, performs training (for example, in the case of a neural network, adjusts the weights and biases of the network), and stores the trained machine learning model in the machine learning model (object detection) DB 116. When performing training, an existing machine learning model in the machine learning model (object detection) DB 116 may be used for training.

[0378] Training is explained as being synonymous with learning. Here, we will explain it as "learning" in supervised learning. Specifically, data given in advance is regarded as "examples (= advice from a teacher)" so to speak, and learning (= fitting to data) is performed using them as a guide.

[0379] In this embodiment, data obtained by manually measuring the crack rate (data in which crack lines and predicted crack lines are superimposed on an image) is used as training data (example problems), and the finite element method is implemented and a machine learning model is trained so that when unknown data is given, it is possible to predict which are the crack lines and which are the predicted lines.

[0380] When training, a machine learning model is trained by setting a predetermined weight for machine learning data owned by a specific user, group, or case, or for machine learning data with a recent upload date and time, and differentiating the importance of the data. It goes without saying that the predetermined weight may be set or varied by implementing the present invention or the AI ​​processing of the present invention.

[0381] If the performance evaluation of the newly trained machine learning model determines that it has higher accuracy than the existing machine learning model, that is, if the newly trained machine learning model can detect objects more accurately than the existing machine learning model for the evaluation dataset in the machine learning data (C) DB804, it may replace the existing machine learning model in the machine learning model (object detection) DB116.

[0382] If the performance evaluation of the newly trained machine learning model determines that it has lower accuracy than the existing machine learning model, that is, if the newly trained machine learning model can detect objects more accurately than the existing machine learning model for the evaluation dataset in the machine learning data (C) DB804, the trained machine learning model may be discarded.

[0383] As described above, the present invention is characterized in that the performance of a newly trained machine learning model is periodically evaluated by a training program for object detection. Also, if the newly trained machine learning model detects an object more correctly than the existing machine learning model, the existing machine learning model in the machine learning model (object detection) DB 116 is replaced with the newly trained machine learning model.

[0384] The periodic execution may be performed after the processing of one item is completed, or may be performed during a period of time when the device is not in use, by monitoring the usage time.

[0385] 35, in the prediction program for object detection, a preprocessing program 807 reads a captured image group 806 and performs preprocessing of the images. The preprocessed images are read into a machine learning (prediction) program 808, and a machine learning model trained by a training program is read from the machine learning model (object detection) DB 116, and object detection is performed. The results of object detection for the captured image group are stored in a position / size result group 809. Next, as shown in FIG. 34(c), the crack 39 and a prediction line 70 along which a crack may occur (may occur) in the future are drawn in the XY cross-sectional image. Using Figure 52, which is an enlarged view of Figure 34(c), the estimation of crack line and crack predictability using the shortest path search will be explained.

[0386] The start point 71 and the end point 72 where the crack 39 may occur at the above-mentioned joint boundary are detected by using an object detection technology based on deep learning YOLO (You Look Only Once). In addition, a geometric method and a finite element method are combined.

[0387] In order to automatically generate one polyline, the start point 71 and end point 72 detected above are used to perform an A* (A-star) path search algorithm (an algorithm for calculating the shortest path from the start node to the goal node), to search for a path through the crack 39, and to generate one polyline.

[0388] When searching for a path, a cost (weight when passing through each node) is set for the area of ​​the solder part 86 and the crack 39 obtained in Fig. 34(a). The shortest path from the start point 71 to the end point 72 is searched for so as to pass through this area preferentially (i.e., by setting the cost of this area low). This makes it possible to generate one polyline.

[0389] The crack lines are connected so that the path from the starting point 71 to the end point 72 is the shortest path that passes through the areas of the solder part 86 and the crack 39. Therefore, if the path is not the shortest path, the crack lines may not be connected (discontinuous). The predicted line 70 is the line that is not a crack line among the shortest paths.

[0390] In order to improve the efficiency of the mesh creation process in the subsequent stage, the Douglas-Peucker algorithm is used to simultaneously simplify (thin out points) the polyline composed of the crack 39 and the predicted line 70 where a crack may occur in the future.

[0391] The Douglas-Peucker algorithm mainly selects the midpoint with the furthest distance when a perpendicular line is dropped to the line connecting the starting point 71 and the end point 72. If the distance is less than a criterion value, it is considered to be a straight line and the points in between are omitted. If it is more than the criterion value, it is divided at the midpoint and the process is repeated recursively.

[0392] The reason for simplifying a polyline (thinning out points) is to make the processed result easier for users to edit later. When editing a polyline, if you want to move some vertices slightly, it becomes difficult to edit if there are too many of those vertices.

[0393] Finally, the automatic generation process of polylines composed of cracks 39 and predicted lines 70 along which cracks may occur in the future is performed on all XY cross-sectional images in the same manner, and crack surfaces and crack prediction surfaces composed of these polylines are generated as meshes.

[0394] The mesh data was converted into 3D data as STL (stereolithography) files, making it possible to display the data using 3D Slicer and other tools.

[0395] On the other hand, since the length of the crack 39 and the length of the predicted line 70 along which a crack may occur in the future can be measured for each XY cross-sectional image, the 3D crack rate can be calculated using (Equation 2).

[0396] 34 has been described taking the crack 39 as an example, but the present invention is not limited to this. It goes without saying that the same can be applied to the void 36. It is sufficient to replace the term "crack" with "void." The feature of the present invention is that it can non-destructively visualize the three-dimensional distribution of cracks and quantitatively evaluate the cracks.

[0397] Using this method, it is possible to quantitatively observe over time how cracks progress in three dimensions in response to mechanical and thermal shock stress, and to analyze the mechanism behind this and how it affects bond strength. The above embodiments of the present invention have been described with respect to the device for non-destructively visualizing and measuring the cracks 39 inside a substance, and the visualization and measurement methods.

[0398] The (image) pattern 40 of voids and the like can also be obtained by the X-ray CT device 53 or the like. Although it is different from the image of the crack 39 and the like, it becomes possible to detect voids and the like by learning it as a void and the like. Therefore, it goes without saying that the matters described as cracks 39 in the examples of this specification can be replaced with voids or the like. In the embodiment of the present invention, the difference is whether the AI ​​processing is made to memorize the pattern as a void or as a crack.

[0399] The pattern 40 can be obtained not only by an X-ray image using an X-ray CT device 53 or the like, but also by using, for example, a camera 60 to obtain the pattern 40 corresponding to the X-ray image. Therefore, it goes without saying that in the device of the present invention, an X-ray device such as an X-ray CT device may be replaced with an ultrasonic device 2 such as an ultrasonic microscope. It goes without saying that the above embodiment can also be applied to images processed using ultrasonic images.

[0400] By combining an X-ray device such as an X-ray CT device 53 with a camera 60 and a height / range measuring device 62 to obtain the pattern 40, the accuracy of detecting cracks 39 and the like is improved.

[0401] The above embodiment is a method or device for detecting cracks 39 in a transmitted X-ray image, but the present invention is not limited to this. It goes without saying that a wide variety of things can be non-destructively analyzed, for example, air bubbles in adhesive resin, spaces in concrete blocks, metal inclusions in iron parts, fat particles in the internal organs of living organisms, etc.

[0402] The present invention is not limited to detecting, treating, and evaluating cracks and the like that occur in the solder portion 86 and the like. For example, the solder portion 86 may be a conductive paste (copper paste, silver paste, etc.), a conductive adhesive, anisotropic adhesive film (Anisotropic Conductive File: ACF), anisotropic conductive paste, etc. A start point 71 and an end point 72 are provided at the interface or connecting portion between the electrode 58 and the pin 57, etc., and treatment is performed. It goes without saying that the present invention can also be applied to plating the surface of a component, detecting cracks and the like that occur in the paint or plated portion, and evaluating the structure.

[0403] Although the examples have been described mainly assuming that cracks occur in solder and that the cracks in the solder are detected, the present invention is not limited to this. For example, the present invention can be applied to a case where an adhesive is applied to a first member, and cracks that occur in the applied adhesive are detected and used for structural evaluation. Also, the present invention can be applied to a case where an adhesive is applied between a first member and a second member, and cracks that occur in the applied adhesive are detected and used for structural evaluation.

[0404] Needless to say, the present invention can also be applied to the detection of cracks and the like that occur at the joint when the first resin member and the second resin member are melted and joined, and to structural evaluation. Needless to say, the present invention can also be applied to the detection of cracks and the like that occur at the interface, and to structural evaluation, as long as the interface is made up of different materials or different compositions.

[0405] By applying the present invention, it is possible to detect, measure and evaluate the undercoat paint film applied to the substrate, the topcoat paint film, and cracks at the interface between them. It is also possible to measure, detect and evaluate the area of ​​cavities (voids, etc.) that have occurred in concrete, and the range or location of cracks.

[0406] Although the detection of cracks 39 in joints and the like has been described as an example, the present invention is not limited to this. It goes without saying that the present invention can also be applied to voids, non-bonded parts, differences in crystal orientation, and the like.

[0407] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is indicated by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. It goes without saying that the matters or contents described in this specification and drawings can be mutually combined. [Explanation of symbols]

[0408] 11 Control section 12 Main memory 13. Communications Department 14 Control section 15 Display Panel 16 Auxiliary storage 17 Network 18 Recording media 21 Images 22 Analysis results (mask images) 31 Pattern Classification Training Program 32 Pattern Classification and Prediction Program 36 Void 37 Classification (processing unit) 39 Crack 40 Patterns 41 Area Detection Training Program 42 Area detection and prediction program 51 Automatic crack detection device 52 Ultrasonic Microscope 53 X-ray CT device 54 Electronic Components (measurement samples) 55 Solder 56 Substrate 57 Lead pin 58 Terminal electrode 59 Joint 60 Cameras 61 X-ray 62 Height and range measuring device (3D macro device, 3D measuring macroscope) 63 Rotating Device 64 Laser Device 65 Laser Light 66 XY Stage 70 Prediction of future cracks 71 Starting point (starting position) 72 End point (end position) 81 Object Detection Training Program 82 Object detection and prediction program 83 X-ray receiver 84 Resist 85 Terminal electrode 86 Solder 87 Mounting electrode 88 X-ray irradiator 89 Printed Circuit Boards 101 User Authentication Program 102 Case Management Program 103 Photo upload program 104 Photographed Images (1) Database (DB) 105 Analysis Results (1) Database (DB) 106 Analysis Report (1) Database (DB) 107 Data for Machine Learning (1) Database (DB) 108 Machine Learning Model (Pattern Classification) Database (DB) 109 Machine learning model (area detection) database (DB) 110 Machine Learning Programs 111 Auto-detection program 112 Analysis result editing and viewing program 113 Automatic Analysis Report Creation Program 114 Analysis Report Notification Program 115 Analysis report editing and viewing program 116 Machine Learning Model (Object Detection) Database (DB) 201 Pattern Classification and Prediction Program 202 Area detection and prediction program for pattern A 203 Area detection and prediction program for pattern B 204 Pattern C Area Detection Prediction Program 301 Photographed Images (A) Database (DB) 302 Pattern (A) Database (DB) 303 Data augmentation preprocessing program 304 Machine Learning Data (A) Database (DB) 305 Machine Learning (Training) Program 306 Images 307 Pre-processing program 308 Machine Learning (Prediction) Program 309 Pattern Classification Results 401 Photographed Images (B) Database (DB) 402 Mask Image (B) Database (DB) 403 Data augmentation preprocessing program 404 Machine Learning Data (B) Database (DB) 405 Machine Learning (Training) Program 406 Images 407 Pre-processing program 408 Machine Learning (Prediction) Program 409 Analysis results (mask images) 801 Photographed Images (C) Database (DB) 802 Position and size (C) Database (DB) 803 Data augmentation pre-processing program 804 Machine Learning Data (C) Database (DB) 805 Machine Learning (Training) Program 806 Images 807 Pre-processing program 808 Machine Learning (Prediction) Program 809 Position and size results group 901 Images 902 Area detection and prediction program 903 Analysis results (mask images) 904 Confirmation status and editing result (mask image) database (DB)

Claims

1. A method for visualizing voids and cracks at a joint between a mounting member and a substrate on which the mounting member is mounted, bonded, or placed, comprising: First X-ray data of the mounting member; At least one fourth data item among design data of the substrate, surface data of the substrate, and second X-ray data of the substrate; acquiring third X-ray data of a state in which the mounting member is mounted, adhered, or placed on the substrate; A method for visualizing voids and cracks in a joint, characterized by extracting or obtaining an image or image data of the joint from the first X-ray data, the fourth data, and the third X-ray data, and visualizing the joint.

2. A method for visualizing voids and cracks at a joint between a mounting member and a substrate on which the mounting member is mounted, bonded, or placed, comprising: First X-ray data of the mounting member; At least one fourth data item among design data of the substrate, surface data of the substrate, and second X-ray data of the substrate; acquiring third X-ray data of a state in which the mounting member is mounted, adhered, or placed on the substrate; extracting or obtaining an image or image data of the joint from the first X-ray data, the fourth data, and the third X-ray data; classifying the extracted or acquired image or image data based on at least one of the type of electronic component, pin shape, shape of terminal electrode to be joined, type of substrate, and material of the electronic component; A method for visualizing voids and cracks in joints, characterized by machine learning the classified images or image patterns and visualizing the joints.

3. A method for visualizing voids and cracks at a joint between a mounting member and a substrate on which the mounting member is mounted, bonded, or placed, comprising: First X-ray data of the mounting member; At least one fourth data item among design data of the substrate, surface data of the substrate, and second X-ray data of the substrate; third X-ray data of a state in which the mounting member is mounted, adhered, or placed on the substrate; acquiring fifth data of at least one of position data, area data, unevenness data, and elevation data of the joint; A method for visualizing voids and cracks in a joint, characterized by extracting or obtaining an image or image data of the joint from the first X-ray data, the fourth data, the third X-ray data, and the fifth data, and visualizing the joint.

4. A method for visualizing voids and cracks in a joint as described in claim 1, claim 2 or claim 3, characterized in that the third X-ray data is X-ray data obtained from multiple angles with respect to the substrate.

5. A method for visualizing voids and cracks in a joint as described in claim 1, claim 2 or claim 3, characterized in that the crack area or void area of ​​the joint is determined and the state of crack or void occurrence is visualized.

6. A device for visualizing voids and cracks at a joint between a mounting member and a substrate on which the mounting member is mounted, bonded, or placed, comprising: a first database that stores first X-ray data of the mounting member; a second database storing at least one fourth data item selected from design data of the substrate, surface data of the substrate, and second X-ray data of the substrate; an X-ray data acquisition device that acquires third X-ray data of the substrate in a state where the mounting member is mounted, adhered, or placed on the substrate; A device for visualizing voids and cracks in a joint, characterized in that an image or image data of the joint is extracted or obtained from the first X-ray data, the fourth data, and the third X-ray data.

7. A device for visualizing voids and cracks at a joint between a mounting member and a substrate on which the mounting member is mounted, bonded, or placed, comprising: a first database that stores first X-ray data of the mounting member; a second database storing at least one fourth data item selected from design data of the substrate, surface data of the substrate, and second X-ray data of the substrate; a rotation device that rotates the substrate on which the mounting member is mounted, bonded, or placed; an X-ray data acquisition device for acquiring third X-ray data of the joint; classifying the third X-ray data of the joint based on at least one of the type of electronic component, the shape of the pin, the shape of the terminal electrode to be joined, the type of substrate, and the material of the electronic component; A device for visualizing voids and cracks in joints, characterized by performing machine learning on the classified images or image patterns and visualizing the joints.

8. A device for visualizing voids and cracks at a joint between a mounting member and a substrate on which the mounting member is mounted, bonded, or placed, comprising: a first database that stores first X-ray data of the mounting member; a second database storing at least one fourth data item selected from design data of the substrate, surface data of the substrate, and second X-ray data of the substrate; an X-ray data acquisition device that acquires third X-ray data of the joint from a plurality of angles; A visualization device for voids and cracks in a joint, characterized by comprising a data measurement device that measures at least one fifth data of the position data, area data, unevenness data, and elevation data of the joint.

9. A visualization device for voids and cracks in a joint as described in claim 6, claim 7 or claim 8, characterized in that it further comprises a temperature measuring device that heats or warms the joint and measures the temperature or temperature distribution of the joint.

10. A visualization device for voids and cracks in joints as described in claim 6, claim 7 or claim 8, characterized in that the X-ray data acquisition device acquires the third X-ray data from the front surface of the substrate and the back surface of the substrate.

Images