Defect inspection device, defect inspection method, and defect inspection program

Abstract

The present invention reduces processing load and improves detection accuracy when detecting a defect. This defect inspection device includes: an image recognition processing unit that performs image recognition processing on captured image data obtained by capturing an image of an inspection target and determines the attributes of regions in the inspection target; a cutting-out unit that cuts out, from the captured image data, partial image data including a region having a specific attribute from among the regions in the inspection target; a prediction unit that predicts the probability of a defect in the partial image data by inputting the partial image data to a trained defect detection model; and a display control unit that superimposes and displays a marker in a display form corresponding to the predicted probability on the position where the partial image data of the captured image data is cut out.

Description

Defect inspection device, defect inspection method, and defect inspection program

[0001] The present disclosure relates to a defect inspection device, a defect inspection method, and a defect inspection program.

[0002] Conventionally, there has been known a defect inspection device that performs an object detection process on photographed image data of an inspection object to detect defects contained in the inspection object. With this defect inspection device, for example, by increasing the resolution of the photographed image data, it is possible to detect finer defects.

[0003] Patent No. 7270314

[0004] On the other hand, increasing the resolution increases the amount of data per image of captured image data, which increases the processing load on the defect inspection device. To address this issue, it is possible to divide and process the captured image data in order to reduce the processing load. Furthermore, when the inspection target is a structure formed from multiple types of components, it is believed that object detection processing can be performed separately for each type of component, rather than performing the object detection processing on the entire captured image data all at once, thereby enabling more accurate detection of minute defects.

[0005] The present disclosure aims to reduce the processing load when detecting defects and improve the detection accuracy.

[0006] A defect inspection device according to a first aspect of the present disclosure includes an image recognition processing unit that performs image recognition processing on photographed image data of an object to be inspected and determines the attributes of each region within the object to be inspected; a cutting unit that cuts out partial image data from the photographed image data, the partial image data including a region having a specific attribute from among the regions within the object to be inspected; a prediction unit that predicts the probability that the partial image data contains a defect by inputting the partial image data into a trained defect detection model; and a display control unit that superimposes and displays a marker in a display mode corresponding to the predicted probability at the position of the photographed image data where the partial image data has been cut out.

[0007] A second aspect of the present disclosure is the defect inspection device according to the first aspect, wherein the region having the specific attribute is a region having an attribute in which occurrence of a defect is predicted.

[0008] A third aspect of the present disclosure is a defect inspection device according to the first or second aspect, wherein the image recognition processing unit determines an area having a first attribute in which a defect is predicted to occur in the inspection object and an area having a second attribute in which a defect is not predicted to occur in the inspection object, and the cut-out unit calculates a position for cutting out partial image data including the area having the first attribute based on an arrangement of the areas having the second attribute.

[0009] A fourth aspect of the present disclosure is a defect inspection device according to the third aspect, wherein the areas having the second attribute are arranged in a grid pattern on the object to be inspected, and the cut-out unit generates a grid based on the center points of the areas having the second attribute, calculates the vertical midpoint positions of the grid points, the horizontal midpoint positions of the grid points, and the diagonal midpoint positions of the grid points as the cut-out positions, and cuts out a rectangular area from the captured image data centered on the calculated cut-out position as the partial image data.

[0010] A fifth aspect of the present disclosure is the defect inspection device according to the fourth aspect, wherein the cutout unit cuts out a square area of ​​the same size centered on the cutout position as the partial image data.

[0011] A sixth aspect of the present disclosure is a defect inspection device according to the fourth or fifth aspect, wherein the inspection object is a semiconductor package substrate, the area having the first attribute is a solder resist area, and the area having the second attribute is a solder joint area.

[0012] A seventh aspect of the present disclosure is a defect inspection device according to any one of the third to sixth aspects, wherein the image recognition processing unit determines the first attribute and the second attribute on a pixel-by-pixel basis using a semantic segmentation model.

[0013] An eighth aspect of the present disclosure is a defect inspection device described in any one of the first to seventh aspects, wherein the display control unit superimposes and displays a rectangular frame indicating the presence of a defect, the rectangular frame corresponding to the size of the cut-out partial image data, at a position where the partial image data where the predicted probability is equal to or greater than a predetermined threshold is cut out.

[0014] A ninth aspect of the present disclosure is a defect inspection device described in any one of the first to eighth aspects, wherein the display control unit displays the predicted probability and a rectangular frame corresponding to the size of the corresponding partial image data, superimposed on the position where the corresponding partial image data is cut out.

[0015] A tenth aspect of the present disclosure is a defect inspection device according to any one of the first to ninth aspects, comprising: an aggregation unit that determines the presence or absence of a defect based on a predicted probability for each of the partial image data and aggregates the results of the determination; and a calculation unit that performs statistical processing using the results of the determination and calculates statistical values, and the display control unit displays the aggregated aggregation results and the calculated statistical values ​​superimposed on the captured image data.

[0016] An eleventh aspect of the present disclosure is a defect inspection device described in any one of the first to tenth aspects, wherein the prediction unit predicts the probability that the partial image data contains a defect by inputting the partial image data into the trained defect detection model trained based on training data including the partial image data and information indicating the type of defect.

[0017] A defect inspection method according to a twelfth aspect of the present disclosure includes an image recognition processing step in which image recognition processing is performed on photographed image data of an object to be inspected to determine the attributes of each region within the object to be inspected; an extraction step in which partial image data including regions having specific attributes among the regions within the object to be inspected is extracted from the photographed image data; a prediction step in which the partial image data is input into a trained defect detection model to predict the probability that the partial image data contains a defect; and a display control step in which a marker having a display mode corresponding to the predicted probability is superimposed and displayed at the position of the photographed image data where the partial image data has been extracted.

[0018] A defect inspection program according to a thirteenth aspect of the present disclosure causes a computer to execute the following steps: an image recognition processing step of performing image recognition processing on photographed image data of an object to be inspected and determining the attributes of each region within the object to be inspected; an extraction step of extracting partial image data from the photographed image data, the partial image data including regions within the object to be inspected that have specific attributes; a prediction step of predicting the probability that the partial image data contains a defect by inputting the partial image data into a trained defect detection model; and a display control step of superimposing and displaying a marker in a display mode corresponding to the predicted probability at the position of the photographed image data where the partial image data has been extracted.

[0019] According to the present disclosure, it is possible to reduce the processing load when detecting defects and improve the detection accuracy.

[0020] FIG. 1 is a diagram showing an example of the system configuration of a defect inspection system. FIG. 2 is a diagram showing an example of an inspection target. FIG. 3 is a diagram showing an example of the hardware configuration of a learning device and a defect inspection device. FIG. 4 is a diagram showing an example of the functional configuration of the learning device. FIG. 5 is a diagram showing a specific example of processing by an image recognition processing unit. FIG. 6 is a diagram showing a specific example of processing by a point coordinate calculation unit. FIG. 7 is a diagram showing a specific example of processing by a partial image clipping unit. FIG. 8 is a diagram showing a specific example of processing by a learning data generation unit. FIG. 9 is a diagram showing a specific example of processing by a learning unit. FIG. 10 is an example of a flowchart showing the flow of learning processing by the learning device. FIG. 11 is a diagram showing an example of the functional configuration of a defect inspection device. FIG. 12 is a diagram showing a specific example of processing by a display control unit. FIG. 13 is an example of a flowchart showing the flow of defect inspection processing by the defect inspection device.

[0021] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configuration are designated by the same reference numerals, and redundant description will be omitted.

[0022] [First Embodiment] <System Configuration of Defect Inspection System> First, the system configuration of a defect inspection system including a defect inspection device according to a first embodiment will be described. Fig. 1 is a diagram showing an example of the system configuration of a defect inspection system. As shown in Fig. 1, the defect inspection system 100 includes an imaging device 110, a learning device 120, a defect inspection device 130, and a display device 140.

[0023] The imaging device 110 captures an image of an inspection object 160 placed on an inspection table 150 and transmits the captured image data to the learning device 120 or the defect inspection device 130 .

[0024] In the learning phase, the learning device 120 performs image recognition processing on the captured image data transmitted from the imaging device 110 and determines the attributes of each region within the inspection object 160 on a pixel-by-pixel basis. Furthermore, the learning device 120 extracts partial image data from the captured image data, including regions with specific attributes that predict the occurrence of defects, among the regions whose attributes have been determined. The learning device 120 also acquires, from the display device 140, inspection result information obtained when an inspector 170 performs a visual inspection of the captured image data displayed on the display device 140, and stores the information as learning data in association with the partial image data. Furthermore, the learning device 120 performs a learning process on the defect detection model using the learning data to generate a trained defect detection model.

[0025] In the inspection phase, the defect inspection device 130 performs image recognition processing on the captured image data transmitted from the imaging device 110, and determines the attribute of each region in the inspection object 160 on a pixel-by-pixel basis. In the first embodiment, among multiple types of members that form the inspection object 160, regions of the same type of member are determined to have the same attribute.

[0026] The defect inspection device 130 also extracts partial image data from the captured image data, including areas with specific attributes that predict the occurrence of defects among the areas whose attributes have been determined. The defect inspection device 130 also inputs the extracted partial image data into a trained defect detection model to predict the probability that the partial image data contains a defect. The defect inspection device 130 then superimposes a marker in a display mode corresponding to the predicted probability at the position where the partial image data was extracted from the captured image data, and transmits the superimposed marker to the display device 140.

[0027] In the learning phase, display device 140 acquires image data captured by imaging device 110 via defect inspection device 130 and displays the data to inspector 170. This allows inspector 170 to perform a visual inspection and transmit inspection result information to learning device 120.

[0028] Furthermore, in the inspection phase, display device 140 receives from defect inspection device 130 the captured image data in which markers in a display mode according to the predicted probability are superimposed at the respective positions where the partial image data has been cut out, and displays the data to inspector 170. This allows inspector 170 to perform a visual inspection, for example, by focusing on partial image data whose predicted probability is equal to or greater than a predetermined threshold.

[0029] In this way, the defect inspection device 130 according to the first embodiment extracts partial image data from the captured image data and predicts the probability of defects in the extracted partial image data. As a result, according to the first embodiment, the processing load on the defect inspection device 130 can be reduced compared to when batch processing is performed on the entire captured image data.

[0030] Furthermore, the defect inspection device 130 according to the first embodiment performs image recognition processing to determine the attributes of each region and extracts regions having specific attributes that predict the occurrence of defects as partial image data. As a result, according to the first embodiment, even if the inspection target is a structure formed of multiple members, it is possible to extract regions of members that predict the occurrence of defects as partial image data.

[0031] In other words, regions of components where defects are not predicted to occur are excluded from the prediction target, and therefore the processing load on the defect inspection device 130 can be further reduced compared to when all regions are extracted as partial image data. Furthermore, the extracted partial image data contains a large proportion of specific attributes that are predicted to cause defects, and therefore the accuracy of detecting minute defects can be improved compared to when minute defects are detected from captured image data that includes regions of multiple types of components.

[0032] <Example of Inspection Object> Next, a specific example of the inspection object 160 inspected by the defect inspection device 130 according to the first embodiment will be described. FIG. 2 is a diagram showing an example of the inspection object. As shown in FIG. 2, the inspection object 160 is a structure formed by a base 210 and a plurality of members 220. More specifically, the inspection object 160 is a semiconductor package substrate, and the example in FIG. 2 shows the semiconductor package substrate as viewed from above. Therefore, the base 210 is a solder resist, and the plurality of members 220 are solder joints.

[0033] As shown in FIG. 2, when the inspection object 160 is a semiconductor package substrate, the solder joints, which are multiple members 220, have roughly the same size and shape and are arranged at equal intervals in a grid pattern on the solder resist, which is the base 210.

[0034] In the first embodiment, it is predicted that minute defects will occur in the region of the base 210 of the inspection object 160. Furthermore, the imaging device 110 has a resolution sufficient to detect the minute defects when they occur. Therefore, the amount of captured image data obtained by capturing an image of the entire inspection object 160 is large, and attempting to process the captured image data in a batch increases the processing load on the defect inspection device 130.

[0035] Therefore, as described above, the defect inspection device 130 according to the first embodiment extracts partial image data from the captured image data. In this case, the defect inspection device 130 according to the first embodiment extracts the regions of the base 210 located between the adjacent members 220 as partial image data.

[0036] <Hardware Configuration of Learning Device and Defect Inspection Device> Next, we will explain the hardware configurations of the learning device 120 and the defect inspection device 130. Note that the hardware configuration of the learning device 120 and the hardware configuration of the defect inspection device 130 are generally the same, so they will be explained together here using FIG. 3.

[0037] 3 is a diagram showing an example of the hardware configuration of the learning device and the defect inspection device. As shown in Fig. 3, the learning device 120 and the defect inspection device 130 have a processor 301, a memory 302, an auxiliary storage device 303, an input / output device 304, a communication device 305, and a drive device 306. The hardware components of the learning device 120 and the defect inspection device 130 are connected to each other via a bus 307.

[0038] The processor 301 has various arithmetic devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 301 reads various programs into the memory 302 and executes them. The various programs here include, for example, a learning program or a defect inspection program, which will be described later.

[0039] The memory 302 includes a main storage device such as a read-only memory (ROM) or a random access memory (RAM). The processor 301 and the memory 302 form a so-called computer. The computer realizes various functions by the processor 301 executing various programs read onto the memory 302.

[0040] The auxiliary storage device 303 stores various programs and various data used when the various programs are executed by the processor 301. For example, a training data storage unit 411, a trained model storage unit 412, and a trained model storage unit 1111, which will be described later, are realized in the auxiliary storage device 303.

[0041] The input / output device 304 is a connection device that connects to an operation device 311, which is an example of a user interface device, and a display device 312. The communication device 305 is a communication device for communicating with an external device 320 via a network. The external device 320 here includes the imaging device 110, the display device 140, etc.

[0042] The drive device 306 is a device for loading a recording medium 330. The recording medium 330 here includes media that optically, electrically, or magnetically record information, such as CD-ROMs, flexible disks, and magneto-optical disks. The recording medium 330 may also include media such as semiconductor memories that electrically record information, such as ROMs and flash memories.

[0043] The various programs to be installed in the auxiliary storage device 303 are installed, for example, by setting the distributed recording medium 330 in the drive device 306 and reading out the various programs recorded on the recording medium 330. Alternatively, the various programs to be installed in the auxiliary storage device 303 may be installed by being downloaded from a network via the communication device 305.

[0044] <Functional Configuration of Learning Device> Next, the functional configuration of the learning device 120 will be described. Fig. 4 is a diagram showing an example of the functional configuration of the learning device. As described above, a learning program is installed in the learning device 120. By executing the learning program, the learning device 120 functions as an image data acquisition unit 401, an image recognition processing unit 402, a point coordinate calculation unit 403, a partial image clipping unit 404, an inspection result information acquisition unit 405, a learning data generation unit 406, and a learning unit 407.

[0045] The image data acquisition unit 401 acquires captured image data from the imaging device 110. The image data acquisition unit 401 notifies the image recognition processing unit 402 and the partial image clipping unit 404 of the acquired captured image data.

[0046] The image recognition processing unit 402 performs image recognition processing on the captured image data to determine the attributes of each region included in the captured image data on a pixel-by-pixel basis. In the first embodiment, the image recognition processing unit 402 inputs the captured image data into, for example, a semantic segmentation model, thereby recognizing the region of the base 210 and the region of the member 220 on a pixel-by-pixel basis. Note that the semantic segmentation model used by the image recognition processing unit 402 is, for example, "DeepLabV3" or the like.

[0047] The point coordinate calculation unit 403 calculates a grid vector from the captured image data in which attributes are determined pixel by pixel. The grid vector is a vector for generating a grid that specifies the position coordinates of the components 220 arranged in a grid pattern. The point coordinate calculation unit 403 determines the start position and direction of the grid vector so that the sum of the distances between the grid points and the center points of the components 220 is minimized. By determining the grid vector in this way, even if one of the components 220 cannot be recognized during image recognition processing, a grid can be generated as long as the other components 220 can be recognized. This allows the point coordinate calculation unit 403 to appropriately calculate the position coordinates of all the components 220.

[0048] The point coordinate calculation unit 403 generates a grid based on the determined grid vector, calculates the position coordinates of the midpoint between adjacent grid points in the generated grid, and notifies the partial image clipping unit 404 of the calculated position coordinates of the midpoint.

[0049] The partial image cropping unit 404 specifies a rectangular area centered on the position coordinates of the midpoint notified by the point coordinate calculation unit 403 in the captured image data notified by the image data acquisition unit 401, and crops it as partial image data. The shape and size of the rectangular area cropped by the partial image cropping unit 404 as partial image data are assumed to be determined in advance.

[0050] The partial image data cut out by the partial image cutout unit 404 is sequentially notified to the learning data generation unit 406 .

[0051] The test result information acquisition unit 405 acquires test result information from the display device 140 and notifies the learning data generation unit 406 of the information.

[0052] The learning data generation unit 406 generates learning data by associating the partial image data notified by the partial image clipping unit 404 with the inspection result information notified by the inspection result information acquisition unit 405, and stores the learning data in the learning data storage unit 411. The inspection result information notified by the inspection result information acquisition unit 405 includes the type of defect and the position coordinates of the defect. The learning data generation unit 406 generates learning data by associating the partial image data corresponding to the position coordinates of the defect with the type of defect.

[0053] The learning unit 407 performs a learning process on the defect detection model using the learning data stored in the learning data storage unit 411, and generates a trained defect detection model. The learning unit 407 stores the generated trained defect detection model in the trained model storage unit 412.

[0054] <Specific example of processing by each unit of the learning device> Next, a specific example of processing by each unit of the learning device 120 will be described. Here, specific examples of processing by the image recognition processing unit 402, the point coordinate calculation unit 403, the partial image clipping unit 404, the learning data generation unit 406, and the learning unit 407 will be described as processing by each unit of the learning device 120.

[0055] (1) Specific Example of Processing by the Image Recognition Processing Unit 402 First, a specific example of processing by the image recognition processing unit 402 will be described. Fig. 5 is a diagram showing a specific example of processing by the image recognition processing unit. In Fig. 5, reference numeral 510 indicates captured image data. However, due to space limitations, the example in Fig. 5 shows only a partial area of ​​the captured image data obtained by capturing an image of the inspection object 160.

[0056] As described above, the image recognition processing unit 402 performs image recognition processing on the captured image data indicated by the reference numeral 510, and determines the area of ​​the base 210 and the area of ​​the member 220 on a pixel-by-pixel basis.

[0057] 5, reference numeral 520 indicates the image recognition processing result obtained by the image recognition processing unit 402 determining the area of ​​the base 210 and the area of ​​the member 220 on a pixel-by-pixel basis. In the example of reference numeral 520, pixels determined by the image recognition processing unit 402 to be the area of ​​the base 210 are indicated in white. In addition, in the example of reference numeral 520, pixels determined by the image recognition processing unit 402 to be the area of ​​the member 220 are indicated in black.

[0058] The example of the image recognition processing result indicated by the reference numeral 520 shows that some of the multiple members 220 were not recognized.

[0059] (2) Specific Example of Processing by Point Coordinate Calculation Unit 403 Next, a specific example of processing by the point coordinate calculation unit 403 will be described. Fig. 6 is a diagram showing a specific example of processing by the point coordinate calculation unit. As shown in Fig. 6, when the image recognition processing result indicated by reference numeral 520 is input, the point coordinate calculation unit 403 determines a lattice vector 611 as indicated by reference numeral 610.

[0060] As described above, the point coordinate calculation unit 403 determines the start position and direction of the lattice vector 611 so as to minimize the sum of the distances between the lattice points and the center points of the components 220. The solid lines shown within the reference numeral 610 indicate a lattice generated based on the determined lattice vector 611, etc. By generating a lattice in this way, even if some areas of the multiple components 220 are not recognized, the point coordinate calculation unit 403 can properly calculate the position coordinates of all of the components 220.

[0061] As described above, the point coordinate calculation unit 403 calculates the position coordinates of the midpoints between adjacent grid points in the generated grid. In Fig. 6, the black circles in the figure 620 indicate that the point coordinate calculation unit 403 has calculated: the position coordinates of the midpoints between vertically adjacent grid points, the position coordinates of the midpoints between horizontally adjacent grid points, and the position coordinates of the midpoints between diagonally adjacent grid points.

[0062] (3) Specific Example of Processing by the Partial Image Cutout Unit 404 Next, a specific example of processing by the partial image cutout unit 404 will be described. Fig. 7 is a diagram showing a specific example of processing by the partial image cutout unit. As shown in Fig. 7, when captured image data indicated by reference numeral 510 and the position coordinates of the midpoint indicated by reference numeral 620 are input, the partial image cutout unit 404 cuts out the partial image data as indicated by reference numeral 710.

[0063] As described above, the partial image cropping unit 404 crops a rectangular area centered on the position coordinates of the midpoint from the captured image data as partial image data. Indicated by reference numeral 710, partial image data 711 shows a rectangular area cropped from the captured image data, the area being centered on the position coordinates of the midpoint 621. Similarly, indicated by reference numeral 710, partial image data 712 shows a rectangular area cropped from the captured image data, the area being centered on the position coordinates of the midpoint 622.

[0064] 7, the rectangular regions are square, and the sizes of the extracted rectangular regions are the same. This is to facilitate the learning process when inputting the rectangular regions into a defect detection model. Also, by rotating the extracted partial image data by 90 degrees or 180 degrees, it is possible to store the partial image data as different partial image data from the partial image data before rotation in the learning data, thereby increasing the amount of learning data.

[0065] (4) Specific Example of Processing by the Learning Data Generator 406 Next, a description will be given of a specific example of processing by the learning data generator 406. Fig. 8 is a diagram showing a specific example of processing by the learning data generator.

[0066] 8, reference numeral 810 denotes an example of inspection result information transmitted from the display device 140. As indicated by reference numeral 810, the inspection result information includes information items such as a "defect identifier," "defect position coordinates," and "defect type."

[0067] "Defect identifier" stores an identifier for identifying a defect detected by visual inspection by inspector 170. "Defect position coordinates" stores information calculated by inspector 170 inputting the circumscribing rectangle of the defect, and information for specifying the position and size of the circumscribing rectangle. Specifically, the position coordinates of the upper left and lower right of the circumscribing rectangle are stored. "Defect type" stores the type of defect determined by visual inspection by inspector 170.

[0068] 8, when the inspection result information indicated by the reference numeral 810 and the partial image data 711, 712, etc. indicated by the reference numeral 710 are input, the learning data generation unit 406 generates learning data 820. The learning data 820 includes the following information items: "partial image identifier," "partial image," "defect identifier," "defect position coordinates," and "defect type."

[0069] An identifier for identifying partial image data is stored in "Partial image identifier." Partial image data is stored in "Partial image." Corresponding inspection result information is stored in "Defect identifier," "Defect position coordinates," and "Defect type."

[0070] In the example of Fig. 8, the partial image data with "partial image identifier" = "IM1" has no defect, and therefore no corresponding inspection result information. Therefore, the "defect identifier" and "defect position coordinates" are left blank, and "no defect" is stored in the "defect type".

[0071] On the other hand, the partial image data with "partial image identifier" = "IM20" has a defect and has corresponding inspection result information. Therefore, the "defect identifier", "defect position coordinates", and "defect type" are "ID1", "(x １１ , y １１ ), (x １２ , y １２ ) and "defect a" are stored. Similarly, the partial image data with "partial image identifier" = "IM32" has a defect, and there is corresponding inspection result information. Therefore, "defect identifier", "defect position coordinates", and "defect type" are stored as "ID2", "(x ２１ , y ２１ ), (x２２ , y ２２ ) and "Defect a" are stored.

[0072] The learning data generation unit 406 associates the partial image data with the inspection result information based on the relationship between the position coordinates of the extracted partial image data and the position coordinates stored in the "defect position coordinates" of the inspection result information. The learning data generation unit 406 stores the generated learning data 820 in the learning data storage unit 411.

[0073] (5) Specific Example of Processing by the Learning Unit 407 Next, a specific example of processing by the learning unit 407 will be described. Fig. 9 is a diagram showing a specific example of processing by the learning unit. As shown in Fig. 9, the learning unit 407 includes a defect detection model 901 and a comparison / modification unit 902. The learning unit 407 performs a learning process on the defect detection model 901 using the learning data 820 stored in the learning data storage unit 411.

[0074] Specifically, the learning unit 407 inputs each piece of partial image data stored in the "partial image" of the learning data 820 to the defect detection model 901. As a result, the defect detection model 901 outputs the probability that each piece of partial image data has a defect as output data. The learning unit 407 also inputs the type of defect stored in the "defect type" of the learning data 820 to the comparison and modification unit 902. As a result, the comparison and modification unit 902 compares the output data output from the defect detection model 901 with the type of defect stored in the "defect type."

[0075] For example, in the case of partial image data with a "partial image identifier" = "IM1", the comparison modification unit 902 compares the output data output from the defect detection model 901 with "0%", which indicates "no defect", which is the defect type stored in the "defect type", and calculates an error. Similarly, in the case of partial image data with a "partial image identifier" = "IM20", the comparison modification unit 902 compares the output data output from the defect detection model 901 with "100%", which indicates "defect a", which is the defect type stored in the "defect type", and calculates an error. Similarly, in the case of partial image data with a "partial image identifier" = "IM32", the comparison modification unit 902 compares the output data output from the defect detection model 901 with "100%", which indicates "defect a", which is the defect type stored in the "defect type", and calculates an error.

[0076] Furthermore, the comparison and modification unit 902 updates the model parameters of the defect detection model 901 in accordance with the calculated error. As a result, the learning unit 407 can generate a learned defect detection model.

[0077] The learning unit 407 stores the generated trained defect detection model in the trained model storage unit 412.

[0078] <Learning Process Flow by Learning Device> Next, a description will be given of the learning process flow by the learning device 120. Fig. 10 is an example of a flowchart showing the learning process flow by the learning device. The flowchart shown in Fig. 10 is executed in the learning phase.

[0079] In step S1001 , the learning device 120 acquires captured image data captured by the imaging device 110 .

[0080] In step S1002, the learning device 120 acquires test result information from the display device 140.

[0081] In step S1003, the learning device 120 performs image recognition processing on the captured image data and determines the attributes of each region on a pixel-by-pixel basis.

[0082] In step S1004, the learning device 120 determines a lattice vector based on the position coordinates of the center point of the member 220.

[0083] In step S1005, the learning device 120 generates a grid based on the determined grid vector, and calculates the position coordinates of the midpoint between adjacent grid points.

[0084] In step S1006, the learning device 120 inputs "1" to counter i, which counts the midpoints whose position coordinates have been calculated.

[0085] In step S1007, the learning device 120 cuts out a rectangular area having the position coordinates of the i-th midpoint as the partial image data.

[0086] In step S1008, the learning device 120 determines whether or not there is test result information corresponding to the partial image data. If it is determined in step S1008 that there is corresponding test result information (YES in step S1008), the process proceeds to step S1009.

[0087] In step S1009, the learning device 120 stores the partial image data and the test result information in association with each other in the learning data.

[0088] On the other hand, if it is determined in step S1008 that there is no corresponding test result information (NO in step S1008), the process proceeds to step S1010.

[0089] In step S1010, the learning device 120 associates the partial image data with "no defect" and stores it in the learning data.

[0090] In step S1011, the learning device 120 determines whether the processes in steps S1007 to S1010 have been performed for all partial image data extracted from the captured image data. If it is determined in step S1011 that there is partial image data for which processing has not been performed (NO in step S1011), the learning device 120 proceeds to step S1012.

[0091] In step S1012, the learning device 120 increments the counter i and returns to step S1007.

[0092] On the other hand, if it is determined in step S1011 that processing has been performed on all partial image data (YES in step S1011), the process proceeds to step S1013.

[0093] In step S1013, the learning device 120 reads the learning data, performs a learning process on the defect detection model, and generates a trained defect detection model.

[0094] In step S1014, the learning device 120 stores the generated trained defect detection model in the trained model storage unit 412.

[0095] <Functional Configuration of Defect Inspection Apparatus> Next, the functional configuration of the defect inspection apparatus 130 will be described. Fig. 11 is a diagram showing an example of the functional configuration of the defect inspection apparatus. As described above, a defect inspection program is installed in the defect inspection apparatus 130. By executing the defect inspection program, the defect inspection apparatus 130 functions as an image data acquisition unit 1101, an image recognition processing unit 1102, a point coordinate calculation unit 1103, a partial image clipping unit 1104, a prediction unit 1105, a counting unit 1106, and a display control unit 1107.

[0096] The image data acquisition unit 1101 acquires captured image data from the imaging device 110. The image data acquisition unit 1101 notifies the image recognition processing unit 1102 and the partial image clipping unit 1104 of the acquired captured image data.

[0097] The image recognition processing unit 1102 performs image recognition processing on the captured image data to determine the attributes of each region included in the captured image data on a pixel-by-pixel basis. In the first embodiment, the image recognition processing unit 1102 inputs the captured image data into, for example, a semantic segmentation model, thereby recognizing the region of the base 210 and the region of the member 220 on a pixel-by-pixel basis. Note that the semantic segmentation model used by the image recognition processing unit 1102 is, for example, "DeepLabV3" or the like.

[0098] The point coordinate calculation unit 1103 calculates a grid vector from the captured image data in which attributes are determined pixel by pixel. The grid vector is a vector for generating a grid that specifies the position coordinates of the components 220 arranged in a grid pattern. The point coordinate calculation unit 1103 determines the starting position and direction of the grid vector so that the sum of the distances between the grid points and the center points of the components 220 is minimized. By determining the grid vector in this way, even if one of the components 220 cannot be recognized during image recognition processing, a grid can be generated as long as the other components 220 can be recognized. This allows the point coordinate calculation unit 403 to appropriately calculate the position coordinates of all the components 220.

[0099] The point coordinate calculation unit 1103 generates a grid based on the determined grid vector, calculates the position coordinates of the midpoint between adjacent grid points in the generated grid, and notifies the partial image clipping unit 1104 of the calculated position coordinates of the midpoint.

[0100] The partial image cropping unit 1104 specifies a rectangular area centered on the position coordinates of the midpoint notified by the point coordinate calculation unit 1103 in the captured image data notified by the image data acquisition unit 1101, and crops it as partial image data. The shape and size of the rectangular area cropped by the partial image cropping unit 1104 as partial image data are assumed to be determined in advance.

[0101] The partial image data cut out by the partial image cutout unit 1104 is input to the prediction unit 1105 in sequence.

[0102] The prediction unit 1105 predicts the probability that the partial image data has a defect by inputting the partial image data into a trained defect detection model read from the trained model storage unit 1111. The prediction unit 1105 also notifies the counting unit 1106 and the display control unit 1107 of the predicted probability.

[0103] The counting unit 1106 acquires the probability that the partial image data has a defect from the prediction unit 1105 and determines whether the acquired probability is equal to or greater than a predetermined threshold. If the counting unit 1106 determines that the probability is equal to or greater than the predetermined threshold, it determines that the corresponding partial image data has a defect. On the other hand, if the counting unit 1106 determines that the probability is less than the predetermined threshold, it determines that the corresponding partial image data does not have a defect.

[0104] The counting unit 1106 determines whether or not there is a defect in all partial image data extracted from the captured image data, counts the determination results, performs statistical processing, and calculates statistical values. The counting unit 1106 also notifies the display control unit 1107 of the counting results and statistical values.

[0105] The display control unit 1107 acquires the probability that the partial image data has a defect from the prediction unit 1105, and superimposes a marker in a display mode corresponding to the acquired probability at the position of the corresponding partial image data in the captured image data. The display control unit 1107 also superimposes the counting results and statistical values ​​notified by the counting unit 1106 on the captured image data.

[0106] Furthermore, the display control unit 1107 controls the display device 140 to display the captured image data on which the sign, the counting result, and the statistical value are superimposed.

[0107] <Specific example of processing by each unit of the defect inspection device> Next, a specific example of processing by each unit of the defect inspection device 130 will be described. Note that, among the processing by each unit of the defect inspection device 130, the processing from the image data acquisition unit 1101 to the partial image cutout unit 1104 is similar to the processing from the image data acquisition unit 401 to the partial image cutout unit 404 of the learning device 120, and therefore a description thereof will be omitted here. Here, only a specific example of processing by the display control unit 1107 will be described.

[0108] Fig. 12 is a diagram showing a specific example of processing by the display control unit. In Fig. 12, reference numeral 1210 indicates a state in which the display control unit 1107 displays a marker in a display mode according to the probability of a defect at the position of the corresponding partial image data in the captured image data.

[0109] The example of reference numeral 1210 shows a state in which a thick solid rectangular frame is displayed as an indicator at the cut-out position of partial image data that has been determined to have a defect because the probability of a defect is equal to or greater than a predetermined threshold. According to the display mode shown by reference numeral 1210, by referring to the thick solid rectangular frame 1211 to the thick solid rectangular frame 1214, the inspector 170 can easily grasp the position in the captured image data that has been determined to have a defect.

[0110] The example of reference numeral 1220 shows a state in which the predicted probability for each partial image data is displayed at the position where each partial image data is cut out. The example of reference numeral 1220 also shows a state in which a rectangular frame corresponding to the predicted probability for each partial image data is displayed at the position where each partial image data is cut out.

[0111] The example of reference numeral 1220 indicates that when the probability of a defect is equal to or less than the first threshold, a dotted rectangular frame and the probability of a defect are displayed at the position where the corresponding partial image data is cut out. The example of reference numeral 1220 also indicates that when the probability of a defect is greater than the first threshold and equal to or less than the second threshold, a thin solid rectangular frame and the probability of a defect are displayed at the position where the corresponding partial image data is cut out. Furthermore, the example of reference numeral 1220 indicates that when the probability of a defect is greater than the second threshold, a thick solid rectangular frame and the probability of a defect are displayed at the position where the corresponding partial image data is cut out.

[0112] According to the display mode of the reference numeral 1220, by referring to each rectangular frame, it is possible to easily understand what predictions the trained defect detection model has made about the partial image data.

[0113] In the example of reference numeral 1220, due to space limitations, a rectangular frame is displayed in a display mode corresponding to the probability of a defect for some of the extracted partial image data. However, a rectangular frame may be similarly displayed in a display mode corresponding to the probability of a defect for other partial image data.

[0114] In the example of reference numeral 1220, the display mode according to the probability of a defect is changed by changing the line type and the line thickness of the rectangular frame, but the display mode according to the probability of a defect is not limited to this. For example, the color, shape, and size of the rectangular frame may be changed.

[0115] Furthermore, in the example of reference numeral 1220, the display of the counting results and statistical values ​​is omitted due to space limitations, but the counting results and statistical values ​​may be displayed at any position.

[0116] <Flow of Defect Inspection Process by Defect Inspection Apparatus> Next, a flow of defect inspection process by the defect inspection apparatus 130 will be described. Fig. 13 is an example of a flowchart showing the flow of defect inspection process by the defect inspection apparatus. The flowchart shown in Fig. 13 is executed in the inspection phase.

[0117] In step S1301 , the defect inspection device 130 acquires captured image data captured by the imaging device 110 .

[0118] In step S1302, the defect inspection device 130 performs image recognition processing on the captured image data and determines the attribute of each region on a pixel-by-pixel basis.

[0119] In step S1303, the defect inspection device 130 determines a lattice vector based on the position coordinates of the center point of the component.

[0120] In step S1304, the defect inspection device 130 generates a grid based on the determined grid vector, and calculates the position coordinates of the midpoint between adjacent grid points.

[0121] In step S1305, the defect inspection device 130 inputs "1" to a counter i that counts the midpoints whose position coordinates have been calculated.

[0122] In step S1306, the defect inspection device 130 cuts out a rectangular area having the position coordinates of the i-th midpoint as the center as partial image data.

[0123] In step S1307, the defect inspection device 130 inputs the partial image data into the trained defect detection model and predicts the probability that the partial image data contains a defect.

[0124] In step S1308, the defect inspection device 130 determines whether the probability of defects has been predicted for all partial image data.

[0125] In step S1308, if it is determined that there is partial image data for which the probability of a defect has not been predicted (NO in step S1308), the process proceeds to step S1309.

[0126] In step S1309, the defect inspection device 130 increments the counter i, and the process returns to step S1306.

[0127] On the other hand, if it is determined in step S1308 that the probability of defects has been predicted for all partial image data (YES in step S1308), the process proceeds to step S1310.

[0128] In step S1310, the defect inspection device 130 determines whether or not there is a defect in the partial image data based on the predicted probability. The defect inspection device 130 also tallies and statistically processes the determination results to calculate statistical values.

[0129] In step S1311, the defect inspection device 130 superimposes, on the captured image data, the markers, the counting results, and the statistical values ​​according to the probabilities.

[0130] In step S1312, the defect inspection device 130 displays on the display device 140 the captured image data on which the markers, the counting results, and the statistical values ​​corresponding to the probabilities are superimposed.

[0131] <Summary> As is clear from the above explanation, the defect inspection device 130 according to the first embodiment: - performs image recognition processing on photographed image data of the inspection object 160, and determines the attributes of each region in the inspection object 160 on a pixel-by-pixel basis; - cuts out, from the photographed image data, partial image data that includes a region of the base 210 where the occurrence of a defect is predicted, from each region in the inspection object 160; - predicts the probability that the partial image data contains a defect by inputting the partial image data into a trained defect detection model; - superimposes and displays a marker in a display format according to the predicted probability at the position where the partial image data was cut out in the photographed image data of the inspection object.

[0132] In this way, the defect inspection device 130 according to the first embodiment extracts partial image data from the captured image data and predicts the probability of defects in the extracted partial image data. As a result, according to the first embodiment, the processing load on the defect inspection device 130 can be reduced compared to when batch processing is performed on the entire captured image data.

[0133] Furthermore, the defect inspection device 130 according to the first embodiment performs image recognition processing to determine the attributes of each region and extracts regions having specific attributes that predict the occurrence of defects as partial image data. As a result, according to the first embodiment, even if the inspection target is a structure formed of multiple members, it is possible to extract regions of members that predict the occurrence of defects as partial image data.

[0134] In other words, regions of components where defects are not predicted to occur are excluded from the prediction target, and therefore the processing load on the defect inspection device 130 can be further reduced compared to when all regions are extracted as partial image data. Furthermore, the extracted partial image data contains a large proportion of specific attributes that are predicted to cause defects, and therefore the accuracy of detecting minute defects can be improved compared to when minute defects are detected from captured image data that includes regions of multiple types of components.

[0135] Second Embodiment In the above-described first embodiment, a case has been described in which the defect inspection device 130 performs defect inspection processing on captured image data obtained by capturing an image of a semiconductor package substrate to be inspected from above. For this reason, in the above-described first embodiment, it has been described that an area having a specific attribute (first attribute) in which a defect is predicted to occur is a solder resist area, and an area having an attribute (second attribute) in which a defect is not predicted to occur is a solder joint area.

[0136] However, this is not limited to the photographed image data acquired by the defect inspection device 130. Furthermore, the method of cutting out partial image data by the defect inspection device 130 is not limited to this.

[0137] For example, defect inspection processing may be performed by acquiring photographed image data obtained by photographing the semiconductor package substrate to be inspected from the side. In the case of photographed image data obtained by photographing the semiconductor package substrate from the side, multiple layers of glass fiber impregnated with resin are included. Therefore, the image recognition processing unit 402, 1102, for example, determines the first layer region as a region having a first attribute, and determines the second layer region as another region having a first attribute. Furthermore, the cutout unit, for example, cuts out partial image data including the first layer region and partial image data including the second layer region. In this case, the region other than the layer regions is the region having the second attribute.

[0138] That is, areas determined by the image recognition processing units 402, 1102 to have a specific attribute that predicts the occurrence of a defect include: areas of a specific type of component as in the first embodiment, and areas of a specific group of components made up of multiple types of components as in the second embodiment. Specifically, areas with a specific attribute include areas where the type of component is one specific type, and areas where multiple types of components are included but the combination of component type and arrangement is one specific type.

[0139] In this way, by configuring the second embodiment to extract partial image data for each area that contains multiple types of components but has a specific combination of component types and arrangements, it is possible to achieve the same effects as in the first embodiment.

[0140] [Third Embodiment] In the first embodiment, the case where there is one type of defect has been described, but there may be multiple types of defects. When there are multiple types of defects, the display control unit 1107 may be configured to display each type of defect in a different display mode. Furthermore, the counting unit 1106 may be configured to count the determination results for each type of defect.

[0141] Although the first embodiment does not mention details of the method for counting the determination results or the method for statistical processing, the counting unit 1106 may, for example, count the number of partial image data determined to have a defect. Alternatively, if partial image data determined to have a defect are consecutive, the counting unit 1106 may count the number of consecutive partial image data. Alternatively, the counting unit 1106 may calculate the proportion of partial image data determined to have a defect by dividing the number of partial image data determined to have a defect by the total number of partial image data extracted from the captured image data. Alternatively, the counting unit 1106 may generate a histogram based on the probability of defect.

[0142] In the first embodiment, a grid is generated by determining a grid vector, but the grid generation method is not limited to this. In the first embodiment, a rectangular area having a center at the position coordinate of the midpoint of the generated grid is cut out as partial image data, but the method of cutting out partial image data is not limited to this.

[0143] In addition, in the first embodiment described above, both partial image data determined to have defects and partial image data determined to have no defects are used as learning data, but it is also possible to use only one of the partial image data as learning data.

[0144] Furthermore, in the first embodiment described above, the learning device 120 and the defect inspection device 130 are configured as separate devices, but the learning device 120 and the defect inspection device 130 may also be configured as an integrated device.

[0145] The present disclosure is not limited to the configurations described in the above embodiments, and may be combined with other elements, etc. These aspects may be changed without departing from the spirit of the present disclosure, and may be appropriately determined depending on the application form.

[0146] This application claims priority based on Japanese Patent Application No. 2023-108434, filed on June 30, 2023, the entire contents of which are incorporated herein by reference.

[0147] 100: Defect inspection system 110: Imaging device 120: Learning device 130: Defect inspection device 140: Display device 210: Base 220: Member 401: Image data acquisition unit 402: Image recognition processing unit 403: Point coordinate calculation unit 404: Partial image clipping unit 405: Inspection result information acquisition unit 406: Learning data generation unit 407: Learning unit 611: Lattice vector 621, 622: Midpoint 711, 712: Partial image data 820: Learning data 901: Defect detection model 1101: Image data acquisition unit 1102: Image recognition processing unit 1103: Point coordinate calculation unit 1104: Partial image clipping unit 1105: Prediction unit 1106: Aggregation unit 1107: Display control unit

Claims

1. A defect inspection device having an image recognition processing unit that performs image recognition processing on captured image data of an inspection object and determines the attributes of each area within the inspection object; an extraction unit that extracts partial image data from the captured image data, the partial image data including an area having a specific attribute from among each area within the inspection object; a prediction unit that predicts the probability that the partial image data contains a defect by inputting the partial image data into a trained defect detection model; and a display control unit that superimposes and displays a sign in a display mode corresponding to the predicted probability at the position where the partial image data of the captured image data is extracted.

2. The defect inspection device according to claim 1, wherein the area having the specific attribute is an area having an attribute in which occurrence of a defect is predicted.

3. A defect inspection device as described in claim 1 or 2, wherein the image recognition processing unit determines areas having a first attribute in which the occurrence of a defect is predicted in the inspection object and areas having a second attribute in which the occurrence of a defect is not predicted in the inspection object, and the cut-out unit calculates a position for cutting out partial image data including the areas having the first attribute based on an arrangement of the areas having the second attribute.

4. The defect inspection device described in claim 3, wherein the areas having the second attribute are arranged in a lattice pattern in the object to be inspected, and the cut-out unit generates a lattice based on center points of the areas having the second attribute, calculates the vertical midpoint position of the lattice points, the horizontal midpoint position of the lattice points, and the diagonal midpoint position of the lattice points as the cut-out positions, and cuts out a rectangular area from the captured image data centered on the calculated cut-out position as the partial image data.

5. The defect inspection device according to claim 4, wherein said cut-out section cuts out a square area of ​​the same size centered on said cut-out position as said partial image data.

6. The defect inspection device according to claim 4 or 5, wherein the inspection object is a semiconductor package substrate, the area having the first attribute is a solder resist area, and the area having the second attribute is a solder joint area.

7. The defect inspection device according to any one of claims 3 to 6, wherein the image recognition processing unit determines the first attribute and the second attribute on a pixel-by-pixel basis using a semantic segmentation model.

8. A defect inspection device as described in any one of claims 1 to 7, wherein the display control unit superimposes a rectangular frame indicating the presence of a defect, the rectangular frame corresponding to the size of the cut-out partial image data, at a position where the partial image data where the predicted probability is equal to or greater than a predetermined threshold is cut out.

9. A defect inspection device as described in any one of claims 1 to 8, wherein the display control unit displays the predicted probability and a rectangular frame according to the size of the corresponding partial image data by superimposing it on the position where the corresponding partial image data is cut out.

10. A defect inspection device as described in any one of claims 1 to 9, comprising: a counting unit that judges the presence or absence of a defect based on the probability predicted for each of the partial image data and counts the judgment results; and a calculation unit that performs statistical processing using the judgment results and calculates statistical values, wherein the display control unit displays the counted result and the calculated statistical values ​​by superimposing them on the captured image data.

11. A defect inspection device as described in any one of claims 1 to 10, wherein the prediction unit predicts the probability that the partial image data contains a defect by inputting the partial image data into the trained defect detection model trained based on learning data including the partial image data and information indicating the type of defect.

12. A defect inspection method in which a computer executes the following steps: an image recognition processing step of performing image recognition processing on photographed image data of an object to be inspected and determining the attributes of each area within the object to be inspected; an extraction step of extracting partial image data from the photographed image data, the partial image data including an area having a specific attribute from among each area within the object to be inspected; a prediction step of predicting the probability that the partial image data contains a defect by inputting the partial image data into a trained defect detection model; and a display control step of superimposing and displaying a sign having a display mode according to the predicted probability at the position where the partial image data of the photographed image data is extracted.

13. A defect inspection program for causing a computer to execute the following steps: an image recognition processing step for performing image recognition processing on photographed image data of an object to be inspected and determining the attributes of each area within the object to be inspected; an extraction step for extracting partial image data from the photographed image data, the partial image data including an area having a specific attribute from among each area within the object to be inspected; a prediction step for predicting the probability that the partial image data contains a defect by inputting the partial image data into a learned defect detection model; and a display control step for superimposing and displaying a sign having a display mode according to the predicted probability at the position where the partial image data of the photographed image data is extracted.

Images