Image generation device, image generation method, and computer-readable program

Abstract

To create a learned model capable of appropriately classifying a short circuit defect whose luminance is close to that of a background part.SOLUTION: The image generation device 5 generates an image used when the learned model is created. In the target image 61 showing the defect candidate of the substrate, the representative value of the pixel values of the pattern portion is larger than the representative value of the pixel values of the background portion. The difference enhancement unit 54 generates a difference-enhanced image in which the difference between the target image and the reference image is enhanced. A pixel position at which the pixel value of the target image is a representative value of a pattern portion and the pixel value of the reference image is a representative value of a background portion is assumed as a reference pixel position, and each pixel position at which a difference value obtained by subtracting the pixel value of the reference image from the pixel value of the target image is a smaller positive value than the difference value at the reference pixel position is set as a pixel position of interest. A value of a ratio obtained by dividing a pixel value at each pixel-of-interest position in the difference-enhanced image by a pixel value at the reference pixel position is closer to 1 than a value of a ratio obtained by dividing a difference value at the pixel-of-interest position between the target image and the reference image by a difference value at the reference pixel position.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a technique for generating images. [Background technology]

[0002] Conventionally, systems have been known that detect defect candidates from images of a substrate and determine whether the defect candidates are real defects or false reports (pseudo defects) (see, for example, Patent Documents 1 and 2). Generally, a rule-based algorithm is used to achieve high-speed processing for defect candidate detection, and a subsequent classification unit utilizes a method that applies pattern recognition. For example, pattern recognition involves learning-based recognition, in which case a boundary for recognition or discrimination is derived using example learning samples. In such systems, to avoid overlooking real defects, excessive defect candidate detection is performed using the rule base, while false reports are suppressed by the subsequent classification unit.

[0003] In recent years, highly accurate classification has become possible through methods using deep learning in pattern recognition. In classification using deep learning, the feature extraction for classification itself can be considered as learning, and highly accurate classification performance is achieved by acquiring and learning classification features that humans (algorithm designers) could never have imagined.

[0004] Patent Document 3 discloses a method for multi-value-encoding a binary master image obtained by converting design data (CAD data) of a printed circuit board. Specifically, representative pixel values ​​of the background and pattern parts of the printed circuit board are identified from a histogram of pixel values ​​in a captured image of the printed circuit board, and the representative pixel values ​​are assigned to the corresponding areas of the master image (simple multi-value-encoding). Furthermore, a spatial smoothing process using a variance calculated from the histogram is applied to the master image, thereby generating a multi-value master image that approximates the captured image. Patent Document 4 discloses a shaking comparison process. In this process, an image to be inspected obtained by capturing an image of a board and a reference image corresponding to the image to be inspected are prepared. The reference image is then positioned at a shaking position shifted two-dimensionally by a fixed amount in each peripheral direction relative to the image to be inspected, and an absolute difference image indicating the absolute value of the difference between the reference image and the image to be inspected at each shaking position is obtained. A maximum value image is obtained by performing maximum value filtering on the absolute value image of the difference at each shaking position, and a defect image is obtained by detecting the minimum pixel value at the same position from the maximum value images at multiple shaking positions. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-99625 [Patent Document 2] Japanese Patent Application Laid-Open No. 2006-98151 [Patent Document 3] Japanese Patent Application Laid-Open No. 2000-199709 [Patent Document 4] Japanese Patent Application Laid-Open No. 2002-310928 Summary of the Invention [Problem to be solved by the invention]

[0006] Deep learning has made remarkable improvements in classification accuracy, but misclassification still occurs. When classifying true defects and false alarms in pattern inspection, it is necessary to eliminate as many misclassifications as possible, in which a defect candidate that is actually a true defect is classified as a false alarm (i.e., overlooking a true defect). While it is possible to create a trained model that reduces the oversight of true defects by adjusting the weighting coefficients for defect classification during training, it is difficult to completely eliminate oversight of true defects.

[0007] One type of defect that may be overlooked is a defect called a "dark short." Generally, a pattern short defect occurs when the pattern material is not sufficiently removed during the etching process, leaving an unwanted pattern. In the case of a dark short defect, the pattern material is not completely removed, leaving a thin layer of material between patterns, resulting in a short defect that creates electrical continuity between the patterns. In captured images, the brightness of dark short defects is lower than that of normal short defects. Possible causes of this lower brightness include the thinness of the pattern in the dark short defect, which blocks the reflected light from the pattern by adjacent patterns. Thus, in captured images, the pixel values ​​of dark short defects are close to those of the background, making it difficult to distinguish between the two. Furthermore, dark short defects tend to occur less frequently than normal short defects, making them insufficient for use as training data. Therefore, there is a need for a trained model that can appropriately classify dark short defects, whose brightness is close to that of the background, or a method for appropriately classifying dark short defects using a trained model.

[0008] The present invention has been made in consideration of the above-mentioned problems, and aims to create a trained model that can appropriately classify dark short defects whose brightness is close to the background, or to appropriately classify dark short defects using a trained model. [Means for solving the problem]

[0009] A first aspect of the present invention is an image generation device that generates an image used when creating a trained model for classifying defects on a substrate, or an image that is input to the trained model when classifying the defects, the image generation device including: a storage unit that stores a target image that is an image showing defect candidates on the surface of the substrate, in which a representative value of pixel values ​​in an area showing a pattern portion is greater than a representative value of pixel values ​​in an area showing a background portion; and a difference enhancement unit that generates a difference-enhanced image that highlights the difference between the target image and a reference image corresponding to the target image, the difference-enhanced image being an image in which pixel values ​​of the target image and pixel values ​​of the reference image are equal to the representative value of the pattern portion and the representative value of the background portion, respectively. a pixel value of each pixel position in the difference-enhanced image from the pixel value of the target image and the pixel value of the reference image at each pixel position where the difference value obtained by subtracting the pixel value of the reference image from the pixel value of the target image is a positive value smaller than the difference value at the reference pixel position is taken as the pixel position of interest; and the difference enhancement unit obtains the pixel value of each pixel position in the difference-enhanced image from the pixel value of the target image and the pixel value of the reference image at each pixel position so that the ratio obtained by dividing the pixel value of each pixel position of interest in the difference-enhanced image by the pixel value of the reference pixel position is closer to 1 than the ratio obtained by dividing the difference value of each pixel position of interest between the target image and the reference image by the difference value at the reference pixel position.

[0010] A second aspect of the present invention is an image generating device of the first aspect, wherein the difference enhancement unit obtains the pixel values ​​of each pixel position of the difference enhanced image from the pixel values ​​of the target image and the pixel values ​​of the reference image at each pixel position using a predetermined conversion formula or table.

[0011] A third aspect of the present invention is an image generating device of the first or second aspect, further comprising a visibility improvement unit that generates a visibility-enhancing image that shows the target image and indicates areas in the difference-enhanced image that are identified as the differences between the target image and the reference image in a manner that makes them identifiable from other areas.

[0012] A fourth aspect of the present invention is an image generating device of the third aspect, wherein in the difference-enhanced image, different pixel value ranges are assigned to a first group of pixel positions where the difference value obtained by subtracting the pixel values ​​of the reference image from the pixel values ​​of the target image is negative, and a second group of pixel positions where the difference value is positive, and the visibility improvement unit is equipped with an intermediate image generation unit that generates a first intermediate image in which pixel values ​​based on the difference-enhanced image are assigned to the first group of pixel positions, and a second intermediate image in which pixel values ​​based on the difference-enhanced image are assigned to the second group of pixel positions, and an image synthesis unit that generates the visibility-enhanced image in which the first group of pixel positions and the second group of pixel positions are distinguishable by synthesizing the first intermediate image, the second intermediate image, and the target image.

[0013] A fifth aspect of the present invention is an image generating device of the fourth aspect, further comprising a display unit for displaying the visibility-enhanced image and an input unit for receiving input from an operator, wherein the intermediate image generating unit regenerates the first intermediate image and the second intermediate image from the difference-enhanced image while performing gradation correction based on the input, and the image synthesis unit synthesizes the regenerated first intermediate image and the second intermediate image with the target image to regenerate the visibility-enhanced image.

[0014] Aspect 6 of the present invention is an image generating device of aspect 4 (which may be aspect 4 or 5), wherein the difference enhancement unit moves the reference image relatively to a plurality of shaking positions within a predetermined range with respect to the target image, and generates the difference enhanced image at each shaking position; the intermediate image generation unit generates the first intermediate image and the second intermediate image at each shaking position, obtains a final first intermediate image from the plurality of first intermediate images at the plurality of shaking positions, and obtains a final second intermediate image from the plurality of second intermediate images at the plurality of shaking positions; and the image synthesis unit generates the visibility-enhanced image by synthesizing the final first intermediate image, the final second intermediate image, and the target image.

[0015] A seventh aspect of the present invention is an image generation method for generating an image used when creating a trained model for classifying defects on a substrate, or an image input to the trained model when classifying the defects, comprising the steps of: (a) preparing a target image, which is an image showing defect candidates on the surface of the substrate, in which a representative value of pixel values ​​in an area showing a pattern portion is greater than a representative value of pixel values ​​in an area showing a background portion, and a reference image corresponding to the target image; and (b) generating a difference-enhanced image that highlights the difference between the target image and the reference image, in which the pixel values ​​of the target image and the pixel values ​​of the reference image are equal to the representative value of the pattern portion and the representative value of the background portion, respectively. When a pixel position is assumed to be a reference pixel position and each pixel position at which a difference value obtained by subtracting the pixel value of the reference image from the pixel value of the target image is a positive value smaller than the difference value at the reference pixel position is assumed to be a pixel position of interest, in step b), the pixel value of each pixel position in the difference-enhanced image is obtained from the pixel value of the target image and the pixel value of the reference image at each pixel position so that the ratio value obtained by dividing the pixel value of each pixel position of interest in the difference-enhanced image by the pixel value of the reference pixel position is closer to 1 than the ratio value obtained by dividing the difference value of each pixel position of interest between the target image and the reference image by the difference value at the reference pixel position.

[0016] An eighth aspect of the present invention is a computer-readable program that causes a computer to generate images to be used when creating a trained model for classifying defects on a substrate, or images to be input to the trained model when classifying the defects, wherein execution of the program by a computer causes the computer to execute the steps of: a) preparing a target image that is an image showing defect candidates on the surface of the substrate, in which a representative value of pixel values ​​in an area showing a pattern portion is greater than a representative value of pixel values ​​in an area showing a background portion, and a reference image corresponding to the target image; and b) generating a difference-enhanced image that highlights the difference between the target image and the reference image, wherein the pixel values ​​of the target image and the pixel values ​​of the reference image are respectively higher than the representative value of pixel values ​​in the area showing the pattern portion. a pixel position where the representative value of the background portion is equal to the representative value of the background portion is assumed to be a reference pixel position, and each pixel position where a difference value obtained by subtracting the pixel value of the reference image from the pixel value of the target image is a positive value smaller than the difference value at the reference pixel position is assumed to be a pixel position of interest, in step b), the pixel value of each pixel position of the difference-enhanced image is obtained from the pixel value of the target image and the pixel value of the reference image at each pixel position so that the ratio value obtained by dividing the pixel value of each pixel position of interest in the difference-enhanced image by the pixel value of the reference pixel position is closer to 1 than the ratio value obtained by dividing the difference value of each pixel position of interest between the target image and the reference image by the difference value at the reference pixel position. [Effects of the Invention]

[0017] According to the present invention, it is possible to generate a difference-enhanced image in which pixel values ​​of dark short defects, whose brightness is close to that of the background in a target image, are approximated to pixel values ​​of normal short defects. As a result, it is possible to create a trained model that can appropriately classify dark short defects, or to appropriately classify dark short defects using a trained model. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an inspection device. [Figure 2]FIG. 1 illustrates a configuration of a computer. [Figure 3] FIG. 2 is a block diagram showing a functional configuration realized by a computer. [Figure 4] FIG. 10 is a diagram for explaining a dark short defect. [Figure 5] FIG. 10 is a diagram showing a processing flow relating to creation of a classifier. [Figure 6A] FIG. 10 is a diagram showing an overlaid image and a difference-enhanced image side by side. [Figure 6B] FIG. 10 is a diagram showing an overlaid image and a difference-enhanced image side by side. [Figure 7A] FIG. 10 is a diagram showing an overlaid image and a difference-enhanced image side by side. [Figure 7B] FIG. 10 is a diagram showing an overlaid image and a difference-enhanced image side by side. [Figure 8] FIG. 10 is a diagram illustrating hue conversion. [Figure 9] FIG. 10 is a diagram showing a histogram of pixel values ​​of a target image. [Figure 10] FIG. 10 is a diagram showing a change in hue difference of a short-circuit defect with respect to pixel values ​​of a target image [Figure 11] FIG. 10 is a diagram showing a change in hue difference of an open defect with respect to pixel values ​​of a target image. [Figure 12] FIG. 10 is a diagram illustrating a configuration of an image generating unit related to annotation assistance. [Figure 13] FIG. 10 is a diagram showing a flow of a process for assisting annotation. [Figure 14] 4A and 4B are diagrams for explaining generation of a first intermediate image and a second intermediate image. [Figure 15A] FIG. 10 is a diagram showing a target image and a visibility-enhancing image side by side. [Figure 15B] FIG. 10 is a diagram showing a target image and a visibility-enhancing image side by side. [Figure 16] 10A and 10B are diagrams showing images displayed side by side on a display unit in the process of the comparative example; [Figure 17] FIG. [Figure 18]FIG. 10 is a diagram showing a histogram of pixel values ​​of a target image. [Figure 19] FIG. 10 is a diagram illustrating an example of tone correction. [Figure 20] FIG. 10 is a diagram illustrating another example of tone correction. [Figure 21] FIG. 10 is a diagram showing a superimposed image and a visibility-enhancing image side by side. [Figure 22] FIG. 10 is a diagram for explaining a swing position. [Figure 23] FIG. 10 is a diagram illustrating another example of the difference emphasis unit. DETAILED DESCRIPTION OF THE INVENTION

[0019] 1 is a diagram showing the configuration of an inspection device 1 according to one embodiment of the present invention. The inspection device 1 is a device that inspects the appearance of a printed circuit board 9. The printed circuit board 9 has wiring patterns and the like formed on the surface of a resin base material using a conductive material such as copper.

[0020] The inspection apparatus 1 includes an apparatus main body 2 and a computer 3. The computer 3 controls the overall operation of the inspection apparatus 1 and also realizes an image generation unit, etc., which will be described later. The apparatus main body 2 includes an imaging device 21, a stage 22, and a stage driving unit 23. The imaging device 21 captures an image of the printed circuit board 9 to acquire (data of) a multi-valued captured image. The imaging device 21 includes an illumination unit 211, an optical system 212, and an imaging unit 213. The illumination unit 211 emits illumination light. The optical system 212 guides the illumination light to the printed circuit board 9 and guides light from the printed circuit board 9 to the imaging unit 213. The imaging unit 213 converts the image of the printed circuit board 9 formed by the optical system 212 into an electrical signal. The stage 22 holds the printed circuit board 9. The stage driving unit 23 moves the stage 22 relatively to the imaging device 21. The stage driving unit 23 is composed of a ball screw, a guide rail, a motor, etc. The computer 3 controls the stage driving unit 23 and the imaging device 21 so that an image of a predetermined area of ​​the printed circuit board 9 is captured.

[0021] FIG. 2 shows the configuration of the computer 3. The computer 3 has a typical computer system configuration including a CPU 31, a ROM 32, a RAM 33, a storage device 34, a display unit (display) 35, an input unit 36, a reading device 37, a communication unit 38, a GPU 39, and a bus 30. The CPU 31 performs various arithmetic operations. The GPU 39 performs various arithmetic operations related to image processing and the like. The ROM 32 stores basic programs. The RAM 33 and the storage device 34 store various types of information. The display unit 35 displays various types of information such as images. The input unit 36 ​​includes a keyboard 36a and a mouse 36b for receiving input from an operator. The reading device 37 reads information from a computer-readable recording medium C1 such as an optical disk, a magnetic disk, a magneto-optical disk, or a memory card. The communication unit 38 transmits and receives signals to and from the device main body 2 and the like. The bus 30 is a signal circuit that connects the CPU 31, the GPU 39, the ROM 32, the RAM 33, the storage device 34, the display unit 35, the input unit 36, the reading device 37, and the communication unit 38. The computer 3 may be provided with a touch panel, and the input unit 36 ​​and the display unit 35 may be realized by the touch panel.

[0022] In the computer 3, the program 340 is read in advance from the recording medium C1 via the reading device 37 and stored in the storage device 34. The program 340 may be stored in the storage device 34 via a network. The CPU 31 and the GPU 39 execute arithmetic processing using the RAM 33 and the storage device 34 in accordance with the program 340. The CPU 31 and the GPU 39 function as arithmetic units in the computer 3. Other components functioning as arithmetic units may be employed in addition to the CPU 31 and the GPU 39.

[0023] 3 is a block diagram showing the functional configuration realized by the computer 3. In the inspection apparatus 1, the computer 3 executes arithmetic processing and the like in accordance with a program 340, thereby realizing a defect candidate detection unit 41, an image generation unit 5, a learning unit 42, and a classifier 43. That is, the CPU 31, GPU 39, ROM 32, RAM 33, storage device 34, and the like of the computer 3 realize the defect candidate detection unit 41, the image generation unit 5, the learning unit 42, and the classifier 43. The defect candidate detection unit 41 executes a predetermined inspection process on the captured image to detect a defect candidate area. The defect candidate detection unit 41 also extracts an area including the defect candidate area from the captured image and inputs it to the image generation unit 5 as a target image. Various well-known processes may be performed as the inspection process in the defect candidate detection unit 41.

[0024] The image generation unit 5 includes an image storage unit 51, a reference image generation unit 52, a registration unit 53, a difference enhancement unit 54, and an input image creation unit 55. The image storage unit 51 stores the target image 61 and a master image 69 (described later). The reference image generation unit 52 generates a multi-valued reference image from the binary master image 69. The registration unit 53 performs registration processing to align the position of the reference image with the target image 61. The difference enhancement unit 54 generates a difference-enhanced image (described later) from the target image 61 and the reference image. The input image creation unit 55 creates an input image from the target image 61, the reference image, and the difference-enhanced image. The learning unit 42 uses the input image to create a classifier 43. The classifier 43 is a trained model (AI (artificial intelligence) model) that classifies defect candidates in an image as either true defects or false alarms. All or part of the defect candidate detection unit 41, image generation unit 5, learning unit 42, and classifier 43 may be realized by dedicated electric circuits, or each function may be realized by an individual program.Furthermore, the defect candidate detection unit 41, image generation unit 5, learning unit 42, and classifier 43 may be realized by multiple computers.

[0025] Here, the dark short defect will be explained. FIG. 4 is a diagram for explaining the dark short defect. The upper part of FIG. 4 shows three images 71 to 73 that are part of a captured image showing the pattern of the printed circuit board 9, and the lower part shows changes in pixel values ​​(brightness values) of multiple pixels along dashed lines L1 to L3 in each of the images 71 to 73, using lines L1 to L3 with the same symbol. In the upper part of FIG. 4, the image 71 on the left does not contain a defect, the image 72 in the center contains a normal short defect D1, and the image 73 on the right contains a dark short defect D2. In the upper part of FIG. 4, the narrower the spacing between the parallel diagonal lines, the greater the pixel value, as in other images showing the printed circuit board 9. In this specification, the greater the pixel value, the higher the brightness (brightness).

[0026] Images 71 to 73 include an area A1 (hereinafter simply referred to as "pattern area A1") that indicates a pattern area of ​​the printed circuit board 9 and an area A2 (hereinafter simply referred to as "background area A2") that indicates a background area. The pixel value of the pattern area A1 is larger than the pixel value of the background area A2. In image 71, the two pattern areas A1 are disconnected, and the pixel value change along the dashed-dotted line L1 indicates that the pixel value of the area between the two pattern areas A1 is a small value that indicates the background area A2. In image 72, the two pattern areas A1 are connected by a normal short defect D1. The pixel value change along the dashed-dotted line L2 indicates that the pixel value of the area between the two pattern areas A1 is a large value, the same as that of the pattern area A1. In image 73, the two pattern areas A1 are connected by a dark short defect D2. In the change in pixel values ​​along the dashed line L3, the pixel values ​​in the region between the two pattern portions A1 are between the pixel values ​​of the pattern portion A1 and the pixel values ​​of the background portion A2, and are closer to the pixel values ​​of the background portion A2 than to the pixel values ​​of the pattern portion A1. Such dark short defects D2 are difficult to distinguish from the background portion A2, and are prone to misclassification when a classifier classifies them as true defects or false alarms. Below, we will explain a method that can suppress misclassification of dark short defects D2 by a classifier.

[0027] FIG. 5 is a diagram showing the flow of processing related to the creation of the classifier 43 in the inspection device 1. In the processing related to the creation of the classifier 43, first, a target image 61 is acquired from a captured image by the defect candidate detection unit 41 in FIG. 3, and is stored and prepared in the image storage unit 51 (step S11). As described above, the target image 61 is a multi-valued image indicating a defect candidate, and in this processing example, it is a monochromatic image (grayscale image). In reality, a large number of target images 61 are stored in the image storage unit 51, but FIG. 3 shows only one target image 61.

[0028] The reference image generation unit 52 reads out the target image 61 and the master image 69 from the image storage unit 51. The master image 69 is a binary image generated from design data (CAD data) of the printed circuit board, and is stored in advance in the image storage unit 51. In the master image 69, for example, one value is assigned to each pixel included in the pattern portion A1, and another value is assigned to each pixel included in the background portion A2. In practice, the reference image generation unit 52 reads out an area of ​​the master image 69 (hereinafter simply referred to as the "master image 69") that shows approximately the same area of ​​the printed circuit board 9 shown in the target image 61. Then, the master image 69 is multi-valued in accordance with the target image 61, and a monochrome reference image is generated (step S12).

[0029] The method of Japanese Patent Laid-Open No. 2000-199709 (the above-mentioned Patent Document 3) is used to convert the master image 69 to multilevel values. For example, a discriminant analysis method is used to identify the class of the background portion A2 and the class of the pattern portion A1 in a histogram of pixel values ​​of the target image 61. Then, a representative value of the pixel values ​​in the class of the background portion A2 is assigned to the pixels of the background portion A2 of the master image 69, and a representative value of the pixel values ​​in the class of the pattern portion A1 is assigned to the pixels of the pattern portion A1 of the master image 69. In this way, a reference image is generated and prepared. The representative value is a value that indicates approximately the center of the class, such as the average, median, or mode. The reference image may be subjected to a spatial smoothing process using a variance value, etc.

[0030] The reference image and target image 61 are input to the alignment unit 53, which aligns the reference image with the target image 61 (step S13). For example, the reference image is moved to a plurality of shaking positions within a predetermined range (a predetermined range in the row and column directions) with respect to the target image 61 (see FIG. 22 described later), and the sum of absolute differences in pixel values ​​between the reference image and the target image 61 is calculated at each shaking position. Then, the position of the reference image where the sum of absolute differences is smallest is determined as the position after alignment with the target image 61. Note that the alignment unit 53 may align the target image 61 with the reference image. The alignment between the reference image and the target image 61 may be performed only as needed.

[0031] The difference emphasis unit 54 generates a difference emphasis image that emphasizes the difference between the reference image (hereinafter simply referred to as the "reference image") after alignment and the target image 61 (step S14). Conceptually, first, a color image (hereinafter referred to as the "overlaid image") is generated in which the reference image is set in the R (red) plane and the target image 61 is set in the G (green) plane. No image is set in the B (blue) plane of the overlaid image.

[0032] 6A and 6B are diagrams showing an overlaid image 62 and a difference-enhanced image 63 side by side. In FIGS. 6A and 6B, the overlaid image 62 is shown on the left side, and the difference-enhanced image 63 is shown on the right side (similar to FIGS. 7A and 7B described below). The overlaid image 62 in FIG. 6A is generated from a target image 61 containing a normal short defect D1, and the overlaid image 62 in FIG. 6B is generated from a target image 61 containing a dark short defect D2.

[0033] For example, the images 71 to 73 in the upper part of FIG. 4 can be considered to be a reference image, a target image 61 including a normal short defect D1, and a target image 61 including a dark short defect D2, respectively. In most of the superimposed image 62, the pixel values ​​of the R plane in which the reference image is set and the pixel values ​​of the G plane in which the target image 61 is set are almost the same, resulting in a yellow color. In reality, in both the reference image and the target image 61, the pixel values ​​of the pattern portion A1 are greater than the pixel values ​​of the background portion A2, so the pattern portion A1 in the superimposed image 62 is a brighter yellow than the background portion A2. The yellow areas are non-defective areas (also called normal areas).

[0034] On the other hand, in the case of a normal short defect D1 and a dark short defect D2 (hereinafter collectively referred to as "short-type defects"), the pixel value of the G plane, in which the target image 61 is set, is greater than the pixel value of the R plane, in which the reference image is set (see the lower part of Figure 4), and therefore the color is green. In the target image 61, the pixel value of the normal short defect D1 is greater than the pixel value of the dark short defect D2, and therefore in the superimposed image 62, the normal short defect D1 appears a brighter green than the dark short defect D2. In reality, due to the influence of noise, misalignment between images, etc., there are also green areas that are not short-type defects. Note that in Figures 6A and 6B, yellow is indicated by cross-hatching, and green is indicated by parallel diagonal lines extending from the upper right to the lower left.

[0035] 6A and 6B show examples including short-circuit defects, but the target image 61 may also include open defects. An open defect is a defect in which a portion of a pattern portion on the printed circuit board 9 is missing. In a typical open defect, the pattern portion is almost completely missing, and the pixel value in the captured image is the same small value as the background portion A2. There may also be portions where the pattern portion on the printed circuit board 9 is partially incomplete (for example, a portion where the pattern portion is thinner than other portions, hereinafter referred to as a "light open portion"). In a light open portion, the pixel value in the captured image is, for example, between the pixel value of the pattern portion A1 and the pixel value of the background portion A2, and is closer to the pixel value of the pattern portion A1 than the pixel value of the background portion A2. Note that a light open portion can be considered similar to noise caused by unevenness on the surface of the pattern portion A1, and does not need to be treated as a defect.

[0036] 7A and 7B are diagrams showing a superimposed image 62 and a difference-enhanced image 63 side by side. The superimposed image 62 in FIG. 7A is generated from a target image 61 including a normal open defect D3, and the superimposed image 62 in FIG. 7B is generated from a target image 61 including a light-open portion D4. The normal open defect D3 and the light-open portion D4 (hereinafter collectively referred to as "open defects") are colored red because the pixel values ​​of the R plane, in which the reference image is set, are greater than the pixel values ​​of the G plane, in which the target image 61 is set. In the target image 61, the pixel values ​​of the normal open defect D3 are smaller than the pixel values ​​of the light-open portion D4, whereas in the reference image, the pixel values ​​of both are the same. As a result, in the superimposed image 62, the normal open defect D3 and the light-open portion D4 have different red colors. In reality, due to the influence of noise, misalignment between images, and the like, there are also red areas that are not open defects.

[0037] Next, a hue conversion is performed on the superimposed image 62 to generate a monochrome difference-enhanced image 63. Fig. 8 is a diagram for explaining the hue conversion. The upper part of Fig. 8 shows the relationship between the color and the type of defect in the superimposed image 62, the middle part of Fig. 8 shows the formula for the hue conversion, and the lower part of Fig. 8 shows the gradation range in the difference-enhanced image 63.

[0038] In the hue conversion, for each pixel position in the superimposed image 62, a hue value (here, an angle in the hue circle) is calculated from the pixel values ​​of the R plane, G plane, and B plane, and a difference-enhanced image 63 is generated in which the hue value is the pixel value of the pixel position. In this processing example in which no image is set in the B plane of the superimposed image 62, the pixel value of the R plane (pixel value of the reference image) V R is the pixel value of the G plane (pixel value of the target image 61) V G At pixel locations where the pixel size is larger than α(V G / V R The hue value can be calculated by the following equation: ) (where α>0). The range of hue values ​​that the pixel position can take is 0 or more and less than α. The pixel value V of the G plane G is the pixel value of the R plane V R At pixel locations where the error is greater than α(2-V R / V G The range of hue values ​​that the pixel position can take is greater than α and less than or equal to 2α. The pixel value V of the R plane R and the pixel value of the G plane V G At pixel locations where the values ​​are equal, the hue value is α.

[0039] Here, the characteristics of hue conversion will be explained. FIG. 9 is a diagram showing a histogram of pixel values ​​of a target image 61. For ease of explanation, in FIG. 9, the pixel values ​​are assumed to range from 0 to 1. A histogram of the target image 61 showing a printed circuit board 9 usually exhibits a bimodal distribution. In the example of FIG. 9, the representative value in the distribution of pixel values ​​of pixels included in the background portion A2 (hereinafter simply referred to as the "representative value of the background portion A2"; the same applies to the pattern portion A1) is 0.1, and the representative value of the pattern portion A1 is 0.8. The representative value of the pattern portion A1 is larger than the representative value of the background portion A2.

[0040] As described above, short-circuit defects in the target image 61 appear in the background portion A2, and therefore have large pixel values, even though the pixel values ​​are originally small (dark). For example, a normal short-circuit defect D1 is a group of pixels in the background portion A2 that have pixel values ​​approximately the same as those in the pattern portion A1. A dark short-circuit defect D2 is a group of pixels in the background portion A2 that have pixel values ​​slightly larger (brighter) than those in the surrounding area. On the other hand, open-circuit defects in the target image 61 are in the opposite state to short-circuit defects. For example, a normal open-circuit defect D3 is a group of pixels in the pattern portion A1 that have pixel values ​​approximately the same as those in the background portion A2. A light open-circuit portion D4 is a group of pixels in the pattern portion A1 that have pixel values ​​slightly smaller than those in the surrounding area.

[0041] As mentioned above, the pixel values ​​of the reference image are V R , the pixel value of the target image 61 is V G At the pixel position included in the short-circuit defect, α(2-V R / V G ) to calculate the hue value, and V R is approximately the representative value of the background area A2. G =V R From this state, V G -V R As α(V G / V R ) to calculate the hue value, and V R is approximately the representative value of the pattern part A1.R =V G From this state, V R -V G As V increases, the hue value approaches 0 from α. Therefore, in the difference-enhanced image 63, the hue value of each defect and the hue value of the normal region (V R =V G The absolute value of the difference between the hue value α of the open defect and the hue value α of the open defect (hereinafter referred to as "hue difference") can be regarded as the degree of the defect. If α is 0.5, the hue difference of the open defect is 0.5(1-(V G / V R )), and the hue difference of short-circuit defects is 0.5(1-(V R / V G The range that the hue value can take, that is, the range of pixel values ​​in the difference emphasized image 63, is 0 to 1.

[0042] FIG. 10 is a diagram showing changes in the hue difference of short-circuit defects with respect to pixel values ​​of the target image 61. The vertical axis of FIG. 10 represents the hue difference or the simple difference described below, and the horizontal axis represents the pixel value of the target image 61. As described above, short-circuit defects are defects that occur in the background portion A2, and in FIG. 10, line M1 represents the hue difference of short-circuit defects with respect to each pixel value of the target image 61 when the pixel value of the background portion A2 in the reference image is set to 0.1 according to the pixel value histogram of FIG. 9. Line M2 also represents the absolute value of the difference between each pixel value of the target image 61 and the pixel value of the background portion A2 (hereinafter referred to as the "simple difference").

[0043] As shown in FIG. 10, the simple difference increases linearly with an increase in pixel value of the target image 61. In contrast, the hue difference increases nonlinearly with an increase in pixel value of the target image 61. In the range of pixel values ​​of the target image 61 that is slightly larger than the background portion A2, i.e., the range corresponding to the dark short defect D2, the hue difference increases rapidly. Therefore, it can be said that the hue difference emphasizes the difference between the target image 61 and the reference image. Furthermore, when the pixel values ​​of the target image 61 are near the representative value (0.8) of the pattern portion, i.e., in the range corresponding to the normal short defect D1, the change in the hue difference increases gradually. As a result, in the difference-enhanced image 63 obtained by the hue conversion, the dark short defect D2 and the normal short defect D1 have pixel values ​​that are close to each other. Note that by adjusting the value of α in the hue conversion, it is possible to make the gradation range of the short-circuit type defect in the difference-enhanced image 63 the same as the gradation range of the difference image between the target image 61 and the reference image. Furthermore, using FIG. 10 and the conversion formula α(2-V R / V G ), the magnitude relationship between the simple difference and the magnitude relationship between the hue difference coincide at any two pixel positions where the pixel value of the target image 61 is greater than the pixel value of the reference image.

[0044] Here, a pixel position where the pixel value of the target image 61 and the pixel value of the reference image are equal to the representative value of the pattern portion and the representative value of the background portion, respectively, is assumed as a reference pixel position. The reference pixel position in the target image 61 corresponds to a normal short defect D1. Furthermore, each pixel position where the difference value obtained by subtracting the pixel value of the reference image from the pixel value of the target image 61 is a positive value smaller than the difference value at the reference pixel position is considered as a pixel of interest. The dark short defect D2 in the target image 61 is the pixel of interest. As described above, in the difference-enhanced image 63 generated by hue conversion, the pixel value of the pixel of interest is emphasized. Therefore, in the difference-enhanced image 63, the ratio obtained by dividing the pixel value of each pixel of interest by the pixel value of the reference pixel position is closer to 1 than the ratio obtained by dividing the difference value at the pixel of interest between the target image 61 and the reference image by the difference value at the reference pixel position. As a result, as will be described later, when creating a classifier 43 using the difference-enhanced image 63, the normal short defect D1 and the dark short defect D2 are more likely to be learned as defects with similar characteristics.

[0045] FIG. 11 is a diagram showing changes in the hue difference of an open defect with respect to the pixel value of the target image 61. The vertical axis of FIG. 11 indicates the hue difference or simple difference, and the horizontal axis indicates the pixel value of the target image 61. As described above, an open defect is a defect that occurs on the pattern portion A1. In FIG. 11, line M3 indicates the hue difference of the open defect with respect to each pixel value of the target image 61 when the pixel value of the pattern portion A1 in the reference image is set to 0.8 according to the pixel value histogram of FIG. 9. Line M4 also indicates the absolute value of the difference (i.e., simple difference) between each pixel value of the target image 61 and the pixel value of the pattern portion A1.

[0046] As shown in FIG. 11, both the hue difference and the simple difference decrease linearly with an increase in pixel value of the target image 61. Therefore, for open defects, the hue difference and the simple difference (i.e., the difference-enhanced image 63 and the difference image) can be said to be equivalent. As mentioned above, the light-open portion D4 in the target image 61 is considered to be equivalent to noise, etc. in the pattern portion A1, and therefore does not require any special enhancement processing. Furthermore, by adjusting the value of α in the hue conversion, it is possible to make the gradation range of the open defects in the difference-enhanced image 63 equivalent to that of the difference image. FIG. 11 and the conversion formula α(V G / V R ), for any two pixel positions where the pixel value of the reference image is greater than the pixel value of the target image 61, the magnitude relationship between the simple difference and the magnitude relationship between the hue difference coincide.

[0047] Once the difference-enhanced image 63 is generated, the reference image, the target image 61, and the difference-enhanced image 63 are input to the input image creation unit 55. The input image creation unit 55 creates an input image in which the reference image, the target image 61, and the difference-enhanced image 63 are set in the R plane, the G plane, and the B plane, respectively (step S15). Furthermore, an operator checks the target image 61, etc., displayed on the display unit 35 to determine whether the defect candidate indicated by the target image 61 (input image) is a real defect or a false alarm. When the operator inputs an input specifying a real defect or a false alarm via the input unit 36, a label indicating a real defect or a false alarm is assigned to the input image. That is, an annotation indicating the type of defect candidate indicated by the target image 61 is performed. In this way, the input image labeled as a real defect or a false alarm is created as learning data (step S16). In practice, the above steps S12 to S16 are performed on a large number of target images 61 stored in the image storage unit 51, and a learning data set including a large number of input images is created.

[0048] Once the learning dataset is prepared, the learning unit 42 performs machine learning so that the output of the classifier for each input image is approximately the same as the label (true defect or false alarm) of the input image, and a classifier 43 is created (step S17). The classifier 43 is a trained model that classifies defect candidates indicated by an image into true defects or false alarms, and the creation of the classifier 43 involves determining the values ​​of parameters included in the classifier 43 and the structure of the classifier 43. The machine learning is performed, for example, by deep learning using a neural network. The machine learning may also be performed by well-known methods other than deep learning. This completes the process of creating the classifier 43.

[0049] When classifying unknown defect candidates detected by the defect candidate detection unit 41 using the classifier 43, the same processes as in steps S12 to S15 are performed to create an input image indicating the defect candidates. The input image is then input to the classifier 43, whereby the defect candidates are classified as true defects or false alarms. The difference-enhanced image 63 included in the input image together with the target image 61 and the reference image is an image used when creating the classifier 43 that classifies defects in the printed circuit board 9, and is also an image input to the classifier 43 when classifying unknown defects (defect candidates). Note that only the difference-enhanced image 63 may be used when creating the classifier 43 and when classifying unknown defects.

[0050] Here, we describe the results of an experiment to create a classifier using difference-enhanced images. In this experiment, 200 target images showing normal short defects and 60 target images showing dark short defects were prepared, and the classifier was trained. When cross-validation was performed in the comparative example processing using only the target images and reference images, no oversights (misclassifications) occurred for normal short defects, but one or two samples for dark short defects were overlooked. In contrast, in the processing using the target images, reference images, and difference-enhanced images, all samples were correctly classified for both normal short defects and dark short defects.

[0051] As described above, the image generating device (image generating unit 5 in the above) includes an image storage unit 51 and a difference emphasizing unit 54. The image storage unit 51 stores a target image 61, which is an image showing defect candidates on the surface of the printed circuit board 9, in which a representative value of pixel values ​​in a region showing a pattern portion is greater than a representative value of pixel values ​​in a region showing a background portion. The difference emphasizing unit 54 generates a difference-enhanced image 63 that emphasizes the difference between the target image 61 and a reference image corresponding to the target image 61. Here, a pixel position where the pixel value of the target image 61 and the pixel value of the reference image are equal to the representative value of the pattern portion A1 and the representative value of the background portion A2 in the target image 61, respectively, is assumed to be a reference pixel position, and each pixel position where the difference value obtained by subtracting the pixel value of the reference image from the pixel value of the target image 61 is a positive value smaller than the difference value at the reference pixel position is assumed to be a pixel of interest position. In this case, the difference emphasis unit 54 obtains the pixel value of each pixel position in the difference emphasis image 63 from the pixel value of the target image 61 and the pixel value of the reference image at that pixel position so that the ratio value obtained by dividing the pixel value of each target pixel position in the difference emphasis image 63 by the pixel value of the reference pixel position is closer to 1 than the ratio value obtained by dividing the difference value of the target pixel position between the target image 61 and the reference image by the difference value of the reference pixel position.

[0052] This makes it possible to generate a difference-enhanced image 63 in which the pixel values ​​of the dark short defects D2, whose brightness is close to that of the background portion A2 in the target image 61, are approximated to the pixel values ​​of the normal short defects D1. By using the difference-enhanced image 63 in which the dark short defects D2, which occur infrequently, are converted to have the same characteristics (brightness) as the normal short defects D1, for which learning data is easy to prepare, the classifier 43 can be efficiently created. As a result, it becomes possible to create a classifier 43 that can appropriately classify the dark short defects D2, which are fatal defects that occur infrequently, or to appropriately classify the dark short defects D2 using the classifier 43. Note that the relationship in which the representative value of the pattern portion A1 in the target image 61 is greater than the representative value of the background portion A2 is merely for convenience. For example, in an image in which the pixel values ​​of the background portion A2 are greater than the pixel values ​​of the pattern portion A1, the gradation may be inverted and then the same processing as described above may be performed.

[0053] In a preferred image generating device, the difference emphasis unit 54 obtains the pixel value of each pixel position in the difference emphasis image 63 from the pixel value of the target image 61 and the pixel value of the reference image at that pixel position using a predetermined conversion formula (a hue conversion formula in the above example). This makes it possible to easily generate the difference emphasis image 63. The difference emphasis unit 54 may determine the pixel values ​​of the difference emphasis image 63 using a conversion formula other than the hue conversion formula. Alternatively, a table may be used that indicates the pixel values ​​of the difference emphasis image 63 corresponding to the pixel values ​​of the target image 61 and the pixel values ​​of the reference image.

[0054] Creating a highly accurate classifier requires accurate annotation of each defect candidate in a large number of learning images (e.g., 10,000 or more images). Annotation requires an operator to check each image one by one. The dark short defect D2 has a pixel value similar to that of the background A2 (see pixel value V2 in Figure 9), making it less visible. This increases the possibility that the operator will make a mistake in determining whether the defect is a real defect or a false alarm. This reduces the accuracy of the created classifier.

[0055] Next, a process in which the image generation unit 5 assists the operator in making accurate annotations will be described. FIG. 12 is a diagram showing the configuration of the image generation unit 5 related to annotation assistance, and only shows the configuration from the difference enhancement unit 54 onwards. The image generation unit 5 in FIG. 12 adds a visibility improvement unit 56. The visibility improvement unit 56 includes an intermediate image generation unit 561 and an image synthesis unit 562. FIG. 12 also shows the display unit 35 and the input unit 36. Other configurations of the image generation unit 5 are the same as those in FIG. 3.

[0056] FIG. 13 is a diagram showing the flow of processing by the image generation unit 5 to assist annotation, and shows the processing performed in step S16 of FIG. 5. The processing shown by the dashed blocks in FIG. 13 (steps S24 and S25) is performed in a processing example described later, but is not performed in this processing example. The difference-enhanced image 63 generated in step S14 of FIG. 5 is input to the visibility improvement unit 56. The intermediate image generation unit 561 generates a first intermediate image and a second intermediate image from the difference-enhanced image 63 (step S21).

[0057] FIG. 14 is a diagram for explaining generation of the first intermediate image and the second intermediate image. As described above, an open defect is a pixel position group (hereinafter referred to as the "first pixel position group") where the difference value obtained by subtracting the pixel value of the reference image from the pixel value of the target image 61 is negative. A short defect is a pixel position group (hereinafter referred to as the "second pixel position group") where the difference value obtained by subtracting the pixel value of the reference image from the pixel value of the target image 61 is positive. Furthermore, for an open defect, the hue value in the difference-enhanced image 63 is equal to or greater than 0 and smaller than α, and for a short defect, the hue value is greater than α and equal to or smaller than 2α. Thus, in the difference-enhanced image 63, different pixel value ranges are assigned to the first pixel position group and the second pixel position group.

[0058] The intermediate image generation unit 561 stores a lookup table (hereinafter referred to as "LUT") that associates each pixel value of the difference emphasized image 63 with a pixel value of the first intermediate image or a pixel value of the second intermediate image. As shown in FIG. 14 , in the LUT, the output pixel value linearly decreases from 255 to 0 as the pixel value in the difference emphasized image 63 increases from 0 to α, and the output pixel value linearly increases from 0 to 255 as the pixel value in the difference emphasized image 63 increases from α to 2α. By referencing the LUT, an output pixel value is obtained for each pixel value at each pixel position in the difference emphasized image 63. A monochrome first intermediate image is generated by assigning the output pixel value indicated by the LUT to the first pixel position group and assigning a pixel value of 0 to the other pixel positions. A monochrome second intermediate image is generated by assigning the output pixel value indicated by the LUT to the second pixel position group and assigning a pixel value of 0 to the other pixel positions.

[0059] The first intermediate image is an image that shows only a first group of pixel positions in the difference emphasized image 63, and pixel values ​​based on the difference emphasized image 63 are assigned to the first group of pixel positions. The second intermediate image is an image that shows only a second group of pixel positions in the difference emphasized image 63, and pixel values ​​based on the difference emphasized image 63 are assigned to the second group of pixel positions. For any two pixel positions in the first group of pixel positions, the magnitude relationship of the pixel values ​​in the first intermediate image matches the magnitude relationship of the absolute values ​​of the difference obtained by subtracting the pixel value of the reference image from the pixel value of the target image 61. For any two pixel positions in the second group of pixel positions, the magnitude relationship of the pixel values ​​in the second intermediate image matches the magnitude relationship of the absolute values ​​of the difference. The first and second intermediate images are input to the image synthesis unit 562. The target image is also input to the image synthesis unit 562.

[0060] An open defect is a dark area in the target image 61 where an originally bright area (pattern portion A1) has become dark, and is also called a "dark defect." A short defect is a bright area in the target image 61 where an originally dark area (background portion A2) has become bright, and is also called a "bright defect." Therefore, the first intermediate image and the second intermediate image can be regarded as a dark defect image and a bright defect image, respectively.

[0061] The image synthesis unit 562 synthesizes the target image 61, the first intermediate image, and the second intermediate image to generate a color visibility-enhanced image (step S22). For example, for each pixel position, the pixel value of the target image 61 is set to Obj, the pixel value of the first intermediate image is set to DMapD, and the pixel value of the second intermediate image is set to DMapL, and the pixel value R' of the R plane, the pixel value G' of the G plane, and the pixel value B' of the B plane of the visibility-enhanced image at that pixel position are calculated using Equation 1.

[0062]

number

[0063] As shown in Equation 1, the R plane of the visibility-enhanced image represents an image obtained by combining the target image 61 and the first intermediate image, the G plane represents an image obtained by combining the target image 61 and the second intermediate image, and the B plane represents the target image 61. Therefore, in the visibility-enhanced image, open defects (dark defects) appear red and short defects (bright defects) appear green in the target image 61, which is expressed in grayscale. In this way, the visibility-enhanced image is an image in which the first pixel position group and the second pixel position group are distinguishable.

[0064] 15A and 15B are diagrams showing a target image 61 and a visibility-enhanced image 64 side by side. In FIGS. 15A and 15B, the target image 61 is shown on the left, and the visibility-enhanced image 64 generated from the target image 61 is shown on the right. The target image 61 in FIG. 15A includes a normal short defect D1, and the target image 61 in FIG. 15B includes a dark short defect D2. As described above, in the difference-enhanced image 63, the dark short defect D2 and the normal short defect D1 have pixel values ​​that are similar to each other. Similarly, in the second intermediate image, the dark short defect D2 and the normal short defect D1 have pixel values ​​that are similar to each other. Therefore, the normal short defect D1 in the visibility-enhanced image 64 in FIG. 15A and the dark short defect D2 in the visibility-enhanced image 64 in FIG. 15B are colored in greens of similar brightness.

[0065] On the other hand, for open defects, as explained with reference to FIG. 11 , simple difference and hue difference are equivalent, so normal open defects D3 in the visibility-enhanced image 64 are colored a brighter red than light-open portions D4. As mentioned above, light-open portions are considered to be equivalent to noise in the pattern portion A1, and therefore do not need to be treated as defects. As described above, the visibility enhancement unit 56 generates a visibility-enhanced image 64 that shows areas identified as differences between the target image 61 and the reference image in the difference-enhanced image 63 in a manner that makes them distinguishable from other areas.

[0066] Once the visibility-enhanced image 64 is generated, annotation is performed to indicate the type of defect candidate indicated by the target image 61, and learning data is created (step S23). At this time, the visibility-enhanced image 64 is displayed on the display unit 35. The dark short defect D2, which is difficult to visually determine, is highlighted in the difference-enhanced image 63, so that the visibility of the dark short defect D2 is improved in the visibility-enhanced image 64, just like the normal short defect D1. By checking the visibility-enhanced image 64 on the display unit 35, the operator can appropriately determine whether the defect candidate is a real defect or a false alarm. This completes the process of the image generation unit 5 assisting with annotation.

[0067] As described above, once a training data set corresponding to a large number of target images 61 is created, in step S17 of FIG. 5, the classifier 43 is created using the training data set. In this processing example, the input images created by the input image creation unit 55 (see FIG. 3) are included in the training data, but the visibility-enhanced images 64 are not included. That is, the visibility-enhanced images 64 are used only for annotation. In this case, too, the visibility-enhanced images 64 can be said to be images used when training the classifier 43. Of course, the training data may include the visibility-enhanced images 64. In this case, when classifying unknown defect candidates using the classifier 43, the visibility-enhanced images 64 are also generated and input to the classifier 43.

[0068] Here, a comparative example of processing for assisting annotation will be described. In the comparative example of processing, as shown in FIG. 16, a target image 91 and an image 92 in which a defect candidate 921 is highlighted by a predetermined method are displayed side by side on a display unit. In printed circuit boards and semiconductor boards, information about the periphery of the defect candidate is important as a criterion for determining whether the defect is a true defect or a false alarm. For example, if the defect candidate is located in an area between wiring patterns, it is determined to be a defect with a high criticality (a true defect), whereas if the defect candidate is located in another area, it is determined to be a defect with a low criticality (a false alarm). Therefore, simply arranging the images 91 and 92 side by side, as in the comparative example of processing, makes it difficult to determine where the highlighted defect candidate 921 is located in the target image 91, which may make it difficult for the operator to determine whether the defect is a true defect or a false alarm.

[0069] 12 , the visibility enhancement unit 56 displays the target image 61 and generates a visibility-enhanced image 64 that displays areas identified as differences between the target image 61 and the reference image in the difference-enhanced image 63 so that they can be distinguished from other areas. In the visibility-enhanced image 64 of this processing example, defect candidates in the actual target image 61 are colored, making it possible to easily grasp the positional relationship between the surrounding patterns and the defect candidates, and thus enabling efficient and accurate annotation of the defect candidates. As a result, the classification performance of the classifier 43 is improved, i.e., a classifier 43 with high classification accuracy can be created.

[0070] In the difference-enhanced image 63, different pixel value ranges are assigned to a first pixel position group for which the difference value obtained by subtracting the pixel values ​​of the reference image from the pixel values ​​of the target image 61 is negative, and a second pixel position group for which the difference value is positive. An intermediate image generation unit 561 generates a first intermediate image by assigning pixel values ​​based on the difference-enhanced image 63 to the first pixel position group, and a second intermediate image by assigning pixel values ​​based on the difference-enhanced image 63 to the second pixel position group. An image synthesis unit 562 synthesizes the first intermediate image, the second intermediate image, and the target image 61 to generate a visibility-enhanced image 64 in which the first pixel position group and the second pixel position group are distinguishable. This makes it possible to generate a preferable visibility-enhanced image 64 in which dark defects (first pixel position group) and bright defects (second pixel position group) are distinguishable.

[0071] While it is conceivable that annotation can be performed efficiently by pre-classifying defect candidates using rule-based processing and then having an operator review the classified images, there is a possibility that an operator may determine that a defect candidate with low visibility is a false alarm, even though it is a real defect. In addition to annotation, for example, when evaluating image recognition AI, a human must judge from the image whether the AI's classification of defect candidates is correct. However, even if the AI ​​performs the correct classification, it is difficult to determine whether the classification is correct for defect candidates with low visibility. Even in such cases, generating the above-described visibility-enhanced image 64 makes it easy to objectively determine whether the defect candidate is a real defect or a false alarm.

[0072] In the above process of generating the visibility-enhanced image 64 by the matrix operation of Equation 1, if for some reason it is desired to check the target image 61 before coloring, the target image 61 can be generated and checked by using its inverse matrix, even if the target image 61 is not separately stored. Note that the visibility improvement unit 56 may color dark defects and bright defects the same color. For example, by changing (1 1 0) in the first row to (1 1 1) and (1 0 1) in the second row to (1 0 0) in the matrix operation of Equation 1, it is possible to color dark defects and bright defects red in the visibility-enhanced image 64.

[0073] FIG. 17 is a diagram showing a target image 61a, and FIG. 18 is a diagram showing a histogram of pixel values ​​of the target image 61a. When the printed circuit board 9 is, for example, a flexible board having a film-like substrate and patterns such as wiring patterns are formed on both sides of the substrate, a show-through region A21, where the pattern portion on the side opposite the imaging device 21 shows through, may appear in the background region A2 of the target image 61a. In this case, in the histogram of FIG. 18, a peak P2 in the background region A2 excluding the show-through region A21 and a peak P3 in the show-through region A21 may appear adjacent to each other, and the pixel value of the dark short defect D2 may be close to the pixel value of the show-through region A21. Furthermore, in the target image 61a of FIG. 17, the noise in the pattern region A1 is greater than that in the background region A2, and the width of the peak P1 in the pattern region A1 is wider in the histogram of FIG. 18. In the difference-enhanced image 63, noise in the show-through area A21 and the pattern portion A1 is also emphasized in addition to the defect candidates, making it difficult to recognize the defect candidates in the visibility-enhanced image 64 obtained by the LUT shown in Fig. 14. As a result, it may be difficult to determine whether each defect candidate is a true defect or a false alarm.

[0074] Next, a process for suppressing the emphasis of noise in the show-through region A21 and the pattern portion A1 in the visibility-enhanced image 64 will be described with reference to Fig. 13. In this process example, the processes indicated by the dashed line blocks in Fig. 13 (steps S24 and S25) are also executed. When the visibility-enhanced image 64 is generated in the visibility improving unit 56 (step S22), the visibility-enhanced image 64 is displayed on the display unit 35. If the operator checks the visibility-enhanced image 64 on the display unit 35 and determines that unnecessary portions such as noise in the show-through region A21 and the pattern portion A1 have been emphasized, an input command for gradation correction is made via the input unit 36 ​​(step S24).

[0075] FIG. 19 is a diagram illustrating an example of gradation correction, showing output pixel values ​​for a pixel value range from α to 2α in the difference-enhanced image 63 (second intermediate image of a short-circuit defect). FIG. 19 illustrates gamma curves for multiple gamma values. The operator inputs a desired gamma value, and the intermediate image generation unit 561 adjusts the LUT (step S25). The intermediate image generation unit 561 also functions as an LUT adjustment unit. Next, a first intermediate image and a second intermediate image are regenerated from the difference-enhanced image 63 using the adjusted LUT (step S21). After that, the image synthesis unit 562 synthesizes the regenerated first intermediate image and second intermediate image with the target image 61, thereby regenerating a visibility-enhanced image 64 (step S22). The regenerated visibility-enhanced image 64 is displayed on the display unit 35.

[0076] The above steps S25, S21, and S22 are repeated until a visibility-enhanced image 64 that satisfies the operator (hereinafter referred to as the "final visibility-enhanced image 64") is obtained (step S24). Thereafter, annotation is performed based on the final visibility-enhanced image 64 (steps S24 and S23). The example in FIG. 19 shows gamma curves with multiple gamma values ​​that are equal to or less than 1, but of course, the gamma value may be greater than 1. For example, a gamma value less than 1 is used to suppress the emphasis of non-defects, and a gamma value greater than 1 is used to further emphasize defects.

[0077] As described above, in the visibility improvement unit 56, the intermediate image generation unit 561 regenerates the first intermediate image and the second intermediate image from the difference-enhanced image 63 while performing gradation correction based on input from the operator. Furthermore, the image synthesis unit 562 synthesizes the regenerated first intermediate image and the second intermediate image with the target image 61 to regenerate the visibility-enhanced image 64. In this way, by performing gradation correction when generating the first intermediate image and the second intermediate image (by adjusting the LUT in the above example) and dynamically changing the emphasis or suppression of defective or non-defective areas, it is possible to generate the visibility-enhanced image 64 in which the difference between defects and non-defects is easily distinguishable. As a result, accurate annotation can be achieved.

[0078] When gamma correction is performed as the gradation correction, the only required parameter is the gamma value, allowing the operator to intuitively adjust the LUT. The gradation correction may be other than gamma correction. For example, as shown in FIG. 20 , the pixel value of a difference-enhanced image 63 having an output pixel value of 0 may be set to be greater than α. Similarly, the pixel value of a difference-enhanced image 63 having an output pixel value of 255 may be set to be less than 2α. As described above, the learning data may include a visibility-enhanced image 64. In this case, when classifying unknown defect candidates using the classifier 43, the visibility-enhanced image 64 is generated by the same gradation correction as the final visibility-enhanced image 64 used in annotation, and is input to the classifier 43.

[0079] FIG. 21 is a diagram showing a superimposed image 62 and a visibility-enhanced image 64 side by side. The superimposed image 62 and the visibility-enhanced image 64 in FIG. 21 are generated from a target image 61 in which the line width of the pattern portion A1 differs from that of the reference image. Specifically, of the two pattern portions A1 in the target image 61, one has a line width larger than that of the reference image, and the other has a line width smaller than that of the reference image. As a result, in the visibility-enhanced image 64 in FIG. 21, a green linear region B1, the same as a normal short defect D1, appears along the upper edge of the upper pattern portion A1, and a red linear region B2, the same as a normal open defect D3, appears along the lower edge of the lower pattern portion A1 (similarly in the superimposed image 62). Hereinafter, these regions B1 and B2 will be referred to as "edge detection regions B1 and B2."

[0080] Similarly, if there is a positional misalignment between the target image 61 and the reference image, a green edge detection region B1 or a red edge detection region B2 appears along the edge of the pattern portion A1 in the visibility-enhanced image 64. If there is a difference in the shape of the pattern portion A1, a green edge detection region B1 or a red edge detection region B2 also appears. These edge detection regions B1 and B2 are often judged to be non-defects, depending on their size, shape, etc., and are therefore over-detected regions. However, the presence of the emphasized edge detection regions B1 and B2 reduces the visibility of defect candidates.

[0081] Next, an image generation process for suppressing the appearance of the edge detection regions B1 and B2 will be described. In this processing example, a method conforming to Japanese Patent Application Laid-Open No. 2002-310928 (the above-mentioned Patent Document 4), so-called multi-value wiggling comparison, is used. Specifically, as shown in FIG. 22, the difference enhancement unit 54 positions the reference image 68 at wiggling positions shifted two-dimensionally by a fixed amount in each peripheral direction relative to the target image 61, and generates a difference-enhanced image 63 from the reference image 68 and the target image 61 at each wiggling position. In the example of FIG. 22, 3×3 wiggling positions are set, but the number of wiggling positions may be determined arbitrarily. Typically, (2n+1)×(2n+1) wiggling positions are set, where n is an integer greater than or equal to 1. The target image 61 may be moved relative to the reference image 68 to generate a difference-enhanced image 63 at each wiggling position.

[0082] The intermediate image generation unit 561 generates a first intermediate image and a second intermediate image from the difference-enhanced image 63 at each shaking position. Once the first intermediate image and the second intermediate image have been generated at all shaking positions, the minimum value in all the first intermediate images is identified for each pixel position and assigned to that pixel position, thereby obtaining a final first intermediate image. Similarly, the minimum value in all the second intermediate images is identified for each pixel position and assigned to that pixel position, thereby obtaining a final second intermediate image. In the final first intermediate image and the final second intermediate image, the edge detection areas B1 and B2 are reduced.

[0083] Thereafter, the image synthesis unit 562 sets, for each pixel position, the pixel value of the target image 61 as Obj, the pixel value of the final first intermediate image as DMapD, and the pixel value of the final second intermediate image as DMapL, and calculates the pixel value R' of the R plane, the pixel value G' of the G plane, and the pixel value B' of the B plane of the visibility-enhanced image 64 at that pixel position using the above-mentioned equation 1. As a result, the visibility-enhanced image 64 is generated in which the appearance of the edge detection regions B1 and B2 is suppressed. In the visibility-enhanced image 64, defect candidates that attract the operator's attention are emphasized, thereby improving the visibility of the defect candidates.

[0084] As described above, in this processing example, the difference enhancement unit 54 moves the reference image 68 relative to the target image 61 to multiple rocking positions within a predetermined range, generating a difference-enhanced image 63 at each rocking position. The intermediate image generation unit 561 generates a first intermediate image and a second intermediate image at each rocking position. The intermediate image generation unit 561 also acquires a final first intermediate image from the multiple first intermediate images at the multiple rocking positions, and acquires a final second intermediate image from the multiple second intermediate images at the multiple rocking positions. The image synthesis unit 562 synthesizes the final first intermediate image, the final second intermediate image, and the target image 61 to generate a visibility-enhanced image 64. This prevents areas along the edges of the pattern portion A1 from being emphasized in the visibility-enhanced image 64 due to variations in line width or slight differences in shape of the pattern portion on the printed circuit board 9, misalignment between the target image 61 and the reference image, etc. As a result, a visibility-enhanced image 64 can be generated in which the difference between defects and non-defects is easily distinguishable, enabling accurate annotation. Of course, when classifying unknown defect candidates using the classifier 43, the above-mentioned method may be used to generate the visibility-enhanced image 64 to be input to the classifier 43.

[0085] The image generating device and image generating method described above can be modified in various ways.

[0086] The difference emphasis unit 54 that generates the difference emphasized image 63 may be realized by various configurations. For example, in the difference emphasis unit 54a of FIG. 23 , the target image 61 and the reference image are input to a difference calculation unit 541. The difference calculation unit 541 calculates a difference value by subtracting the pixel value of the target image 61 from the pixel value of the reference image for each pixel position. If the difference value is greater than 0, the difference value is input to a first emphasis unit 543. If the difference value is equal to or less than 0, the value 0 is input to the first emphasis unit 543. Furthermore, a difference value is calculated by subtracting the pixel value of the reference image from the pixel value of the target image 61 for each pixel position. If the difference value is greater than 0, the difference value is input to a second emphasis unit 544. If the difference value is equal to or less than 0, the value 0 is input to the second emphasis unit 544. Furthermore, the target image 61 and the reference image are input to a determination unit 542. If the pixel value of the reference image for each pixel position is greater than the pixel value of the target image 61, the determination unit 542 activates the first emphasis unit 543, and the value input from the difference calculation unit 541 is multiplied by a fixed value and output to the integration unit 545. If the pixel value of the reference image is less than or equal to the pixel value of the target image 61, the determination unit 542 activates the second emphasis unit 544, and the value input from the difference calculation unit 541 is multiplied by a fixed value and output to the integration unit 545.

[0087] The integrating unit 545 assigns a value input from the first emphasizing unit 543 or the second emphasizing unit 544 to each pixel position, thereby acquiring a difference-emphasized image 63. At this time, values ​​greater than the maximum value in the gradation range set for the difference-emphasized image 63 are converted to the maximum value. As a result, in the difference-emphasized image 63 acquired by the difference emphasizing unit 54a, the pixel value of the dark short defect D2, whose brightness in the target image 61 is close to that of the background portion A2, approximates the pixel value of the normal short defect D1. The difference emphasizing unit 54a may change the value to be multiplied in the emphasizing units 543 and 544 based on the pixel values ​​of the target image 61 and the pixel values ​​of the reference image, or based on the difference value between the two images, thereby generating a difference-emphasized image 63 in which the pixel values ​​of the dark short defects approximate the pixel values ​​of the normal short defects.

[0088] In the visibility-enhanced image 64, the areas of the short-circuit defect D1, the dark short-circuit defect D2, etc. are colored, but the areas may be made identifiable by surrounding them with lines or hatching, etc.

[0089] In the above embodiment, a classifier 43 (trained model) is created that classifies defect candidates in an image into either true defects or false defects, but the classifier 43 may also classify the detailed defect types of defect candidates (short defects, open defects, foreign matter, etc.).

[0090] The reference image may be any image that shows a printed circuit board 9 that does not contain any defects, and does not necessarily have to be generated from the master image 69. For example, an image obtained by averaging captured images of a plurality of printed circuit boards 9 may be used as the reference image.

[0091] The target image 61 may be an image showing defect candidates on the surface of a substrate other than the printed circuit board 9 (for example, a semiconductor substrate, a glass substrate, or the like).

[0092] The configurations in the above-described embodiment and each modification may be combined as appropriate as long as they are not mutually contradictory. [Explanation of symbols]

[0093] 3. Computer 5. Image generation unit 9 Printed Circuit Board 35 Display section 36 Input section 43 Classifier 51 Image storage unit 54,54a Difference emphasis part 56 Visibility improvement part 61,61a Target image 63 Difference-enhanced images 64 Visibility Improvement Images 68 Reference Images 340 Programs 561 Intermediate Image Generation Unit 562 Image Synthesis Unit A1 Pattern section A2 background part Steps S11~S17, S21~S25

Claims

1. An image generation device that generates an image used when creating a trained model that classifies defects on a substrate, or an image that is input to the trained model when classifying the defects, a storage unit that stores a target image that is an image showing defect candidates on the surface of a substrate, in which a representative value of pixel values ​​in an area showing a pattern portion is larger than a representative value of pixel values ​​in an area showing a background portion; a difference emphasizing unit that generates a difference-emphasized image that emphasizes the difference between the target image and a reference image corresponding to the target image; Equipped with an image generating device in which a pixel position where the pixel value of the target image and the pixel value of the reference image are equal to the representative value of the pattern portion and the representative value of the background portion, respectively, is assumed to be a reference pixel position, and each pixel position where a difference value obtained by subtracting the pixel value of the reference image from the pixel value of the target image is a positive value smaller than the difference value at the reference pixel position is assumed to be a focus pixel position, the difference emphasis unit obtains pixel values ​​of each pixel position in the difference emphasis image from the pixel values ​​of the target image and the pixel values ​​of the reference image at each pixel position so that a ratio value obtained by dividing the pixel value of each focus pixel position in the difference emphasis image by the pixel value of the reference pixel position is closer to 1 than a ratio value obtained by dividing the difference value of each focus pixel position between the target image and the reference image by the difference value at the reference pixel position.

2. 2. The image generating device according to claim 1, an image generating device in which the difference emphasis unit obtains pixel values ​​at each pixel position of the difference emphasis image from pixel values ​​of the target image and pixel values ​​of the reference image at each pixel position using a predetermined conversion formula or table.

3. 3. The image generating device according to claim 1, The image generating device further includes a visibility enhancement unit that generates a visibility-enhanced image that shows the target image and indicates areas in the difference-enhanced image that are identified as the differences between the target image and the reference image in a manner that makes them distinguishable from other areas.

4. 4. The image generating device according to claim 3, In the difference-enhanced image, different pixel value ranges are assigned to a first group of pixel positions where the difference values ​​obtained by subtracting pixel values ​​of the reference image from pixel values ​​of the target image are negative, and a second group of pixel positions where the difference values ​​are positive, The visibility improving portion is an intermediate image generating unit that generates a first intermediate image in which pixel values ​​based on the difference emphasized image are assigned to the first pixel position group, and a second intermediate image in which pixel values ​​based on the difference emphasized image are assigned to the second pixel position group; an image synthesis unit that synthesizes the first intermediate image, the second intermediate image, and the target image to generate the visibility-enhanced image in which the first pixel position group and the second pixel position group are distinguishable; An image generating device comprising:

5. 5. The image generating device according to claim 4, a display unit that displays the visibility-enhancing image; an input unit that receives input from an operator; Furthermore, the intermediate image generation unit regenerates the first intermediate image and the second intermediate image from the difference emphasized image while performing gradation correction based on the input; The image generating device wherein the image synthesis unit synthesizes the regenerated first intermediate image and the regenerated second intermediate image with the target image to regenerate the visibility-enhanced image.

6. 5. The image generating device according to claim 4, the difference emphasizing unit generates the difference emphasized image at each shaking position while moving the reference image relatively to the target image to a plurality of shaking positions within a predetermined range, the intermediate image generation unit generates the first intermediate image and the second intermediate image at each of the shaking positions, acquires a final first intermediate image from the plurality of first intermediate images at the plurality of shaking positions, and acquires a final second intermediate image from the plurality of second intermediate images at the plurality of shaking positions; The image generating device wherein the image synthesis unit synthesizes the final first intermediate image, the final second intermediate image, and the target image to generate the visibility-enhanced image.

7. An image generation method for generating an image used when creating a trained model for classifying defects on a substrate, or an image to be input to the trained model when classifying the defects, a) preparing a target image which is an image showing defect candidates on a surface of a substrate, in which a representative value of pixel values ​​in an area showing a pattern portion is larger than a representative value of pixel values ​​in an area showing a background portion, and a reference image corresponding to the target image; b) generating a difference-enhanced image that emphasizes the difference between the target image and the reference image; Equipped with a pixel position where the pixel value of the target image and the pixel value of the reference image are equal to the representative value of the pattern portion and the representative value of the background portion, respectively, is assumed to be a reference pixel position, and each pixel position where a difference value obtained by subtracting the pixel value of the reference image from the pixel value of the target image is a positive value smaller than the difference value at the reference pixel position is assumed to be a pixel position of interest, wherein in step b), the pixel value of each pixel position in the difference-enhanced image is obtained from the pixel value of the target image and the pixel value of the reference image at each pixel position so that a ratio obtained by dividing the pixel value of each pixel position of interest in the difference-enhanced image by the pixel value of the reference pixel position is closer to 1 than a ratio obtained by dividing the difference value of each pixel position of interest between the target image and the reference image by the difference value at the reference pixel position.

8. A computer-readable program that causes a computer to generate an image used in creating a trained model for classifying defects in a substrate, or an image that is input to the trained model when classifying the defects, wherein execution of the program by a computer causes the computer to: a) preparing a target image which is an image showing defect candidates on a surface of a substrate, in which a representative value of pixel values ​​in an area showing a pattern portion is larger than a representative value of pixel values ​​in an area showing a background portion, and a reference image corresponding to the target image; b) generating a difference-enhanced image that emphasizes the difference between the target image and the reference image; Execute a pixel position where the pixel value of the target image and the pixel value of the reference image are equal to the representative value of the pattern portion and the representative value of the background portion, respectively, is assumed to be a reference pixel position, and each pixel position where a difference value obtained by subtracting the pixel value of the reference image from the pixel value of the target image is a positive value smaller than the difference value at the reference pixel position is assumed to be a pixel position of interest, wherein in step b), the pixel value of each pixel position in the difference-enhanced image is obtained from the pixel values ​​of the target image and the pixel values ​​of the reference image at each pixel position so that a ratio obtained by dividing the pixel value of each pixel position of interest in the difference-enhanced image by the pixel value of the reference pixel position is closer to 1 than a ratio obtained by dividing the difference value of each pixel position of interest between the target image and the reference image by the difference value at the reference pixel position.

Images