Method, program, and information processing system for generating learning models

Abstract

To provide a method for generating a learning model that can output reference labels or distance labels for autofocus using a learning model based on images of a substrate. [Solution] One aspect of the method for generating a learning model is to acquire first training data including an image of a substrate taken at a focal point position by an imaging device 2 and a reference label, acquire second training data having multiple sets according to distance based on a set of a distance label indicating the distance from the focal point position and an image of the substrate taken by the imaging device 2 after moving to the distance, and then execute a process to train a learning model that outputs a reference label indicating the focal point position or a distance label indicating the distance from the focal point position when an image of the substrate taken by the imaging device 2 is input, based on the acquired first training data and second training data.

Description

[[Technical Field]]

[0001] The present invention relates to a learning model generation method, a program, and an information processing system. [[Background Art]]

[0002] In recent years, autofocus technology has been actively developed to improve the accuracy of defect inspection on inspection objects. For example, Patent Document 1 discloses a focus position adjustment method for an inspection apparatus, which adjusts the height of a pattern formation surface of an evaluation substrate, measures the light amount at front and rear focal positions to calculate an autofocus signal, specifies a height position where a difference between the signals is equal to or less than a threshold value, adjusts a substrate to be inspected, captures an optical image, and inspects for defects. [[Prior Art Documents]] [[Patent Documents]]

[0003] [[Patent Document 1]] Japanese Unexamined Patent Publication No. 2024-119099 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] However, the invention according to Patent Document 1 has a problem that it cannot output a reference label or a distance label for autofocus using a learning model based on an image of a substrate.

[0005] In one aspect, an object of the present invention is to provide a learning model generation method, a program, and the like that can output a reference label or a distance label for autofocus using a learning model based on an image of a substrate. [[Means for Solving the Problem]]

[0006] A method for generating a learning model relating to one aspect is characterized by acquiring first training data including an image of a substrate taken at a focal point position by an imaging device and a reference label, acquiring second training data having multiple sets depending on the distance based on a set of a distance label indicating the distance from the focal point position and an image of the substrate taken by the imaging device moved to the distance, and executing a process to train a learning model that outputs a reference label indicating the focal point position or a distance label indicating the distance from the focal point position when an image of the substrate taken by the imaging device is input, based on the acquired first training data and second training data.

[0007] The program for one aspect acquires an image of a circuit board captured by a camera, and when the acquired image is input to a learning model that outputs a reference label indicating the focal point position or a distance label indicating the distance from the focal point position, the program causes the computer to execute a process to output the reference label or distance label. [Effects of the Invention]

[0008] In one respect, it becomes possible to output reference labels or distance labels for autofocus using a learning model based on images of the substrate. [Brief explanation of the drawing]

[0009] [Figure 1] This is an explanatory diagram showing an overview of the focusing system. [Figure 2] This is a block diagram showing an example of a control device configuration. [Figure 3] This is an explanatory diagram showing an example of the record layout for the training data database and the reference information database. [Figure 4] This is an explanatory diagram illustrating the relationship between the focus value and the point of focus. [Figure 5] This is an explanatory diagram illustrating an example of image changes due to variations in the focal point. [Figure 6] This is an explanatory diagram illustrating the operation of the focusing system. [Figure 7] This is a flowchart showing the steps involved in generating the focusing model. [Figure 8] This flowchart shows the processing procedure for outputting drive commands according to distance labels. [Figure 9] This flowchart shows the processing procedure for outputting a drive command according to the distance label in Embodiment 2. [Modes for carrying out the invention]

[0010] The present invention will be described in detail below with reference to the drawings illustrating its embodiments.

[0011] (Embodiment 1) Embodiment 1 relates to a configuration for generating a learning model based on an image of a substrate. A substrate is a base material or structure in an electronic device or circuit for mounting components, and plays a role in providing mechanical support and electrical connections. The substrate includes, but is not limited to, printed circuit boards, semiconductor substrates, glass substrates, metal substrates, etc. The substrate may also include, for example, ceramic substrates, flexible substrates, or laminated substrates.

[0012] In this embodiment, an example of a substrate is described, but the invention is not limited to this. It can be applied to any object photographed by the imaging device. The object includes objects of various shapes or materials, such as electronic components, integrated circuits, optical components, mechanical parts, or biological samples.

[0013] Conventionally, when inspecting substrates (e.g., semiconductor substrates), it is necessary to perform autofocus (AF) on the imaging device multiple times before inspection, as the height of the substrate varies depending on the inspection position. Therefore, in AF processing using image features, if the position of the imaging device differs while the value of the same features is used, the direction of movement becomes unclear, and AF takes time. As a result, multiple focusing is required, which can prolong inspection time and reduce overall productivity. Furthermore, when high-precision focusing is required, ensuring accuracy becomes difficult.

[0014] In order to solve such problems, the introduction of more efficient focusing techniques is required. In the present embodiment, a learning model is generated using training data created based on images of a substrate. By using the generated learning model, focusing on the substrate can be performed with higher accuracy and efficiency.

[0015] FIG. 1 is an explanatory diagram showing an outline of a focusing system. The system of the present embodiment includes an information processing apparatus 1 and an imaging apparatus 2, and each apparatus transmits and receives information via a bus or a network such as the Internet.

[0016] The information processing apparatus 1 is an information processing apparatus that performs processing, storage, and transmission / reception for various types of information. The information processing apparatus 1 is an apparatus that performs visual inspection, shape measurement, defect detection, dimension measurement, or the like on substrates in manufacturing or quality control. The information processing apparatus 1 is, for example, a control apparatus, a server apparatus, a personal computer, or the like. In the present embodiment, for the sake of brevity, the information processing apparatus 1 is hereinafter read as the control apparatus 1.

[0017] The imaging apparatus 2 is an imaging apparatus (optical instrument) such as a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera. The imaging apparatus 2 includes a control unit (a first control unit). The imaging apparatus 2 is attached to a stage on which a work (e.g., a substrate) is placed, and the attachment position is fixed. The control unit of the imaging apparatus 2 performs imaging when the stage is moved up and down.

[0018] In the Z-axis direction, the direction in which the stage approaches the imaging apparatus 2 is defined as the "+" direction, and the direction in which the stage moves away from the imaging apparatus 2 is defined as the "-" direction. When imaging a substrate, as an example, after the stage is moved from the in-focus position by a predetermined distance (e.g., -100 µm) in advance, the stage is moved in the +Z direction at equal intervals (e.g., 10 µm) to a predetermined maximum distance (e.g., +100 µm), and the imaging apparatus 2 images a predetermined number of (e.g., 2) substrate images at each position.

[0019] Note that instead of the image capturing apparatus 2, an information processing terminal capable of capturing images may be used. The information processing terminal is, for example, an information processing device such as a smartphone, a mobile phone, a tablet, or a personal computer terminal.

[0020] The control device 1 according to the present embodiment acquires first training data including an image of a substrate captured at an in-focus position by the image capturing apparatus 2 and a reference label. The control device 1 acquires second training data having a plurality of sets corresponding to distances based on a set of a distance label indicating a distance from the in-focus position and an image of the substrate captured by the image capturing apparatus 2 moved to the distance. The control device 1 trains, based on the acquired first training data and second training data, a learning model that outputs a reference label indicating an in-focus position or a distance label indicating a distance from the in-focus position when an image of a substrate captured by the image capturing apparatus 2 is input. Details of the learning model will be described later.

[0021] FIG. 2 is a block diagram showing a configuration example of the control device 1. The control device 1 includes a control unit (second control unit) 11, a storage unit 12, a communication unit 13, an operation unit 14, a mass storage unit 15, a drive unit 16, and a reading unit 17. Each component is connected via a bus B.

[0022] The control unit 11 includes an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), or a quantum processor. The control unit 11 reads out and executes a control program 1P (program product) stored in the storage unit 12, thereby performing various information processing, control processing and the like related to the control device 1. The control program 1P described in the present embodiment may be provided on a recording medium, or may be distributed from an external computer.

[0023] Furthermore, the control program 1P can be deployed on a single computer, at a single site, or distributed across multiple sites and run on multiple computers interconnected by a communication network.

[0024] In Figure 2, the control unit 11 is described as a single processor, but it may be a multiprocessor. Furthermore, the control unit 11 may perform various information processing or control processing using the same processor within the control device 1, or it may perform these processes using different processors within the control device 1.

[0025] The storage unit 12 includes memory elements such as RAM (Random Access Memory) and ROM (Read Only Memory), and stores the control program 1P or data necessary for the control unit 11 to execute processing. The storage unit 12 also temporarily stores data necessary for the control unit 11 to execute arithmetic processing.

[0026] The communication unit 13 is a communication module for performing communication-related processing. The operation unit 14 is an interface (e.g., buttons or a touch panel) for the user to operate the control device 1, and is used for setting inspection conditions or controlling the device.

[0027] The large-capacity storage unit 15 includes a recording medium such as an HDD (Hard disk drive) or an SSD (Solid State Drive). The large-capacity storage unit 15 includes a focusing model (learning model) 151, a training data DB (database) 152, and a reference information DB 153.

[0028] The focusing model 151 is an output device (estimator) that outputs a reference label indicating the focal point position or a distance label indicating the distance from that focal point position, based on an image of the substrate, and is a trained model generated by machine learning. The training data DB 152 stores the training data for constructing (generating) the focusing model 151. The reference information DB 153 stores reference information such as the direction of movement and the distances between them that are equally spaced, associated with the distance labels.

[0029] In this embodiment, the storage unit 12 and the large-capacity storage unit 15 may be configured as a single storage device. Furthermore, the large-capacity storage unit 15 may be composed of multiple storage devices. Moreover, the large-capacity storage unit 15 may be an external storage device connected to the control device 1.

[0030] The drive unit 16 drives a mechanism for moving the stage. If the mounting position of the imaging device 2 is not fixed, the drive unit 16 may also drive a mechanism for moving the imaging device 2.

[0031] The reading unit 17 reads a portable storage medium 1a, including a CD (Compact Disc)-ROM or DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 17 and store it in the large-capacity storage unit 15. Alternatively, the control unit 11 may download the control program 1P from another computer via a network N or the like and store it in the large-capacity storage unit 15. Furthermore, the control unit 11 may also read the control program 1P from the semiconductor memory 1b.

[0032] The control device 1 may perform various information processing and control processing using a single computer, or it may be performed in a distributed manner using multiple computers. Furthermore, the control device 1 may be implemented using multiple virtual machines located on a single server, or it may be implemented using a cloud server.

[0033] Figure 3 is an explanatory diagram showing an example of the record layout of the training data DB152 and the reference information DB153. The training data DB152 includes a training data ID column, a type column, an input data column, and an output data column. The training data ID column stores the ID of each training data, which is uniquely identified to identify each training data. The type column stores the type of training data. The types of training data include first training data and second training data.

[0034] The first training data consists of combinations of images of the substrate captured by the imaging device 2 at the focal point position and reference labels. The reference label is an indicator for showing the optimal focal point position (the position where the focus is most accurately achieved), representing the position where the substrate is captured most clearly. In other words, the reference label is a label that indicates where the imaging device 2 focuses relative to the focal point position of the substrate.

[0035] For example, if the focal point is set to 0 μm, the reference label will be "00 (±0 μm)". Since the image of the substrate taken based on the reference label is the clearest, the focus is adjusted based on that reference label.

[0036] The second training data consists of multiple sets of data depending on the distance, based on a set of distance labels indicating the distance from the focal point and an image of the substrate taken by the imaging device 2 moved to that distance. Images of the substrate are taken at multiple distances by moving the imaging device at equal intervals in the direction away from or towards the substrate, relative to the focal point.

[0037] Distance labels indicate how far the current position is from the focal point. Distance labels are used to indicate movement in front of or behind the focal point. For example, if the distance label is "01 (+20μm)", it indicates movement in the direction of moving closer to the substrate at a distance of +20μm from the focal point. If the distance label is "02 (-20μm)", it indicates movement in the direction of moving away from the substrate at a distance of -20μm from the focal point.

[0038] The input data sequence stores either an image of a circuit board in the first training data set, or images of multiple circuit boards in the second training data set. The output data sequence stores either a reference label in the first training data set, or distance labels in the second training data set.

[0039] Reference information DB153 includes a distance label column, a movement direction column, and a movement distance column. The distance label column stores the names of distance labels indicating the distance from the focal point position. The movement direction column stores the direction in which the imaging device 2 is moved. The movement direction includes the direction away from the substrate with respect to the focal point position, or the direction towards the substrate with respect to the focal point position. The movement distance column stores the distance the imaging device 2 is moved from the focal point position.

[0040] The storage configurations described above for each database are merely examples; other storage configurations are also acceptable as long as the relationships between the data are maintained.

[0041] Figure 4 is an explanatory diagram illustrating the relationship between focus value and focal point. Graph 11a is a graph showing the relationship between focus value and focal point.

[0042] The horizontal axis shows the position of the imaging device 2 along the Z axis, moving from "WD (Working Distance) increase" to "WD decrease". The vertical axis shows the focus value, fluctuating in the range of 0 to 100. WD (Working Distance) is the distance between the lens and the object being photographed in the imaging device, specifically the linear distance between the front of the lens (or the foremost part of the optical system) and the object being photographed. The object being photographed is, for example, a substrate.

[0043] As shown in the diagram, the focus value varies depending on the focal point position within the range of 0 to 100. The focal point position increases or decreases depending on the position on the Z axis. In particular, the focus value is high at the Z axis position "0" (near the center), and optimal focus is obtained at this position, allowing for the capture of the optimal image (for example, the original image and differential image shown in number 2). At this position, the image is sharpest and in perfect focus.

[0044] Alternatively, if the WD is increased when the Z-axis is at "-40", the focus value becomes "40". At this position, the substrate is slightly away from the lens, and the original image and differential image shown in 1 can be captured clearly. Furthermore, if the WD is decreased when the Z-axis is at "30", the focus value becomes "40". In this case, the substrate is closer to the lens, so the focus is readjusted, and the original image and differential image shown in 3 can be captured. In addition, the original image and differential image shown in 1 and the original image and differential image shown in 3 are determined to be the same image based on the image features.

[0045] Thus, as the working distance (WD) increases or decreases, the focal point shifts, causing the focus value to change. Therefore, to obtain the optimal focus value, the focal point must be precisely adjusted.

[0046] Figure 5 is an explanatory diagram illustrating an example of image change due to variations in the focal point position. As shown in the figure, the object to be photographed (for example, a substrate) is photographed using the focal point position (the position where the Z axis is "0") as a reference. The image quality of the substrate photographed at the focal point position is the sharpest and the image is perfectly in focus.

[0047] Furthermore, when the working width (WD) is reduced, the imaging device 2 moves closer to the substrate relative to the focal point, resulting in a decrease in the clarity of the captured image. In this case, if the distance to the substrate is too close, it may become difficult to focus.

[0048] Furthermore, increasing the working distance (WD) causes the imaging device 2 to move away from the substrate relative to the focal point, resulting in a decrease in the clarity of the captured image. In this case, the longer focal length may make it easier to deviate from the focal point.

[0049] As shown in the diagram, the degree of image blurring differs depending on the increase or decrease in WD. Image blurring refers to a state where part of the image appears blurred because it is out of focus. When the imaging device 2 approaches the substrate, the light rays emitted from the substrate do not converge on the camera sensor but intersect behind the camera sensor. When the imaging device 2 moves away from the substrate, the light rays emitted from the substrate intersect and diffuse before reaching the camera sensor. As a result, the degree of image blurring differs depending on the position of the imaging device 2 relative to the focal point, making it difficult to estimate the position and direction of movement of the imaging device 2.

[0050] Conventional AF methods using focus values ​​have a problem: since the direction of focus cannot be determined from the focus value in a single shot, it requires approximately 5 to 10 shots and movement of the shooting device 2 using methods such as hill climbing or regression, which is time-consuming. If AF takes a long time, it affects the overall inspection cycle (the time or frequency of inspections). In this embodiment, we provide a focusing system that utilizes the differences in how these images appear to determine how far away the focus position is from the image itself, rather than using image features.

[0051] Figure 6 is an explanatory diagram illustrating the operation of the focusing system. The imaging device 2 photographs the substrate to be photographed. The imaging device 2 transmits the image of the substrate to the control device 1. The control device 1 receives the image of the substrate transmitted from the imaging device 2. The control device 1 inputs the received image of the substrate to the focusing model 151 and outputs a reference label indicating the focal point position, or a distance label indicating the distance from the focal point position.

[0052] The focusing model 151 is used as a program module that is part of the artificial intelligence software. The focusing model 151 is a pre-built neural network estimator that takes an image of the substrate as input and outputs a reference label indicating the focal point, or a distance label indicating the distance from that focal point.

[0053] The control device 1 generates a focusing model 151 by performing deep learning to learn the features of the substrate within the image of the substrate. The focusing model 151 is, for example, a VGG16 learning model. The VGG16 learning model is a type of deep learning model used for image recognition or classification, and is one of the Convolutional Neural Network architectures.

[0054] The focusing model 151 includes an input layer that accepts an image of the substrate, an output layer that outputs a reference label indicating the focal point or a distance label indicating the distance from the focal point, and an intermediate layer that has been trained by backpropagation. The intermediate layer consists of 13 convolutional layers and 3 fully connected layers.

[0055] The input layer receives an image of the substrate. Each convolutional layer extracts features from the image using a small filter of a predetermined size (e.g., 3x3). After each convolutional layer, an activation function called ReLU (Rectified Linear Unit) is applied to introduce nonlinearity. The convolutional layers also use Max Pooling to reduce the image resolution and compress the features. Finally, three fully connected layers (FC layers) integrate the features obtained from the convolutional layers. The output layer outputs the final classification result (reference label or distance label).

[0056] Alternatively, you may use any algorithm other than VGG16, such as CNN (Convolutional Neural Network), RCNN (Regions with Convolutional Neural Network), SSD (Single Shot Multibook Detector), YOLO (You Only Look Once), SVM (Support Vector Machine), Bayesian networks, Transformer networks, or regression trees.

[0057] For example, the control device 1 generates a focusing model 151 using the training data stored in the training data DB 152. The training data includes first training data and second training data. The first training data is data of combinations in which an image of the substrate taken by the imaging device 2 at the focal point position (the position where the Z axis is "0") is associated with a reference label (for example, "00 (±0 μm)").

[0058] The second training data consists of multiple sets of data depending on the distance, based on a set of distance labels indicating the distance from the focal point and an image of the substrate taken by the imaging device 2 moved to that distance. The images of the substrate included in the second training data are taken at multiple distances by moving the imaging device 2 at equal intervals (e.g., 10 μm) in the direction away from the substrate (-Z) or in the direction towards the substrate (+Z) relative to the focal point.

[0059] Regarding the generation process of the second training data, as an example, first, the control device 1 moves the stage by a predetermined distance (e.g., -100 μm) from the focal point position of the substrate via the drive unit 16, and sets the position after the move as the starting position where the substrate is first photographed. The imaging device 2 photographs the substrate at the starting position. The control device 1 moves the stage by equal intervals (e.g., 10 μm) in the +Z direction from the starting position via the drive unit 16 until it reaches a predetermined maximum distance (e.g., +100 μm). As the stage moves at equal intervals, the imaging device 2 photographs the substrate at each position, so the substrate is photographed at multiple distances. This makes it possible to acquire multiple images of the substrate at different distances from the focal point position. In addition, a predetermined number of images (e.g., 2 images) can be taken at each movement position.

[0060] Next, each captured image is assigned a distance label corresponding to its respective shooting position. For example, if the stage starts at -100 μm and moves in the +Z direction every 10 μm, each image may be assigned a label corresponding to the distance moved (-100 μm, -90 μm, -80 μm, ... +100 μm). Based on the images obtained at each shooting position using the procedure described above, and the distance labels indicating the distance from the focal point, a second set of training data consisting of multiple sets depending on the distance can be generated.

[0061] Furthermore, the captured images may be preprocessed, such as by noise reduction or normalization, and converted into a format suitable for training. The first and second training data may also be manually created.

[0062] The control device 1 performs learning using the acquired first and second training data. Specifically, the control device 1 inputs images of the circuit board, which are the first and second training data, into the input layer and extracts features contained in the circuit board images. The control device 1 passes the extracted features to 13 convolutional layers. The control device 1 processes the images passed from the input layer through the 13 convolutional layers.

[0063] Specifically, in the convolutional layers, each layer extracts features from the image using a small filter of a predetermined size (e.g., 3x3). After each convolutional layer, an activation function called ReLU is applied to introduce nonlinearity, allowing the focusing model 151 to learn more complex patterns. After the convolutional layers, a max pooling layer is applied to reduce the image resolution.

[0064] The control device 1 passes the processed image features from the convolutional layer to three fully connected layers (FC layers). The fully connected layers integrate the features obtained from the convolutional layer and generate the final information necessary for prediction. The control device 1 passes the final information generated by the fully connected layers to the output layer. Through the output layer, the control device 1 estimates (predicts) a reference label (focus position) or a distance label (distance from the focus position).

[0065] The control device 1 compares the estimation result output from the output layer with the reference label or distance label, i.e., the ground truth value, that labeled the substrate image in the first and second training data, and optimizes the parameters used in the calculation processing in the hidden layer so that the output value from the output layer approaches the ground truth value. These parameters are, for example, the weights (connection coefficients) between neurons, or the coefficients of the activation function used in each neuron. The method of parameter optimization is not particularly limited, but for example, the control device 1 optimizes various parameters using backpropagation.

[0066] The control device 1 performs the above processing on the images of each substrate included in the first training data and the second training data, and generates a focusing model 151. As a result, for example, the control device 1 trains the focusing model 151 using the first training data and the second training data, thereby generating a learning model capable of outputting a reference label or a distance label. The control device 1 stores the generated focusing model 151 in the large-capacity storage unit 15.

[0067] The focusing model 151 may also be generated by an external information processing device, a cloud server, or dedicated hardware device.

[0068] For example, for an image of a circuit board, a reference label of "00 (±0 μm)" or a distance label of "01 (+10 μm)" is output.

[0069] The control device 1 outputs the direction and distance to move the imaging device 2 toward the focal point position (reference label) based on the outputted reference label or distance label. Specifically, the control device 1 refers to the reference information DB153, which stores the direction of movement and the distances that are equally spaced relative to each other, in correspondence with the distance label, and determines the direction of movement (e.g., +Z direction or -Z direction) and the distance of movement required for the imaging device 2 to reach the focal point position.

[0070] The control device 1 generates a drive command based on the direction and distance to move the imaging device 2 toward the output focal point position. For example, if the direction of movement is in the +Z direction and the distance of movement is 20 μm, the control device 1 generates a drive command to move the imaging device 2 a distance of 20 μm in the +Z direction in order to reach the focal point position.

[0071] The control device 1 outputs the generated drive command to the imaging device 2 or the stage on which the imaging device 2 is mounted, via the drive unit 16. For example, if a drive command is output to the imaging device 2, the imaging device 2 moves toward the focal point in the commanded direction and distance according to the drive command output from the control device 1. Alternatively, if a drive command is output to the stage, the stage moves toward the focal point in the commanded direction and distance according to the drive command output from the control device 1.

[0072] In the process described above, an example was explained in which the direction and distance are output by referring to the reference information DB153, but this is not the only way. For example, the control device 1 may calculate the direction of movement (+Z direction or -Z direction) and the distance traveled (e.g., 20 μm) based on the distance label. Specifically, the control device 1 determines the direction of movement based on the value of the distance label. If the distance label is a positive value, the control device 1 sets the direction of movement to the +Z direction and uses the distance traveled as the value. Alternatively, if the distance label is a negative value, the control device 1 sets the direction of movement to the -Z direction and uses the absolute value of the distance traveled as the value.

[0073] For example, if the reference label is "0 μm" and the distance label is "+20 μm", the control device 1 outputs a drive command to move the imaging device 2 20 μm in the +Z direction. Alternatively, if the reference label is "0 μm" and the distance label is "-20 μm", the control device 1 outputs a drive command to move the imaging device 2 20 μm in the -Z direction.

[0074] Figure 7 is a flowchart showing the procedure for generating the focusing model 151. The control unit 11 of the control device 1 generates first training data, which is a combination of images of the substrate captured at the focal point position by the imaging device 2 and reference labels (step S101).

[0075] Regarding the generation process of the first training data, for example, the imaging device 2 outputs (transmits) an image of the circuit board captured at the focal point to the control device 1. The image of the circuit board may be transferred directly to the control device 1 via a bus, or it may be transmitted to the control device 1 via a wireless communication unit built into the imaging device 2. Furthermore, the image of the circuit board may be transferred to the control device 1 using a memory medium such as USB (Universal Serial Bus). The control unit 11 of the control device 1 then assigns a reference label (for example, "00 (±0 μm)") to the image of the circuit board output from the imaging device 2.

[0076] The control unit 11 generates a second set of training data, which consists of multiple sets depending on the distance, based on a set of distance labels indicating the distance from the focal point position and an image of the substrate taken by the imaging device 2 which has been moved to that distance (step S102).

[0077] As an example of the process for generating the second training data, first, the control unit 11 of the control device 1 moves the stage by a predetermined distance (e.g., -100 μm) from the focal point position of the substrate via the drive unit 16, and sets the position after the move as the starting position where the substrate is first photographed. The imaging device 2 photographs the substrate at the starting position. The control unit 11 moves the stage from the starting position of imaging in the +Z direction at equal intervals (e.g., 10 μm) via the drive unit 16 until it reaches a predetermined maximum distance (e.g., +100 μm).

[0078] As the stage moves at equal intervals, the imaging device 2 photographs the substrate at each position, resulting in the substrate being photographed at multiple distances. This allows for the acquisition of multiple images of the substrate at different distances from the focal point. Furthermore, a predetermined number of images (e.g., 2) can be captured at each moving position.

[0079] Next, the control unit 11 of the control device 1 assigns a distance label (for example, "01 (+10μm)") corresponding to each acquisition position to each acquired image. For example, if the stage starts at -100μm and moves in the +Z direction every 10μm, labels corresponding to the distance moved (-100μm, -90μm, -80μm, ... +100μm) may be assigned to each image. Based on the images at each acquisition position obtained by the procedure described above and the distance labels indicating the distance from the focal point position, a second set of training data having multiple sets according to distance can be generated.

[0080] The control unit 11 stores the generated first training data and second training data in the training data DB 152 of the large-capacity storage unit 15 (step S103).

[0081] Furthermore, once the above-described process (steps S101 to S103) has been executed, it can be omitted from the next time onward.

[0082] The control unit 11 obtains the generated first training data from the training data DB 152 (step S104). The control unit 11 obtains the generated second training data from the training data DB 152 (step S105).

[0083] The control unit 11 uses the acquired first training data and second training data to generate a focusing model 151 that takes an image of the substrate as input and outputs a distance label or distance label (step S106). The control unit 11 stores the generated focusing model 151 in the large-capacity storage unit 15 (step S107) and terminates the series of processes.

[0084] Figure 8 is a flowchart showing the processing procedure when outputting a drive command according to the distance label. The imaging device 2 photographs the substrate to be photographed (step S211). The imaging device 2 outputs (transmits) the image of the photographed substrate to the control device 1 (step S212).

[0085] The control unit 11 of the control device 1 acquires an image of the substrate output from the imaging device 2 (step S111). The control unit 11 inputs the acquired image of the substrate to the focusing model 151 (step S112) and outputs a reference label or distance label (step S113).

[0086] The control unit 11 generates a drive command that includes the direction and distance to move the imaging device 2 toward the focal point position (reference label) based on the output distance label (step S114). Specifically, the control unit 11 refers to the reference information DB 153 in the large-capacity storage unit 15 and identifies the direction and distance of movement associated with the distance label. The control unit 11 generates a drive command that includes the identified direction and distance of movement.

[0087] The control unit 11 outputs the generated drive command to the stage on which the imaging device 2 is mounted via the drive unit 16 (step S115). The stage moves toward the focal point in the commanded direction and distance according to the drive command output from the control unit 11. Alternatively, the control unit 11 may output the generated drive command directly to the imaging device 2 via the drive unit 16. The control unit 11 then terminates processing.

[0088] According to this embodiment, it is possible to generate a focusing model 151 using the first training data and the second training data.

[0089] According to this embodiment, by outputting a reference label or distance label using the focusing model 151, it is possible to predict the optimal focal position based on the characteristics of the substrate, eliminate ambiguity in the direction of movement, and achieve rapid focusing.

[0090] According to this embodiment, by utilizing the focusing model 151, substrate inspection is accelerated and inspection time is reduced, thereby improving the efficiency and quality of inspection.

[0091] (Embodiment 2) Embodiment 2 relates to a configuration in which each of the divided images obtained by dividing the image of the substrate is input to the focusing model 151, and a reference label or distance label is output for each divided image. Note that explanations of content that overlaps with Embodiment 1 are omitted.

[0092] The control device 1 divides the image of the circuit board into smaller images of a predetermined size (for example, 100 x 100). The image division process involves dividing the image of the circuit board vertically and horizontally into predetermined numbers of pixels, and then extracting each smaller image to obtain multiple divided images. The number of divided images is not particularly limited and can be any number as needed.

[0093] The control device 1 inputs each of the divided images obtained from the image of the substrate into the focusing model 151 and outputs a reference label or distance label for each divided image. The control device 1 obtains the reference label or distance label for each divided image output from the learning model. The control device 1 calculates the average value of the reference labels based on the obtained reference labels for each divided image. Alternatively, the control device 1 calculates the average value of the distance labels based on the obtained distance labels for each divided image.

[0094] The control device 1 outputs a drive command to control the imaging device 2 based on the calculated average value of the reference labels or the average value of the distance labels. In this embodiment, an example of the average value has been described, but it is not limited to this. For example, statistical methods such as majority voting, median, or weighted average may be used.

[0095] Furthermore, a drive command can be output using a predetermined threshold. Specifically, the control device 1 determines whether the average value of the calculated distance labels is less than or equal to the predetermined threshold. If the average value of the distance labels is less than or equal to the predetermined threshold, the control device 1 does not output a drive command to adjust the position of the imaging device 2.

[0096] If the average value of the distance labels is a decimal, the control device 1 rounds it to a predetermined number of decimal places to convert the average value used for determination into a form close to an integer. If the average value of the distance labels after rounding is less than or equal to a predetermined threshold, the control device 1 does not output a drive command to adjust the position of the imaging device 2.

[0097] Alternatively, if the average value of the distance labels is greater than a predetermined threshold, the control device 1 generates a drive command that includes the direction and distance to move the imaging device 2 toward the reference label (focus position) based on the average value of the distance labels. The control device 1 outputs the generated drive command to the imaging device 2 or the stage on which the imaging device 2 is mounted.

[0098] If the average value of the distance labels is a decimal, the control device 1 rounds it to a predetermined number of decimal places to convert the average value used for determination into a form close to an integer. If the average value of the distance labels after rounding is greater than a predetermined threshold, the control device 1 generates a drive command that includes the direction and distance to move the imaging device 2 toward the reference label, based on the average value of the distance labels after rounding.

[0099] Furthermore, in addition to rounding the average of the decimal distance labels, other methods such as truncation or rounding up may also be used.

[0100] Figure 9 is a flowchart showing the processing procedure for outputting a drive command according to the distance label in Embodiment 2. Note that components that overlap with those in Figure 8 are denoted by the same reference numerals and their explanations are omitted. After executing the process in step S111, the control unit 11 of the control device 1 divides the acquired image of the substrate into smaller images of a predetermined size (for example, 100 x 100) (step S121).

[0101] The control unit 11 inputs each segmented image obtained by the segmentation process into the focusing model 151 (step S122) and outputs a reference label or distance label for each segmented image (step S123). Based on the reference label or distance label for each segmented image output from the focusing model 151, the control unit 11 calculates the average value of the reference label or distance label (step S124).

[0102] The control unit 11 generates a drive command to control the imaging device 2 based on the calculated average value of the reference labels or distance labels (step S125). The drive command includes the average value of the reference labels (focus position), the direction and distance to move the imaging device 2 toward the focus position. The control unit 11 then executes the process of step S115.

[0103] According to this embodiment, by outputting a reference label or distance label using the focusing model 151 based on segmented images obtained by dividing the image of the substrate, it is possible to improve the accuracy and efficiency of substrate inspection.

[0104] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the invention is indicated by the claims, not in the sense described above, and all modifications within the sense and scope equivalent to the claims are intended.

[0105] The matters described in each embodiment can be combined with each other. Furthermore, the independent and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. In addition, the claims use a form in which claims referencing two or more other claims (multi-claim form), but are not limited to this. A form in which multi-claims referencing at least one multi-claim (multi-multi-claim) may also be used. [Explanation of Symbols]

[0106] 1. Information Processing Device (Control Device) 11 Control Unit 12 Storage section 13 Communications Department 14 Control section 15 Mass storage 151 Focusing Model (Learning Model) 152 Training Data DB 153 Reference Information Database 16 Drive unit 17 Reading Unit 1a Portable storage medium 1b Semiconductor memory 1P Control Program 2. Imaging device B Bus N Network

Claims

1. First training data is acquired, which includes an image of the substrate taken at the focal point position using an imaging device, and a reference label. Based on a set of distance labels indicating the distance from the focal point and an image of the substrate taken by the imaging device moved to the distance, a second set of training data is acquired, having multiple sets depending on the distance. Based on the acquired first and second training data, a learning model is trained that outputs a reference label indicating the focal point or a distance label indicating the distance from the focal point when an image of the substrate captured by the imaging device is input. Methods for generating learning models.

2. The image of the substrate included in the second training data is By moving the imaging device at equal intervals in the direction away from the substrate with respect to the aforementioned focal point position, images are taken at multiple distances. A method for generating a learning model according to claim 1.

3. The image of the substrate included in the second training data is By moving the imaging device at equal intervals in the direction approaching the substrate with respect to the aforementioned focal point position, images are taken at multiple distances. A method for generating a learning model according to claim 1.

4. The image of the circuit board is acquired using the imaging device. By inputting an acquired image into a learning model that outputs a reference label indicating the focal point or a distance label indicating the distance from the focal point when an image is input, the model outputs either a reference label or a distance label. A program that instructs a computer to perform a process.

5. Based on the outputted distance label, the direction and distance to move the imaging device toward the focal point are output. The program according to claim 4.

6. The aforementioned learning model, The system is trained based on first training data, which includes an image of the substrate taken at the focal point by the imaging device and a reference label, and second training data, which has multiple sets depending on the distance, based on a set of distance labels indicating the distance from the focal point and an image of the substrate taken by the imaging device after moving it to the distance. The program according to claim 4 or 5.

7. Referencing a storage unit that stores the direction of movement and the distances that are equally spaced relative to each other, in association with the aforementioned distance labels, the direction of movement and the distance are output. The program according to claim 4 or 5.

8. By inputting each of the divided images obtained by dividing the image of the substrate into the learning model, the model outputs a reference label or distance label for each divided image. The program according to claim 4 or 5.

9. The imaging device is controlled based on the average value of the reference label or distance label of each segmented image output from the learning model. The program according to claim 8.

10. An information processing system comprising a photographing device having a first control unit and a control device having a second control unit, The first control unit is, Take a picture of the circuit board placed on the stage, The captured image of the circuit board is transmitted to the control device. The second control unit is, The image of the circuit board transmitted from the first control unit is received. By inputting an acquired image into a learning model that outputs a reference label indicating the focal point or a distance label indicating the distance from the focal point when an image is input, the model outputs either a reference label or a distance label. A drive command corresponding to the outputted distance label is output to the imaging device or the stage. Information processing system.

Images