Focus position estimation model generation system, focal position estimation model generation method, focal position estimation model generation program, and focal position estimation model

Abstract

It is possible to generate a trained model used for image-based estimation, such as estimating focus position, through short-term training. [Solution] The feature output model generation system 20 is a system that receives image-based information as input and generates a feature output model that outputs the features of the image, and includes a training image acquisition unit 21 that acquires multiple training images associated with focus position information relating to the focus position at the time of image capture, and a feature output model generation unit 22 that generates a feature output model from the acquired training images through machine learning, and the feature output model generation unit 22 compares the features of two different training images according to the focus position information associated with the two training images, and performs machine learning based on the comparison results.

Description

[Technical Field]

[0001] The present invention relates to a feature output model generation system, a feature output model generation method, a feature output model generation program, and the feature output model, which receives image-based information as input and generates a feature output model that outputs features of the image. [Background technology]

[0002] Conventionally, virtual slide scanners have been used, which capture (scan) images of glass slides and use the resulting images as virtual microscope images. Such devices require that the focal position be on the sample when capturing the image. In response to this, it has been proposed to estimate an appropriate focal position based on an image of the sample. For example, Patent Document 1 discloses an estimation using a machine learning algorithm. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-50713 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when generating a trained model that estimates an appropriate focal position, simply performing machine learning using an image and information indicating the appropriate focal position as training data requires a long time for training. Therefore, it is difficult to estimate an appropriate focal position in a short time, and there has been a demand for estimation in a short time.

[0005] An embodiment of the present invention has been made in consideration of the above, and aims to provide a feature output model generation system, a feature output model generation method, a feature output model generation program, and the feature output model, which are capable of generating a trained model used for image-based estimation such as focus position estimation through short-time training. [Means for solving the problem]

[0006] In order to achieve the above object, a feature output model generation system according to one embodiment of the present invention is a feature output model generation system that receives image-based information as input and generates a feature output model that outputs features of the image, and includes: a training image acquisition means for acquiring a plurality of training images each associated with focus position information relating to the focus position at the time of image capture; and a feature output model generation means for generating a feature output model by machine learning from the training images acquired by the training image acquisition means, wherein the feature output model generation means compares the features of two different training images according to the focus position information associated with the two training images, and performs machine learning based on the comparison results.

[0007] In a feature output model generation system according to one embodiment of the present invention, a feature output model that outputs image features is generated by machine learning. In this case, the features of two different training images are compared according to focus position information associated with the two training images, and machine learning is performed based on the comparison results. This generation method allows a feature output model to be generated in a short time, for example, by using an existing trained model that inputs image-based information. In other words, the feature output model generation system according to one embodiment of the present invention allows a trained model used for image-based estimation, such as focus position estimation, to be generated in a short time.

[0008] The feature output model generation means may perform machine learning so that, when two different training images have the same focal position, the difference in the feature amounts between the two training images becomes small, and, when two different training images have different focal positions, the difference in the feature amounts between the two training images becomes large. With this configuration, a feature output model can be generated reliably and appropriately.

[0009] The learning image acquisition means may acquire, as the learning images, images in which radiation from the image capture object is detected, images in which light from the image capture object is detected when light is irradiated onto the image capture object, or images in which electrical characteristics of the image capture object are detected when light is irradiated onto the image capture object. Furthermore, the learning image acquisition means may acquire, as the learning images, images in which light of a specific wavelength is irradiated onto the image capture object. With these configurations, it is possible to generate an appropriate feature output model according to the type of image to be used.

[0010] The training image acquisition means may acquire focus position information related to the focus position when in focus corresponding to each acquired training image, and the feature output model generation system may further include focus position estimation model generation means that, from the focus position information acquired by the training image acquisition means, inputs features output from the feature output model generated by the feature output model generation means by machine learning and generates a focus position estimation model that estimates the focus position when in focus corresponding to the image related to the features. This configuration makes it possible to generate a focus position estimation model that estimates the focus position when in focus from an image. In other words, this configuration makes it possible to generate, in addition to the feature output model, a trained model used for estimating the focus position based on an image through short training.

[0011] Incidentally, one embodiment of the present invention can be described not only as an invention of a feature output model generation system as described above, but also as inventions of a feature output model generation method and a feature output model generation program as described below. These are essentially the same invention, just in different categories, and achieve similar actions and effects.

[0012] That is, a feature output model generation method according to one embodiment of the present invention is a feature output model generation method that inputs information based on an image and generates a feature output model that outputs features of the image, and includes a training image acquisition step of acquiring a plurality of training images each associated with focus position information related to the focus position at the time of image capture, and a feature output model generation step of generating a feature output model by machine learning from the training images acquired in the training image acquisition step, in which the features of two different training images are compared according to the focus position information associated with the two training images, and machine learning is performed based on the comparison results.

[0013] In the feature output model generation step, machine learning may be performed so that if two different training images have the same focus position, the difference in the features of the two training images is small, and if two different training images have different focus positions, the difference in the features of the two training images is large.

[0014] In the learning image acquisition step, an image in which radiation from the image capture object is detected, an image in which light from the image capture object is detected when light is irradiated onto the image capture object, or an image in which electrical characteristics of the image capture object are detected when light is irradiated onto the image capture object may be acquired as the learning image. In the learning image acquisition step, an image in which light of a specific wavelength is irradiated onto the image capture object may be acquired as the learning image.

[0015] In the training image acquisition step, focus position information relating to the focus position at the time of focusing corresponding to each of the acquired training images is acquired, and the feature output model generation method may further include a focus position estimation model generation step in which, from the focus position information acquired in the training image acquisition step, features output from the feature output model generated in the feature output model generation step are input by machine learning to generate a focus position estimation model that estimates the focus position at the time of focusing corresponding to the image related to the features.

[0016] Furthermore, a feature output model generation program according to one embodiment of the present invention is a feature output model generation program that causes a computer to operate as a feature output model generation system that receives image-based information as input and generates a feature output model that outputs features of the image, and causes the computer to function as a training image acquisition means that acquires a plurality of training images associated with focus position information relating to the focus position at the time of image capture, and a feature output model generation means that generates a feature output model by machine learning from the training images acquired by the training image acquisition means, and the feature output model generation means compares the features of two different training images according to the focus position information associated with the two training images, and performs machine learning based on the comparison results.

[0017] The feature output model generation means may perform machine learning so that, when two different training images have the same focus position, the difference in the features of the two training images is small, and, when two different training images have different focus positions, the difference in the features of the two training images is large.

[0018] The learning image acquisition means may acquire, as the learning image, an image obtained by detecting radiation from the image capture object, an image obtained by detecting light from the image capture object when light is irradiated onto the image capture object, or an image obtained by detecting electrical characteristics of the image capture object when light is irradiated onto the image capture object. The learning image acquisition means may acquire, as the learning image, an image obtained by irradiating the image capture object with light of a specific wavelength.

[0019] The learning image acquisition means may acquire focus position information relating to the focus position at the time of focusing corresponding to each acquired learning image, and the feature output model generation program may also cause the computer to function as focus position estimation model generation means that inputs, through machine learning, features output from the feature output model generated by the feature output model generation means from the focus position information acquired by the learning image acquisition means, and generates a focus position estimation model that estimates the focus position at the time of focusing corresponding to the image related to the features.

[0020] Furthermore, the feature output model itself generated by the feature output model generation system according to one embodiment of the present invention is also an invention having a novel configuration. That is, the feature output model according to one embodiment of the present invention is a feature output model for inputting image-based information and causing a computer to function to output features of the image, and is generated by machine learning from a plurality of training images associated with focus position information relating to the focus position at the time of image capture, and the features of the two training images that are different from each other are compared according to the focus position information associated with the two training images, and machine learning is performed based on the comparison results.

[0021] The feature output model may be generated by performing machine learning so that when two different training images have the same focus position, the difference in the features of the two training images is small, and when two different training images have different focus positions, the difference in the features of the two training images is large.

[0022] The feature output model may be generated by performing machine learning using, as training images, an image in which radiation from an image capture object is detected, an image in which light from the image capture object is detected when the image capture object is irradiated with light, or an image in which electrical characteristics of the image capture object are detected when the image capture object is irradiated with light.The feature output model may be generated by performing machine learning using, as training images, an image in which light of a specific wavelength is irradiated onto the image capture object. [Effects of the Invention]

[0023] According to one embodiment of the present invention, it is possible to generate a trained model to be used for image-based estimation, such as estimation of focus position, through short-term training. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a diagram illustrating the configuration of a feature output model generation system and a focus position estimation system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram illustrating an example of a partial configuration of an inspection device. [Figure 3] 10 is an example of an image captured at each focal position. [Figure 4] FIG. 10 is a diagram for explaining generation of a feature output model by machine learning. [Figure 5] FIG. 10 is a diagram illustrating an existing trained model used to generate a feature output model. [Figure 6] 1A and 1B are diagrams illustrating examples of a defocused image and a focused image. [Figure 7] 1 is a flowchart illustrating a feature output model generating method that is a process executed in the feature output model generating system according to an embodiment of the present invention. [Figure 8] 3 is a flowchart illustrating a focus position estimation method, which is a process executed by the focus position estimation system according to the embodiment of the present invention. [Figure 9] FIG. 10 is a diagram schematically illustrating a generated focus map. [Figure 10]FIG. 1 is a diagram showing a configuration of a feature output model generation program according to an embodiment of the present invention, together with a recording medium. [Figure 11] FIG. 2 is a diagram showing the configuration of a focus position estimation program according to an embodiment of the present invention, together with a recording medium. DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, embodiments of a feature output model generation system, a feature output model generation method, a feature output model generation program, and a feature output model according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated explanations will be omitted.

[0026] FIG. 1 shows a computer 10 that is a feature output model generation system according to this embodiment. The computer 10 is a device (system) that performs information processing on images. Specifically, the computer 10 performs information processing on images captured by at least one of an inspection device 40 and an observation device 50. Note that the computer 10 may also perform information processing on images captured by devices other than the inspection device 40 and the observation device 50. In other words, the computer 10 may be applied to devices that capture images other than the inspection device 40 and the observation device 50.

[0027] The inspection apparatus 40 is an apparatus that captures an image of a semiconductor device and inspects the semiconductor device based on the captured image. For example, the inspection apparatus 40 performs failure analysis of the semiconductor device. The semiconductor device to be inspected is, for example, a wafer covered with μLEDs (Light Emitting Diodes). The failure analysis is performed by, for example, light emission analysis, heat generation analysis, analysis using a pattern image, or laser analysis (OBIRCH, OBIC, DALS, etc.). The inspection apparatus 40 may be a conventional inspection apparatus. All of the components described below may be included in a conventional inspection apparatus.

[0028] FIG. 2 shows, for example, the configuration of a portion of an inspection device 40 that performs optical emission analysis. As shown in FIG. 2, the inspection device 40 includes a camera 41, a mounting unit 42, a light source 43, an optical system 44, an objective lens 45, and a stage 46. The camera 41 is an imaging device that captures an image of a semiconductor device, which is an object to be imaged. The camera 41 is, for example, an InGaAs camera. The mounting unit 42 is configured to mount the semiconductor device, which is an object to be imaged. In FIG. 2, a standard sample 60 mounted on the mounting unit 42 is not an object to be inspected, but corresponds to an image corresponding to a focal position, which will be described later. The standard sample 60 is a sample on which an artificial pattern (for example, a radial stripe pattern as shown in FIG. 2) is applied.

[0029] The light source 43 is a device that outputs illumination light to be irradiated onto the object to be imaged. The light source 43 may output illumination light of a specific wavelength (for example, a standard wavelength of 1100 nm, or wavelengths other than the standard wavelength of 900 nm and 1300 nm). Alternatively, the wavelength of the illumination light may be switched, for example, by providing a plurality of light sources 43. The optical system 44 is an optical system for irradiating the illumination light output from the light source 43 onto the semiconductor device, which is the object to be imaged. The objective lens 45 is an objective lens used for imaging by the camera 41, such as a solid immersion lens (SIL). The stage 46 is a member for adjusting the focal position during imaging by the camera 41. The stage 46 may be movable not only in the imaging direction (focal position direction, Z-axis direction) but also in any direction in three dimensions (i.e., an XYZ stage). The focal position during imaging in the inspection device 40 is controlled by the computer 10, as described below. The inspection device 40 includes an inspection unit that inspects semiconductor devices using images obtained by the above-described configuration.

[0030] The configuration of the inspection device 40 does not need to be as described above, and a different configuration may be used depending on the analysis method. For example, in heat generation analysis, an InSb camera may be used as the imaging device (camera 41) without the need for illumination (light source 43). Furthermore, when performing analysis using a pattern image, an incoherent light source or a coherent light source may be used as the illumination (light source 43), and a two-dimensional detector, or an optical scanning device and a photodiode may be used as the imaging device. When performing laser analysis, an incoherent light source or a coherent light source may be used as the illumination (light source 43), and an electrical characteristic acquisition device for semiconductor devices may be used as the imaging device.

[0031] The observation device 50 is a device that captures an image of a biological sample placed on a slide glass and observes the captured image of the biological sample. The observation device 50 may be a conventional observation device. For example, the observation device 50 is the conventional virtual slide scanner described above. The observation device 50 includes an imaging device that captures an image of the biological sample, which is the object to be imaged, a mounting section on which the biological sample, which is the object to be imaged, is placed, and an observation section that observes the biological sample using the captured image. The focal position during imaging in the observation device 50 is controlled by the computer 10, as described below.

[0032] Imaging by the inspection device 40 and the observation device 50 must be performed with the object being imaged in focus. This is to ensure proper inspection or observation of the object being imaged. FIG. 3 shows examples of images captured according to the focal position. This image was captured of the standard sample 60 shown in FIG. 2. FIG. 3(a) is an image captured at the in-focus focal position. FIG. 3(b) is an image captured when the focal position is 100 μm away from the in-focus focal position. FIG. 3(c) is an image captured when the focal position is further away than in FIG. 3(b). FIG. 3(d) is an image captured when the focal position is 500 μm away from the in-focus focal position, further away than in FIG. 3(c). That is, FIG. 3(a) is an example of a focused image, and FIGS. 3(b) to 3(d) are examples of defocused images.

[0033] The computer 10 performs information processing to ensure that imaging by the inspection device 40 and the observation device 50 is performed with the object to be imaged in focus. The computer 10 functionally includes a feature output model generation system 20 that generates a trained model through machine learning, and a focal position estimation system 30 that performs information processing to enable the above-mentioned imaging using the trained model generated by the feature output model generation system 20. As described in detail below, the feature output model generation system 20 is a system (apparatus) that receives image-based information and generates a feature output model that outputs features of the image. As described in detail below, the focal position estimation system 30 is a system (apparatus) that estimates a focused focal position corresponding to an estimation target image. Note that in this embodiment, the inspection device 40 and the observation device 50 are shown as imaging devices, but other devices (systems) that capture images with the object to be imaged in focus may also be used.

[0034] The computer 10 is a conventional computer including hardware such as a CPU (Central Processing Unit), memory, and a communication module. The computer 10 may also be a computer system including multiple computers. The computer 10 may also be configured using cloud computing. The functions of the computer 10, which will be described later, are realized by these components operating through programs or the like. The computer 10, the inspection device 40, and the observation device 50 are connected to each other so as to be able to send and receive information.

[0035] Next, we will explain the functions of the feature output model generation system 20 and the focal position estimation system 30 included in the computer 10 according to this embodiment. As shown in Fig. 1, the feature output model generation system 20 is configured to include a learning image acquisition unit 21, a feature output model generation unit 22, and a focal position estimation model generation unit 23.

[0036] Before describing each function of the feature output model generation system 20, we will explain the trained models generated by the feature output model generation system 20. The trained models generated by the feature output model generation system 20 are a feature output model and a focus position estimation model.

[0037] The feature output model is a model that receives image-based information as input and outputs the feature of the image. The image used as input to the feature output model is an image captured by the inspection device 40 and the observation device 50, or a partial image thereof. The feature output from the feature output model is information indicating the feature of the input image. In this embodiment, the feature reflects the focal position when the image was captured. In other words, the feature output model is an optical model related to optical features. The feature is, for example, a vector with a predetermined number of dimensions (for example, 1024 dimensions). The feature is used as input to a focal position estimation model, as will be described later.

[0038] The feature output model includes, for example, a neural network. The neural network may be multi-layered. That is, the feature output model may be generated by deep learning. The neural network may also be a convolutional neural network (CNN).

[0039] The feature output model has neurons for inputting information based on an image to its input layer. For example, the information input to the feature output model is the pixel value of each pixel of the image. In this case, the input layer has neurons equal to the number of pixels in the image, and each neuron receives the pixel value of the corresponding pixel. As will be described later, the image related to the information input to the feature output model is an image of a preset size (for example, an image of 224 × 224 pixels).

[0040] Note that the information input to the feature output model may be other than the pixel values ​​of each pixel as long as it is based on the image. For example, the information may be a feature to be input to the feature output model, obtained by performing preprocessing such as conventional image processing on the image to reduce the influence of the imaging environment. By performing such preprocessing, it is possible to improve the efficiency of machine learning and the accuracy of the generated feature output model.

[0041] The feature output model is provided with neurons for outputting features to the output layer, for example, neurons equal in number to the dimension of the feature vector.

[0042] The focus position estimation model is a model that inputs a feature output from the feature output model and estimates a focus position at the time of focusing corresponding to an image related to the feature. The focus position estimation model, for example, outputs information indicating the difference between the focus position when the image related to the input feature was captured and the focus position at the time of focusing as an estimated result of the focus position at the time of focusing. The difference is, for example, a value obtained by subtracting the focal length corresponding to the focus position when the image related to the feature was captured from the focal length corresponding to the focus position at the time of focusing. That is, in this case, the output value is a value indicating the focus position when the image in a coordinate system in which the focus position at the time of focusing is set to 0 is captured. The focus position at the time of focusing is a focus position for capturing an image by focusing on an object captured in the image related to the input feature. A focused image can be captured by changing the focus position by the above difference from the focus position when a defocused image related to the input feature was captured.

[0043] In this case, candidates for the difference may be set in advance, and the focus position estimation model may output a value indicating the degree to which the candidate is appropriate. For example, the candidate differences may be +50 μm, 0 μm, −50 μm, −100 μm, etc., and the focus position estimation model may output a value indicating the degree to which each candidate is appropriate. For example, the candidate with the highest value is set as the difference. Alternatively, the focus position estimation model may output the value of the difference itself.

[0044] Alternatively, the focus position estimation model may output information indicating the focus position itself when in focus (for example, the focal length corresponding to the focus position when in focus). In this case, candidates for the focus position itself when in focus may be set in advance, and the focus position estimation model may output a value indicating the degree to which the candidate is valid. Alternatively, the focus position estimation model may output the value of the focus position itself when in focus.

[0045] The focus position estimation model includes, for example, a neural network. The neural network may be multi-layered. That is, the focus position estimation model may be generated by deep learning. The neural network may also be a convolutional neural network (CNN).

[0046] The focus position estimation model has neurons for inputting feature quantities to an input layer. For example, the input layer has neurons corresponding to the neurons provided in the output layer of the feature quantity output model. That is, the input layer has the same number of neurons as the output layer of the feature quantity output model. The focus position estimation model has neurons for outputting the above-mentioned estimation result of the focus position when in focus. For example, the model has neurons equal to the number of candidates (when a value for each candidate is output) or one neuron (when the above-mentioned difference or the focus position itself when in focus is output).

[0047] The feature output model and the focus position estimation model may be configured by something other than a neural network.

[0048] The feature output model and the focus position estimation model are expected to be used as program modules that are part of artificial intelligence software. The feature output model and the focus position estimation model are used, for example, in a computer having a CPU and memory, and the CPU of the computer operates according to instructions from the model stored in the memory. For example, the CPU of the computer operates according to the instructions to input information to the model, perform calculations according to the model, and output results from the model. Specifically, the CPU of the computer operates according to the instructions to input information to an input layer of a neural network, perform calculations based on parameters such as trained weighting coefficients in the neural network, and output results from the output layer of the neural network.

[0049] The training image acquisition unit 21 is a training image acquisition means that acquires a plurality of training images associated with focal position information relating to the focal position at the time of image capture. The training image acquisition unit 21 may acquire, as training images, images in which radiation from an image capture object is detected, images in which light from an image capture object is detected when light is irradiated onto the image capture object, or images in which electrical characteristics of an image capture object are detected when light is irradiated onto the image capture object. The training image acquisition unit 21 may acquire, as training images, images in which light of a specific wavelength is irradiated onto the image capture object. The training image acquisition unit 21 acquires focal position information relating to the focal position at the time of focus corresponding to each acquired training image.

[0050] For example, the training image acquisition unit 21 acquires an image captured by the inspection device 40 or the observation device 50. This image shows an imaged object for the training image. The imaged object for the training image may be, for example, the standard sample 60 shown in FIG. 2. Alternatively, the imaged object for the training image may be something else (for example, an object captured by the inspection device 40 or the observation device 50). For example, as shown in FIG. 4, the training image acquisition unit 21 cuts out a partial image from the acquired image 70 to set it as a training image 71. The training image 71 is an image used in machine learning to generate the two trained models described above. Therefore, the training image acquisition unit 21 acquires a number of training images 71 sufficient to generate an appropriate trained model.

[0051] In this embodiment, the focus position at the time of focusing is estimated from a defocused image, so the multiple learning images 71 include a defocused image. Furthermore, there may be multiple defocused images, and the focus positions associated with the defocused images may be various positions. In other words, the focal lengths corresponding to the focus positions may be various distances. Furthermore, the multiple learning images 71 may include a focused image. FIG. 4 shows an example in which three images 70 are acquired. The vertical direction of the portion showing the images 70 in FIG. 4 is the imaging direction (the focus position direction, the Z-axis direction).

[0052] The training image 71 corresponds to an image used as input to the feature output model. In this case, the feature output model inputs information based on a partial image of the image captured by the inspection device 40 or the observation device 50 that is the same size as the training image 71, rather than the entire image. Therefore, the training image acquisition unit 21 cuts out the training image 71, which is a partial image of a predetermined size used as input to the feature output model, from the acquired image 70. The position from which the training image 71 is cut out in the image 70 is a portion that shows the captured object. However, the training image 71 may include a training image 71 that does not show the captured object. The position from which the training image 71 is cut out in the image 70 may be set in advance. Alternatively, the position from which the training image 71 is cut out may be determined by performing image recognition on the image 70 and estimating that the captured object is shown.

[0053] 4, the learning image acquisition unit 21 may extract multiple learning images 71 from one image 70. When extracting multiple learning images 71 from the image 70, the positions of the learning images 71 may overlap.

[0054] In the inspection device 40 or the observation device 50, imaging is performed to generate images that are the basis for the learning images 71. In this case, for example, in the inspection device 40 or the observation device 50, the position (XY) during imaging is fixed except for the imaging direction (Z-axis direction), and imaging is performed multiple times in succession with different focal positions. In this case, as shown in FIG. 4, the focal position is changed at regular intervals (steps) (ΔZ). Imaging for the learning images 71 in the inspection device 40 or the observation device 50 may be performed by methods other than those described above.

[0055] Furthermore, the learning image acquisition unit 21 may use the image 70 itself captured by the inspection device 40 or the observation device 50 as the learning image. In this case, the image used for input to the feature output model also has the same size as the image 70. Furthermore, the learning image acquisition unit 21 may acquire the image 70 from a device other than the inspection device 40 or the observation device 50.

[0056] As described above, the training image 71 may be an image in which radiation from an object to be imaged is detected (an image used for light emission / heat generation analysis), an image in which light from the object to be imaged is detected when the object is irradiated with light (an image used for pattern analysis), or an image in which electrical characteristics of the object to be imaged when the object is irradiated with light (an image used for laser analysis). The training image 71 may also be an image in which the object to be imaged is irradiated with light of a specific wavelength (for example, light of a specific wavelength used for inspection). These are types of images typically used in the inspection device 40 or the observation device 50. However, when generating a set of feature output models and focal position estimation models, only images of one of the types may be used as the training image 71. In this case, the generated set of feature output models and focal position estimation models corresponds to a specific image type.

[0057] The wavelength of light used in the optical emission analysis varies depending on the driving voltage and design rules of the inspection device 40. Furthermore, in an actual optical system, focus deviations occur for each wavelength due to adjustments and the characteristics of optical elements. Furthermore, the detection sensitivity may be maximized at a position different from the focal position at which the pattern image (picture) is observed. Taking these factors into consideration, a feature output model for each wavelength may be generated using images for each specific wavelength, as described above. For example, a feature output model for each wavelength may be generated using images for the standard wavelength (1100 nm) and wavelengths other than the standard wavelength (900 nm, 1300 nm).

[0058] Furthermore, when generating a set of feature output models and focal position estimation models, only images (including partial images) captured by any model (type) of the inspection device 40 or observation device 50 may be used as the training images 71. In this case, the generated set of feature output models and focal position estimation models corresponds to a specific model of the inspection device 40 or observation device 50. In other words, the feature output models and focal position estimation models reflect the characteristics of the inspection device 40 or observation device 50. By making the training images correspond to a specific type of image or a specific model of the inspection device 40 or observation device 50 in this way, a trained model with higher accuracy can be obtained. Alternatively, the focal position estimation model may be common to multiple image types or models.

[0059] Each training image 71 is associated with focal position information relating to the focal position when the training image 71 was captured. The focal position information is, for example, information indicating the above-mentioned focal position. However, the focal position information may be information relating to the focal position and may be other than the above information as long as it can be used to generate the trained model described above. The focal position information is obtained, for example, as information when the image was captured by the inspection device 40 or the observation device 50. For example, the training image acquisition unit 21 receives and acquires the image associated with the focal position information from the inspection device 40 or the observation device 50.

[0060] Furthermore, the learning image acquisition unit 21 acquires focus position information relating to the focus position when in focus corresponding to each acquired learning image. The focus position information is, for example, information indicating the focus position when in focus. However, the focus position information may be information relating to the focus position when in focus and may be other than the above information as long as it is information relating to the focus position when in focus and can be used to generate the trained model described above. The focus position information is obtained by a conventional method of measuring the focus position when in focus. For example, the learning image acquisition unit 21 acquires the focus position information by receiving an input operation by the user to the computer 10 of the focus position information obtained by measurement.

[0061] The learning image acquisition unit 21 outputs the acquired information to the feature amount output model generation unit 22 and the focal position estimation model generation unit 23.

[0062] The feature output model generation unit 22 is a feature output model generation means that generates a feature output model by machine learning from the training images 71 acquired by the training image acquisition unit 21. The feature output model generation unit 22 compares the features of two different training images 71 according to focus position information associated with the two different training images 71, and performs machine learning based on the comparison result. The feature output model generation unit 22 may perform machine learning so that when the two different training images 71 have the same focus position, the difference in the features of the two different training images 71 becomes small, and so that when the two different training images 71 have different focus positions, the difference in the features of the two different training images 71 becomes large.

[0063] The feature output model generation unit 22 generates a feature output model as follows. The feature output model generation unit 22 receives training images 71 and focal position information related to the training images 71 from the training image acquisition unit 21. The feature output model generation unit 22 performs machine learning for generating a feature output model using two training images 71 selected from the input training images 71 as one set. The set used for machine learning includes both a set of training images 71 having the same focal position and a set of training images 71 having different focal positions. For example, the set of training images 71 having the same focal position may be training images 71 cut out from the same image 70, as shown in FIG. 4. The set of training images 71 may be selected using a predetermined method that satisfies the above conditions. The set of training images 71 may also be selected from images 70 that have the same position (XY) at the time of imaging other than the imaging direction (Z-axis direction).

[0064] The feature output model generation unit 22 performs machine learning using information based on the selected set of training images 71 as input to the feature output model. As shown in FIG. 4, when each of the training images 71 in a set is input to a feature output model 80, a feature is obtained as output for each training image 71. In FIG. 4, the value of each element of a vector, which is a feature, is shown by a bar graph. In this case, the feature output model 80 to which one training image 71 is input is used as the training target, and the feature output model 80 to which the other training image 71 is input is used as the comparison target. However, these feature output models 80 are the same and are currently undergoing training.

[0065] The feature output model generation unit 22 compares the two output feature amounts with reference to the focus position information and performs machine learning based on the comparison result. When the focus positions of the two training images 71 indicated by the focus position information are the same (i.e., on the same plane), the feature output model generation unit 22 performs machine learning so that the difference in the feature amounts of the two training images 71 becomes smaller. When the focus positions of the two training images 71 indicated by the focus position information are different (i.e., different Z positions), the feature output model generation unit 22 performs machine learning so that the difference in the feature amounts of the two training images 71 becomes larger. Note that when the two training images 71 are cut out from the same image, the focus positions of the two training images 71 are the same. Furthermore, when the focus positions of the two training images 71 are close enough to be considered to be the same, the focus positions of the two training images 71 may also be considered to be the same.

[0066] That is, the feature amounts of partial images cut out from images on the same focal plane are made to have a high correlation regardless of the cut-out position. On the other hand, the feature amounts of partial images cut out from images on different focal planes are made to have a low correlation. By performing machine learning in this way, the feature amounts output from the feature output model reflect characteristics according to the focal position.

[0067] Specifically, when the focus positions of the two learning images 71 are the same, the feature output model generation unit 22 performs machine learning using the following loss_xy as the loss function.

number

[0068] When the focus positions of the two learning images 71 are different from each other, the feature output model generation unit 22 performs machine learning using the following loss_z as the loss function.

number

[0069] The feature output model generation unit 22 generates a feature output model by repeatedly selecting a set of learning images 71 and performing machine learning. For example, the feature output model generation unit 22 generates a feature output model by repeating the above process until the generation of the feature output model converges based on preset conditions or a preset number of times, as in the conventional case.

[0070] The feature output model generation unit 22 may generate the feature output model using an existing trained model generated by machine learning. As the existing trained model, a model that inputs image-based information is used, similar to the feature output model according to this embodiment. That is, an existing trained model that shares input with the feature output model according to this embodiment may be used. The existing trained model is, for example, a model for performing image recognition, specifically, ResNet, VGG, Mobile Net, etc. A part of the existing trained model is used to generate the feature output model. As shown in FIG. 5 , the output layer of the existing trained model 81 is deleted, and the part of the existing trained model 81 up to the intermediate layer is used to generate the feature output model. The existing trained model 81 used to generate the feature output model may include all of the intermediate layer, or only a part of the intermediate layer.

[0071] The feature output model generation unit 22 inputs the above-mentioned part of the existing trained model and sets it as the feature output model at the start of machine learning. That is, the feature output model generation unit 22 performs fine tuning using the above-mentioned part of the existing trained model as initial parameters of the feature output model. Furthermore, a new output layer may be added to the output side of the above-mentioned part of the trained model, and set it as the feature output model at the start of machine learning. Furthermore, when a new output layer is added, a new intermediate layer may be added between the output side of the above-mentioned part of the trained model and the new output layer, and set it as the feature output model at the start of machine learning.

[0072] The feature output model generation unit 22 may generate a feature output model without using an existing trained model. For example, a model with random values ​​as initial parameters, similar to conventional machine learning, may be used as the feature output model at the start of machine learning.

[0073] Using an existing trained model to generate a feature output model has the following advantages: The training time can be significantly reduced. A highly accurate feature output model can be generated even with a small number of training images 71, that is, a feature output model that can output more appropriate features. The above-mentioned existing trained model has already acquired the ability to separate features with low abstraction. Therefore, it is only necessary to train the model using new training images 71, focusing on features with high abstraction.

[0074] The feature output model generation unit 22 outputs the generated feature output model to the focus position estimation model generation unit 23 and the focus position estimation system 30. The generated feature output model may be used for purposes other than those described in this embodiment. In that case, for example, the feature output model generation unit 22 transmits or outputs the feature output model to another device or module that uses the feature output model. Alternatively, the feature output model generation unit 22 may store the generated feature output model in the computer 10 or another device so that it can be used by another device or module that uses the feature output model.

[0075] The focus position estimation model generation unit 23 is a focus position estimation model generation means that generates a focus position estimation model by machine learning from the focus position information acquired by the learning image acquisition unit 21. The focus position estimation model receives as input the feature amounts output from the feature amount output model generated by the feature amount output model generation unit 22 as described above, and estimates the focus position at the time of focusing that corresponds to the image related to the feature amounts.

[0076] The focus position estimation model generation unit 23 generates a focus position estimation model as follows: The focus position estimation model generation unit 23 receives the training image 71 and the focus position information related to the training image 71 from the training image acquisition unit 21. The focus position estimation model generation unit 23 receives the feature amount output model from the feature amount output model generation unit 22.

[0077] The focus position estimation model generation unit 23 inputs information based on the training image 71 into a feature output model and acquires the feature of the training image 71, which is output from the feature output model. The focus position estimation model generation unit 23 performs machine learning using the acquired feature as input to the focus position estimation model and information based on the focus position information of the training image 71 related to the input feature as output of the focus position estimation model. The information based on the focus position information is information corresponding to the output of the focus position estimation model. If the focus position estimation model outputs a value for each of the above-mentioned candidates, the information based on the focus position information is, for example, a value for each candidate (one-hot vector) in which a value of 1 is assigned to a candidate corresponding to the focus position information and a value of 0 is assigned to a candidate not corresponding to the focus position information. If the focus position estimation model outputs the above-mentioned difference or a value of the focus position itself, the information based on the focus position information is the focus position information itself or a value calculated from the focus position information. Before performing machine learning, the focus position estimation model generation unit 23 generates information based on the focus position information corresponding to the output of the focus position estimation model.

[0078] The machine learning itself, i.e., the updating of the parameters of the focus position estimation model, may be performed in the same manner as in the conventional method. The focus position estimation model generation unit 23 generates the focus position estimation model by repeating the machine learning process until the generation of the focus position estimation model converges based on preset conditions or for a predetermined number of times, as in the conventional method.

[0079] The focus position estimation model generation unit 23 outputs the generated focus position estimation model to the focus position estimation system 30. The generated focus position estimation model may be used for purposes other than those described in this embodiment. In that case, for example, the focus position estimation model generation unit 23 transmits or outputs the focus position estimation model to another device or module that uses the focus position estimation model. Alternatively, the focus position estimation model generation unit 23 may store the generated focus position estimation model in the computer 10 or another device so that it can be used by another device or module that uses the focus position estimation model. These are the functions of the feature output model generation system 20.

[0080] Next, a description will be given of the functions of the focal position estimation system 30 according to this embodiment. As shown in FIG. 1, the focal position estimation system 30 includes an estimation target image acquisition unit 31, a focal position estimation unit 32, and a control unit 33.

[0081] The focal position estimation system 30 estimates the focal position at the time of focusing when an image of an object to be imaged is captured by the inspection device 40 or the observation device 50. This estimation is performed by first capturing an image of the object to be imaged (imaging for focusing) by the inspection device 40 or the observation device 50. In this imaging, the focal position does not necessarily have to be the one at which the object to be imaged is in focus, i.e., the focal position at the time of focusing. Therefore, the image obtained by this imaging may be a defocused image. For example, this image may be a defocused image as shown in FIG. 6(a). FIG. 6 is an image of a semiconductor device. The defocused image shown in FIG. 6(a) is an image obtained when the focal position at the time of imaging is +5 nm from the focal position at the time of focusing.

[0082] The focal position estimation system 30 estimates the focal position when in focus from this image. Using the estimated focal position when in focus, an image can be captured in the inspection device 40 or the observation device 50, thereby obtaining an image in which the object to be captured is in focus, i.e., a focused image. For example, a focused image such as that shown in FIG. 6(b) can be obtained. The focused image shown in FIG. 6(b) corresponds to the defocused image in FIG. 6(a).

[0083] The estimation target image acquisition unit 31 is an estimation target image acquisition means for acquiring an estimation target image. The estimation target image is an image used to estimate the focal position when in focus in the focal position estimation system 30. The estimation target image is also an image used as input to the feature output model. In other words, the estimation target image corresponds to the learning image 71 described above.

[0084] The estimation target image acquisition unit 31 acquires an image captured by the inspection device 40 or the observation device 50. The image captured by the inspection device 40 or the observation device 50 at this time is the image captured for focusing as described above. For example, the estimation target image acquisition unit 31 cuts out a partial image from the acquired image and sets it as the estimation target image.

[0085] The estimation target image acquisition unit 31 cuts out an estimation target image, which is a partial image of a preset size used as input for the feature output model, from the acquired image. The position in the image from which the estimation target image is cut out is a portion in which the captured object is shown. The position in the image from which the estimation target image is cut out may be set in advance. Alternatively, the position from which the estimation target image is cut out may be determined by performing image recognition on the image and estimating that the captured object is shown.

[0086] If the image used as input to the feature output model is the same size as the image captured by the inspection device 40 or the observation device 50, the estimation target image acquisition unit 31 may use the image captured by the inspection device 40 or the observation device 50 itself as the estimation target image.

[0087] The type of the estimation target image is the same as the type of the learning image described above. For example, the estimation target image may be an image in which radiation from the imaging target is detected, an image in which light from the imaging target is detected when light is irradiated onto the imaging target, or an image in which electrical characteristics of the imaging target are detected when light is irradiated onto the imaging target. The estimation target image may also be an image in which light of a specific wavelength (e.g., light of a specific wavelength used for inspection) is irradiated onto the imaging target.

[0088] The estimation target image acquisition unit 31 outputs the acquired estimation target image to the focus position estimation unit 32.

[0089] The focus position estimation unit 32 is a focus position estimation means that uses a feature amount output model to output feature amounts of the estimation target image from the estimation target image acquired by the estimation target image acquisition unit 31, and estimates a focus position at the time of focusing corresponding to the estimation target image from the output feature amount. The focus position estimation unit 32 may use a focus position estimation model to estimate a focus position at the time of focusing corresponding to the estimation target image from the feature amount output from the feature amount output model.

[0090] The focal position estimation unit 32 receives the feature output model and the focal position estimation model generated by the feature output model generation system 20, stores them, and uses them for estimation. The focal position estimation unit 32 receives the estimation target image from the estimation target image acquisition unit 31.

[0091] The focus position estimation unit 32 inputs information based on the estimation target image to a feature output model and acquires feature amounts of the estimation target image, which are output from the feature output model. The focus position estimation unit 32 inputs the acquired feature amounts to the focus position estimation model and acquires information indicating the focus position when in focus corresponding to the estimation target image, which is output from the focus position estimation model, as an estimation result of the focus position. The focus position estimation unit 32 outputs the information indicating the focus position when in focus corresponding to the acquired estimation target image to the control unit 33.

[0092] The control unit 33 is a control means that controls the focal position during imaging of the imaging target, based on the focal position estimated by the focal position estimation unit 32. The control unit 33 inputs information indicating the focal position when in focus corresponding to the estimation target image from the focal position estimation unit 32. The control unit 33 controls the inspection device 40 or the observation device 50 so that the focal position during imaging becomes the focal position when in focus indicated by the input information. The inspection device 40 or the observation device 50 that has received the control adjusts the focal position during imaging, for example, by operating the stage 46. As a result, the image captured by the inspection device 40 or the observation device 50 becomes a focused image. In this way, the control unit 33 controls the autofocus in the inspection device 40 or the observation device 50. The above is the configuration of the focal position estimation system 30.

[0093] Next, the processing executed by the computer 10 according to this embodiment (the operating method performed by the computer 10) will be described using the flowcharts of Fig. 7 and Fig. 8. First, the processing executed when generating a feature output model and a focus position estimation model, that is, the feature output model generation method which is the processing executed by the feature output model generation system 20 according to this embodiment, will be described using the flowchart of Fig. 7.

[0094] In this process, first, the training image acquisition unit 21 acquires a plurality of training images associated with focus position information relating to the focus position at the time of image capture (S01, training image acquisition step). The training image acquisition unit 21 also acquires focus position information relating to the focus position at the time of focusing corresponding to each of the training images. Next, the feature output model generation unit 22 generates a feature output model from the training images by machine learning (S02, feature output model generation step). At this time, the feature amounts of the two training images 71 are compared according to the focus position information associated with the two different training images, and machine learning is performed based on the comparison result. Next, the focus position estimation model generation unit 23 generates a focus position estimation model from the focus position information by machine learning (S03, focus position estimation model generation step).

[0095] The generated feature output model and focus position estimation model are output from the feature output model generation system 20 to the focus position estimation system 30. The focus position estimation system 30 stores the feature output model and focus position estimation model and uses them in the following processes. The above is the feature output model generation method, which is the process executed by the feature output model generation system 20 according to this embodiment.

[0096] Next, using the flowchart of Figure 8, we will explain the process performed when estimating the focus position at the time of focusing corresponding to the estimation target image, i.e., the focus position estimation method, which is the process performed by the focus position estimation system 30 of this embodiment.

[0097] In this process, first, the estimation target image acquisition unit 31 acquires an estimation target image (S11, estimation target image acquisition step). The estimation target image is based on an image acquired by imaging for focusing by the inspection device 40 or the observation device 50. Next, the focus position estimation unit 32 uses the feature output model to output feature amounts of the estimation target image from the estimation target image. Next, the focus position estimation unit 32 uses the focus position estimation model to estimate a focus position at the time of focusing corresponding to the estimation target image from the feature amounts of the estimation target image (S12, focus position estimation step).

[0098] Next, the control unit 33 controls the focal position when the inspection device 40 or the observation device 50 captures a new image of the object to be imaged, based on the estimated focal position (S13, control step). As a result, the image captured by the inspection device 40 or the observation device 50 becomes a focused image. The above is the focal position estimation method, which is processing executed by the focal position estimation system 30 according to this embodiment.

[0099] In this embodiment, a feature output model that outputs image features is generated by machine learning. At this time, the features of two different training images are compared according to focus position information associated with the two training images, and machine learning is performed based on the comparison results. According to this generation, for example, by using an existing trained model that inputs image-based information, a feature output model can be generated in a short time. In other words, according to this embodiment, a feature output model that is a trained model used for image-based estimation, such as focus position estimation, can be generated by short-time learning.

[0100] However, as described above, it is not necessary to use an existing trained model to generate a feature output model. Even in this case, it is possible to generate a feature output model that can output appropriate features according to the focus position.

[0101] Furthermore, as described above, machine learning for generating a feature output model may be performed so that when two different training images have the same focus position, the difference in the features of the two training images is small, and when two different training images have different focus positions, the difference in the features of the two training images is large. This configuration allows for reliable and appropriate generation of a feature output model. However, machine learning does not necessarily have to be performed as described above, and may be performed based on the results of comparing the features of the two training images.

[0102] As described above, the learning images and estimation target images may be images in which radiation from an imaging object is detected, images in which light from the imaging object is detected when the imaging object is irradiated with light, or images in which electrical characteristics of the imaging object are detected when the imaging object is irradiated with light. Furthermore, the learning images and estimation target images may be images in which light of a specific wavelength is irradiated onto the imaging object. These configurations enable the generation and use of an appropriate feature output model according to the type of image used. However, the learning images and estimation target images are not limited to those described above, and may be any images according to the focal position.

[0103] Furthermore, as in the present embodiment, the feature output model generation system 20 may further include a focus position estimation model generation unit 23 that generates a focus position estimation model. With this configuration, it is possible to generate a focus position estimation model that estimates the focus position at the time of focusing from an image. That is, with this configuration, it is possible to generate, in addition to the feature output model, a trained model used for estimating the focus position based on an image through short training.

[0104] However, the feature amount output model generation system 20 does not have to include the focus position estimation model generation unit 23. That is, the feature amount output model generation system 20 may be configured to generate only the feature amount output model. Furthermore, the generated feature amount output model may be used for purposes other than estimating the focus position when in focus.

[0105] In the focus position estimation system 30 according to this embodiment, the above-described feature output model is used to estimate the focus position when in focus. Therefore, according to this embodiment, it is possible to estimate the focus position when in focus based on an image with a short preparation time. Furthermore, according to the focus position estimation system 30 according to this embodiment, it is possible to estimate the focus position when in focus by capturing an image for focusing once. Therefore, it is possible to estimate the focus position when in focus more quickly than when capturing an image multiple times while changing the focus position to search for the focus position when in focus.

[0106] Furthermore, the focus position estimation model described above may be used to estimate the focus position when in focus. With this configuration, the focus position when in focus can be reliably and appropriately estimated. However, the focus position estimation model described above does not need to be used to estimate the focus position when in focus; estimation can be performed from the feature values ​​output from the feature value output model.

[0107] Furthermore, as in the present embodiment, the focal position estimation system 30 may further include a control unit 33 that controls the focal position when the object is imaged in the inspection device 40 or the observation device 50, based on the estimated focal position. With this configuration, the object can be imaged in focus in the inspection device 40 or the observation device 50. However, the focal position estimation system 30 does not have to include the control unit 33. In other words, the focal position estimation system 30 only needs to estimate the focal position when in focus.

[0108] Furthermore, the focal position estimation system 30 according to this embodiment and the above-described inspection apparatus 40 or observation apparatus 50 can be configured as a series of systems. That is, the semiconductor inspection system according to this embodiment may be a system including the focal position estimation system 30 and the inspection apparatus 40. Furthermore, the living body observation system according to this embodiment may be a system including the focal position estimation system 30 and the observation apparatus 50.

[0109] The focus position estimation system 30 may estimate the focus position when in focus for multiple positions on the image captured by the inspection device 40 or the observation device 50. In this case, the estimation target image acquisition unit 31 acquires multiple estimation target images by cutting them out from multiple positions on the image captured by the inspection device 40 or the observation device 50. For example, the estimation target image acquisition unit 31 cuts out estimation target images from multiple portions of the image in which the captured object is captured. The focus position estimation unit 32 estimates the focus position when in focus for each of the multiple estimation target images. For example, the focus position estimation unit 32 may generate a focus map that indicates the focus position when in focus for each position. The control unit 33 may control the inspection device 40 or the observation device 50 to adjust the focus position for each position and perform imaging based on the focus position when in focus for each position.

[0110] Fig. 9 shows the focus position (focus map) estimated for each position when in focus. Fig. 9(a) shows the focus position for each position of the image capture object 90 when in focus on a plane (XY plane) perpendicular to the imaging direction (Z-axis direction). Fig. 9(b) shows the focus position for each position of the image capture object 90 when in focus on a plane (XZ plane) when the imaging direction (Z-axis direction) is viewed from the side (Y-axis). Each value in Fig. 9 indicates the difference between the focus position when the estimation target image was captured and the focus position when in focus.

[0111] When inspecting a wafer covered with μLEDs, even if the focal position is aligned at the center of the wafer, blurring occurs near the edges of the wafer. Therefore, by generating and controlling the focus map as described above, it is possible to capture images that can be used appropriately for inspection at each position. Note that the above configuration may also be used for imaging objects other than semiconductor devices, such as biological samples.

[0112] In this embodiment, the computer 10 includes the feature output model generation system 20 and the focus position estimation system 30, but the feature output model generation system 20 and the focus position estimation system 30 may be implemented independently.

[0113] Next, a description will be given of a feature output model generation program and a focus position estimation program for executing the above-described series of processes by the feature output model generation system 20 and the focus position estimation system 30. As shown in Fig. 10, the feature output model generation program 200 is stored in a program storage area 211 formed in a computer-readable recording medium 210 that is inserted into a computer and accessed, or that is provided in the computer. The recording medium 210 may be a non-transitory recording medium.

[0114] The feature output model generation program 200 includes a learning image acquisition module 201, a feature output model generation module 202, and a focal position estimation model generation module 203. Functions realized by executing the learning image acquisition module 201, the feature output model generation module 202, and the focal position estimation model generation module 203 are similar to the functions of the learning image acquisition unit 21, the feature output model generation unit 22, and the focal position estimation model generation unit 23 of the feature output model generation system 20 described above, respectively.

[0115] 11 , the focus position estimation program 300 is stored in a program storage area 311 formed on a computer-readable recording medium 310 that is inserted into a computer and accessed, or that is provided in the computer. The recording medium 310 may be a non-transitory recording medium. Note that the recording medium 310 may be the same as the recording medium 210.

[0116] The focus position estimation program 300 includes an estimation target image acquisition module 301, a focus position estimation module 302, and a control module 303. Functions realized by executing the focus position estimation module 302 and the control module 303 are similar to the functions of the estimation target image acquisition unit 31, the focus position estimation unit 32, and the control unit 33 of the focus position estimation system 30 described above, respectively.

[0117] The feature output model generating program 200 and the focus position estimating program 300 may be configured so that part or all of them are transmitted via a transmission medium such as a communication line, and are received and recorded (including installed) by another device. Furthermore, each module of the feature output model generating program 200 and the focus position estimating program 300 may be installed on one of multiple computers, rather than on a single computer. In this case, the above-described series of processes are performed by a computer system including the multiple computers.

[0118] The feature output model generation system, feature output model generation method, feature output model generation program, and feature output model disclosed herein have the following configurations. [1] A feature output model generation system that receives image-based information as input and generates a feature output model that outputs features of the image, a learning image acquisition means for acquiring a plurality of learning images each associated with focus position information relating to a focus position at the time of image capture; a feature output model generation means for generating a feature output model by machine learning from the learning images acquired by the learning image acquisition means, The feature output model generation means compares the features of two different learning images according to focus position information associated with the two learning images, and performs machine learning based on the comparison results. [2] The feature output model generation system described in [1], wherein the feature output model generation means performs machine learning so that, when two different training images have the same focus position, the difference in the features of the two training images becomes small, and, when two different training images have different focus positions, the difference in the features of the two training images becomes large. [3] The feature output model generation system according to [1] or [2], wherein the learning image acquisition means acquires, as learning images, images that detect radiation from the object to be imaged, images that detect light from the object to be imaged when light is irradiated onto the object to be imaged, or images that detect the electrical characteristics of the object to be imaged when light is irradiated onto the object to be imaged. [4] The feature output model generation system according to [3], wherein the learning image acquisition means acquires, as a learning image, an image obtained by irradiating a target object with light of a specific wavelength. [5] The learning image acquisition means acquires focus position information relating to a focus position at the time of focusing corresponding to each of the acquired learning images, The feature output model generation system according to any one of [1] to [4], further comprising a focus position estimation model generation means that, by machine learning, inputs a feature output from the feature output model generated by the feature output model generation means from the focus position information acquired by the learning image acquisition means, and generates a focus position estimation model that estimates the focus position when in focus corresponding to the image related to the feature. [6] A feature output model generation method for generating a feature output model that receives image-based information as input and outputs a feature of the image, comprising: a learning image acquisition step of acquiring a plurality of learning images associated with focus position information relating to focus positions at the time of image capture; a feature output model generating step of generating a feature output model by machine learning from the learning images acquired in the learning image acquiring step, In the feature output model generation step, the feature values ​​of two different training images are compared according to focus position information associated with the two training images, and machine learning is performed based on the comparison results. [7] The feature output model generation method described in [6], wherein in the feature output model generation step, machine learning is performed so that if two different training images have the same focus position, the difference in the features of the two training images becomes small, and if two different training images have different focus positions, the difference in the features of the two training images becomes large. [8] A feature output model generation method according to [6] or [7], wherein in the learning image acquisition step, an image in which radiation from the object to be imaged is detected, an image in which light from the object to be imaged is detected when light is irradiated onto the object to be imaged, or an image in which the electrical characteristics of the object to be imaged when light is irradiated onto the object to be imaged are detected is acquired as a learning image. [9] The feature output model generation method according to [8], wherein in the learning image acquisition step, an image obtained by irradiating the object to be imaged with light of a specific wavelength is acquired as the learning image.

[10] In the learning image acquisition step, focus position information relating to a focus position at the time of focusing corresponding to each of the acquired learning images is acquired, The feature output model generation method according to any one of [6] to [9], further comprising a focus position estimation model generation step of inputting, by machine learning, feature values ​​output from the feature output model generated in the feature output model generation step from the focus position information acquired in the learning image acquisition step, and generating a focus position estimation model that estimates the focus position when in focus corresponding to the image related to the feature values.

[11] A feature output model generation program that causes a computer to operate as a feature output model generation system that receives image-based information as input and generates a feature output model that outputs a feature of the image, The computer, a learning image acquisition means for acquiring a plurality of learning images each associated with focus position information relating to a focus position at the time of image capture; a feature output model generating means for generating a feature output model by machine learning from the learning images acquired by the learning image acquiring means; The feature output model generation means is a feature output model generation program that compares the features of two different learning images according to focus position information associated with the two learning images, and performs machine learning based on the comparison results.

[12] The feature output model generation program described in

[11] , wherein the feature output model generation means performs machine learning so that, when two different training images have the same focus position, the difference in the features of the two training images becomes small, and, when two different training images have different focus positions, the difference in the features of the two training images becomes large.

[13] The feature output model generation program described in

[11] or

[12] , wherein the learning image acquisition means acquires, as learning images, images that detect radiation from the object to be imaged, images that detect light from the object to be imaged when light is irradiated onto the object to be imaged, or images that detect the electrical characteristics of the object to be imaged when light is irradiated onto the object to be imaged.

[14] The feature output model generation program according to

[13] , wherein the learning image acquisition means acquires an image obtained by irradiating a target object with light of a specific wavelength as a learning image.

[15] The learning image acquisition means acquires focus position information relating to a focus position at the time of focusing corresponding to each of the acquired learning images, The computer The feature output model generation program according to any one of

[11] to

[14] , which also functions as a focus position estimation model generation means that inputs, by machine learning, features output from a feature output model generated by the feature output model generation means from focus position information acquired by the learning image acquisition means, and generates a focus position estimation model that estimates the focus position at the time of focusing corresponding to the image related to the features.

[16] A feature output model for inputting image-based information and causing a computer to function to output features of the image, generated by machine learning from a plurality of learning images associated with focus position information relating to the focus position at the time of image capture; A feature output model in which the features of two different learning images are compared according to focus position information associated with the two learning images, and machine learning is performed based on the comparison results.

[17] The feature output model described in

[16] is generated by performing machine learning so that, when two different training images have the same focus position, the difference in the features of the two training images is small, and, when two different training images have different focus positions, the difference in the features of the two training images is large.

[18] A feature output model according to

[16] or

[17] , which is generated by performing machine learning using as training images an image in which radiation from an object to be imaged is detected, an image in which light from the object to be imaged is detected when light is irradiated onto the object to be imaged, or an image in which electrical characteristics of the object to be imaged when light is irradiated onto the object to be imaged are detected.

[19] A feature output model according to

[18] , which is generated by performing machine learning on an image obtained by irradiating a target object with light of a specific wavelength as a learning image. [Explanation of symbols]

[0119] 10...computer, 20...feature output model generation system, 21...learning image acquisition unit, 22...feature output model generation unit, 23...focal position estimation model generation unit, 30...focal position estimation system, 31...estimation target image acquisition unit, 32...focal position estimation unit, 33...control unit, 40...inspection device, 41...camera, 42...mounting unit, 43...light source, 44...optical system, 45...objective lens, 46...stage, 50...observation device, 200...feature output model generation program, 201...learning image acquisition module, 202...feature output model generation module, 203...focal position estimation model generation module, 210...recording medium, 211...program storage area, 300...focal position estimation program, 301...estimation target image acquisition module, 302...focal position estimation module, 303...control module, 310...recording medium, 311...program storage area.

Claims

1. 1. A focus position estimation model generation system that receives a feature amount of an image acquired based on the image and generates a focus position estimation model that estimates a focus position at a time of focusing corresponding to the image related to the feature amount, a learning image acquisition means for acquiring a plurality of learning images and focus position information relating to the focus positions at the time of focusing corresponding to each of the learning images; a focus position estimation model generation means for acquiring feature amounts of the learning images based on the learning images acquired by the learning image acquisition means, and for generating a focus position estimation model by machine learning from the acquired feature amounts and the focus position information acquired by the learning image acquisition means; Equipped with A focus position estimation model generation system in which the feature is obtained by inputting information based on an image relating to the feature into a feature output model generated by machine learning.

2. The focal position estimation model generation system according to claim 1 , wherein the feature amount reflects the focal position when the image related to the feature amount was captured.

3. The focus position estimation model generation system according to claim 1 , wherein the feature amount is a vector with a predetermined number of dimensions.

4. The focus position estimation model generation system according to any one of claims 1 to 3, wherein the learning image acquisition means acquires, as learning images, images that detect radiation from the object to be imaged, images that detect light from the object to be imaged when light is irradiated onto the object to be imaged, or images that detect the electrical characteristics of the object to be imaged when light is irradiated onto the object to be imaged.

5. 5. The focal position estimation model generation system according to claim 4, wherein the learning image acquisition means acquires, as the learning image, an image obtained when the object to be imaged is irradiated with light of a specific wavelength.

6. 1. A focus position estimation model generation method for generating a focus position estimation model by inputting a feature amount of an image acquired based on the image and estimating a focus position at a time of focusing corresponding to the image related to the feature amount, comprising: a learning image acquisition step of acquiring a plurality of learning images and focus position information relating to focus positions at the time of focusing corresponding to each of the learning images; a focus position estimation model generation step of acquiring feature amounts of the learning images based on the learning images acquired in the learning image acquisition step, and generating a focus position estimation model by machine learning from the acquired feature amounts and the focus position information acquired in the learning image acquisition step; Including, A focus position estimation model generation method in which the feature is obtained by inputting information based on an image relating to the feature into a feature output model generated by machine learning.

7. a focus position estimation model generation program that causes a computer to operate as a focus position estimation model generation system that receives a feature amount of an image acquired based on the image and generates a focus position estimation model that estimates a focus position at the time of focusing that corresponds to the image related to the feature amount, The computer a learning image acquisition means for acquiring a plurality of learning images and focus position information relating to the focus positions at the time of focusing corresponding to each of the learning images; a focus position estimation model generation means for acquiring feature amounts of the learning images based on the learning images acquired by the learning image acquisition means, and for generating a focus position estimation model by machine learning from the acquired feature amounts and the focus position information acquired by the learning image acquisition means; It functions as A focus position estimation model generation program in which the feature is obtained by inputting information based on an image relating to the feature into a feature output model generated by machine learning.

8. A focus position estimation model for inputting a feature amount of an image acquired based on the image, and causing a computer to function to perform a calculation according to the input and output information, A focus position estimation model generated by the focus position estimation model generation system according to any one of claims 1 to 5.

Images