Tilt estimation system, tilt estimation method, tilt estimation program, semiconductor inspection system, and biological observation system

Abstract

The tilt of an object to be imaged can be estimated with a simple configuration in a short time. [Solution] The tilt estimation system 30 is a system that estimates the tilt of an imaged object captured in an image, and includes an estimated object image acquisition unit 31 that acquires estimated object images, which are multiple partial images, from the image, a focus position estimation unit 32 that outputs features from the estimated object images using a feature output model and estimates the focus position at the time of focusing corresponding to each of the multiple estimated object images, and a tilt estimation unit 33 that estimates the tilt of the imaged object from these focus positions at the time of focusing. The feature output model is generated by machine learning from multiple training images that are associated with focus position information related to the focus position at the time of imaging, and the features of two different training images are compared according to the focus position information associated with the two training images, and machine learning is performed based on the comparison results.

Description

[Technical Field]

[0001] The present invention relates to a tilt estimation system, a tilt estimation method, a tilt estimation program, a semiconductor inspection system, and a living body observation system, which estimate the tilt of an object captured in an image. [Background technology]

[0002] It has been proposed to estimate the tilt of an object to be imaged, for example, the tilt with respect to the imaging direction, when imaging the object. For example, Patent Document 1 discloses estimating the tilt of a semiconductor device when imaging the semiconductor device for semiconductor device inspection. The estimated tilt is used to adjust the attitude of the semiconductor device. In Patent Document 1, the tilt of the object to be imaged is estimated using an optical system including a dedicated lens (for example, a relay lens system) and a scanner. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 154895 Summary of the Invention [Problem to be solved by the invention]

[0004] When estimating the tilt of an object to be imaged using a special optical system as disclosed in Patent Document 1, a configuration (e.g., a lens turret) and space for providing the special optical system within the device are required. Furthermore, preparing the special optical system requires costs. Furthermore, the method disclosed in Patent Document 1 requires processes such as turret switching, laser scanning, and alignment, which makes it take a long time to estimate the tilt.

[0005] One embodiment of the present invention has been made in consideration of the above, and aims to provide a tilt estimation system, a tilt estimation method, and a tilt estimation program that are capable of estimating the tilt of an object to be imaged in a short time with a simple configuration, as well as a semiconductor inspection system and a biological observation system related thereto. [Means for solving the problem]

[0006] In order to achieve the above object, a tilt estimation system according to one embodiment of the present invention is a tilt estimation system that estimates the tilt of an imaged object captured in an image, and includes: an estimation target image acquisition means that acquires an image containing the imaged object and acquires estimation target images, which are multiple partial images, from the image; a focus position estimation means that uses a feature output model that inputs image-based information and outputs feature values ​​of the image to output feature values ​​for each of the multiple estimation target images acquired by the estimation target image acquisition means, and estimates a focus position at the time of focusing corresponding to each of the multiple estimation target images from the output feature values; and a tilt estimation means that estimates the tilt of the imaged object captured in the image from the focus position at the time of focusing corresponding to each of the multiple estimation target images estimated by the focus position estimation means. The feature output model is generated by machine learning from multiple training images associated with focus position information related to the focus position at the time of capturing, and the feature values ​​of the two different training images are compared according to the focus position information associated with the two training images, and machine learning is performed based on the comparison results.

[0007] In a tilt estimation system according to an embodiment of the present invention, the inclination of an imaged object is estimated by estimating the in-focus focal position corresponding to each of the estimation target images, which are each a plurality of partial images of an image. Therefore, once an image of the imaged object is obtained, the inclination of the imaged object can be estimated in a short time. Furthermore, the feature output model used for estimating the focal position can output feature values ​​appropriate for estimating the focal position, and by using this, the inclination of the imaged object can be appropriately estimated. Therefore, the tilt estimation system according to an embodiment of the present invention can estimate the inclination of an imaged object with a simple configuration and in a short time.

[0008] The focus position estimation means may input the feature amounts output from the feature amount output model and estimate the focus positions at the time of focusing corresponding to each of the plurality of estimation target images using a focus position estimation model that estimates the focus positions at the time of focusing corresponding to the images related to the feature amounts, and the focus position estimation model may be generated by machine learning from focus position information related to the focus positions at the time of focusing corresponding to each of the training images. With this configuration, the focus position estimation model can be used to reliably and appropriately estimate the focus position. As a result, the tilt of the imaged object can be reliably and appropriately estimated.

[0009] The tilt estimation system may further include a control unit that controls the tilt of the object when capturing an image based on the tilt of the object estimated by the tilt estimation unit. With this configuration, the object can be captured at an appropriate tilt.

[0010] A semiconductor inspection system and a living-body observation system according to one embodiment of the present invention can be configured to include the tilt estimation system described above. That is, a semiconductor inspection system according to one embodiment of the present invention includes the tilt estimation system described above, a mounting unit on which a semiconductor device is placed as an imaging target for the tilt estimation system, and an inspection unit that inspects the semiconductor device. Also, a living-body observation system according to one embodiment of the present invention includes the tilt estimation system described above, a mounting unit on which a biological sample is placed as an imaging target for the tilt estimation system, and an observation unit that observes the biological sample.

[0011] Incidentally, one embodiment of the present invention can be described as an invention of a tilt estimation system as described above, and can also be described as an invention of a tilt estimation method and a tilt estimation program as described below.

[0012] That is, a tilt estimation method according to one embodiment of the present invention is a tilt estimation method for estimating the tilt of an imaged object captured in an image, and includes: an estimation target image acquisition step of acquiring an image containing the imaged object and acquiring estimation target images, which are multiple partial images, from the image; a focus position estimation step of using a feature output model that inputs image-based information and outputs feature values ​​of the image to output feature values ​​for each of the multiple estimation target images acquired in the estimation target image acquisition step, and estimating a focus position at the time of focusing corresponding to each of the multiple estimation target images from the output feature values; and a tilt estimation step of estimating the tilt of the imaged object captured in the image from the focus positions at the time of focusing corresponding to each of the multiple estimation target images estimated in the focus position estimation step, wherein the feature output model is generated by machine learning from multiple training images associated with focus position information related to the focus position at the time of imaging, and the feature values ​​of the two different training images are compared according to the focus position information associated with the two training images, and machine learning is performed based on the comparison results.

[0013] In the focus position estimation step, the feature output from the feature output model is input, and the focus position at the time of focusing corresponding to each of the multiple estimation target images is estimated using a focus position estimation model that estimates the focus position at the time of focusing corresponding to the image related to the feature, and the focus position estimation model may be generated by machine learning from focus position information related to the focus position at the time of focusing corresponding to each of the training images.

[0014] The tilt estimation method may further include a control step of controlling the tilt of the object to be imaged at the time of image capture, based on the tilt of the object to be imaged estimated in the tilt estimation step.

[0015] Furthermore, a tilt estimation program according to one embodiment of the present invention is a tilt estimation program that causes a computer to operate as a tilt estimation system that estimates the tilt of an imaged object captured in an image. The tilt estimation program causes the computer to function as: an estimation target image acquisition means that acquires an image containing the imaged object and acquires estimation target images, which are multiple partial images, from the image; a focus position estimation means that uses a feature output model that inputs image-based information and outputs feature values ​​for each of the multiple estimation target images acquired by the estimation target image acquisition means, and outputs feature values ​​for each of the multiple estimation target images from the output feature values; and a tilt estimation means that estimates the tilt of the imaged object captured in the image from the focus positions at the time of focus corresponding to each of the multiple estimation target images estimated by the focus position estimation means. The feature output model is generated by machine learning from multiple training images associated with focus position information related to the focus position at the time of capture, and the feature values ​​of the two different training images are compared according to the focus position information associated with the two training images, and machine learning is performed based on the comparison results.

[0016] The focus position estimation means inputs the feature output from the feature output model and estimates the focus position at the time of focusing corresponding to each of the multiple estimation target images using a focus position estimation model that estimates the focus position at the time of focusing corresponding to the image related to the feature, and the focus position estimation model may be generated by machine learning from focus position information related to the focus position at the time of focusing corresponding to each of the training images.

[0017] The tilt estimation program may also cause the computer to function as a control unit that controls the tilt of the object to be imaged during imaging, based on the tilt of the object to be imaged estimated by the tilt estimation unit. [Effects of the Invention]

[0018] According to one embodiment of the present invention, the inclination of an object to be imaged can be estimated with a simple configuration in a short time. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a diagram illustrating a configuration of a feature output model generation system and a tilt estimation system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram illustrating an example of a partial configuration of an inspection device. [Figure 3] 10 is an example of an image captured at each focal position. [Figure 4] FIG. 10 is a diagram for explaining generation of a feature output model by machine learning. [Figure 5] FIG. 10 is a diagram illustrating an existing trained model used to generate a feature output model. [Figure 6] 1A and 1B are diagrams illustrating examples of a defocused image and a focused image. [Figure 7] FIG. 10 is a diagram illustrating an example of an estimation target image for an image. [Figure 8] FIG. 10 is a diagram for explaining estimation of the tilt of an image capture object. [Figure 9] 1 is a flowchart illustrating a feature output model generating method that is a process executed in the feature output model generating system according to an embodiment of the present invention. [Figure 10] 3 is a flowchart illustrating a tilt estimation method that is a process executed by the tilt estimation system according to the embodiment of the present invention. [Figure 11] FIG. 1 is a diagram showing a configuration of a feature output model generation program according to an embodiment of the present invention, together with a recording medium. [Figure 12] FIG. 2 is a diagram showing the configuration of an inclination estimation program according to an embodiment of the present invention, together with a recording medium. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of a tilt estimation system, a tilt estimation method, a tilt estimation program, a semiconductor inspection system, and a living body observation system according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated explanations will be omitted.

[0021] FIG. 1 shows a computer 10 that is a tilt estimation system according to this embodiment. The computer 10 is a device (system) that performs information processing on images. Specifically, the computer 10 performs information processing on images captured by at least one of an inspection device 40 and an observation device 50. Note that the computer 10 may also perform information processing on images captured by devices other than the inspection device 40 and the observation device 50. In other words, the computer 10 may be applied to devices that capture images other than the inspection device 40 and the observation device 50.

[0022] The inspection apparatus 40 is an apparatus that captures an image of a semiconductor device and inspects the semiconductor device based on the captured image. For example, the inspection apparatus 40 performs failure analysis of the semiconductor device. The semiconductor device to be inspected is, for example, a wafer covered with μLEDs (Light Emitting Diodes). The failure analysis is performed by, for example, light emission analysis, heat generation analysis, analysis using a pattern image, or laser analysis (OBIRCH, OBIC, DALS, etc.). The inspection apparatus 40 may be a conventional inspection apparatus. All of the components described below may be included in a conventional inspection apparatus.

[0023] FIG. 2 shows, for example, the configuration of a portion of an inspection device 40 that performs optical emission analysis. As shown in FIG. 2, the inspection device 40 includes a camera 41, a mounting unit 42, a light source 43, an optical system 44, an objective lens 45, and a stage 46. The camera 41 is an imaging device that captures an image of a semiconductor device, which is an object to be imaged. The camera 41 is, for example, an InGaAs camera. The mounting unit 42 is configured to mount the semiconductor device, which is an object to be imaged. In FIG. 2, a standard sample 60 mounted on the mounting unit 42 is not an object to be inspected, but corresponds to an image corresponding to a focal position, which will be described later. The standard sample 60 is a sample on which an artificial pattern (for example, a radial stripe pattern as shown in FIG. 2) is applied.

[0024] The mounting unit 42 is configured to be able to control the tilt (posture) of the image capture object placed on it with respect to the imaging direction. For example, the mounting unit 42 is configured so that the tilt of the mounting surface on which the image capture object is placed with respect to the imaging direction is variable. In this way, the mounting unit 42 can perform tilt correction of the image capture object. A conventional mounting unit that can control the tilt of the image capture object can be used as the mounting unit 42. The tilt of the image capture object when it is imaged is controlled by the computer 10, as will be described later.

[0025] The light source 43 is a device that outputs illumination light to be irradiated onto the object to be imaged. The light source 43 may output illumination light of a specific wavelength (for example, a standard wavelength of 1100 nm, or wavelengths other than the standard wavelength of 900 nm and 1300 nm). Alternatively, the wavelength of the illumination light may be switched, for example, by providing a plurality of light sources 43. The optical system 44 is an optical system for irradiating the illumination light output from the light source 43 onto the semiconductor device, which is the object to be imaged. The objective lens 45 is an objective lens used for imaging by the camera 41, such as a solid immersion lens (SIL). The stage 46 is a member for adjusting the focal position during imaging by the camera 41. The stage 46 may be movable not only in the imaging direction (focal position direction, Z-axis direction) but also in any direction in three dimensions (i.e., an XYZ stage). The focal position during imaging in the inspection device 40 is controlled by the computer 10, as described below. The inspection device 40 includes an inspection unit that inspects semiconductor devices using images obtained by the above-described configuration.

[0026] The configuration of the inspection device 40 does not need to be as described above, and a different configuration may be used depending on the analysis method. For example, in heat generation analysis, an InSb camera may be used as the imaging device (camera 41) without the need for illumination (light source 43). Furthermore, when performing analysis using a pattern image, an incoherent light source or a coherent light source may be used as the illumination (light source 43), and a two-dimensional detector, or an optical scanning device and a photodiode may be used as the imaging device. When performing laser analysis, an incoherent light source or a coherent light source may be used as the illumination (light source 43), and an electrical characteristic acquisition device for semiconductor devices may be used as the imaging device.

[0027] The observation device 50 is a device that captures an image of a biological sample placed on a slide glass and observes the captured image of the biological sample. The observation device 50 may be a conventional observation device. For example, the observation device 50 is the conventional virtual slide scanner described above. The observation device 50 includes an imaging device that captures an image of the biological sample, which is the object to be imaged, a mounting section on which the biological sample, which is the object to be imaged, is placed, and an observation section that observes the biological sample using the captured image. The focal position during imaging in the observation device 50 is controlled by the computer 10, as will be described later. Like the mounting section 42 of the inspection device 40 described above, the mounting section of the observation device 50 is also configured to be able to control the inclination (attitude) of the imaged object placed thereon with respect to the imaging direction.

[0028] Imaging by the inspection device 40 and the observation device 50 must be performed with the object to be imaged properly tilted. An appropriate tilt of the object to be imaged means, for example, a state in which the surface of the object to be imaged is perpendicular to the imaging direction, i.e., a state in which the object to be imaged is not tilted relative to the imaging direction. This is to ensure proper inspection or observation of the object to be imaged. For example, a semiconductor device that is the subject of failure analysis has its backside polished during processing. Warping and distortion occur on the surface of the semiconductor device sample due to distortion during polishing and process-induced stress. If the semiconductor device is tilted, problems may occur, for example, with contact with the solid immersion lens, and therefore it is necessary to correct the tilt during imaging, i.e., tilt correction. Note that tilt of the object to be imaged may also occur for reasons other than those mentioned above.

[0029] Imaging by the inspection device 40 and the observation device 50 must be performed with the object being imaged in focus. This is to ensure proper inspection or observation of the object being imaged. FIG. 3 shows examples of images captured according to the focal position. This image was captured of the standard sample 60 shown in FIG. 2. FIG. 3(a) is an image captured at the in-focus focal position. FIG. 3(b) is an image captured when the focal position is 100 μm away from the in-focus focal position. FIG. 3(c) is an image captured when the focal position is further away than in FIG. 3(b). FIG. 3(d) is an image captured when the focal position is 500 μm away from the in-focus focal position, further away than in FIG. 3(c). That is, FIG. 3(a) is an example of a focused image, and FIGS. 3(b) to 3(d) are examples of defocused images.

[0030] The computer 10 performs information processing to ensure that imaging by the inspection device 40 and the observation device 50 is performed with the tilt of the object being imaged appropriately. The computer 10 may also perform information processing to ensure that imaging by the inspection device 40 and the observation device 50 is performed with the object being imaged in focus. The computer 10 functionally includes a feature output model generation system 20 that generates a trained model through machine learning, and a tilt estimation system 30 that performs information processing to enable the above imaging using the trained model generated by the feature output model generation system 20. As will be described in detail later, the feature output model generation system 20 is a system (apparatus) that inputs information based on an image and generates a feature output model that outputs features of the image. As will be described in detail later, the tilt estimation system 30 is a system (apparatus) that estimates the tilt of the object being imaged in an image. Note that in this embodiment, the inspection device 40 and the observation device 50 are shown as imaging devices, but other devices (systems) that capture images of the object being imaged may also be used.

[0031] The computer 10 is a conventional computer including hardware such as a CPU (Central Processing Unit), memory, and a communication module. The computer 10 may also be a computer system including multiple computers. The computer 10 may also be configured using cloud computing. The functions of the computer 10, which will be described later, are realized by these components operating through programs or the like. The computer 10, the inspection device 40, and the observation device 50 are connected to each other so as to be able to send and receive information.

[0032] Next, the functions of the feature output model generation system 20 and the tilt estimation system 30 included in the computer 10 according to this embodiment will be described. As shown in FIG. 1, the feature output model generation system 20 includes a learning image acquisition unit 21, a feature output model generation unit 22, and a focus position estimation model generation unit 23.

[0033] Before describing each function of the feature output model generation system 20, we will explain the trained models generated by the feature output model generation system 20. The trained models generated by the feature output model generation system 20 are a feature output model and a focus position estimation model.

[0034] The feature output model is a model that receives image-based information as input and outputs feature quantities of the image. The images used as input to the feature output model are partial images of images captured by the inspection device 40 and the observation device 50. The feature quantities output from the feature output model are information indicating the features of the input image. In this embodiment, the features reflect the focal position when the image was captured. In other words, the feature output model is an optical model related to optical features. The feature quantities are, for example, vectors with a preset number of dimensions (for example, 1024 dimensions). The feature quantities are used as input to a focal position estimation model, as will be described later.

[0035] The feature output model includes, for example, a neural network. The neural network may be multi-layered. That is, the feature output model may be generated by deep learning. The neural network may also be a convolutional neural network (CNN).

[0036] The feature output model has neurons for inputting information based on an image to its input layer. For example, the information input to the feature output model is the pixel value of each pixel of the image. In this case, the input layer has neurons equal to the number of pixels in the image, and each neuron receives the pixel value of the corresponding pixel. As will be described later, the image related to the information input to the feature output model is an image of a preset size (for example, an image of 224 × 224 pixels).

[0037] Note that the information input to the feature output model may be other than the pixel values ​​of each pixel as long as it is based on the image. For example, the information may be a feature to be input to the feature output model, obtained by performing preprocessing such as conventional image processing on the image to reduce the influence of the imaging environment. By performing such preprocessing, it is possible to improve the efficiency of machine learning and the accuracy of the generated feature output model.

[0038] The feature output model is provided with neurons for outputting features to the output layer, for example, neurons equal in number to the dimension of the feature vector.

[0039] The focus position estimation model is a model that inputs a feature output from the feature output model and estimates a focus position at the time of focusing corresponding to an image related to the feature. The focus position estimation model, for example, outputs information indicating the difference between the focus position when the image related to the input feature was captured and the focus position at the time of focusing as an estimated result of the focus position at the time of focusing. The difference is, for example, a value obtained by subtracting the focal length corresponding to the focus position when the image related to the feature was captured from the focal length corresponding to the focus position at the time of focusing. That is, in this case, the output value is a value indicating the focus position when the image in a coordinate system in which the focus position at the time of focusing is set to 0 is captured. The focus position at the time of focusing is a focus position for capturing an image by focusing on an object captured in the image related to the input feature. A focused image can be captured by changing the focus position by the above difference from the focus position when a defocused image related to the input feature was captured.

[0040] In this case, candidates for the difference may be set in advance, and the focus position estimation model may output a value indicating the degree to which the candidate is appropriate. For example, the candidate differences may be +50 μm, 0 μm, −50 μm, −100 μm, etc., and the focus position estimation model may output a value indicating the degree to which each candidate is appropriate. For example, the candidate with the highest value is set as the difference. Alternatively, the focus position estimation model may output the value of the difference itself.

[0041] Alternatively, the focus position estimation model may output information indicating the focus position itself when in focus (for example, the focal length corresponding to the focus position when in focus). In this case, candidates for the focus position itself when in focus may be set in advance, and the focus position estimation model may output a value indicating the degree to which the candidate is valid. Alternatively, the focus position estimation model may output the value of the focus position itself when in focus.

[0042] The focus position estimation model includes, for example, a neural network. The neural network may be multi-layered. That is, the focus position estimation model may be generated by deep learning. The neural network may also be a convolutional neural network (CNN).

[0043] The focus position estimation model has neurons for inputting feature quantities to an input layer. For example, the input layer has neurons corresponding to the neurons provided in the output layer of the feature quantity output model. That is, the input layer has the same number of neurons as the output layer of the feature quantity output model. The focus position estimation model has neurons for outputting the above-mentioned estimation result of the focus position when in focus. For example, the model has neurons equal to the number of candidates (when a value for each candidate is output) or one neuron (when the above-mentioned difference or the focus position itself when in focus is output).

[0044] The feature output model and the focus position estimation model may be configured by something other than a neural network.

[0045] The feature output model and the focus position estimation model are expected to be used as program modules that are part of artificial intelligence software. The feature output model and the focus position estimation model are used, for example, in a computer having a CPU and memory, and the CPU of the computer operates according to instructions from the model stored in the memory. For example, the CPU of the computer operates according to the instructions to input information to the model, perform calculations according to the model, and output results from the model. Specifically, the CPU of the computer operates according to the instructions to input information to an input layer of a neural network, perform calculations based on parameters such as trained weighting coefficients in the neural network, and output results from the output layer of the neural network.

[0046] The training image acquisition unit 21 is a training image acquisition means that acquires a plurality of training images associated with focal position information relating to the focal position at the time of image capture. The training image acquisition unit 21 may acquire, as training images, images in which radiation from an image capture object is detected, images in which light from an image capture object is detected when light is irradiated onto the image capture object, or images in which electrical characteristics of an image capture object are detected when light is irradiated onto the image capture object. The training image acquisition unit 21 may acquire, as training images, images in which light of a specific wavelength is irradiated onto the image capture object. The training image acquisition unit 21 acquires focal position information relating to the focal position at the time of focus corresponding to each acquired training image.

[0047] For example, the training image acquisition unit 21 acquires an image captured by the inspection device 40 or the observation device 50. This image shows an imaged object for the training image. The imaged object for the training image may be, for example, the standard sample 60 shown in FIG. 2. Alternatively, the imaged object for the training image may be something else (for example, an object captured by the inspection device 40 or the observation device 50). For example, as shown in FIG. 4, the training image acquisition unit 21 cuts out a partial image from the acquired image 70 to set it as a training image 71. The training image 71 is an image used in machine learning to generate the two trained models described above. Therefore, the training image acquisition unit 21 acquires a number of training images 71 sufficient to generate an appropriate trained model.

[0048] In this embodiment, the focus position at the time of focusing is estimated from a defocused image, so the multiple learning images 71 include a defocused image. Furthermore, there may be multiple defocused images, and the focus positions associated with the defocused images may be various positions. In other words, the focal lengths corresponding to the focus positions may be various distances. Furthermore, the multiple learning images 71 may include a focused image. FIG. 4 shows an example in which three images 70 are acquired. The vertical direction of the portion showing the images 70 in FIG. 4 is the imaging direction (the focus position direction, the Z-axis direction).

[0049] The training image 71 corresponds to an image used as input to the feature output model. In this case, the feature output model inputs information based on a partial image of the image captured by the inspection device 40 or the observation device 50 that is the same size as the training image 71, rather than the entire image. Therefore, the training image acquisition unit 21 cuts out the training image 71, which is a partial image of a predetermined size used as input to the feature output model, from the acquired image 70. The position from which the training image 71 is cut out in the image 70 is a portion that shows the captured object. However, the training image 71 may include a training image 71 that does not show the captured object. The position from which the training image 71 is cut out in the image 70 may be set in advance. Alternatively, the position from which the training image 71 is cut out may be determined by performing image recognition on the image 70 and estimating that the captured object is shown.

[0050] 4, the learning image acquisition unit 21 may extract multiple learning images 71 from one image 70. When extracting multiple learning images 71 from the image 70, the positions of the learning images 71 may overlap.

[0051] In the inspection device 40 or the observation device 50, imaging is performed to generate images that are the basis for the learning images 71. In this case, for example, in the inspection device 40 or the observation device 50, the position (XY) during imaging is fixed except for the imaging direction (Z-axis direction), and imaging is performed multiple times in succession with different focal positions. In this case, as shown in FIG. 4, the focal position is changed at regular intervals (steps) (ΔZ). Imaging for the learning images 71 in the inspection device 40 or the observation device 50 may be performed by methods other than those described above.

[0052] Furthermore, the learning image acquisition unit 21 may acquire the images 70 from a device other than the inspection device 40 or the observation device 50.

[0053] As described above, the training image 71 may be an image in which radiation from an object to be imaged is detected (an image used for light emission / heat generation analysis), an image in which light from the object to be imaged is detected when the object is irradiated with light (an image used for pattern analysis), or an image in which electrical characteristics of the object to be imaged when the object is irradiated with light (an image used for laser analysis). The training image 71 may also be an image in which the object to be imaged is irradiated with light of a specific wavelength (for example, light of a specific wavelength used for inspection). These are types of images typically used in the inspection device 40 or the observation device 50. However, when generating a set of feature output models and focal position estimation models, only images of one of the types may be used as the training image 71. In this case, the generated set of feature output models and focal position estimation models corresponds to a specific image type.

[0054] The wavelength of light used in the optical emission analysis varies depending on the driving voltage and design rules of the inspection device 40. Furthermore, in an actual optical system, focus deviations occur for each wavelength due to adjustments and the characteristics of optical elements. Furthermore, the detection sensitivity may be maximized at a position different from the focal position at which the pattern image (picture) is observed. Taking these factors into consideration, a feature output model for each wavelength may be generated using images for each specific wavelength, as described above. For example, a feature output model for each wavelength may be generated using images for the standard wavelength (1100 nm) and wavelengths other than the standard wavelength (900 nm, 1300 nm).

[0055] Furthermore, when generating a set of feature output models and focal position estimation models, only images (including partial images) captured by any model (type) of the inspection device 40 or observation device 50 may be used as the training images 71. In this case, the generated set of feature output models and focal position estimation models corresponds to a specific model of the inspection device 40 or observation device 50. In other words, the feature output models and focal position estimation models reflect the characteristics of the inspection device 40 or observation device 50. By making the training images correspond to a specific type of image or a specific model of the inspection device 40 or observation device 50 in this way, a trained model with higher accuracy can be obtained. Alternatively, the focal position estimation model may be common to multiple image types or models.

[0056] Each training image 71 is associated with focal position information relating to the focal position when the training image 71 was captured. The focal position information is, for example, information indicating the above-mentioned focal position. However, the focal position information may be information relating to the focal position and may be other than the above information as long as it can be used to generate the trained model described above. The focal position information is obtained, for example, as information when the image was captured by the inspection device 40 or the observation device 50. For example, the training image acquisition unit 21 receives and acquires the image associated with the focal position information from the inspection device 40 or the observation device 50.

[0057] Furthermore, the learning image acquisition unit 21 acquires focus position information relating to the focus position when in focus corresponding to each acquired learning image. The focus position information is, for example, information indicating the focus position when in focus. However, the focus position information may be information relating to the focus position when in focus and may be other than the above information as long as it is information relating to the focus position when in focus and can be used to generate the trained model described above. The focus position information is obtained by a conventional method of measuring the focus position when in focus. For example, the learning image acquisition unit 21 acquires the focus position information by receiving an input operation by the user to the computer 10 of the focus position information obtained by measurement.

[0058] The learning image acquisition unit 21 outputs the acquired information to the feature output model generation unit 22 and the focal position estimation model generation unit 23.

[0059] The feature output model generation unit 22 is a feature output model generation means that generates a feature output model by machine learning from the training images 71 acquired by the training image acquisition unit 21. The feature output model generation unit 22 compares the features of two different training images 71 according to focus position information associated with the two different training images 71, and performs machine learning based on the comparison result. The feature output model generation unit 22 may perform machine learning so that when the two different training images 71 have the same focus position, the difference in the features of the two different training images 71 becomes small, and so that when the two different training images 71 have different focus positions, the difference in the features of the two different training images 71 becomes large.

[0060] The feature output model generation unit 22 generates a feature output model as follows. The feature output model generation unit 22 receives training images 71 and focal position information related to the training images 71 from the training image acquisition unit 21. The feature output model generation unit 22 performs machine learning for generating a feature output model using two training images 71 selected from the input training images 71 as one set. The set used for machine learning includes both a set of training images 71 having the same focal position and a set of training images 71 having different focal positions. For example, the set of training images 71 having the same focal position may be training images 71 cut out from the same image 70, as shown in FIG. 4. The set of training images 71 may be selected using a predetermined method that satisfies the above conditions. The set of training images 71 may also be selected from images 70 that have the same position (XY) at the time of imaging other than the imaging direction (Z-axis direction).

[0061] The feature output model generation unit 22 performs machine learning using information based on the selected set of training images 71 as input to the feature output model. As shown in FIG. 4, when each of the training images 71 in a set is input to a feature output model 80, a feature is obtained as output for each training image 71. In FIG. 4, the value of each element of a vector, which is a feature, is shown by a bar graph. In this case, the feature output model 80 to which one training image 71 is input is used as the training target, and the feature output model 80 to which the other training image 71 is input is used as the comparison target. However, these feature output models 80 are the same and are currently undergoing training.

[0062] The feature output model generation unit 22 compares the two output feature amounts with reference to the focus position information and performs machine learning based on the comparison result. When the focus positions of the two training images 71 indicated by the focus position information are the same (i.e., on the same plane), the feature output model generation unit 22 performs machine learning so that the difference in the feature amounts of the two training images 71 becomes smaller. When the focus positions of the two training images 71 indicated by the focus position information are different (i.e., different Z positions), the feature output model generation unit 22 performs machine learning so that the difference in the feature amounts of the two training images 71 becomes larger. Note that when the two training images 71 are cut out from the same image, the focus positions of the two training images 71 are the same. Furthermore, when the focus positions of the two training images 71 are close enough to be considered to be the same, the focus positions of the two training images 71 may also be considered to be the same.

[0063] That is, the feature amounts of partial images cut out from images on the same focal plane are made to have a high correlation regardless of the cut-out position. On the other hand, the feature amounts of partial images cut out from images on different focal planes are made to have a low correlation. By performing machine learning in this way, the feature amounts output from the feature output model reflect characteristics according to the focal position.

[0064] Specifically, when the focus positions of the two learning images 71 are the same, the feature output model generation unit 22 performs machine learning using the following loss_xy as the loss function.

number

[0065] When the focus positions of the two learning images 71 are different from each other, the feature output model generation unit 22 performs machine learning using the following loss_z as the loss function.

number

[0066] The feature output model generation unit 22 generates a feature output model by repeatedly selecting a set of learning images 71 and performing machine learning. For example, the feature output model generation unit 22 generates a feature output model by repeating the above process until the generation of the feature output model converges based on preset conditions or a preset number of times, as in the conventional case.

[0067] The feature output model generation unit 22 may generate the feature output model using an existing trained model generated by machine learning. As the existing trained model, a model that inputs image-based information is used, similar to the feature output model according to this embodiment. That is, an existing trained model that shares input with the feature output model according to this embodiment may be used. The existing trained model is, for example, a model for performing image recognition, specifically, ResNet, VGG, Mobile Net, etc. A part of the existing trained model is used to generate the feature output model. As shown in FIG. 5 , the output layer of the existing trained model 81 is deleted, and the part of the existing trained model 81 up to the intermediate layer is used to generate the feature output model. The existing trained model 81 used to generate the feature output model may include all of the intermediate layer, or only a part of the intermediate layer.

[0068] The feature output model generation unit 22 inputs the above-mentioned part of the existing trained model and sets it as the feature output model at the start of machine learning. That is, the feature output model generation unit 22 performs fine tuning using the above-mentioned part of the existing trained model as initial parameters of the feature output model. Furthermore, a new output layer may be added to the output side of the above-mentioned part of the trained model, and set it as the feature output model at the start of machine learning. Furthermore, when a new output layer is added, a new intermediate layer may be added between the output side of the above-mentioned part of the trained model and the new output layer, and set it as the feature output model at the start of machine learning.

[0069] The feature output model generation unit 22 may generate a feature output model without using an existing trained model. For example, a model with random values ​​as initial parameters, similar to conventional machine learning, may be used as the feature output model at the start of machine learning.

[0070] Using an existing trained model to generate a feature output model has the following advantages: The training time can be significantly reduced. A highly accurate feature output model can be generated even with a small number of training images 71, that is, a feature output model that can output more appropriate features. The above-mentioned existing trained model has already acquired the ability to separate features with low abstraction. Therefore, it is only necessary to train the model using new training images 71, focusing on features with high abstraction.

[0071] The feature output model generation unit 22 outputs the generated feature output model to the focus position estimation model generation unit 23 and the tilt estimation system 30. The generated feature output model may be used for purposes other than those described in this embodiment. In that case, for example, the feature output model generation unit 22 transmits or outputs the feature output model to another device or module that uses the feature output model. Alternatively, the feature output model generation unit 22 may store the generated feature output model in the computer 10 or another device so that it can be used by another device or module that uses the feature output model.

[0072] The focus position estimation model generation unit 23 is a focus position estimation model generation means that generates a focus position estimation model by machine learning from the focus position information acquired by the learning image acquisition unit 21. The focus position estimation model receives as input the feature amounts output from the feature amount output model generated by the feature amount output model generation unit 22 as described above, and estimates the focus position at the time of focusing that corresponds to the image related to the feature amounts.

[0073] The focus position estimation model generation unit 23 generates a focus position estimation model as follows: The focus position estimation model generation unit 23 receives the training image 71 and the focus position information related to the training image 71 from the training image acquisition unit 21. The focus position estimation model generation unit 23 receives the feature amount output model from the feature amount output model generation unit 22.

[0074] The focus position estimation model generation unit 23 inputs information based on the training image 71 into a feature output model and acquires the feature of the training image 71, which is output from the feature output model. The focus position estimation model generation unit 23 performs machine learning using the acquired feature as input to the focus position estimation model and information based on the focus position information of the training image 71 related to the input feature as output of the focus position estimation model. The information based on the focus position information is information corresponding to the output of the focus position estimation model. If the focus position estimation model outputs a value for each of the above-mentioned candidates, the information based on the focus position information is, for example, a value for each candidate (one-hot vector) in which a value of 1 is assigned to a candidate corresponding to the focus position information and a value of 0 is assigned to a candidate not corresponding to the focus position information. If the focus position estimation model outputs the above-mentioned difference or a value of the focus position itself, the information based on the focus position information is the focus position information itself or a value calculated from the focus position information. Before performing machine learning, the focus position estimation model generation unit 23 generates information based on the focus position information corresponding to the output of the focus position estimation model.

[0075] The machine learning itself, i.e., the updating of the parameters of the focus position estimation model, may be performed in the same manner as in the conventional method. The focus position estimation model generation unit 23 generates the focus position estimation model by repeating the machine learning process until the generation of the focus position estimation model converges based on preset conditions or for a predetermined number of times, as in the conventional method.

[0076] The focus position estimation model generation unit 23 outputs the generated focus position estimation model to the tilt estimation system 30. The generated focus position estimation model may be used for purposes other than those described in this embodiment. In that case, for example, the focus position estimation model generation unit 23 transmits or outputs the focus position estimation model to another device or module that uses the focus position estimation model. Alternatively, the focus position estimation model generation unit 23 may store the generated focus position estimation model in the computer 10 or another device so that it can be used by another device or module that uses the focus position estimation model. These are the functions of the feature output model generation system 20.

[0077] Next, we will explain the functions of the tilt estimation system 30 according to this embodiment. As shown in Figure 1, the tilt estimation system 30 includes an estimation target image acquisition unit 31, a focal position estimation unit 32, a tilt estimation unit 33, and a control unit 34.

[0078] The tilt estimation system 30 estimates the tilt of an object captured in an image captured by the inspection device 40 or the observation device 50. To do so, the tilt estimation system 30 estimates the focal position at the time of focusing when the object is imaged by the inspection device 40 or the observation device 50. This estimation is performed by first capturing an image of the object (image capture for tilt estimation) by the inspection device 40 or the observation device 50. In this image capture, the focal position does not necessarily have to be the focal position at the time of focusing on the object, i.e., the focal position at the time of focusing. Therefore, the image obtained by this image capture may be a defocused image. For example, this image may be a defocused image as shown in FIG. 6(a). FIG. 6 is an image of a semiconductor device. The defocused image shown in FIG. 6(a) is an image captured when the focal position at the time of capturing is +5 nm from the focal position at the time of focusing.

[0079] The tilt estimation system 30 estimates the focal positions at multiple positions in the image when in focus from this image. The tilt estimation system 30 estimates the tilt of the object captured in the image from the estimated focal positions at multiple positions in focus. Furthermore, by capturing an image in the inspection device 40 or observation device 50 using the estimated focal positions at focus, an image in which the object is in focus, i.e., a focused image, can be obtained. For example, a focused image such as that shown in FIG. 6(b) can be obtained. The focused image shown in FIG. 6(b) corresponds to the defocused image in FIG. 6(a).

[0080] The estimation target image acquisition unit 31 is an estimation target image acquisition means that acquires an image containing an imaging target and acquires estimation target images, which are multiple partial images, from the image. The estimation target image is an image used to estimate the focal position when in focus in the tilt estimation system 30. The estimation target image is also an image used as input to the feature output model. In other words, the estimation target image corresponds to the learning image 71 described above.

[0081] The estimation target image acquisition unit 31 acquires an image captured by the inspection device 40 or the observation device 50. The image captured by the inspection device 40 or the observation device 50 at this time is image captured for the above-mentioned tilt estimation. For example, the estimation target image acquisition unit 31 cuts out multiple partial images from the acquired image to set as the estimation target image.

[0082] The estimation target image acquisition unit 31 cuts out an estimation target image, which is a partial image of a predetermined size used as input to the feature output model, from the acquired image. The position in the image from which the estimation target image is cut out is a portion in which the captured object is captured. The position in the image from which the estimation target image is cut out may be set in advance. For example, as shown in FIG. 7, the estimation target image acquisition unit 31 may set, as estimation target images 91, a plurality of partial images obtained by dividing an image 90 into portions of a predetermined size. In the example shown in FIG. 7, nine estimation target images 91, each of which is 3×3, are cut out from the image. When the estimation target image 91 is cut out as described above, if the captured object is captured in the entire image 90, the estimation target image 91 may be cut out from the entire image 90. Alternatively, if the captured object is captured in a portion of the image 90, the estimation target image 91 may be cut out from that portion. Alternatively, a position where the captured object is estimated to be captured by performing image recognition on the image may be set as the position from which the estimation target image is cut out. The tilt estimation system 30 is configured so that the position where the estimation target image is cut out from the image is also used in the functions described below.

[0083] The type of the estimation target image is the same as the type of the learning image described above. For example, the estimation target image may be an image in which radiation from the imaging target is detected, an image in which light from the imaging target is detected when light is irradiated onto the imaging target, or an image in which electrical characteristics of the imaging target are detected when light is irradiated onto the imaging target. The estimation target image may also be an image in which light of a specific wavelength (e.g., light of a specific wavelength used for inspection) is irradiated onto the imaging target.

[0084] The estimation target image acquisition unit 31 outputs the acquired estimation target images to the focal position estimation unit 32.

[0085] The focus position estimation unit 32 is a focus position estimation means that uses a feature output model to output feature amounts of each of the plurality of estimation target images from each of the plurality of estimation target images acquired by the estimation target image acquisition unit 31, and estimates a focus position at the time of focusing corresponding to each of the plurality of estimation target images from the output feature amounts. The focus position estimation unit 32 may use a focus position estimation model to estimate a focus position at the time of focusing corresponding to the estimation target image from the feature amounts output from the feature output model.

[0086] The focal position estimation unit 32 receives the feature output model and the focal position estimation model generated by the feature output model generation system 20, stores them, and uses them for estimation. The focal position estimation unit 32 receives a plurality of estimation target images from the estimation target image acquisition unit 31.

[0087] The focus position estimation unit 32 inputs information based on the estimation target image to a feature output model and acquires feature amounts of the estimation target image, which are output from the feature output model. The focus position estimation unit 32 inputs the acquired feature amounts to a focus position estimation model and acquires information indicating the focus position at the time of focusing corresponding to the estimation target image, which is output from the focus position estimation model, as an estimation result of the focus position. The focus position estimation unit 32 acquires information indicating the focus position at the time of focusing corresponding to the estimation target image for each of the multiple estimation target images. The focus position estimation unit 32 outputs information indicating the focus position at the time of focusing corresponding to the multiple acquired estimation target images to the tilt estimation unit 33.

[0088] The tilt estimation unit 33 is a tilt estimation means that estimates the tilt of the imaged object captured in the image from the focal position at the time of focusing corresponding to each of the multiple estimation target images estimated by the focal position estimation unit 32. The tilt estimation unit 33 estimates the tilt of the imaged object with respect to the imaging direction in, for example, the inspection device 40 or the observation device 50. The tilt estimation unit 33 estimates the tilt of the imaged object as follows.

[0089] The tilt estimation unit 33 receives information indicating the focal position at the time of focusing corresponding to each of the multiple estimation target images from the focal position estimation unit 32. In this embodiment, the position in the imaging direction corresponding to the focal position at the time of focusing corresponding to the estimation target image is considered to be the position of the imaging target. Since the estimation target images are cut out from multiple positions in the image, the position of the imaging target in the imaging direction at each position of the estimation target image in the image can be obtained. The tilt of the imaging target is estimated from the positions of the multiple imaging targets.

[0090] For example, the tilt estimation unit 33 estimates the tilt of the imaged object for each coordinate axis on a plane perpendicular to the imaging direction. The tilt estimation unit 33 estimates the tilt of the imaged object for each of the X-axis and Y-axis, which are two coordinate axes parallel to each side of an estimation target image 91 of an image 90 as shown in Fig. 7. For example, the tilt estimation unit 33 estimates an angle θ1 of tilt of the imaged object on the X-axis with respect to a plane perpendicular to the imaging direction (Z-axis direction) (the hatched surface in the figure) as shown in Fig. 8(a), and an angle θ2 of tilt of the imaged object on the Y-axis with respect to a plane perpendicular to the imaging direction (Z-axis direction) (the hatched surface in the figure) as shown in Fig. 8(b).

[0091] FIG. 8 shows information indicating the focal position at the time of focusing estimated from each estimation target image 91 (specifically, the difference between the focal position when the image was captured and the focal position at the time of focusing) (values ​​such as +2, 0, −2, +3, 0, and −3). FIG. 8(a) shows an example in which the object to be imaged is tilted in the X-axis direction, specifically, the right side is lowered. FIG. 8(b) shows an example in which the object to be imaged is tilted in the Y-axis direction, specifically, the front side is lowered.

[0092] The tilt estimation unit 33 calculates the angle θ1 and the angle θ2 using the following equations.

number

[0093] x and y are determined based on the position of the estimation target image as shown in Fig. 7. For example, x is the distance between two predetermined estimation target images P a ,P b The position (for example, the center coordinates P center ) in the X-axis direction. That is, x = |P b -P a |(X-axis component only). y is the distance between two pre-set estimation target images P a ,P c The position (for example, the center coordinates P center ) in the Y-axis direction. That is, y = |P c -P a |(X-axis component only). Note that if the position of the image to be estimated in the image is set in advance, the above x and y will be constant values ​​and may therefore be stored in the tilt estimation unit 33 in advance.

[0094] z1 and z2 are calculated based on the focal position at the time of focusing corresponding to each of the estimation target images as shown in FIG. 8. For example, z1 is calculated based on the focal position at the time of focusing corresponding to each of the estimation target images P a ,P b The focal position Z at the time of focusing corresponds to a ,Z b That is, z1 = |Z b -Z a z1 is the two estimation target images P a ,P c The focal position Z at the time of focusing corresponds to a ,Z c That is, z2 = |Z c -Z a|

[0095] The estimation target image used for the tilt of the imaged object may be other than the above. For example, the estimation target image used for the tilt of the imaged object may be used as follows to estimate the tilt. An image showing the imaged object is divided into a plurality of regions, each containing a plurality of estimation target images. For example, the image is divided into 3x3 rectangular regions. Region (P a ) for each of the multiple estimation target images (P a1 ,P a2 ,P a3 ,…) the corresponding focal position (Z a1 ,Z a2 ,Z a3 , ...) after removing the outliers. The median (Z median ) is taken. The median (Z median ) and the median (Z median ) corresponding to the estimation target image (P amedian ) into the area (P a The inclination is estimated as the focus position and estimation target image when the focus is focused, which represent the image.

[0096] The tilt estimation unit 33 may estimate the tilt by a method other than the above, as long as it estimates the tilt of the image capture object captured in the image from the focal position at the time of focus corresponding to each of the multiple estimation target images estimated by the focal position estimation unit 32. Furthermore, the tilt estimation unit 33 may estimate an angle other than the above-mentioned angles θ1 and θ2 as the tilt of the image capture object. The tilt estimation unit 33 outputs information indicating the estimated tilt of the image capture object to the control unit 34. Furthermore, the tilt estimation unit 33 may also output information indicating the focal position at the time of focus corresponding to the estimation target image used to estimate the tilt of the image capture object to the control unit 34.

[0097] The control unit 34 is a control means that controls the tilt of the object to be imaged when it is imaged, based on the tilt of the object to be imaged estimated by the tilt estimation unit 33. The control unit 34 may also control the focal position of the object to be imaged when it is imaged, based on the focal position estimated by the focal position estimation unit 32.

[0098] The control unit 34 inputs information indicating the tilt of the imaged object from the tilt estimation unit 33. The control unit 34 may also input information indicating the focal position at the time of focusing corresponding to the estimation target image from the tilt estimation unit 33. The control unit 34 controls the inspection device 40 or the observation device 50 so that the tilt of the imaged object indicated by the input information is eliminated during imaging. Specifically, the control unit 34 controls the inspection device 40 or the observation device 50 so that the imaged object is tilted opposite to the tilt of the imaged object indicated by the input information. The inspection device 40 or the observation device 50, upon receiving the control, adjusts the tilt of the imaged object during imaging, for example, by operating the mounting unit. In this way, the control unit 34 controls tilt correction in the inspection device 40 or the observation device 50. As a result, the image captured by the inspection device 40 or the observation device 50 is an image in which the tilt of the imaged object is appropriate.

[0099] Furthermore, the control unit 34 controls the inspection device 40 or the observation device 50 so that the focal position during imaging is the in-focus focal position indicated by the input information. For example, the control unit 34 controls the focal position during imaging to be the in-focus position corresponding to a preset estimation target image among multiple estimation target images. The inspection device 40 or observation device 50 that receives this control then operates the stage 46, for example, to adjust the focal position during imaging. As a result, the image captured by the inspection device 40 or observation device 50 becomes a focused image. In this way, the control unit 34 controls the autofocus in the inspection device 40 or observation device 50. The above is the configuration of the tilt estimation system 30.

[0100] Next, the processing executed by the computer 10 according to this embodiment (the operating method performed by the computer 10) will be described using the flowcharts in Fig. 9 and Fig. 10. First, the processing executed when generating a feature output model and a focus position estimation model, that is, the feature output model generation method which is the processing executed by the feature output model generation system 20 according to this embodiment, will be described using the flowchart in Fig. 9.

[0101] In this process, first, the training image acquisition unit 21 acquires a plurality of training images associated with focus position information relating to the focus position at the time of image capture (S01, training image acquisition step). The training image acquisition unit 21 also acquires focus position information relating to the focus position at the time of focusing corresponding to each of the training images. Next, the feature output model generation unit 22 generates a feature output model from the training images by machine learning (S02, feature output model generation step). At this time, the feature amounts of the two training images 71 are compared according to the focus position information associated with the two different training images, and machine learning is performed based on the comparison result. Next, the focus position estimation model generation unit 23 generates a focus position estimation model from the focus position information by machine learning (S03, focus position estimation model generation step).

[0102] The generated feature output model and focus position estimation model are output from the feature output model generation system 20 to the tilt estimation system 30. The tilt estimation system 30 stores the feature output model and focus position estimation model and uses them in the following processes. The above is the feature output model generation method, which is the process executed by the feature output model generation system 20 according to this embodiment.

[0103] Next, using the flowchart in FIG. 10, we will explain the process executed when estimating the in-focus focus position corresponding to the estimation target image, i.e., the tilt estimation method, which is the process executed by the tilt estimation system 30 according to this embodiment.

[0104] In this process, first, the estimation target image acquisition unit 31 acquires an image containing an image of the imaging target, and acquires estimation target images, which are multiple partial images, from the image (S11, estimation target image acquisition step). The estimation target image is based on an image obtained by imaging for tilt estimation using the inspection device 40 or the observation device 50. Next, the focus position estimation unit 32 uses the feature output model to output feature values ​​for each of the multiple estimation target images. Next, the focus position estimation unit 32 uses the focus position estimation model to estimate a focus position when in focus corresponding to each of the multiple estimation target images from the feature values ​​of each of the multiple estimation target images (S12, focus position estimation step).

[0105] Next, the tilt estimation unit 33 estimates the tilt of the object captured in the images from the focal positions at the time of focusing corresponding to each of the multiple estimation target images (S13, tilt estimation step). Next, the control unit 34 controls the tilt of the object captured by the inspection device 40 or the observation device 50 when a new image is captured, based on the estimated tilt of the object (S14, control step). As a result, the image captured by the inspection device 40 or the observation device 50 is an image in which the tilt of the object is appropriate. At this time, the control unit 34 may also control the focal position of the object captured by the inspection device 40 or the observation device 50 based on the estimated focal positions. As a result, the image captured by the inspection device 40 or the observation device 50 is a focused image. The tilt estimation method, which is processing executed by the tilt estimation system 30 according to this embodiment, has been described above.

[0106] In this embodiment, a feature output model that outputs image features is generated by machine learning. At this time, the features of two different training images are compared according to focus position information associated with the two training images, and machine learning is performed based on the comparison results. According to this generation, for example, by using an existing trained model that inputs image-based information, a feature output model can be generated in a short time. In other words, according to this embodiment, a feature output model that is a trained model used for image-based estimation, such as focus position estimation, can be generated by short-time learning.

[0107] However, as described above, it is not necessary to use an existing trained model to generate a feature output model. Even in this case, it is possible to generate a feature output model that can output appropriate features according to the focus position.

[0108] Furthermore, as described above, machine learning for generating a feature output model may be performed so that when two different training images have the same focus position, the difference in the features of the two training images is small, and when two different training images have different focus positions, the difference in the features of the two training images is large. This configuration allows for reliable and appropriate generation of a feature output model. However, machine learning does not necessarily have to be performed as described above, and may be performed based on the results of comparing the features of the two training images.

[0109] As described above, the learning images and estimation target images may be images in which radiation from an imaging object is detected, images in which light from the imaging object is detected when the imaging object is irradiated with light, or images in which electrical characteristics of the imaging object are detected when the imaging object is irradiated with light. Furthermore, the learning images and estimation target images may be images in which light of a specific wavelength is irradiated onto the imaging object. These configurations enable the generation and use of an appropriate feature output model according to the type of image used. However, the learning images and estimation target images are not limited to those described above, and may be any images according to the focal position.

[0110] Furthermore, as in the present embodiment, the feature output model generation system 20 may further include a focus position estimation model generation unit 23 that generates a focus position estimation model. With this configuration, it is possible to generate a focus position estimation model that estimates the focus position at the time of focusing from an image. That is, with this configuration, it is possible to generate, in addition to the feature output model, a trained model used for estimating the focus position based on an image through short training.

[0111] However, the feature amount output model generation system 20 does not have to include the focus position estimation model generation unit 23. That is, the feature amount output model generation system 20 may be configured to generate only the feature amount output model. Furthermore, the generated feature amount output model may be used for purposes other than estimating the focus position when in focus.

[0112] In the tilt estimation system 30 according to the present invention, the inclination of the imaged object is estimated by estimating the in-focus focal position corresponding to each of the multiple estimation target images from each of the multiple estimation target images, which are multiple partial images of the image. Therefore, once an image of the imaged object is obtained, the tilt of the imaged object can be estimated in a short time. This eliminates the need for the configuration (e.g., a lens turret) and space required to install a special optical system within the device, which was necessary when estimating the tilt of the imaged object using a special optical system. This also eliminates the cost of preparing a special optical system. Furthermore, it also eliminates the need for processes such as turret switching, laser scanning, and alignment. Compared to this method, the time required to estimate the tilt of the imaged object can be significantly reduced.

[0113] Furthermore, the feature output model used for estimating the focal position can output feature values ​​appropriate for estimating the focal position, and by using this, the tilt of the imaged object can be appropriately estimated. Therefore, the tilt estimation system according to the present invention can estimate the tilt of the imaged object with a simple configuration in a short time.

[0114] The tilt estimation system 30 according to this embodiment uses the above-described feature output model to estimate the in-focus focal position. Therefore, according to this embodiment, the in-focus focal position can be estimated based on an image with a short preparation time. Furthermore, the inclination estimation system 30 according to this embodiment can estimate the in-focus focal position by capturing an image for inclination estimation once. Therefore, the in-focus focal position can be estimated more quickly than when capturing an image multiple times while changing the focal position to search for the in-focus focal position. As a result, the inclination of the object being imaged can be estimated more quickly.

[0115] Furthermore, the above-described focus position estimation model may be used to estimate the focus position when in focus. With this configuration, the focus position when in focus can be reliably and appropriately estimated. As a result, the tilt of the image capture object can be reliably and appropriately estimated. However, the above-described focus position estimation model does not need to be used to estimate the focus position when in focus; estimation can be performed from the feature amount output from the feature amount output model.

[0116] Furthermore, as in this embodiment, the tilt estimation system 30 may further include a control unit 34 that controls the tilt of the object when it is imaged in the inspection device 40 or observation device 50, based on the estimated tilt of the object. This configuration allows the inspection device 40 or observation device 50 to image the object at an appropriate tilt. However, the tilt estimation system 30 does not necessarily have to include the control unit 34. In other words, the tilt estimation system 30 only needs to estimate the tilt of the object.

[0117] The tilt estimation system 30 according to this embodiment and the inspection device 40 or the observation device 50 described above can be configured as a series of systems. That is, the semiconductor inspection system according to this embodiment may be a system including the tilt estimation system 30 and the inspection device 40. The living body observation system according to this embodiment may be a system including the tilt estimation system 30 and the observation device 50.

[0118] In this embodiment, the computer 10 includes the feature output model generation system 20 and the tilt estimation system 30, but the feature output model generation system 20 and the tilt estimation system 30 may be implemented independently.

[0119] Next, a description will be given of a feature output model generation program and a tilt estimation program for executing the above-mentioned series of processes by the feature output model generation system 20 and the tilt estimation system 30. As shown in Fig. 11, the feature output model generation program 200 is stored in a program storage area 211 formed in a computer-readable recording medium 210 that is inserted into a computer and accessed, or that is provided in the computer. The recording medium 210 may be a non-transitory recording medium.

[0120] The feature output model generation program 200 includes a learning image acquisition module 201, a feature output model generation module 202, and a focal position estimation model generation module 203. Functions realized by executing the learning image acquisition module 201, the feature output model generation module 202, and the focal position estimation model generation module 203 are similar to the functions of the learning image acquisition unit 21, the feature output model generation unit 22, and the focal position estimation model generation unit 23 of the feature output model generation system 20 described above, respectively.

[0121] 12, the tilt estimation program 300 is stored in a program storage area 311 formed on a computer-readable recording medium 310 that is inserted into a computer and accessed, or that is provided in the computer. The recording medium 310 may be a non-transitory recording medium. Note that the recording medium 310 may be the same as the recording medium 210.

[0122] The tilt estimation program 300 includes an estimation target image acquisition module 301, a focal position estimation module 302, a tilt estimation module 303, and a control module 304. Functions realized by executing the focal position estimation module 302, the tilt estimation module 303, and the control module 304 are similar to the functions of the estimation target image acquisition unit 31, the focal position estimation unit 32, the tilt estimation unit 33, and the control unit 34 of the tilt estimation system 30 described above, respectively.

[0123] The feature output model generating program 200 and the inclination estimation program 300 may be configured so that a part or all of them are transmitted via a transmission medium such as a communication line, and are received and recorded (including installed) by another device. Also, each module of the feature output model generating program 200 and the inclination estimation program 300 may be installed on one of multiple computers, rather than on one computer. In that case, the above-described series of processes are performed by a computer system including the multiple computers.

[0124] The tilt estimation system, tilt estimation method, tilt estimation program, semiconductor inspection system, and living body observation system according to the present disclosure have the following configurations. [1] A tilt estimation system for estimating the tilt of an object captured in an image, comprising: an estimation target image acquisition means for acquiring an image including a captured object and acquiring estimation target images, which are a plurality of partial images, from the image; a focus position estimation means for outputting feature amounts of each of the plurality of estimation target images acquired by the estimation target image acquisition means using a feature amount output model that inputs information based on an image and outputs feature amounts of the image, and for estimating a focus position at the time of focusing corresponding to each of the plurality of estimation target images from the output feature amounts; and tilt estimation means for estimating a tilt of an image capture object captured in the images from a focus position at the time of focusing corresponding to each of a plurality of estimation target images estimated by the focus position estimation means, The feature output model is generated by machine learning from a plurality of training images associated with focus position information relating to the focus position at the time of image capture, and the features of two different training images are compared according to the focus position information associated with the two training images, and machine learning is performed based on the comparison results. [2] The focus position estimation means receives the feature amount output from the feature amount output model, and estimates a focus position at a time when the focus is focused corresponding to each of a plurality of estimation target images using a focus position estimation model that estimates a focus position at a time when the focus is focused corresponding to an image related to the feature amount; The tilt estimation system according to [1], wherein the focus position estimation model is generated by machine learning from focus position information relating to the focus position when in focus corresponding to each of the learning images. [3] The tilt estimation system according to [1] or [2], further comprising a control means for controlling the tilt of the object to be imaged during imaging based on the tilt of the object to be imaged estimated by the tilt estimation means. [4] A tilt estimation system according to any one of [1] to [3]; a mounting portion on which a semiconductor device is mounted as an imaging target for the tilt estimation system; an inspection unit that inspects the semiconductor device; A semiconductor inspection system comprising: [5] A tilt estimation system according to any one of [1] to [3]; a placement unit on which a biological sample is placed as an imaging target for the tilt estimation system; an observation unit for observing the biological sample; A living body observation system comprising: [6] A tilt estimation method for estimating the tilt of an object captured in an image, comprising: an estimation target image acquisition step of acquiring an image including a captured object and acquiring estimation target images, which are a plurality of partial images, from the image; a focus position estimation step of outputting feature amounts of each of the plurality of estimation target images from each of the plurality of estimation target images acquired in the estimation target image acquisition step using a feature amount output model that inputs information based on the image and outputs feature amounts of the image, and estimating focus positions at the time of focusing corresponding to each of the plurality of estimation target images from the output feature amounts; an inclination estimation step of estimating an inclination of an image capture object captured in the images from focus positions at the time of focusing corresponding to each of the plurality of estimation target images estimated in the focus position estimation step, The feature output model is generated by machine learning from a plurality of training images associated with focus position information relating to the focus position at the time of image capture, and the feature values ​​of two different training images are compared according to the focus position information associated with the two training images, and machine learning is performed based on the comparison results. [7] In the focus position estimation step, a feature output from the feature output model is input, and a focus position estimation model that estimates a focus position at a time when focused corresponding to an image related to the feature is used to estimate a focus position at a time when focused corresponding to each of a plurality of estimation target images; The tilt estimation method according to [6], wherein the focus position estimation model is generated by machine learning from focus position information relating to the focus position when in focus corresponding to each of the learning images. [8] The tilt estimation method according to [6] or [7], further comprising a control step of controlling the tilt of the object to be imaged during imaging based on the tilt of the object to be imaged estimated in the tilt estimation step. [9] A tilt estimation program that causes a computer to operate as a tilt estimation system that estimates the tilt of an object captured in an image, The computer, an estimation target image acquisition means for acquiring an image including a captured object and acquiring estimation target images, which are a plurality of partial images, from the image; a focus position estimation means for outputting feature amounts of each of the plurality of estimation target images acquired by the estimation target image acquisition means using a feature amount output model that inputs information based on an image and outputs feature amounts of the image, and for estimating a focus position at the time of focusing corresponding to each of the plurality of estimation target images from the output feature amounts; and a tilt estimation means for estimating a tilt of an image capture object captured in the image from a focus position at the time of focusing corresponding to each of a plurality of estimation target images estimated by the focus position estimation means, The feature output model is generated by machine learning from a plurality of training images associated with focus position information relating to the focus position at the time of image capture, and the feature values ​​of two different training images are compared according to the focus position information associated with the two training images, and machine learning is performed based on the comparison results.

[10] The focus position estimation means inputs the feature amount output from the feature amount output model, and estimates the focus position at the time of focusing corresponding to each of a plurality of estimation target images using a focus position estimation model that estimates the focus position at the time of focusing corresponding to the image related to the feature amount; The tilt estimation program according to [9], wherein the focus position estimation model is generated by machine learning from focus position information relating to the focus position when in focus corresponding to each of the learning images.

[11] The computer: The tilt estimation program according to [9] or

[10] , which also functions as a control means for controlling the tilt of the object to be imaged when it is imaged, based on the tilt of the object to be imaged estimated by the tilt estimation means. [Explanation of symbols]

[0125] 10...computer, 20...feature output model generation system, 21...learning image acquisition unit, 22...feature output model generation unit, 23...focal position estimation model generation unit, 30...tilt estimation system, 31...estimation target image acquisition unit, 32...focal position estimation unit, 33...tilt estimation unit, 34...control unit, 40...inspection device, 41...camera, 42...mounting unit, 43...light source, 44...optical system, 45...objective lens, 46...stage, 50...observation device, 200...feature output model generation program, 201...learning image acquisition module, 202...feature output model generation module, 203...focal position estimation model generation module, 210...recording medium, 211...program storage area, 300...tilt estimation program, 301...estimation target image acquisition module, 302...focal position estimation module, 303...tilt estimation module, 304...control module, 310...recording medium, 311...program storage area.

Claims

1. A tilt estimation system for estimating the tilt of an object captured in an image, comprising: an estimation target image acquisition means for acquiring an image including a captured object and acquiring estimation target images, which are a plurality of partial images, from the image; a focus position estimation means for outputting feature amounts of each of the plurality of estimation target images acquired by the estimation target image acquisition means using a feature amount output model that inputs information based on an image and outputs feature amounts of the image, and for estimating a focus position at the time of focusing corresponding to each of the plurality of estimation target images from the output feature amounts; an inclination estimation means for estimating an inclination of an image capture object captured in a plurality of estimation target images from a focus position at the time of focusing corresponding to each of the plurality of estimation target images estimated by the focus position estimation means; A tilt estimation system comprising:

2. The tilt estimation system according to claim 1 , wherein the feature amount reflects a focal position when the image related to the feature amount was captured.

3. The tilt estimation system according to claim 1 or 2, wherein the feature amount is a vector having a predetermined number of dimensions.

4. The tilt estimation system according to any one of claims 1 to 3, wherein the feature is acquired by inputting information based on an image relating to the feature into a feature output model generated by machine learning.

5. the focus position estimation means estimates a focus position at the time of focusing corresponding to each of the plurality of estimation target images from the feature amounts of each of the plurality of estimation target images using a focus position estimation model that estimates a focus position at the time of focusing corresponding to an image related to the feature amounts; The tilt estimation system according to any one of claims 1 to 4, wherein the focus position estimation model is generated by machine learning from a plurality of learning images and focus position information relating to the focus positions at the time of focusing corresponding to each of the learning images.

6. The tilt estimation system according to any one of claims 1 to 5, further comprising a control means for controlling the tilt of the object to be imaged at the time of image capture based on the tilt of the object to be imaged estimated by the tilt estimation means.

7. A tilt estimation system according to any one of claims 1 to 6; a mounting portion on which a semiconductor device is mounted as an imaging target for the tilt estimation system; an inspection unit that inspects the semiconductor device; A semiconductor inspection system comprising:

8. A tilt estimation system according to any one of claims 1 to 6; a placement unit on which a biological sample is placed as an imaging target for the tilt estimation system; an observation unit for observing the biological sample; A living body observation system comprising:

9. A tilt estimation method for estimating the tilt of an object captured in an image, comprising: an estimation target image acquisition step of acquiring an image including a captured object and acquiring estimation target images, which are a plurality of partial images, from the image; a focus position estimation step of outputting feature amounts of each of the plurality of estimation target images from each of the plurality of estimation target images acquired in the estimation target image acquisition step using a feature amount output model that inputs information based on the image and outputs feature amounts of the image, and estimating focus positions at the time of focusing corresponding to each of the plurality of estimation target images from the output feature amounts; an inclination estimation step of estimating an inclination of an image capture object captured in the images from focus positions at the time of focusing corresponding to each of the plurality of estimation target images estimated in the focus position estimation step; A method for estimating a slope including:

10. A tilt estimation program that causes a computer to operate as a tilt estimation system that estimates the tilt of an object captured in an image, The computer, an estimation target image acquisition means for acquiring an image including a captured object and acquiring estimation target images, which are a plurality of partial images, from the image; a focus position estimation means for outputting feature amounts of each of the plurality of estimation target images acquired by the estimation target image acquisition means using a feature amount output model that inputs information based on an image and outputs feature amounts of the image, and for estimating a focus position at the time of focusing corresponding to each of the plurality of estimation target images from the output feature amounts; an inclination estimation means for estimating an inclination of an image capture object captured in a plurality of estimation target images from a focus position at the time of focusing corresponding to each of the plurality of estimation target images estimated by the focus position estimation means; A tilt estimation program that acts as a

Images