Tilt estimation system, tilt estimation method, tilt estimation program, semiconductor inspection system and live observation system

Abstract

This invention can be simply constructed and can estimate the tilt of a photographic object in a short time. The tilt estimation system 30 is a system for estimating the tilt of a photographed object reflected in an image, and includes: a predicted object image acquisition unit 31, which acquires a plurality of partial images, namely predicted object images, from an image; a focus position estimation unit 32, which uses a feature output model to output feature quantities based on the predicted object images and estimates the focus position corresponding to each of the plurality of predicted object images when in focus; and a tilt estimation unit 33, which estimates the tilt of the photographed object based on the focus positions when in focus; and the feature output model is generated by machine learning by establishing a plurality of corresponding learning images based on focus position information related to the focus position during photography, and by comparing the feature quantities of two different learning images based on the focus position information established, and performing machine learning based on the comparison results.

Description

[Technical Field]

[0001] This invention relates to a tilt estimation system, tilt estimation method, tilt estimation program, semiconductor inspection system, and live observation system for estimating the tilt of a photographic object reflected in an image. [Previous Technology]

[0002] Previously, proposals have included estimating the tilt of the object being photographed, such as its tilt relative to the camera direction. For example, Patent Document 1 shows the estimation of the tilt of a semiconductor device when photographing it for inspection. The estimated tilt is used to adjust the orientation of the semiconductor device. In Patent Document 1, the tilt of the object being photographed is estimated using an optical system including a dedicated lens (e.g., a relay lens system) and a scanner. [Prior Art Documents] [Patent Documents]

[0003] Patent Document 1: International Publication No. 2017 / 154895 [Summary of the Invention]

[0004] [The problem the invention aims to solve]

[0005] As shown in Patent Document 1, when estimating the tilt of a photographed object using a special optical system, it is necessary to have the configuration (e.g., a lens rotating stage) and space to install the special optical system within the device. Furthermore, there is a cost to prepare for the special optical system. Also, the method shown in Patent Document 1 requires switching the rotating stage, laser scanning, and alignment processes, and the tilt estimation takes a long time.

[0006] One embodiment of the present invention was made in view of the above, and its object is to provide a tilt estimation system, tilt estimation method, tilt estimation program, and associated semiconductor inspection system and live observation system that can be simply constructed and perform tilt estimation of an object in a short time. [Technical Means for Solving the Problem]

[0007] To achieve the above objective, one embodiment of the tilt estimation system of the present invention estimates the tilt of a photographic object projected in an image, and includes: a tilt estimation object image acquisition mechanism that acquires an image of the photographic object projected in the image, and acquires a plurality of partial images from the image to form tilt estimation objects; a focus position estimation mechanism that uses a feature quantity output model that takes image-based information as input and outputs feature quantities of the image, outputs feature quantities of a plurality of tilt estimation objects based on each of the plurality of tilt estimation object images acquired by the tilt estimation object image acquisition mechanism, and estimates the tilt of the object based on the output feature quantities. The system includes several presumed object images, each corresponding to a focal point position when in focus; a tilt estimation mechanism that estimates the tilt of the object projected in the image based on the focal point positions estimated by the focal point estimation mechanism and corresponding to the focal points of the multiple presumed object images when in focus; and a feature output model that generates multiple learning images by machine learning based on focal point position information related to the focal point position during shooting. The model compares the feature quantities of the two learning images based on the focal point position information corresponding to the two different learning images and performs machine learning based on the comparison results.

[0008] In one embodiment of the tilt estimation system of the present invention, the tilt of the photographed object is estimated by estimating the focal position when the image is in focus, corresponding to each of the plurality of partial images of the image being estimated, from each of the plurality of estimated object images. Therefore, if an image of the photographed object is obtained, the tilt of the photographed object can be estimated in a short time. Furthermore, the feature quantity output model used for estimating the focal position can output a feature quantity suitable for estimating the focal position. By using this feature quantity, the tilt of the photographed object can be appropriately estimated. Therefore, according to one embodiment of the tilt estimation system of the present invention, the tilt of the photographed object can be simply constructed and the tilt of the photographed object can be estimated in a short time.

[0009] Alternatively, the focus position estimation mechanism can be based on the feature quantity output by the feature quantity output model. It uses a focus position estimation model to estimate the focus position at focus corresponding to the image associated with the feature quantity. This model estimates the focus position at focus corresponding to each of a plurality of estimated object images. The focus position estimation model is generated by machine learning from focus position information related to the focus positions at focus corresponding to each of the learning images. Based on this configuration, by using the focus position estimation model, the focus position can be accurately and appropriately estimated. As a result, the tilt of the photographed object can be accurately and appropriately estimated.

[0010] The tilt estimation system may further include a control mechanism that controls the tilt of the object being photographed during photography based on the tilt of the object being photographed as estimated by the tilt estimation mechanism. With this configuration, photography of an object being photographed at an appropriate tilt can be performed.

[0011] One embodiment of the semiconductor inspection system and the live observation system of the present invention may be configured to include the above-described tilt estimation system. That is, one embodiment of the semiconductor inspection system of the present invention includes: the above-described tilt estimation system; a mounting section on which a semiconductor device is mounted as an imaging target related to the tilt estimation system; and an inspection section on which the semiconductor device is inspected. Furthermore, one embodiment of the live observation system of the present invention includes: the above-described tilt estimation system; a mounting section on which a live sample is mounted as an imaging target related to the tilt estimation system; and an observation section on which the live sample is observed.

[0012] However, in addition to being described as an invention of a tilt estimation system as described above, one embodiment of the present invention may also be described as an invention of a tilt estimation method and a tilt estimation program as described below.

[0013] That is, one embodiment of the tilt estimation method of the present invention is used to estimate the tilt of a photographic object projected in an image. It includes: a step of acquiring an estimated object image, which acquires an image of the photographic object and acquires a plurality of partial images from the image, namely, estimated object images; and a step of estimating a focal position, which uses a feature quantity output model that inputs image-based information and outputs feature quantities of the image, outputs feature quantities of each of the plurality of estimated object images acquired in the estimated object acquisition step, and estimates the tilt of the estimated object image based on the output feature quantities. The model calculates the focal position of each object image when it is in focus; and the tilt estimation step, which estimates the tilt of the object image reflected in the image by estimating the focal position of each object image when it is in focus from the multiple estimated object images estimated in the focal position estimation step; and the feature output model is generated by machine learning by establishing multiple corresponding learning images based on the focal position information related to the focal position when the image is captured, and by comparing the feature quantities of the two learning images based on the focal position information established with two different learning images, and performing machine learning based on the comparison results.

[0014] In the focus position estimation step, the feature quantity output from the feature quantity output model can also be input, and the focus position estimation model that estimates the focus position when the image corresponding to the feature quantity is in focus can be used to estimate the focus position when the image is in focus for each of the plurality of estimated object images. The focus position estimation model is generated by machine learning from the focus position information related to the focus position when the image is in focus for each of the learning images.

[0015] The tilt estimation method may further include: a control step, which controls the tilt of the camera object during the shooting process based on the tilt of the camera object estimated in the tilt estimation step.

[0016] Furthermore, in one embodiment of the present invention, the tilt estimation program causes a computer to act as a tilt estimation system for estimating the tilt of a photographic object projected in an image, and the computer functions as the following mechanisms: a tilt estimation object image acquisition mechanism, which acquires an image of the photographic object projected in the image, and acquires a plurality of partial images from the image, which are the tilt estimation object images; a focus position estimation mechanism, which uses a feature quantity output model that inputs image-based information and outputs feature quantities of the image, and outputs feature quantities of a plurality of tilt estimation object images based on each of the plurality of tilt estimation object images acquired by the tilt estimation object image acquisition mechanism, and based on... The output feature quantity estimates the focal position when in focus corresponding to each of the plurality of estimated object images; and the tilt estimation mechanism estimates the tilt of the object projected in the image based on the focal position estimated by the focal position estimation mechanism when in focus corresponding to each of the plurality of estimated object images; and the feature quantity output model is generated by machine learning by establishing a plurality of corresponding learning images based on the focal position information related to the focal position when in focus, and by comparing the feature quantities of the two learning images based on the focal position information established with two different learning images, and performing machine learning based on the comparison results.

[0017] Alternatively, the focus position estimation mechanism can be based on the feature quantity output by the feature quantity output model, and use a focus position estimation model to estimate the focus position when the image corresponding to the feature quantity is in focus. The focus position estimation model is generated by machine learning from the focus position information related to the focus position when the image corresponding to the learning image is in focus.

[0018] The tilt estimation program can also enable the computer to function as a control mechanism, which controls the tilt of the object being photographed based on the tilt of the object estimated by the tilt estimation mechanism. [Effects of the Invention]

[0019] According to one embodiment of the present invention, the tilt estimation of the photographed object can be performed in a short time with a simple configuration.

Implementation Method

[0021] Hereinafter, together with the drawings, embodiments of the tilt estimation system, tilt estimation method, tilt estimation program, semiconductor inspection system, and live observation system of the present invention will be described in detail. Furthermore, in the description of the drawings, the same symbols are used to mark the same elements, and repeated descriptions are omitted.

[0022] Figure 1 shows the tilt estimation system of this embodiment, namely computer 10. Computer 10 is a device (system) for image information processing. Specifically, computer 10 processes images captured by at least one of inspection device 40 and observation device 50. In addition, computer 10 can also process images captured by devices other than inspection device 40 and observation device 50. That is, computer 10 can also be applied to imaging devices other than inspection device 40 and observation device 50.

[0023] The inspection apparatus 40 is an apparatus that captures images of semiconductor devices and inspects them based on the captured images. For example, the inspection apparatus 40 performs fault analysis on the semiconductor devices. The semiconductor device to be inspected is, for example, a wafer covered with μLEDs (Light Emitting Diodes). Fault analysis is performed, for example, by light emission analysis, heat generation analysis, analysis using patterned images, or analysis using lasers (OBIRCH (Optical Beam Induced Resistance Change), OBIC (Optical Beam Induced Current), or DALS (Dynamic Ambient Light Suppression), etc.). The inspection apparatus 40 may also be a previously used inspection apparatus. The configuration described below may also be entirely possessed by previously used inspection apparatuses.

[0024] Figure 2 shows, for example, a partial configuration of an inspection apparatus 40 for performing luminescence analysis. As shown in Figure 2, the inspection apparatus 40 includes a camera 41, a mounting section 42, a light source 43, an optical system 44, an objective lens 45, and a stage 46. The camera 41 is an imaging device for photographing the object being imaged, i.e., a semiconductor device. The camera 41 is, for example, an InGaAs camera. The mounting section 42 is configured to mount the object being imaged, i.e., the semiconductor device. In Figure 2, the standard sample 60 mounted on the mounting section 42 is not the object being inspected, but rather an image corresponding to the focal position described later. The standard sample 60 is a sample with an artificial pattern (for example, a radial stripe pattern as shown in Figure 2).

[0025] The mounting unit 42 is configured to control the tilt (posture) of the mounted object relative to the camera direction. For example, the mounting unit 42 is configured such that the tilt of the mounting surface of the mounted object relative to the camera direction is variable. In this way, the mounting unit 42 can perform tilt correction of the mounted object. As the mounting unit 42, the former which controls the tilt of the mounted object can be used. The tilt of the mounted object during photography is controlled by the computer 10, as described later.

[0026] The light source 43 is a device that outputs illumination light to the object being photographed. The light source 43 may also output illumination light of a specific wavelength (e.g., a standard wavelength of 1100 nm, and wavelengths different from the standard wavelength of 900 nm and 1300 nm). Furthermore, multiple light sources 43 may be prepared in advance to switch the wavelength of the illumination light. The optical system 44 is an optical system used to illuminate the object being photographed, i.e., the semiconductor device, with the illumination light output from the light source 43. The objective lens 45 is the objective lens used by the camera 41 for imaging, such as a solid immersion lens (SIL). The stage 46 is a component used to adjust the focal position of the camera 41 during imaging. The stage 46 may also be movable not only in the imaging direction (focal position direction, Z-axis direction) but also in any three-dimensional direction (i.e., an XYZ stage). The focal position of the inspection device 40 during imaging, as described later, is controlled by the computer 10. The inspection apparatus 40 includes an inspection section that uses images obtained from the above-described configuration to inspect semiconductor devices.

[0027] Furthermore, the configuration of the inspection device 40 need not be as described above; different configurations may be used depending on the analytical method. For example, in thermal analysis, illumination (light source 43) may not be required, and an InSb camera may be used as the imaging device (camera 41). Also, when performing analysis using patterned images, incoherent or coherent light sources may be used as illumination (light source 43), and a two-dimensional detector or a light scanning device and a photodiode may be used as the imaging device. In the case of laser analysis, incoherent or coherent light sources may be used as illumination (light source 43), and a device for obtaining the electrical characteristics of semiconductor devices may be used as the imaging device.

[0028] The observation device 50 is a device for photographing a live specimen placed on a glass slide and observing the photographed image of the live specimen. The observation device 50 can also be a previously described observation device. For example, the observation device 50 is the aforementioned virtual slide scanner. The observation device 50 includes a camera for photographing the object to be photographed, i.e., the live specimen, a mounting part for placing the object to be photographed, i.e., the live specimen, and an observation part for observing the live specimen by means of the photographed image. The focus position of the observation device 50 during imaging is controlled by the computer 10, as described later. The mounting part of the observation device 50 is also configured to control the tilt (posture) of the object to be photographed, which is mounted in the same way as the mounting part 42 of the inspection device 40, relative to the imaging direction.

[0029] The imaging of the inspection device 40 and the observation device 50 needs to be performed in a state where the object being imaged is tilted appropriately. A state where the object is tilted appropriately means, for example, that the surface of the object being imaged is perpendicular to the imaging direction; that is, the object is not tilted relative to the imaging direction. This is for the proper inspection or observation of the object. For example, the object of fault analysis is a semiconductor device whose back surface is polished during the manufacturing process. Due to the strain during polishing and the stress caused by the process, warping and strain occur on the front surface of the semiconductor device used as a sample. When the semiconductor device is tilted, for example, there may be contact problems with the solid-state immersion lens, so it is necessary to correct the tilt during imaging, i.e., tilt correction. Alternatively, the tilt of the object being imaged can also be caused by reasons other than those mentioned above.

[0030] The imaging of the inspection device 40 and the observation device 50 needs to be performed when the object being imaged is in focus. This is for the proper inspection or observation of the object being imaged. Figure 3 shows an example of an image taken corresponding to the focus position. This image is of the standard sample 60 shown in Figure 2. Figure 3(a) is an image taken at the focus position when in focus. Figure 3(b) is an image when the focus position is 100 μm away from the focus position when in focus. Figure 3(c) is an image when the focus position is further away than in Figure 3(b). Figure 3(d) is an image when the focus position is further away than in Figure 3(c), 500 μm away from the focus position when in focus. That is, Figure 3(a) is an example of a focused image, and Figures 3(b) to (d) are examples of defocused images.

[0031] The computer 10 performs information processing for imaging by the inspection device 40 and the observation device 50 when the object being imaged is tilted appropriately. The computer 10 can also perform information processing for imaging by the inspection device 40 and the observation device 50 when the object being imaged is in focus. As a functional component, the computer 10 includes: a feature quantity output model generation system 20, which generates a pre-learning model through machine learning; and a tilt estimation system 30, which uses the pre-learning model generated by the feature quantity output model generation system 20 to perform the aforementioned imaging information processing. Details of the feature quantity output model generation system 20 will be described later; it is a system (device) that generates a feature quantity output model that takes image-based information as input and outputs the feature quantities of the image. Details of the tilt estimation system 30 will be described later; it is a system (device) that estimates the tilt of the object being imaged as reflected in the image. In this embodiment, the display inspection device 40 and the observation device 50 are used as the imaging devices, but if the imaging device (system) is used to image the object being imaged, other than those mentioned above may also be used.

[0032] Computer 10 is, for example, a conventional computer that includes hardware such as a CPU (Central Processing Unit), memory, and a communication module. Alternatively, computer 10 may be a computer system comprising a plurality of computers. Furthermore, computer 10 may be configured for cloud computing. Through these components, computer 10 performs the functions described later according to programs and other operations. Computer 10 is interconnected with inspection device 40 and observation device 50 in a manner capable of transmitting and receiving information.

[0033] Next, the functions of the feature output model generation system 20 and the tilt estimation system 30 included in the computer 10 of this embodiment will be explained. As shown in FIG1, the feature output model generation system 20 is configured to include a learning image acquisition unit 21, a feature output model generation unit 22, and a focus position estimation model generation unit 23.

[0034] Before explaining the functions of the feature output model generation system 20, the pre-learning model generated by the feature output model generation system 20 will be explained. The pre-learning model generated by the feature output model generation system 20 is either a feature output model or a focus position estimation model.

[0035] The feature quantity output model is a model that takes image-based information as input and outputs feature quantities of the image. The image used as input to the feature quantity output model is a portion of the image captured by the inspection device 40 and the observation device 50. The output of the feature quantity output model, namely the feature quantity, is information that displays the features of the image related to the input. In this embodiment, the feature reflects the focal position when the image is captured. That is, the feature quantity output model is an optical model related to optical features. The feature quantity is, for example, a vector of a pre-defined dimension (e.g., 1024 dimensions). As described later, the feature quantity is used as input to the focal position estimation model.

[0036] The feature output model is constructed, for example, by including a neural network. The neural network can also be multi-layered. That is, the feature output model can also be generated by deep learning. Furthermore, the neural network can also be a convolutional neural network (CNN).

[0037] In the feature output model, neurons are set up to input image-based information into the input layer. For example, the information input to the feature output model is the pixel value of each pixel in the image. In this case, neurons representing the number of pixels in the image are set up in the input layer, and the pixel value of the corresponding pixel is input to each neuron. As described later, the image related to the information input to the feature output model is set to an image of a pre-defined size (e.g., an image of 224×224 pixels).

[0038] Furthermore, the information input to the feature output model can be anything other than the pixel values ​​of each pixel, as long as it is image-based. For example, this information can also be a feature quantity used as input to the feature output model, obtained through preprocessing such as prior image processing to mitigate the effects of the camera environment. By performing such preprocessing, the efficiency of machine learning and the accuracy of the generated feature output model can be improved.

[0039] In the feature output model, neurons are set up to output features to the output layer. For example, neurons are set up to determine the dimension of the feature vector.

[0040] The focus position estimation model is a model that takes into account the feature quantity output by the feature quantity output model and estimates the focus position when the image corresponding to the feature quantity is in focus. The focus position estimation model outputs, for example, information such as the difference between the focus position when the image corresponding to the input feature quantity is captured and the focus position when in focus, as the estimation result of the focus position when in focus. The difference is, for example, the value of the focus distance corresponding to the focus position when the image corresponding to the feature quantity is captured, minus the focus distance corresponding to the focus position when the image is captured. That is, in this case, the value of the focus position when the image is captured in the coordinate system when the focus position when in focus is set to 0 becomes the output value. The focus position when in focus is captured is the focus position used to align the focus with the object being photographed in the image corresponding to the input feature quantity. By changing the focus position by the above difference from the focus position when the defocused image corresponding to the input feature quantity is captured, a focused image can be captured.

[0041] In this case, alternatives to the aforementioned difference can also be preset, and the focal position estimation model can output a value representing the appropriateness of the alternative. For example, alternatives to the difference can be set to +50 μm, 0 μm, -50 μm, -100 μm, ..., and the focal position estimation model can output a value representing the appropriateness of each alternative. For example, the alternative with the highest value can be set as the aforementioned difference. Alternatively, the focal position estimation model can also output the aforementioned difference value itself.

[0042] Alternatively, the focus position estimation model may also output information about the focus position itself when in focus (e.g., the focus distance corresponding to the focus position when in focus). In this case, a candidate focus position itself when in focus may also be preset, and the focus position estimation model outputs a value indicating the appropriateness of the candidate for that candidate. Alternatively, the focus position estimation model may also output the focus position value itself when in focus.

[0043] The focal location estimation model may be constructed by including a neural network. The neural network may also be multi-layered. That is, the focal location estimation model may also be generated by deep learning. Furthermore, the neural network may also be a convolutional neural network (CNN).

[0044] In the focal position estimation model, neurons are set up to input feature quantities into the input layer. For example, neurons corresponding to those set in the output layer of the feature quantity output model are set up in the input layer. That is, neurons in the input layer are set up in the same number as those set in the output layer of the feature quantity output model. In the focal position estimation model, neurons are set up to output the estimated result of the focal position when in focus. For example, neurons with a number of candidates (in the case of outputting the value of each candidate) or one neuron (in the case of outputting the difference or the focal position itself when in focus) are set up.

[0045] In addition, the feature output model and the focal position estimation model can also be constructed from sources other than neural networks.

[0046] Assume that the feature output model and focus location estimation model are used as part of an artificial intelligence software program module. The feature output model and focus location estimation model are used, for example, in a computer equipped with a CPU and memory. The computer's CPU operates according to instructions from the model stored in memory. For example, the computer's CPU operates by inputting information into the model according to the instructions, performing calculations corresponding to the model, and outputting results from the model. Specifically, the computer's CPU operates as follows: according to the instructions, it inputs information into the input layer of the neural network, performs calculations based on parameters such as pre-learned weight coefficients in the neural network, and outputs results from the output layer of the neural network.

[0047] The learning image acquisition unit 21 is a learning image acquisition mechanism that acquires focus position information related to the focus position during imaging and establishes a plurality of corresponding learning images. The learning image acquisition unit 21 can also acquire images of radiation detected from the object being imaged, images of light emitted from the object being imaged when light is irradiated onto the object being imaged, or images of the electrical characteristics of the object being imaged when light is irradiated onto the object being imaged, as learning images. The learning image acquisition unit 21 can also acquire images of the object being imaged when light of a specific wavelength is irradiated onto it, as learning images. The learning image acquisition unit 21 acquires focus position information related to the focus position during focusing corresponding to each acquired learning image.

[0048] For example, the learning image acquisition unit 21 acquires an image captured by the inspection device 40 or the observation device 50. This image is a photographic object used to represent the learning image. The photographic object used for the learning image may, for example, be the standard sample 60 shown in FIG2. Alternatively, the photographic object used for the learning image may be something else (e.g., captured by the inspection device 40 or the observation device 50). For example, as shown in FIG4, the learning image acquisition unit 21 cuts out a portion of the image from the acquired image 70 as a learning image 71. The learning image 71 is the image used for machine learning to generate the above two pre-learning models. Therefore, the learning image acquisition unit 21 acquires a plurality of learning images 71 in a number that can appropriately generate a pre-learning model.

[0049] Since this embodiment estimates the focal position when the image is in focus based on the defocused image, the defocused image is included in the plurality of learning images 71. Furthermore, there may be a plurality of defocused images, and their corresponding focal positions may be various. That is, the focal distances corresponding to their corresponding focal positions may also be various distances. Furthermore, the focused image may also be included in the plurality of learning images 71. Figure 4 shows an example of three acquired images 70. The longitudinal direction of the portion of image 70 shown in Figure 4 is the imaging direction (focal position direction, Z-axis direction).

[0050] The learning image 71 corresponds to the image used as input to the feature quantity output model. In this case, the feature quantity output model input is based on a portion of the image captured by the inspection device 40 or the observation device 50 that is the same size as the learning image 71, rather than information from the entire image. Therefore, the learning image acquisition unit 21 cuts out a portion of the image of a preset size used as input to the feature quantity output model from the acquired image 70, which is the learning image 71. The position where the learning image 71 is cut out from the image 70 is the portion that reflects the photographic object. However, the learning image 71 may also include a portion of the learning image 71 that does not reflect the photographic object. The position where the learning image 71 is cut out from the image 70 can also be preset. Alternatively, the position where the image 70 is image-recognized and estimated to reflect the photographic object can be set as the position where the learning image 71 is cut out.

[0051] As shown in Figure 4, the learning image acquisition unit 21 can also cut out a plurality of learning images 71 from one image 70. When a plurality of learning images 71 are cut out from the image 70, the positions of the learning images 71 can also be repeated.

[0052] In the inspection device 40 or the observation device 50, an image is captured to generate an image that forms the basis for the learning image 71. For example, in the inspection device 40 or the observation device 50, the position (XY) during imaging is fixed outside the imaging direction (Z-axis direction), and multiple consecutive images with different focal points are captured. As shown in FIG4, the focal points differ at certain intervals (steps) (△Z). Imaging of the learning image 71 for the inspection device 40 or the observation device 50 can also be performed using methods other than those described above.

[0053] Furthermore, the learning image acquisition unit 21 can also acquire the image 70 from outside the inspection device 40 or the observation device 50.

[0054] Furthermore, as described above, the learning image 71 may also be an image detecting radiation from the photographed object (an image used for luminescence and heat analysis), an image detecting light from the photographed object when light is irradiated onto the photographed object (an image used for pattern analysis), or an image detecting the electrical characteristics of the photographed object when light is irradiated onto the photographed object (an image used for laser analysis). Furthermore, the learning image 71 may also be an image when light of a specific wavelength (e.g., light of a specific wavelength used for inspection) is irradiated onto the photographed object. These are types of images commonly used by the inspection device 40 or the observation device 50. However, when generating a set of feature quantity output models and a focus position estimation model, any type of image may be used as the learning image 71. In this case, the generated set of feature quantity output models and focus position estimation models corresponds to the type of specific image.

[0055] The wavelength of light used in the luminescence analysis varies depending on the driving voltage and design rules of the inspection device 40. Furthermore, in actual optical systems, focal deviations occur at each wavelength due to adjustments and the characteristics of optical elements. Also, there are cases where the detection sensitivity is highest at a position different from the focal position of the observed pattern image. Considering these situations, as described above, specific images for each wavelength can be used to generate characteristic output models for each wavelength. For example, images of the standard wavelength (1100 nm) and each wavelength different from the standard wavelength (900 nm, 1300 nm) can also be used to generate characteristic output models for each wavelength.

[0056] Furthermore, when generating a set of feature quantity output models and focus position estimation models, images (including some images) captured by any type of inspection device 40 or observation device 50 can be used as the learning image 71. In this case, the generated set of feature quantity output models and focus position estimation models corresponds to a specific type of inspection device 40 or observation device 50. That is, the feature quantity output models and focus position estimation models reflect the characteristics of the inspection device 40 or observation device 50. In this way, by making the learning image correspond to a specific type of image or a specific type of inspection device 40 or observation device 50, a higher-precision pre-learning model can be set. Alternatively, the focus position estimation model can be common to multiple image types or types.

[0057] Focus position information related to the focus position during the capture of the learning image 71 is established to correspond with each learning image 71. The focus position information may be, for example, information displaying the aforementioned focus position. However, if the focus position information is focus position-related information that can be used to generate the aforementioned pre-learning model, it may also be something other than that. The focus position information may be obtained, for example, as information captured by the inspection device 40 or the observation device 50 during image capture. For example, the learning image acquisition unit 21 receives an image corresponding to the focus position information from the inspection device 40 or the observation device 50.

[0058] Furthermore, the learning image acquisition unit 21 acquires focus position information related to the focus position when in focus corresponding to each acquired learning image. The focus position information may be, for example, information displaying the focus position when in focus. However, if the focus position information is information related to the focus position when in focus, it can be used for generating the aforementioned pre-learning model, or it may be something other than that. The focus position information is obtained by a previously used method for measuring the focus position when in focus. For example, the learning image acquisition unit 21 acquires the focus position information by accepting input from the user to the computer 10 using the measured focus position information.

[0059] The learning image acquisition unit 21 outputs the acquired information to the feature quantity output model generation unit 22 and the focus position estimation model generation unit 23.

[0060] The feature output model generation unit 22 is a feature output model generation mechanism that generates a feature output model by machine learning based on the learning image 71 acquired by the learning image acquisition unit 21. The feature output model generation unit 22 establishes corresponding focal position information with two different learning images 71, compares the feature quantities of the two learning images 71, and performs machine learning based on the comparison result. The feature output model generation unit 22 can also perform machine learning in a manner that reduces the difference between the feature quantities of the two different learning images 71 when the two different learning images 71 have the same focal position, and in a manner that increases the difference between the feature quantities of the two different learning images 71 when the two different learning images 71 have different focal positions.

[0061] The feature output model generation unit 22 generates a feature output model as described below. The feature output model generation unit 22 inputs a learning image 71 and focal position information related to the learning image 71 from the learning image acquisition unit 21. The feature output model generation unit 22 uses two learning images 71 selected from the input plurality of learning images 71 as a group to perform machine learning for generating the feature output model. The group used for machine learning includes both groups of learning images 71 with the same focal position and groups of learning images 71 with different focal positions. For example, a group of learning images 71 with the same focal position, as shown in FIG4, can also be learning images 71 cut from the same image 70. The selection of the group of learning images 71 can be performed in a preset method as long as the above conditions are met. Furthermore, the group of learning images 71 can also be selected from images 70 with the same position (XY) during imaging, other than the imaging direction (Z-axis direction).

[0062] The feature output model generation unit 22 uses information from the selected set of learning images 71 as input to the feature output model for machine learning. As shown in Figure 4, if each of the learning images 71 is input to the feature output model 80, a feature quantity is obtained as the output for each of the learning images 71. In Figure 4, the values ​​of each element of the feature quantity, i.e., the vector, are displayed in a bar chart. At this time, the feature output model 80 input to one learning image 71 is set as the learning object, and the feature output model 80 input to another learning image 71 is set as the comparison object. However, these feature output models 80 are the same during the learning process.

[0063] The feature output model generation unit 22 compares the two output feature quantities with reference to the focal position information and performs machine learning based on the comparison result. When the focal positions of the two learning images 71 shown by the focal position information are the same (i.e., on the same plane), the feature output model generation unit 22 performs machine learning in a way that the difference between the feature quantities of the two learning images 71 decreases. When the focal positions of the two learning images shown by the focal position information are different (i.e., different Z positions), the feature output model generation unit 22 performs machine learning in a way that the difference between the feature quantities of the two learning images 71 increases. Furthermore, when the two learning images 71 are cut from the same image, the focal positions of the two learning images 71 become the same. Furthermore, when the focal positions of two learning images 71 are considered to be equally close to each other, the focal positions of the two learning images 71 can also be considered to be the same focal positions to each other.

[0064] That is, the feature quantities of partial images cut from images located on the same focal plane are independent of the cut-out position, and the correlation between them increases. On the other hand, the correlation between the feature quantities of partial images cut from images located on different focal planes decreases. Thus, through machine learning, the feature quantities output by the feature quantity output model reflect the features corresponding to the focal position.

[0065] Specifically, when the focal positions of the two learning images 71 are the same, the feature output model generation unit 22 performs machine learning using the following loss_xy as the loss function. [Number 1] Here, i (0~n) is the suffix of the vector of features. The number of channels (dimensionality) of the vector of features is n+1. Ft0~Ftn are the values ​​of each element of the vector of features output by the feature output model 80 of the learning object. Fc0~Fcn are the values ​​of each element of the vector of features output by the feature output model 80 of the comparison object. SDi is the standard deviation of each element i of the feature. By dividing the difference by the standard deviation as described above to calculate errori, the deviation of the difference of each channel is made consistent. The loss is the average (positive square root) of the errors of each channel.

[0066] When the focal positions of the two learning images 71 are different, the feature output model generation unit 22 performs machine learning using the following loss_z as the loss function. [Number 2] That is, the loss function at this time is the reciprocal of the loss function when the focal positions of the two learning images 71 are the same. Based on the machine learning of the loss function itself, the update of the parameters of the feature output model only needs to be performed in the same way as before. Furthermore, the loss function does not need to be as described above, as long as it follows the above benchmark.

[0067] The feature output model generation unit 22 repeatedly performs the selection and machine learning of the group of learning images 71 to generate a feature output model. For example, the feature output model generation unit 22 generates a feature output model by repeating the above process until the generation of the feature output model converges based on a preset condition, or by repeating the above process a preset number of times.

[0068] The feature output model generation unit 22 can also use an existing pre-learning model generated by machine learning to generate a feature output model. As an existing pre-learning model, similar to the feature output model of this embodiment, it uses a model that takes image-based information as input. That is, it can also use an existing pre-learning model that shares the same input as the feature output model of this embodiment. Existing pre-learning models are, for example, models used for image recognition, specifically ResNet, VGG, MobileNet, etc. For the generation of the feature output model, a portion of the existing pre-learning model is used. As shown in FIG5, the output-side layers of the existing pre-learning model 81 are deleted, and the portion up to the intermediate layers of the existing pre-learning model 81 is used for the generation of the feature output model. The existing pre-learning model 81 used to generate the feature output model may contain all intermediate layers, or it may contain only a portion of the intermediate layers.

[0069] The feature output model generation unit 22 inputs a portion of the existing pre-learning model as the feature output model at the start of machine learning. That is, the feature output model generation unit 22 uses a portion of the existing pre-learning model as the initial parameters of the feature output model for fine-tuning. Alternatively, a new output layer can be applied to the output side of the aforementioned portion of the pre-learning model as the feature output model at the start of machine learning. Furthermore, when a new output layer is applied, a new intermediate layer can be applied between the output side of the aforementioned portion of the pre-learning model and the new output layer as the feature output model at the start of machine learning.

[0070] Alternatively, the feature output model generation unit 22 may also generate a feature output model without using an existing pre-learning model. For example, similar to the previous machine learning, a model with random values ​​as initial parameters may be used as the feature output model at the start point of machine learning.

[0071] By using an existing pre-learning model to generate a feature output model, the following advantages are achieved: The learning time can be significantly reduced. Even with fewer learning images 71, a high-precision feature output model can be generated; that is, a feature output model that outputs more appropriate feature quantities can be generated. The aforementioned existing pre-learning model has the ability to separate features with low abstraction. Therefore, this is because only learning centered on high-abstraction features using the new learning image 71 is required.

[0072] The feature quantity output model generation unit 22 outputs the generated feature quantity output model to the focal position estimation model generation unit 23 and the tilt estimation system 30. Furthermore, the generated feature quantity output model can also be used for purposes other than those described in this embodiment. In such cases, for example, the feature quantity output model generation unit 22 may send or output the feature quantity output model to other devices or modules that use the feature quantity output model. Alternatively, the feature quantity output model generation unit 22 may also store the generated feature quantity output model in the computer 10 or other devices, making it usable in other devices or modules that use the feature quantity output model.

[0073] The focus position estimation model generation unit 23 is a focus position estimation model generation mechanism that generates a focus position estimation model by means of machine learning based on the focus position information obtained by the learning image acquisition unit 21. As described above, the focus position estimation model is input from the feature quantity output by the feature quantity output model generated by the feature quantity output model generation unit 22, and estimates the focus position at focus corresponding to the image associated with the feature quantity.

[0074] The focus position estimation model generation unit 23 generates a focus position estimation model as described below. The focus position estimation model generation unit 23 inputs a learning image 71 and related focus position information from the learning image acquisition unit 21. The focus position estimation model generation unit 23 inputs a feature quantity output model from the feature quantity output model generation unit 22.

[0075] The focus position estimation model generation unit 23 inputs information based on the learning image 71 into the feature output model and obtains the output from the feature output model, namely the feature quantity of the learning image 71. The focus position estimation model generation unit 23 uses the obtained feature quantity as input to the focus position estimation model and uses the information of the focus position information of the learning image 71 related to the input feature quantity as the output of the focus position estimation model, and performs machine learning. The information based on the focus position information is set to the information corresponding to the output of the focus position estimation model. When the focus position estimation model outputs each of the above candidate values, the information based on the focus position information is, for example, setting the candidate value that matches the focus position information to 1 and setting the candidate value that does not match the focus position information to 0 (one-hot vector). When the focus position estimation model outputs the above difference or focus position itself value, the information based on the focus position information is the value calculated based on the focus position information itself or the focus position information. Before performing machine learning, the focus position estimation model generation unit 23 generates information based on the focus position information equivalent to the output of the focus position estimation model.

[0076] The machine learning itself, namely the parameter update of the focal position estimation model, can be performed in the same way as before. The focal position estimation model generation unit 23 performs the machine learning process repeatedly until the generation of the focal position estimation model converges based on the preset conditions, or repeatedly for a preset number of times, in the same way as before, to generate the focal position estimation model.

[0077] The focal position estimation model generation unit 23 outputs the generated focal position estimation model to the tilt estimation system 30. Alternatively, the generated focal position estimation model can also be used for purposes other than those described in this embodiment. In this case, for example, the focal position estimation model generation unit 23 may send or output the focal position estimation model to other devices or modules that use the focal position estimation model. Alternatively, the focal position estimation model generation unit 23 may also store the generated focal position estimation model in the computer 10 or other devices, making it usable in other devices or modules that use the focal position estimation model. The above describes the functions of the feature quantity output model generation system 20.

[0078] Next, the function of the tilt estimation system 30 of this embodiment will be explained. As shown in FIG1, the tilt estimation system 30 is configured to include an image acquisition unit 31 for the estimated object, a focus position estimation unit 32, a tilt estimation unit 33, and a control unit 34.

[0079] The tilt estimation system 30 estimates the tilt of the object being photographed as reflected in the image captured by the inspection device 40 or the observation device 50. Therefore, when the object is photographed in the inspection device 40 or the observation device 50, the tilt estimation system 30 estimates the focal point position when the image is in focus. This estimation is first performed in the inspection device 40 or the observation device 50 by photographing the object (photographing for tilt estimation). In this photographing, the focal point position may not necessarily be a focal point on the object being photographed, that is, it does not need to be the focal point position when the image is in focus. Therefore, the image obtained by this photographing can also be a defocused image. For example, the image can also be a defocused image as shown in FIG6(a). FIG6 is an image of a semiconductor device. The defocused image shown in FIG6(a) is an image when the focal point position during the photographing is 5 nm away from the focal point position when the image is in focus.

[0080] The tilt estimation system 30 estimates the focal point position when in focus at a plurality of positions in the image. The tilt estimation system 30 estimates the tilt of the object being photographed in the image based on the estimated focal point position when in focus at a plurality of positions in the image. Furthermore, using the estimated focal point position when in focus, an image is captured in the inspection device 40 or the observation device 50, thereby obtaining an image where the object is in focus, i.e., a focused image. For example, the focused image shown in FIG. 6(b) is obtained. The focused image shown in FIG. 6(b) corresponds to the defocused image in FIG. 6(a).

[0081] The estimated object image acquisition unit 31 is an estimated object image acquisition mechanism that acquires an image of the image of the object being photographed and acquires a plurality of partial images from that image, namely, estimated object images. The estimated object image is an image used in the tilt estimation system 30 to estimate the focal position when in focus. Furthermore, the estimated object image is an image used as input to the feature quantity output model. That is, the estimated object image corresponds to the learning image 71 mentioned above.

[0082] The presumed object image acquisition unit 31 acquires the image captured by the inspection device 40 or the observation device 50. At this time, the camera of the inspection device 40 or the observation device 50 is the aforementioned tilting presumed image. For example, the presumed object image acquisition unit 31 cuts out a plurality of partial images from the acquired image and sets them as the presumed object image.

[0083] The estimated object image acquisition unit 31 extracts a portion of the image of a preset size from the acquired image, which is used as the input of the feature quantity output model. This portion is the estimated object image. The position from which the estimated object image is extracted is the part that reflects the photographed object. The position from which the estimated object is extracted can also be preset. For example, as shown in FIG7, the estimated object image acquisition unit 31 can also divide the image 90 into a plurality of portions of a preset size as estimated object images 91. In the example shown in FIG7, nine estimated object images 91 of 3×3 are extracted and acquired from the image. When extracting the estimated object images 91 as described above, if the photographed object is reflected in the entire image 90, then it is sufficient to extract the estimated object image 91 from the entire image 90; if the photographed object is reflected in only a portion of the image 90, then it is sufficient to extract the estimated object image 91 from that portion. Alternatively, the position of the object to be identified and estimated in the image can be set as the position of the estimated object image. The tilt estimation system 30, configured to extract the estimated object image from the image, is also used for the functions described later.

[0084] Furthermore, the type of the presumed object image is the same as that of the learning image described above. For example, the presumed object image may be an image detecting radiation from the photographed object, an image detecting light from the photographed object when light is irradiated onto the photographed object, or an image detecting the electrical characteristics of the photographed object when light is irradiated onto the photographed object. Also, the presumed object image may be an image obtained when light of a specific wavelength (e.g., light of a specific wavelength used for inspection) is irradiated onto the photographed object.

[0085] The estimated object image acquisition unit 31 outputs the acquired plurality of estimated object images to the focus position estimation unit 32.

[0086] The focus position estimation unit 32 is a focus position estimation mechanism. It uses a feature quantity output model to output a plurality of feature quantities of each of the plurality of estimated object images acquired by the estimated object image acquisition unit 31, and estimates the focus position at focus corresponding to each of the plurality of estimated object images based on the output feature quantities. The focus position estimation unit 32 may also use a focus position estimation model to estimate the focus position at focus corresponding to the estimated object image from the feature quantities output from the feature quantity output model.

[0087] The focal position estimation unit 32 pre-inputs and memorizes the feature quantity output model and focal position estimation model generated by the feature quantity output model generation system 20 for estimation. The focal position estimation unit 32 inputs a plurality of estimation object images from the estimation object image acquisition unit 31.

[0088] The focus position estimation unit 32 inputs information based on the estimated object image to the feature quantity output model and obtains the feature quantity of the estimated object image as the output of the feature quantity output model. The focus position estimation unit 32 inputs the obtained feature quantity to the focus position estimation model and obtains the output of the focus position estimation model, which is the information of the focus position when the image is in focus corresponding to the estimated object image, as the estimation result of the focus position. The focus position estimation unit 32 obtains the information of the focus position when the image is in focus corresponding to each of the plurality of estimated object images. The focus position estimation unit 32 outputs the information of the focus position when the image is in focus corresponding to the plurality of estimated object images to the tilt estimation unit 33.

[0089] The tilt estimation unit 33 is a tilt estimation mechanism that estimates the tilt of the photographic object projected in the image based on the focal point position estimated by the focal point position estimation unit 32 when each of the plurality of estimated object images is in focus. The tilt estimation unit 33, for example, estimates the tilt of the photographic object relative to the imaging direction in the inspection device 40 or the observation device 50. The tilt estimation unit 33 estimates the tilt of the photographic object as described below.

[0090] The tilt estimation unit 33 inputs and displays information from the focus position estimation unit 32 regarding the focus position when each of the plurality of estimated object images is in focus. In this embodiment, the position in the imaging direction corresponding to the focus position when the estimated object image is in focus is the position of the object being photographed. Since the estimated object image is cut out from a plurality of positions in the image, the position of the object being photographed in the imaging direction at each position of the estimated object image in the image can be obtained. The tilt of the object being photographed is estimated from the positions of these plurality of object being photographed.

[0091] For example, the tilt estimation unit 33 estimates the tilt of the object being photographed on each coordinate axis of a plane perpendicular to the imaging direction. The tilt estimation unit 33 estimates the tilt of the object being photographed according to the two coordinate axes, namely the X-axis and the Y-axis, which are parallel to each side of the estimated object image 91 of image 90, as shown in FIG. 7. For example, the tilt estimation unit 33 estimates the tilt angle θ1 of the object being photographed on the X-axis relative to a plane perpendicular to the imaging direction (Z-axis direction) (the plane marked with a shading line in the figure), as shown in FIG. 8(a), and the tilt angle θ2 of the object being photographed on the Y-axis relative to a plane perpendicular to the imaging direction (Z-axis direction) (the plane marked with a shading line in the figure), as shown in FIG. 8(b).

[0092] Figure 8 shows information displaying the focus position estimated from each estimated object image 91 at the time of focusing (specifically, the difference between the focus position at the time of image capture and the focus position at the time of focusing) (values ​​such as +2, 0, -2, +3, 0, and -3). Figure 8(a) shows an example of the object being photographed tilted in the X-axis direction, specifically, an example of the object tilting to the right. Figure 8(b) shows an example of the object being photographed tilted in the Y-axis direction, specifically, an example of the object tilting towards the front.

[0093] The tilt estimation unit 33 calculates angles θ1 and θ2 according to the following formulas. [Number 3] Here, x is the length of the X-axis direction used to calculate the angle θ1 corresponding to the position of the estimated object image in the image. z1 is the magnitude of the deviation of the focal position when focusing in the shooting direction (Z-axis direction) corresponding to x. y is the length of the Y-axis direction used to calculate the angle θ2 corresponding to the position of the estimated object image in the image. z2 is the magnitude of the deviation of the focal position when focusing in the shooting direction (Z-axis direction) corresponding to y.

[0094] As shown in Figure 7, x and y are based on the position of the estimated object image. For example, x is the difference in the X-axis direction between the positions (e.g., the center coordinates Pcenter of the estimated object image) of two predetermined estimated object images Pa and Pb that are far apart from each other in the X-axis direction of the estimation tilt. That is, x = |Pb-Pa| (X-axis component only). y is the difference in the Y-axis direction between the positions (e.g., the center coordinates Pcenter of the estimated object image) of two predetermined estimated object images Pa and Pc that are far apart from each other in the Y-axis direction of the estimation tilt. That is, y = |Pc-Pa| (X-axis component only). Furthermore, when the position of the estimated object image in the image is predetermined, the above x and y are fixed values, and therefore can also be pre-memorized in the tilt estimation section 33.

[0095] z1 and z2, as shown in Figure 8, are calculated by the focal positions corresponding to the focus points of the estimated object images when they are in focus. For example, z1 is the difference between the focal positions Za and Zb corresponding to the focus points of the two estimated object images Pa and Pb when they are in focus. That is, z1 = |Zb - Za|. z2 is the difference between the focal positions Za and Zc corresponding to the focus points of the two estimated object images Pa and Pc when they are in focus. That is, z2 = |Zc - Za|.

[0096] The presumed object image used for the tilt of the photographed object may also be other than those described above. For example, the presumed object image used for the tilt of the photographed object may also be used to presum the tilt as described below. The image showing the photographed object is divided into a plurality of regions, each containing a plurality of presumed object images. For example, the image is divided into 3×3 rectangular regions. The deviation value is removed from the focus position (Za1, Za2, Za3, ...) corresponding to each of the plurality of presumed object images (Pa1, Pa2, Pa3, ...) in each region (Pa). The center value (Zmedian) of the focus position in focus after removing the deviation value is taken. The tilt is presumed by using the center value (Zmedian) and the presumed object image (Pamedian) corresponding to the center value (Zmedian) as the focus position in focus and the presumed object image representing the region (Pa).

[0097] If the tilt estimation unit 33 estimates the tilt of the photographic object projected in the image based on the focus position at the moment of focus corresponding to each of the plurality of estimated object images estimated by the focus position estimation unit 32, then the tilt can also be estimated by methods other than those described above. Furthermore, the tilt estimation unit 33 can also estimate the tilt of the photographic object as other than the aforementioned angles θ1 and θ2. The tilt estimation unit 33 outputs the estimated tilt information of the photographic object to the control unit 34. Furthermore, the tilt estimation unit 33 can also output the information of the focus position at the moment of focus corresponding to the estimated object image used to estimate the tilt of the photographic object to the control unit 34.

[0098] The control unit 34 is a control mechanism that controls the tilt of the object being photographed based on the tilt of the object being photographed as estimated by the tilt estimation unit 33. The control unit 34 may also control the focus position of the object being photographed based on the focus position estimated by the focus position estimation unit 32.

[0099] The control unit 34 inputs and displays information about the tilt of the object being photographed from the tilt estimation unit 33. Furthermore, the control unit 34 can also input and display information about the focus position corresponding to the estimated object image when in focus from the tilt estimation unit 33. The control unit 34 controls the inspection device 40 or the observation device 50 in a manner that eliminates the tilt of the object being photographed as indicated by the input information during image capture. Specifically, the control unit 34 controls the inspection device 40 or the observation device 50 to tilt in the opposite direction to the tilt of the object being photographed as indicated by the input information. The inspection device 40 or the observation device 50 receiving control, for example, actuates its mounting section to adjust the tilt of the object being photographed during image capture. Thus, the control unit 34 performs tilt correction control of the inspection device 40 or the observation device 50. Consequently, the image captured by the inspection device 40 or the observation device 50 becomes an image where the tilt of the object being photographed is in an appropriate state.

[0100] Furthermore, the control unit 34 controls the inspection device 40 or the observation device 50 such that the focus position during image capture becomes the focus position during focusing as indicated by the input information. For example, the control unit 34 controls the focus position to correspond to a pre-set focus position among a plurality of estimated object images. The inspection device 40 or the observation device 50, upon receiving the control, moves the stage 46 to adjust the focus position during image capture. Thus, the image captured by the inspection device 40 or the observation device 50 becomes a focused image. In this way, the control unit 34 performs automatic focusing control of the inspection device 40 or the observation device 50. The above describes the configuration of the tilt estimation system 30.

[0101] Next, using the flowcharts of Figures 9 and 10, the processing performed by the computer 10 of this embodiment (the action method performed by the computer 10) will be explained. First, using the flowchart of Figure 9, the processing performed when generating the feature output model and the focus position estimation model will be explained, that is, the processing performed by the feature output model generation system 20 of this embodiment, that is, the feature output model generation method.

[0102] In this process, firstly, the learning image acquisition unit 21 acquires focus position information related to the focus position during shooting and establishes a plurality of corresponding learning images (S01, learning image acquisition step). Next, the learning image acquisition unit 21 acquires focus position information related to the focus position during focusing corresponding to each of the learning images. Then, the feature output model generation unit 22 generates a feature output model based on the learning images using machine learning (S02, feature output model generation step). At this time, based on the focus position information corresponding to two different learning images, the feature quantities of the two learning images 71 are compared, and machine learning is performed based on the comparison result. Next, the focus position estimation model generation unit 23 generates a focus position estimation model from the focus position information using machine learning (S03, focus position estimation model generation step).

[0103] The generated feature quantity output model and focal position estimation model are output from the feature quantity output model generation system 20 to the tilt estimation system 30. The tilt estimation system 30 stores the feature quantity output model and focal position estimation model for the following processing. The above is the processing performed by the feature quantity output model generation system 20 of this embodiment, that is, the feature quantity output model generation method.

[0104] Next, using the flowchart in FIG10, the process performed when estimating the focus position corresponding to the image being estimated is explained, that is, the process performed by the tilt estimation system 30 of this embodiment, that is, the tilt estimation method.

[0105] In this process, firstly, the image of the object being photographed is acquired by the presumed object image acquisition unit 31, and a plurality of partial images are acquired from that image, which are the presumed object images (S11, presumed object image acquisition step). The presumed object images are based on images acquired by the tilt-prediction camera of the inspection device 40 or the observation device 50. Next, the focus position estimation unit 32 outputs a plurality of feature quantities of each of the plurality of presumed object images from each of the presumed object images using a feature quantity output model. Next, the focus position estimation unit 32 uses a focus position estimation model to estimate the focus position when the presumed object images are in focus from the feature quantities of each of the plurality of presumed object images (S12, focus position estimation step).

[0106] Next, the tilt estimation unit 33 estimates the tilt of the photographed object projected in the image from the focal point position corresponding to each of the plurality of estimated object images when they are in focus (S13, tilt estimation step). Next, the control unit 34, based on the estimated tilt of the photographed object, suppresses the tilt of the inspection device 40 or the observation device 50 when taking a new image of the photographed object (S14, control step). Thus, the image captured by the inspection device 40 or the observation device 50 becomes an image with the photographed object tilted appropriately. Furthermore, at this time, the control unit 34 can also control the focal point position of the inspection device 40 or the observation device 50 when taking an image of the photographed object based on the estimated focal point position. Thus, the image captured by the inspection device 40 or the observation device 50 becomes a focused image. The above describes the processing performed by the tilt estimation system 30 of this embodiment, i.e., the tilt estimation method.

[0107] In this embodiment, a feature output model of the output image's feature quantities is generated through machine learning. At this time, based on the focal position information corresponding to two different learning images, the feature quantities of these two learning images are compared, and machine learning is performed based on the comparison result. According to this generation, for example, by using an existing pre-learning model that uses image-based input information, a feature output model can be generated in a short time. That is, according to this embodiment, a pre-learning model used for image-based estimations such as focal position estimation can be generated in a short time, namely, a feature output model.

[0108] However, as mentioned above, the generation of the feature output model does not require the use of an existing pre-learning model. In this case, a feature output model that can output appropriate feature quantities corresponding to the focal position can also be generated.

[0109] Furthermore, as described above, machine learning for generating a feature output model can be performed in such a way that the difference between the feature quantities of two different learning images decreases when they are at the same focal position, and in such a way that the difference between the feature quantities of two different learning images increases when they are at different focal positions. Based on this configuration, a feature output model can be reliably and appropriately generated. However, machine learning need not be performed in the manner described above; it can be performed simply based on the comparison results of the feature quantities of two learning images.

[0110] Furthermore, as described above, the learning image and the estimated object image can also be images detecting radiation from the photographed object, images detecting light from the photographed object when light is irradiated onto the photographed object, or images detecting the electrical characteristics of the photographed object when light is irradiated onto the photographed object. Moreover, the learning image and the estimated object image can also be images when light of a specific wavelength is irradiated onto the photographed object. Based on these configurations, appropriate feature output models corresponding to the type of image used can be generated, and the feature output models can be used. However, the learning image and the estimated object image are not limited to the above; any image corresponding to the focal point position is acceptable.

[0111] Furthermore, as in this embodiment, the feature output model generation system 20 may also include a focus position estimation model generation unit 23 for generating a focus position estimation model. According to this configuration, a focus position estimation model can be generated from the focus position estimated when the image is in focus. That is, according to this configuration, a pre-learning model for estimating the focus position of an image can be generated in a short time, in conjunction with the feature output model.

[0112] However, the feature quantity output model generation unit 20 may not have a focus position estimation model generation unit 23. That is, the feature quantity output model generation system 20 may also be configured to only generate feature quantity output models. Furthermore, the generated feature quantity output models may also be used for purposes other than focus position estimation during focusing.

[0113] In the tilt estimation system 30 of the present invention, the tilt of the photographed object is estimated by estimating the focal point position corresponding to each of the plurality of partial images of the image, i.e., the estimated object image, when they are in focus. Therefore, if an image of the photographed object is obtained, the tilt of the photographed object can be estimated in a short time. This eliminates the need for a special optical system (e.g., a lens rotating stage) and space required to install such a system within the device, which is necessary when estimating the tilt of the photographed object using a special optical system. Furthermore, the cost of preparing a special optical system is eliminated. Also, there is no need for switching rotating stages, laser scanning, and alignment processes. Compared to this, the tilt estimation time of the photographed object can be significantly shortened.

[0114] Furthermore, the feature quantity output model used for focal position estimation can output feature quantities suitable for estimating the focal position. By using these feature quantities, the tilt of the photographed object can be appropriately estimated. Therefore, the tilt estimation system of the present invention can be simply constructed and the tilt of the photographed object can be estimated in a short time.

[0115] In the tilt estimation system 30 of this embodiment, the aforementioned feature output model is used to estimate the focal position when in focus. Therefore, according to this embodiment, the focal position estimation when in focus based on the image can be performed with a short preparation time. Furthermore, according to the tilt estimation system 30 of this embodiment, the focal position when in focus can be estimated by a single tilt estimation camera. Therefore, compared to the case where multiple cameras are taken while changing the focal position to explore the focal position when in focus, the focal position when in focus can be estimated quickly. As a result, the tilt of the object being photographed can be estimated quickly.

[0116] Furthermore, the aforementioned focus position estimation model can also be used to estimate the focus position when in focus. Based on this configuration, the focus position when in focus can be estimated accurately and appropriately. As a result, the tilt of the photographed object can be estimated accurately and appropriately. However, for estimating the focus position when in focus, it is not necessary to use the aforementioned focus position estimation model; estimation can be performed solely from the feature quantity output by the self-feature quantity output model.

[0117] Furthermore, as in this embodiment, the tilt estimation system 30 may also include a control unit 34, which controls the tilt of the inspection device 40 or the observation device 50 when photographing the object based on the estimated tilt of the object being photographed. With this configuration, the inspection device 40 or the observation device 50 can photograph an object at an appropriate tilt. However, the tilt estimation system 30 may also lack a control unit 34. That is, the tilt estimation system 30 only needs to estimate the tilt of the object being photographed.

[0118] Furthermore, it can be configured as a series of systems including the tilt estimation system 30 of this embodiment and the inspection device 40 or the observation device 50 described above. That is, the semiconductor inspection system of this embodiment can also be configured as a system including the tilt estimation system 30 and the inspection device 40. Furthermore, the live observation system of this embodiment can also be configured as a system including the tilt estimation system 30 and the observation device 50.

[0119] In addition, in this embodiment, the computer 10 includes a feature output model generation system 20 and a tilt estimation system 30, but the feature output model generation system 20 and the tilt estimation system 30 can also be implemented independently.

[0120] Next, the feature quantity output model generation program and the tilt estimation program used to perform the aforementioned series of feature quantity output model generation system 20 and tilt estimation system 30 will be described. As shown in FIG11, the feature quantity output model generation program 200 is stored in a program storage area 211 formed in a computer-readable recording medium 210 inserted into and accessed by a computer. The recording medium 210 may also be a non-transitory recording medium.

[0121] The feature output model generation program 200 is configured to include a learning image acquisition module 201, a feature output model generation module 202, and a focus position estimation model generation module 203. The functions implemented by executing the learning image acquisition module 201, the feature output model generation module 202, and the focus position estimation model generation module 203 are the same as the functions of the learning image acquisition unit 21, the feature output model generation unit 22, and the focus position estimation model generation unit 23 of the feature output model generation system 20 described above.

[0122] As shown in Figure 12, the tilt estimation program 300 is stored in a program storage area 311 formed in a recording medium 310 that is inserted into and accessed by a computer, or that is computer-readable by the computer. The recording medium 310 may also be a non-transitory recording medium. Alternatively, the recording medium 310 may be the same as the recording medium 210.

[0123] The tilt estimation program 300 is configured to include an image acquisition module 301 for the estimated object, a focus position estimation module 302, a tilt estimation module 303, and a control module 304. The functions implemented by executing the focus position estimation module 302, the tilt estimation module 303, and the control module 304 are the same as the functions of the image acquisition unit 31 for the estimated object, the focus position estimation unit 32, the tilt estimation unit 33, and the control unit 34 of the tilt estimation system 30 described above.

[0124] Furthermore, the characteristic output model generator 200 and the tilt estimation program 300 may also be configured to be transmitted, in whole or in part, via a transmission medium such as a communication line, and received and recorded by other machines (including installation). Additionally, each module of the characteristic output model generator 200 and the tilt estimation program 300 may be installed on any of a plurality of computers, rather than on a single computer. In this case, the aforementioned series of processes are performed through the computer systems of the plurality of computers. [Simplified Explanation of the Diagram]

[0020] Figure 1 is a diagram showing the configuration of the feature quantity output model generation system and tilt estimation system of an embodiment of the present invention. Figure 2 is a diagram showing an example of a partial configuration of the inspection device. Figures 3(a) to (d) are examples of images captured at each focal position. Figure 4 is a diagram illustrating the generation of a feature quantity output model using machine learning. Figure 5 is a diagram of an existing pre-learning model used to generate a feature quantity output model. Figures 6(a) and (b) are diagrams showing examples of defocused and focused images. Figure 7 is a diagram showing an example of an image of an estimated object. Figures 8(a) and (b) are diagrams illustrating the tilt estimation of the photographed object. Figure 9 is a flowchart showing the process performed by the feature quantity output model generation system of an embodiment of the present invention, i.e., the feature quantity output model generation method. Figure 10 is a flowchart showing the process performed by the tilt estimation system of an embodiment of the present invention, i.e., the tilt estimation method. Figure 11 is a diagram showing the configuration of the feature quantity output model generation program of an embodiment of the present invention together with a recording medium. Figure 12 is a diagram showing the structure of the tilt estimation program of an embodiment of the present invention together with the recording medium.

Claims

1. A tilt estimation system for estimating the tilt of a photographic object projected in an image, and comprising: an estimation object image acquisition mechanism that acquires an image of the photographic object projected in the image, and acquires a plurality of partial images from the image to form the estimation object image; The focus position estimation mechanism uses a feature output model that takes image-based information as input and outputs the feature quantities of the image. Based on the feature quantities of each of the plurality of estimated object images obtained by the aforementioned estimated object image acquisition mechanism, it outputs the feature quantities of each of the plurality of estimated object images and estimates the focus position at focus corresponding to each of the plurality of estimated object images based on the output feature quantities. The tilt estimation mechanism estimates the tilt of the photographed object reflected in the image based on the focus position at focus corresponding to each of the plurality of estimated object images estimated by the aforementioned focus position estimation mechanism. The feature output model is generated by machine learning by establishing a plurality of corresponding learning images based on focus position information related to the focus position at the time of shooting. It compares the feature quantities of two different learning images based on the focus position information established by the model and performs machine learning based on the comparison results.

2. The tilt estimation system of claim 1, wherein the aforementioned focus position estimation mechanism is input from the feature quantity output by the aforementioned feature quantity output model, and uses a focus position estimation model to estimate the focus position when in focus corresponding to the image associated with the feature quantity, to estimate the focus position when in focus corresponding to each of the plurality of estimation object images, wherein the aforementioned focus position estimation model is generated by machine learning based on focus position information associated with the focus position when in focus corresponding to each of the aforementioned learning images.

3. The tilt estimation system of claim 1 or 2 further includes: a control mechanism that controls the tilt of the object being photographed during the photographing process based on the tilt of the object being photographed as estimated by the tilt estimation mechanism.

4. A semiconductor inspection system comprising: a tilt estimation system according to any one of claims 1 to 3; a mounting section for mounting a semiconductor device as an image object related to the tilt estimation system; and an inspection section for inspecting the semiconductor device.

5. A live observation system comprising: a tilt estimation system according to any one of claims 1 to 3; a mounting section for mounting a live sample as an object to be photographed in relation to the tilt estimation system; and an observation section for observing the live sample.

6. A tilt estimation method, which is an operation of a tilt estimation system that causes the tilt of a photographic object projected in a estimation image to be induced, comprising: a step of obtaining an estimation object image, wherein an image of the photographic object is obtained by means of the tilt estimation system, and a plurality of partial images are obtained from the image to form the estimation object image; The focus position estimation step utilizes the aforementioned tilt estimation system, employing a feature output model that takes image-based information as input and outputs the image's feature quantities. Based on the feature quantities of each of the plurality of estimated object images obtained in the aforementioned estimated object acquisition step, it outputs the feature quantities of each of the plurality of estimated object images. Based on the output feature quantities, it estimates the focus position at focus corresponding to each of the plurality of estimated object images. The tilt estimation step utilizes the aforementioned tilt estimation system, based on the focus positions at focus corresponding to each of the plurality of estimated object images estimated in the aforementioned focus position estimation step, to estimate the tilt of the object projected onto the image. Furthermore, the aforementioned feature output model is generated by machine learning, establishing a plurality of corresponding learning images based on focus position information related to the focus position at the time of shooting. It compares the feature quantities of two different learning images based on their corresponding focus position information, and performs machine learning based on the comparison results.

7. As in request item 6, the method of presuming tilt, wherein, Using the aforementioned tilt estimation system, in the aforementioned focus position estimation step, the feature quantity output from the aforementioned feature quantity output model is input, and a focus position estimation model that estimates the focus position when in focus corresponding to the image associated with the feature quantity is used to estimate the focus position when in focus corresponding to each of the plurality of estimation object images. The aforementioned focus position estimation model is generated by machine learning based on the focus position information associated with the focus position when in focus corresponding to each of the aforementioned learning images.

8. The tilt estimation method of claim 6 or 7 further includes: a control step, which controls the tilt of the camera object during imaging based on the tilt of the camera object estimated in the tilt estimation step by means of the tilt estimation system described above.

9. A tilt estimation program, which causes a computer to act as a tilt estimation system for estimating the tilt of a camera object projected in an image, and causes the computer to function as the following mechanism: a tilt object image acquisition mechanism that acquires an image of the camera object projected in the image, and acquires a plurality of partial images from the image, which are the tilt object images; The focus position estimation mechanism uses a feature output model that takes image-based information as input and outputs the feature quantities of the image. Based on the feature quantities of each of the plurality of estimated object images obtained by the aforementioned estimated object image acquisition mechanism, it outputs the feature quantities of each of the plurality of estimated object images and estimates the focus position at focus corresponding to each of the plurality of estimated object images based on the output feature quantities. The tilt estimation mechanism estimates the tilt of the photographed object reflected in the image based on the focus position at focus corresponding to each of the plurality of estimated object images estimated by the aforementioned focus position estimation mechanism. The feature output model is generated by machine learning by establishing a plurality of corresponding learning images based on focus position information related to the focus position at the time of shooting. It compares the feature quantities of two different learning images based on the focus position information established by the model and performs machine learning based on the comparison results.

10. The tilt estimation program of claim 9, wherein the aforementioned focus position estimation mechanism is input from the feature quantity output by the aforementioned feature quantity output model, and uses a focus position estimation model that estimates the focus position when in focus corresponding to the image associated with the feature quantity to estimate the focus position when in focus corresponding to each of the plurality of estimation object images, wherein the aforementioned focus position estimation model is generated by machine learning based on focus position information associated with the focus position when in focus corresponding to each of the aforementioned learning images.

11. As in request item 9 or 10, the tilt estimation program enables the computer to also function as a control mechanism, which controls the tilt of the object being photographed based on the tilt of the object being photographed as estimated by the tilt estimation mechanism.

Images