Tilt estimation system, tilt estimation method, tilt estimation program, semiconductor inspection system, and biological observation system

Abstract

The tilt of an object to be imaged can be estimated with a simple configuration in a short time. [Solution] The tilt estimation system 30 is a system that estimates the tilt of an imaged object captured in an image, and includes an estimated object image acquisition unit 31 that acquires estimated object images, which are multiple partial images, from the image, a focus position estimation unit 32 that outputs features from the estimated object images using a feature output model and estimates the focus position at the time of focusing corresponding to each of the multiple estimated object images, and a tilt estimation unit 33 that estimates the tilt of the imaged object from these focus positions at the time of focusing. The feature output model is generated by machine learning from multiple training images that are associated with focus position information related to the focus position at the time of imaging, and the features of two different training images are compared according to the focus position information associated with the two training images, and machine learning is performed based on the comparison results.

Description

[Technical Field]

[0001] The present invention relates to a tilt estimation system, a tilt estimation method, and a tilt estimation program for estimating the tilt of an object captured in an image, as well as a semiconductor inspection system and a biological observation system. [Background technology]

[0002] Conventionally, it has been proposed to estimate the tilt of an object to be imaged, for example, its tilt relative to the imaging direction, when imaging the object. For example, Patent Document 1 shows how to estimate the tilt of a semiconductor device when imaging the semiconductor device for inspection. The estimated tilt is used to adjust the orientation of the semiconductor device. In Patent Document 1, the tilt of the object to be imaged is estimated by an optical system including a dedicated lens (for example, a relay lens system) and a scanner. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] International Publication No. 2017 / 154895 [Overview of the project] [Problems that the invention aims to solve]

[0004] As shown in Patent Document 1, when estimating the tilt of an object to be imaged using a special optical system, a configuration (e.g., a lens turret) and space are required within the device to accommodate the special optical system. In addition, there is a cost associated with preparing the special optical system. Furthermore, the method shown in Patent Document 1 requires processes such as turret switching, laser scanning, and alignment, which results in a long time required for tilt estimation.

[0005] One embodiment of the present invention has been made in view of the above, and aims to provide a tilt estimation system, a tilt estimation method, and a tilt estimation program that can estimate the tilt of an object to be imaged with a simple configuration and in a short time, as well as a semiconductor inspection system and a biological observation system related thereto. [Means for solving the problem]

[0006] To achieve the above objective, a tilt estimation system according to one embodiment of the present invention is a tilt estimation system for estimating the tilt of an object captured in an image, comprising: an estimation target image acquisition means that acquires an image of an object and acquires multiple estimation target images, which are partial images, from the image; a focal position estimation means that uses a feature output model that inputs information based on the image and outputs the feature quantities of the image to output the feature quantities of each of the multiple estimation target images acquired by the estimation target image acquisition means, and estimates the focal position at the time of focus corresponding to each of the multiple estimation target images from the output feature quantities; and a tilt estimation means that estimates the tilt of an object captured in an image from the focal position at the time of focus corresponding to each of the multiple estimation target images estimated by the focal position estimation means, wherein the feature output model is generated by machine learning from a plurality of training images to which focal position information related to the focal position at the time of imaging is associated, and the feature quantities of two training images are compared according to the focal position information associated with two different training images, and machine learning is performed based on the comparison result.

[0007] In the tilt estimation system according to one embodiment of the present invention, the focal position at the time of focus corresponding to each of the multiple target images is estimated from each of the multiple target images, and the tilt of the object being photographed is estimated. Therefore, if an image of the object being photographed is obtained, the tilt of the object being photographed can be estimated in a short time. Furthermore, the feature output model used for estimating the focal position can output features appropriate for estimating the focal position, and by using this, the tilt of the object being photographed can be estimated appropriately. Therefore, according to the tilt estimation system according to one embodiment of the present invention, the tilt of the object being photographed can be estimated with a simple configuration and in a short time.

[0008] The focal position estimation means takes features output from a feature output model as input and uses a focal position estimation model that estimates the focal position at the time of focus corresponding to the image related to those features to estimate the focal position at the time of focus for each of the multiple target images. The focal position estimation model may be generated by machine learning from the focus position information related to the focal position at the time of focus for each training image. With this configuration, the focal position can be estimated reliably and appropriately by using the focal position estimation model. As a result, the tilt of the object being imaged can be estimated reliably and appropriately.

[0009] The tilt estimation system may further include control means for controlling the tilt of the object being imaged during imaging, based on the tilt of the object estimated by the tilt estimation means. This configuration allows for imaging of the object at an appropriate tilt.

[0010] A semiconductor inspection system and a biological observation system according to an embodiment of the present invention can be configured to include the above-described tilt estimation system. That is, a semiconductor inspection system according to an embodiment of the present invention includes the above tilt estimation system, a placement unit on which a semiconductor device is placed as an imaging object related to the tilt estimation system, and an inspection unit that inspects the semiconductor device. Further, a biological observation system according to an embodiment of the present invention includes the above tilt estimation system, a placement unit on which a biological sample is placed as an imaging object related to the tilt estimation system, and an observation unit that observes the biological sample.

[0011] Incidentally, an embodiment of the present invention can be described not only as an invention of the tilt estimation system as described above, but also as an invention of a tilt estimation method and a tilt estimation program as follows.

[0012] That is, a tilt estimation method according to an embodiment of the present invention is a tilt estimation method for estimating the tilt of an imaging object shown in an image, including an estimated target image acquisition step of acquiring an image in which the imaging object is shown and acquiring an estimated target image, which is a plurality of partial images, from the image, and a feature amount output step of using a feature amount output model that inputs information based on the image and outputs a feature amount of the image to output feature amounts of each of the plurality of estimated target images acquired in the estimated target image acquisition step, and estimating a focus position at the time of focusing corresponding to each of the plurality of estimated target images from the output feature amounts, and a tilt estimation step of estimating the tilt of the imaging object shown in the image from the focus positions at the time of focusing corresponding to each of the plurality of estimated target images estimated in the focus position estimation step. The feature amount output model is generated by machine learning from a plurality of learning images associated with focus position information related to the focus position at the time of imaging, and the feature amounts of two different learning images are compared according to the focus position information associated with the two different learning images, and machine learning is performed based on the comparison result.

[0013] In the focal position estimation step, using a focal position estimation model that estimates the focal position at the time of focusing corresponding to an image related to the feature amount by inputting the feature amount output from the feature amount output model, estimate the focal position at the time of focusing corresponding to each of a plurality of target images to be estimated. The focal position estimation model may be generated by machine learning from the focusing position information related to the focal position at the time of focusing corresponding to each learning image.

[0014] The inclination estimation method may further include a control step of controlling the inclination at the time of imaging the imaging object based on the inclination of the imaging object estimated in the inclination estimation step.

[0015] Further, an inclination estimation program according to an embodiment of the present invention is an inclination estimation program that causes a computer to operate as an inclination estimation system that estimates the inclination of an imaging object shown in an image. The computer is caused to function as an estimation target image acquisition means that acquires an image in which the imaging object is shown and acquires an estimation target image that is a plurality of partial images from the image, and using a feature amount output model that inputs information based on the image and outputs the feature amount of the image, outputs the feature amount of each of the plurality of estimation target images from each of the plurality of estimation target images acquired by the estimation target image acquisition means, and estimates the focal position at the time of focusing corresponding to each of the plurality of estimation target images from the output feature amount. A focal position estimation means, and an inclination estimation means for estimating the inclination of the imaging object shown in the image from the focal positions at the time of focusing corresponding to each of the plurality of estimation target images estimated by the focal position estimation means. The feature amount output model is generated by machine learning from a plurality of learning images associated with focal position information related to the focal position at the time of imaging, and the feature amounts of the two learning images are compared according to the focal position information associated with two different learning images, and machine learning is performed based on the comparison result.

[0016] The focus position estimation means takes features output from a feature output model as input and uses a focus position estimation model that estimates the focal position at the time of focus corresponding to the image related to the said features to estimate the focal position at the time of focus for each of the multiple target images to be estimated. The focus position estimation model may be generated by machine learning from focus position information related to the focal position at the time of focus corresponding to each of the training images.

[0017] The tilt estimation program may also function as a control means that controls the tilt of the object being imaged during imaging, based on the tilt of the object estimated by the tilt estimation means. [Effects of the Invention]

[0018] According to one embodiment of the present invention, the inclination of the object being imaged can be estimated with a simple configuration and in a short amount of time. [Brief explanation of the drawing]

[0019] [Figure 1] This figure shows the configuration of a feature output model generation system and a gradient estimation system according to an embodiment of the present invention. [Figure 2] This figure shows an example of some of the components of an inspection device. [Figure 3] These are examples of images captured at different focal points. [Figure 4] This diagram illustrates the generation of feature output models using machine learning. [Figure 5] This figure shows an existing pre-trained model used to generate a feature output model. [Figure 6] This figure shows examples of defocused and focused images. [Figure 7] This figure shows an example of an image to be estimated for a given image. [Figure 8] This is a diagram to explain the estimation of the tilt of the object being imaged. [Figure 9] This flowchart shows a feature output model generation method, which is a process performed by the feature output model generation system according to an embodiment of the present invention. [Figure 10] This is a flowchart showing a slope estimation method, which is a process performed by the slope estimation system according to an embodiment of the present invention. [Figure 11] This figure shows the configuration of a feature output model generation program according to an embodiment of the present invention, along with the recording medium. [Figure 12] This figure shows the configuration of a slope estimation program according to an embodiment of the present invention, along with a recording medium. [Modes for carrying out the invention]

[0020] Hereinafter, embodiments of the tilt estimation system, tilt estimation method, tilt estimation program, semiconductor inspection system, and biological observation system according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant explanations are omitted.

[0021] Figure 1 shows a computer 10, which is a tilt estimation system according to this embodiment. Computer 10 is a device (system) that performs information processing on images. Specifically, computer 10 performs information processing on images captured by at least one of the inspection device 40 and the observation device 50. Computer 10 may also perform information processing on images captured by devices other than the inspection device 40 and the observation device 50. In other words, computer 10 may be applied to imaging devices other than the inspection device 40 and the observation device 50.

[0022] The inspection device 40 is a device that images a semiconductor device and inspects the semiconductor device based on the image it captures. For example, the inspection device 40 performs failure analysis of a semiconductor device. The semiconductor device to be inspected is, for example, a wafer covered with μLEDs (Light Emitting Diodes). Failure analysis is performed, for example, by emission analysis, heat analysis, analysis using pattern images, or laser analysis (OBIRCH, OBIC, or DALS, etc.). The inspection device 40 may be a conventional inspection device. All of the configurations described below may also be those of a conventional inspection device.

[0023] Figure 2 shows, for example, a part of the configuration of an inspection apparatus 40 that performs emission analysis. As shown in Figure 2, the inspection apparatus 40 comprises a camera 41, a mounting unit 42, a light source 43, an optical system 44, an objective lens 45, and a stage 46. The camera 41 is an imaging device that images a semiconductor device, which is the object to be imaged. The camera 41 is, for example, an InGaAs camera. The mounting unit 42 is a configuration for mounting the semiconductor device, which is the object to be imaged. In Figure 2, the standard sample 60 placed on the mounting unit 42 is not the object to be inspected, but corresponds to an image corresponding to the focal position, which will be described later. The standard sample 60 is a sample with an artificial pattern (for example, a radial stripe pattern as shown in Figure 2).

[0024] The mounting unit 42 is configured to control the tilt (orientation) of the mounted object with respect to the imaging direction. For example, the mounting unit 42 is configured so that the tilt of the mounting surface on which the object is mounted is variable with respect to the imaging direction. In this way, the mounting unit 42 can perform tilt correction of the object being imaged. A conventional mounting unit 42 capable of controlling the tilt of the object being imaged can be used. The tilt of the object being imaged during imaging is controlled by the computer 10, as will be described later.

[0025] The light source 43 is a device that outputs illumination light to be irradiated onto the object to be imaged. The light source 43 may output illumination light of a specific wavelength (for example, the standard wavelength of 1100 nm, and wavelengths other than the standard wavelength, such as 900 nm and 1300 nm). Alternatively, multiple light sources 43 may be provided to allow switching of the illumination light wavelength. The optical system 44 is an optical system for irradiating the semiconductor device, which is the object to be imaged, with the illumination light output from the light source 43. The objective lens 45 is an objective lens used for imaging by the camera 41, and is, for example, a solid immersion lens (SIL). The stage 46 is a component for adjusting the focal position during imaging by the camera 41. The stage 46 may be movable not only in the imaging direction (focal position direction, Z-axis direction) but also in any direction in three dimensions (i.e., an XYZ stage). The focal position during imaging in the inspection apparatus 40 is controlled by the computer 10, as described later. The inspection apparatus 40 includes an inspection unit that performs inspection of the semiconductor device using the image obtained by the above configuration.

[0026] The configuration of the inspection device 40 does not have to be as described above, and different configurations may be used depending on the analysis method. For example, in thermal analysis, illumination (light source 43) may be omitted, and an InSb camera may be used as the imaging device (camera 41). Also, when performing analysis using pattern images, an incoherent or coherent light source may be used as the illumination (light source 43), and a two-dimensional detector, or a device that performs optical scanning and a photodiode may be used as the imaging device. When performing laser analysis, an incoherent or coherent light source may be used as the illumination (light source 43), and an electrical characteristic acquisition device for semiconductor devices may be used as the imaging device.

[0027] The observation device 50 is a device that images a biological sample placed on a glass slide and observes the image of the biological sample. The observation device 50 may be a conventional observation device. For example, the observation device 50 is the conventional virtual slide scanner described above. The observation device 50 comprises an imaging device that images the biological sample, which is the object to be imaged, a mounting unit on which the biological sample is placed, and an observation unit that observes the biological sample using the captured image. The focal position during imaging in the observation device 50 is controlled by the computer 10, as will be described later. The mounting unit of the observation device 50 is also configured to control the inclination (posture) of the object to be imaged relative to the imaging direction, similar to the mounting unit 42 of the inspection device 40 described above.

[0028] The imaging performed by the inspection device 40 and the observation device 50 must be carried out with the object being imaged in an appropriate inclination. An appropriate inclination of the object being imaged means, for example, that the surface of the object being imaged is perpendicular to the imaging direction, that is, that the object being imaged is not tilted relative to the imaging direction. This is to ensure that the inspection or observation of the object being imaged is carried out appropriately. For example, a semiconductor device that is the subject of failure analysis has its back surface polished during the process. Due to distortion during polishing and stress caused by the process, warping and distortion occur on the surface of the semiconductor device sample. If the semiconductor device is tilted, for example, there will be problems with the contact of the immersion lens, so it is necessary to correct the inclination during imaging, i.e., tilt correction. Note that the inclination of the object being imaged may be caused by reasons other than those mentioned above.

[0029] Image acquisition by the inspection device 40 and observation device 50 must be performed with the object being imaged in focus. This is to ensure proper inspection or observation of the object being imaged. Figure 3 shows examples of images acquired according to the focal position. These images were taken of the standard sample 60 shown in Figure 2. Figure 3(a) is an image taken at the focal position when in focus. Figure 3(b) is an image when the focal position is 100 μm away from the focal position when in focus. Figure 3(c) is an image when the focal position is even further away than in Figure 3(b). Figure 3(d) is an image when the focal position is even further away than in Figure 3(c), and is 500 μm away from the focal position when in focus. In other words, Figure 3(a) is an example of a focused image, and Figures 3(b) to (d) are examples of defocused images.

[0030] Computer 10 performs information processing to ensure that imaging by the inspection device 40 and observation device 50 is performed with the object being imaged in an appropriate inclination. Computer 10 may also perform information processing to ensure that imaging by the inspection device 40 and observation device 50 is performed with the object being imaged in focus. Functionally, computer 10 includes a feature output model generation system 20 that generates a trained model by machine learning, and a tilt estimation system 30 that performs information processing to enable the above imaging using the trained model generated by the feature output model generation system 20. The feature output model generation system 20 is a system (device) that takes image-based information as input and outputs the features of the image, as will be described in detail later. The tilt estimation system 30 is a system (device) that estimates the inclination of the object being imaged in the image, as will be described in detail later. In this embodiment, the inspection device 40 and observation device 50 are shown as the devices that perform imaging, but other devices (systems) that perform imaging of the object being imaged may be used.

[0031] Computer 10 is a conventional computer that includes hardware such as a CPU (Central Processing Unit), memory, and communication modules. Computer 10 may also be a computer system comprising multiple computers. Furthermore, Computer 10 may be configured using cloud computing. The various functions of Computer 10, described later, are performed by the operation of these components through programs and the like. Computer 10, the inspection device 40, and the observation device 50 are connected to each other in a manner that enables the transmission and reception of information.

[0032] Next, the functions of the feature output model generation system 20 and the slope estimation system 30 included in the computer 10 according to this embodiment will be described. As shown in Figure 1, the feature output model generation system 20 is configured to include a training image acquisition unit 21, a feature output model generation unit 22, and a focus position estimation model generation unit 23.

[0033] Before explaining each function of the feature output model generation system 20, we will explain the pre-trained models generated by the feature output model generation system 20. The pre-trained models generated by the feature output model generation system 20 consist of two parts: a feature output model and a focus location estimation model.

[0034] The feature output model is a model that takes image-based information as input and outputs the features of that image. The images used as input to the feature output model are partial images of images captured by the inspection device 40 and the observation device 50. The features output from the feature output model are information that indicates the features of the input image. In this embodiment, these features reflect the focal position when the image was captured. That is, the feature output model is an optical model relating to optical features. These features are, for example, vectors with a predetermined number of dimensions (for example, 1024 dimensions). These features are used as input to the focal position estimation model, as will be described later.

[0035] The feature output model is constructed, for example, by including a neural network. The neural network may be multi-layered. That is, the feature output model may be generated by deep learning. The neural network may also be a convolutional neural network (CNN).

[0036] The feature output model includes neurons in its input layer for inputting image-based information. For example, the information input to the feature output model is the pixel value of each pixel in an image. In this case, the input layer has as many neurons as there are pixels in the image, and each neuron is input with the pixel value of the corresponding pixel. As will be described later, the image related to the information input to the feature output model is an image of a predetermined size (for example, a 224x224 pixel image).

[0037] Furthermore, the information input to the feature output model does not have to be the pixel values ​​of each pixel, as long as it is image-based. For example, this information may be features for input to the feature output model obtained by performing preprocessing such as conventional image processing on the image to mitigate the effects of the imaging environment. Performing such preprocessing can improve the efficiency of machine learning and the accuracy of the generated feature output model.

[0038] A feature output model includes neurons in its output layer to output features. For example, it may have a number of neurons equal to the number of dimensions of the feature vector.

[0039] The focal position estimation model is a model that takes the features output from the feature output model as input and estimates the focal position at the time of focus corresponding to the image related to those features. For example, the focal position estimation model outputs information showing the difference between the focal position when the image related to the input features was captured and the focal position at the time of focus as the estimated focal position at the time of focus. The difference is, for example, the value obtained by subtracting the focal length corresponding to the focal position when the image related to the features was captured from the focal length corresponding to the focal position at the time of focus. In other words, in this case, the output value is the value that shows the focal position when the image was captured in a coordinate system where the focal position at the time of focus is set to 0. The focal position at the time of focus is the focal position required to capture an image in focus on the object being captured in the image related to the input features. A focused image can be captured by changing the focal position by the above difference from the focal position when a defocused image related to the input features was captured and then capturing the image.

[0040] In this case, the candidate differences mentioned above may be pre-set, and the focal position estimation model may output a value indicating the degree to which each of these candidates is reasonable. For example, if the candidate differences are +50μm, 0μm, -50μm, -100μm, ..., the focal position estimation model may output a value indicating the degree to which each candidate is reasonable. For example, the candidate with the highest value may be selected as the difference mentioned above. Alternatively, the focal position estimation model may output the difference value itself.

[0041] Alternatively, the focal position estimation model may output information indicating the focal position itself at the time of focus (for example, the focal length corresponding to the focal position at the time of focus). In this case, candidate focal positions at the time of focus may be set in advance, and the focal position estimation model may output a value indicating the degree to which each candidate is reasonable. Alternatively, the focal position estimation model may output the focal position value itself at the time of focus as described above.

[0042] The focus location estimation model is constructed, for example, by including a neural network. The neural network may be multi-layered. That is, the focus location estimation model may be generated by deep learning. The neural network may also be a convolutional neural network (CNN).

[0043] The focal position estimation model includes neurons in its input layer for inputting features. For example, the input layer includes neurons corresponding to those in the output layer of the feature output model. That is, the input layer includes the same number of neurons as the output layer of the feature output model. The focal position estimation model also includes neurons for outputting the estimated focal position at the time of focus, as described above. For example, there may be a number of neurons corresponding to the number of candidates (when outputting a value for each candidate) or a single neuron (when outputting the difference or the focal position at the time of focus itself).

[0044] Furthermore, the feature output model and the focus location estimation model may be constructed using methods other than neural networks.

[0045] The feature output model and focus location estimation model are intended for use as program modules that are part of artificial intelligence software. The feature output model and focus location estimation model are used, for example, in a computer equipped with a CPU and memory, and the computer's CPU operates according to instructions from the model stored in memory. For example, the computer's CPU operates according to the instructions, inputting information into the model, performing calculations according to the model, and outputting results from the model. Specifically, the computer's CPU operates according to the instructions, inputting information into the input layer of a neural network, performing calculations based on parameters such as trained weight coefficients in the neural network, and outputting results from the output layer of the neural network.

[0046] The learning image acquisition unit 21 is a learning image acquisition means that acquires multiple learning images to which focal position information related to the focal position at the time of imaging is associated. The learning image acquisition unit 21 may acquire as learning images images in which radiation from the object to be imaged is detected, images in which light from the object to be imaged is detected when light is irradiated onto the object to be imaged is detected, or images in which the electrical characteristics of the object to be imaged are detected when light is irradiated onto the object to be imaged is detected. The learning image acquisition unit 21 may acquire as learning images images when light of a specific wavelength is irradiated onto the object to be imaged is detected. The learning image acquisition unit 21 acquires focal position information related to the focal position at the time of focusing that corresponds to each learning image to be acquired.

[0047] For example, the training image acquisition unit 21 acquires an image captured by the inspection device 40 or the observation device 50. This image contains an object to be captured for use as a training image. The object to be captured for the training image may be, for example, the standard sample 60 shown in Figure 2. Alternatively, the object to be captured for the training image may be something else (for example, something captured by the inspection device 40 or the observation device 50). For example, as shown in Figure 4, the training image acquisition unit 21 extracts a partial image from the acquired image 70 to create a training image 71. The training image 71 is an image used for machine learning to generate the two trained models described above. Therefore, the training image acquisition unit 21 acquires a number of training images 71 sufficient to generate a properly trained model.

[0048] This embodiment estimates the focal position at the time of focus from a defocused image; therefore, the multiple training images 71 include defocused images. Furthermore, there are multiple defocused images, and the focal positions related to them may be at various positions. That is, the focal lengths corresponding to the focal positions related to them may be at various distances. In addition, the multiple training images 71 may include in-focus images. Figure 4 shows an example where three images 70 are acquired. The vertical direction in the portion of Figure 4 showing image 70 is the imaging direction (focal position direction, Z-axis direction).

[0049] The training image 71 corresponds to the image used as input to the feature output model. In this case, the feature output model receives information based on a partial image of the image captured by the inspection device 40 or observation device 50 that is the same size as the training image 71, rather than the entire image captured by the inspection device 40 or observation device 50. Therefore, the training image acquisition unit 21 extracts the training image 71, which is a partial image of a predetermined size used as input to the feature output model, from the acquired image 70. The position in image 70 from which the training image 71 is extracted is the part in which the object being captured is depicted. However, the training image 71 may include training images 71 that do not depict the object being captured. The position in image 70 from which the training image 71 is extracted may be predetermined. Alternatively, the position in image 70 from which the training image 71 is extracted may be the position in which the object being captured is estimated to be depicted after performing image recognition on the image 70.

[0050] As shown in Figure 4, the training image acquisition unit 21 may extract multiple training images 71 from a single image 70. When multiple training images 71 are extracted from image 70, the positions of the training images 71 may overlap.

[0051] In the inspection device 40 or observation device 50, imaging is performed to generate images that will serve as the basis for the training images 71. In this process, for example, in the inspection device 40 or observation device 50, the position (XY) during imaging is fixed except for the imaging direction (Z axis direction), and multiple consecutive images with different focal positions are taken. In this process, as shown in Figure 4, the focal positions are changed at regular intervals (steps) (ΔZ). Imaging for the training images 71 in the inspection device 40 or observation device 50 may be performed by methods other than those described above.

[0052] Furthermore, the learning image acquisition unit 21 may acquire images 70 from sources other than the inspection device 40 or the observation device 50.

[0053] Furthermore, as described above, the training image 71 may be an image in which radiation from the object being imaged is detected (an image used for emission / heating analysis), an image in which light from the object being imaged is detected when light is shone on the object being imaged (an image used for pattern analysis), or an image in which the electrical properties of the object being imaged are detected when light is shone on the object being imaged (an image used for laser analysis). Alternatively, the training image 71 may be an image taken when light of a specific wavelength (for example, light of a specific wavelength used for inspection) is shone on the object being imaged. These are the types of images that are normally used in the inspection device 40 or the observation device 50. However, when generating a set of feature output models and focal position estimation models, only one type of image may be used as the training image 71. In this case, the set of feature output models and focal position estimation models that are generated will correspond to the specific type of image.

[0054] The wavelength of light used in emission analysis varies depending on the drive voltage and design rules of the inspection device 40. Furthermore, in actual optical systems, wavelength-specific focus shifts occur due to adjustments and the characteristics of the optical elements. Also, the detection sensitivity may be maximized at a position different from the focal point where the pattern image is observed. Taking these factors into consideration, a wavelength-specific feature output model may be generated using images for specific wavelengths, as described above. For example, a wavelength-specific feature output model may be generated using images for the standard wavelength (1100 nm) and wavelengths other than the standard wavelength (900 nm, 1300 nm).

[0055] Furthermore, when generating a set of feature output models and focus position estimation models, only images (including partial images) captured by either the inspection device 40 or the observation device 50 may be used as training images 71. In this case, the generated set of feature output models and focus position estimation models will correspond to a specific model of the inspection device 40 or observation device 50. That is, the feature output models and focus position estimation models will reflect the characteristics of the inspection device 40 or observation device 50. By making the training images correspond to a specific type of image or a specific model of the inspection device 40 or observation device 50, a more accurate trained model can be obtained. Alternatively, the focus position estimation model may be common to multiple image types or models.

[0056] Each training image 71 is associated with focal position information relating to the focal position at the time the training image 71 was captured. Focal position information is, for example, information indicating the focal position as described above. However, the focal position information is information relating to the focal position and may be other than the above as long as it can be used to generate the trained model described above. Focal position information can be obtained, for example, as information at the time of image capture in the inspection device 40 or observation device 50. For example, the training image acquisition unit 21 receives and acquires images to which focal position information is associated from the inspection device 40 or observation device 50.

[0057] Furthermore, the training image acquisition unit 21 acquires focus position information relating to the focal position at the time of focus for each training image acquired. Focus position information is, for example, information indicating the focal position at the time of focus. However, the focus position information is information relating to the focal position at the time of focus, and may be other than the above as long as it can be used to generate the trained model described above. Focus position information can be obtained by conventional methods for measuring the focal position at the time of focus. For example, the training image acquisition unit 21 acquires focus position information by accepting input operations from the user to the computer 10 regarding the focus position information obtained by measurement.

[0058] The learning image acquisition unit 21 outputs each acquired piece of information to the feature output model generation unit 22 and the focus position estimation model generation unit 23.

[0059] The feature output model generation unit 22 is a feature output model generation means that generates a feature output model by machine learning from training images 71 acquired by the training image acquisition unit 21. The feature output model generation unit 22 compares the features of two different training images 71 according to the focal position information associated with the two training images 71, and performs machine learning based on the comparison result. The feature output model generation unit 22 may perform machine learning such that the difference in the features of the two training images 71 becomes small when the two different training images relate to the same focal position, and such that the difference in the features of the two training images 71 becomes large when the two different training images 71 relate to different focal positions.

[0060] The feature output model generation unit 22 generates a feature output model as follows: The feature output model generation unit 22 receives training images 71 and focal position information related to the training images 71 from the training image acquisition unit 21. The feature output model generation unit 22 uses two training images 71 selected from the input training images 71 as a set to perform machine learning for feature output model generation. The set used for machine learning includes both a set of training images 71 related to the same focal position and a set of training images 71 related to different focal positions. For example, a set of training images 71 related to the same focal position may be training images 71 extracted from the same image 70, as shown in Figure 4. The selection of a set of training images 71 can be done in a pre-configured manner that satisfies the above conditions. In addition, a set of training images 71 may be selected from an image 70 that has the same position (XY) at the time of imaging, other than the imaging direction (Z axis direction).

[0061] The feature output model generation unit 22 performs machine learning using information based on the selected set of training images 71 as input to the feature output model. As shown in Figure 4, when each of the training images 71 in a set is input to the feature output model 80, features are obtained as output for each training image 71. In Figure 4, the values ​​of each element of the feature vector are shown as a bar graph. In this case, the feature output model 80 that inputs one set of training images 71 is used as the training target, and the feature output model 80 that inputs the other set of training images 71 is used as the comparison target. However, these feature output models 80 are the same model in the process of being trained.

[0062] The feature output model generation unit 22 refers to the focal position information, compares the two output features, and performs machine learning based on the comparison result. If the focal positions of the two training images 71 indicated by the focal position information are the same (i.e., they are on the same plane), the feature output model generation unit 22 performs machine learning so that the difference between the features of the two training images 71 is small. If the focal positions of the two training images indicated by the focal position information are different (i.e., their Z positions are different), the feature output model generation unit 22 performs machine learning so that the difference between the features of the two training images 71 is large. Note that if the two training images 71 are extracted from the same image, the focal positions of the two training images 71 will be the same. Also, if the focal positions of the two training images 71 are close enough to be considered the same, the focal positions of the two training images 71 may be considered the same.

[0063] In other words, the feature vectors of sub-images extracted from an image at the same focal plane are designed to have a high correlation regardless of the extraction position. On the other hand, the feature vectors of sub-images extracted from an image at different focal planes are designed to have a low correlation. Through this machine learning process, the feature vectors output by the feature output model reflect features corresponding to the focal position.

[0064] Specifically, if the focal positions of the two training images 71 are the same, the feature output model generation unit 22 performs machine learning using the following loss_xy as the loss function.

number

[0065] If the focal positions of the two training images 71 are different, the feature output model generation unit 22 performs machine learning using the following loss_z as the loss function.

number

[0066] The feature output model generation unit 22 generates a feature output model by repeatedly selecting a set of training images 71 and performing machine learning. For example, the feature output model generation unit 22 generates a feature output model by repeating the above process until the generation of the feature output model converges based on pre-set conditions, or for a predetermined number of pre-set times, as in the conventional method.

[0067] The feature output model generation unit 22 may generate a feature output model using an existing pre-trained model generated by machine learning. The existing pre-trained model used is one that takes image-based information as input, similar to the feature output model in this embodiment. That is, an existing pre-trained model with the same input as the feature output model in this embodiment may be used. The existing pre-trained model is, for example, a model for image recognition, specifically ResNet, VGG, MobileNet, etc. A portion of the existing pre-trained model is used to generate the feature output model. As shown in Figure 5, the output layer of the existing pre-trained model 81 is removed, and the portion of the existing pre-trained model 81 up to the intermediate layer is used to generate the feature output model. The existing pre-trained model 81 used to generate the feature output model may include all of the intermediate layers, or it may include only a portion of the intermediate layers.

[0068] The feature output model generation unit 22 takes a portion of the existing trained model as input to create the feature output model at the start of machine learning. That is, the feature output model generation unit 22 uses a portion of the existing trained model as the initial parameters of the feature output model to perform fine tuning. Alternatively, a new output layer may be added to the output side of the portion of the trained model to create the feature output model at the start of machine learning. Furthermore, if a new output layer is added, a new hidden layer may be added between the output side of the portion of the trained model and the new output layer to create the feature output model at the start of machine learning.

[0069] The feature output model generation unit 22 may generate a feature output model without using an existing pre-trained model. For example, a model with random values ​​as initial parameters, similar to conventional machine learning, may be used as the feature output model at the start of machine learning.

[0070] Using an existing pre-trained model to generate a feature output model offers the following advantages: Training time can be significantly reduced. A highly accurate feature output model, i.e., a feature output model capable of outputting more appropriate features, can be generated even with a small number of training images (71). The aforementioned existing pre-trained model has already acquired the ability to separate low-level features. Therefore, it is only necessary to train it on high-level features using new training images (71).

[0071] The feature output model generation unit 22 outputs the generated feature output model to the focus position estimation model generation unit 23 and the tilt estimation system 30. The generated feature output model may be used for purposes other than those described in this embodiment. In that case, for example, the feature output model generation unit 22 transmits or outputs the feature output model to other devices or modules that use the feature output model. Alternatively, the feature output model generation unit 22 may store the generated feature output model in the computer 10 or other device so that it can be used by other devices or modules that use the feature output model.

[0072] The focus position estimation model generation unit 23 is a focus position estimation model generation means that generates a focus position estimation model using machine learning from the focus position information acquired by the training image acquisition unit 21. The focus position estimation model takes the features output from the feature output model generated by the feature output model generation unit 22 as described above as input and estimates the focal position at the time of focus corresponding to the image related to those features.

[0073] The focus position estimation model generation unit 23 generates a focus position estimation model as follows: The focus position estimation model generation unit 23 receives a training image 71 and focus position information related to the training image 71 from the training image acquisition unit 21. The focus position estimation model generation unit 23 receives a feature output model from the feature output model generation unit 22.

[0074] The focus position estimation model generation unit 23 inputs information based on the training image 71 into the feature output model and obtains the features of the training image 71, which are the output from the feature output model. The focus position estimation model generation unit 23 uses the obtained features as input to the focus position estimation model and performs machine learning using information based on the focus position information of the training image 71 related to the input features as the output of the focus position estimation model. The information based on the focus position information is considered to be the information corresponding to the output of the focus position estimation model. If the focus position estimation model outputs the above-mentioned values ​​for each candidate, the information based on the focus position information is, for example, the value of the candidate corresponding to the focus position information is set to 1, and the value of the candidate that does not correspond to the focus position information is set to 0 (one-hot vector) for each candidate. If the focus position estimation model outputs the above-mentioned difference or the value of the focus position itself, the information based on the focus position information is the focus position information itself or the value calculated from the focus position information. The focus position estimation model generation unit 23 generates information based on the focus position information that corresponds to the output of the focus position estimation model before performing machine learning.

[0075] The machine learning process itself, that is, updating the parameters of the focal position estimation model, can be carried out in the same way as before. The focal position estimation model generation unit 23 generates the focal position estimation model by repeating the machine learning process until the generation of the focal position estimation model converges based on pre-set conditions, or for a predetermined number of pre-set times, as in the conventional method.

[0076] The focal position estimation model generation unit 23 outputs the generated focal position estimation model to the tilt estimation system 30. The generated focal position estimation model may be used for purposes other than those described in this embodiment. In that case, for example, the focal position estimation model generation unit 23 transmits or outputs the focal position estimation model to another device or module that uses the focal position estimation model. Alternatively, the focal position estimation model generation unit 23 may store the generated focal position estimation model in the computer 10 or other device so that it can be used by other devices or modules that use the focal position estimation model. The above describes the functions of the feature output model generation system 20.

[0077] Next, the functions of the tilt estimation system 30 according to this embodiment will be described. As shown in Figure 1, the tilt estimation system 30 is configured to include an estimation target image acquisition unit 31, a focal position estimation unit 32, a tilt estimation unit 33, and a control unit 34.

[0078] The tilt estimation system 30 estimates the tilt of an object captured in an image taken by the inspection device 40 or observation device 50. To this end, the tilt estimation system 30 estimates the focal position at the time of focus when the object is imaged by the inspection device 40 or observation device 50. This estimation is performed by first imaging the object (imaging for tilt estimation) in the inspection device 40 or observation device 50. In this imaging, the focal position does not necessarily have to be the one at which the object was in focus, i.e., the focal position at the time of focus. Therefore, the image obtained in this imaging may be a defocused image. For example, this image may be a defocused image as shown in Figure 6(a). Figure 6 is an image of a semiconductor device. The defocused image shown in Figure 6(a) is an image where the focal position at the time of imaging is +5 nm from the focal position at the time of focus.

[0079] The tilt estimation system 30 estimates the focal point positions at multiple locations in the image when the image is in focus. From the estimated focal point positions at multiple locations in the image, the tilt estimation system 30 estimates the tilt of the object being imaged. Furthermore, by using the estimated focal point positions when the image is in focus, imaging is performed in the inspection device 40 or observation device 50, thereby obtaining an image in focus on the object being imaged, i.e., a focused image. For example, a focused image like the one shown in Figure 6(b) can be obtained. The focused image shown in Figure 6(b) corresponds to the defocused image in Figure 6(a).

[0080] The estimation target image acquisition unit 31 is an estimation target image acquisition means that acquires an image of the object to be captured and obtains multiple partial images, which are estimation target images, from that image. The estimation target images are images used in the tilt estimation system 30 to estimate the focal position when in focus. Furthermore, the estimation target images are images used as input to the feature output model. In other words, the estimation target images correspond to the training images 71 described above.

[0081] The estimation target image acquisition unit 31 acquires images captured by the inspection device 40 or observation device 50. The imaging performed by the inspection device 40 or observation device 50 is for the purpose of tilt estimation. For example, the estimation target image acquisition unit 31 extracts multiple partial images from the acquired image to form the estimation target image.

[0082] The estimation target image acquisition unit 31 extracts the estimation target image, which is a partial image of a predetermined size, from the acquired image to be used as input for the feature output model. The position in the image from which the estimation target image is extracted is the part in which the object being photographed is depicted. The position in the image from which the estimation target image is extracted may be predetermined. For example, as shown in Figure 7, the estimation target image acquisition unit 31 may use multiple partial images obtained by dividing the image 90 into predetermined sizes as the estimation target image 91. In the example shown in Figure 7, nine 3x3 estimation target images 91 are extracted and acquired from the image. When the estimation target image 91 is extracted as described above, if the object being photographed is depicted in the entire image 90, the estimation target image 91 should be extracted from the entire image 90; if the object being photographed is depicted in a part of the image 90, the estimation target image 91 should be extracted from that part. Alternatively, the position from which the estimation target image is extracted may be the position in which the object being photographed is estimated to be depicted by performing image recognition on the image. The tilt estimation system 30 is configured such that the position from which the target image for estimation is extracted in the image is also used in the functions described later.

[0083] Furthermore, the type of image to be estimated is the same as the type of training image described above. For example, the image to be estimated may be an image in which radiation from the object being imaged is detected, an image in which light from the object being imaged is detected when light is shone on the object being imaged, or an image in which the electrical characteristics of the object being imaged are detected when light is shone on the object being imaged. Alternatively, the image to be estimated may be an image taken when light of a specific wavelength (for example, light of a specific wavelength used for inspection) is shone on the object being imaged.

[0084] The target image acquisition unit 31 outputs the acquired multiple target images to the focus position estimation unit 32.

[0085] The focus position estimation unit 32 is a focus position estimation means that uses a feature output model to output feature quantities for each of the multiple target images acquired by the target image acquisition unit 31, and estimates the focal position at the time of focus corresponding to each of the multiple target images from the output feature quantities. The focus position estimation unit 32 may also use a focus position estimation model to estimate the focal position at the time of focus corresponding to the target image from the feature quantities output from the feature output model.

[0086] The focal position estimation unit 32 inputs and stores the feature output model and focal position estimation model generated by the feature output model generation system 20, and uses them for estimation. The focal position estimation unit 32 receives multiple images to be estimated from the image acquisition unit 31.

[0087] The focal position estimation unit 32 inputs information based on the target image to the feature output model and obtains the feature quantities of the target image, which are the output from the feature output model. The focal position estimation unit 32 inputs the obtained feature quantities to the focal position estimation model and obtains information indicating the focal position at the time of focus corresponding to the target image as the focal position estimation result, which are the output from the focal position estimation model. For each of the multiple target images to be estimated, the focal position estimation unit 32 obtains information indicating the focal position at the time of focus corresponding to the target image to be estimated. The focal position estimation unit 32 outputs the information indicating the focal position at the time of focus corresponding to the multiple obtained target images to the tilt estimation unit 33.

[0088] The tilt estimation unit 33 is a tilt estimation means that estimates the tilt of an object captured in an image from the focal position at the time of focus corresponding to each of the multiple estimated target images estimated by the focal position estimation unit 32. The tilt estimation unit 33 estimates, for example, the tilt of an object captured with respect to the imaging direction in the inspection device 40 or observation device 50. The tilt estimation unit 33 estimates the tilt of the object captured as follows.

[0089] The tilt estimation unit 33 receives information from the focal position estimation unit 32 indicating the focal position at the time of focus corresponding to each of the multiple estimation target images. In this embodiment, the position in the imaging direction corresponding to the focal position at the time of focus corresponding to the estimation target image is assumed to be the position of the object being imaged. Since the estimation target images are extracted from multiple positions in the image, the position of the object being imaged in the imaging direction at each position of the estimation target image in the image can be obtained. The tilt of the object being imaged is estimated from these multiple positions of the object being imaged.

[0090] For example, the tilt estimation unit 33 estimates the tilt of the object being imaged for each coordinate axis on a plane perpendicular to the imaging direction. The tilt estimation unit 33 estimates the tilt of the object being imaged for each of the two coordinate axes, the X axis and the Y axis, which are parallel to each side of the target image 91 of the image 90 as shown in Figure 7. For example, the tilt estimation unit 33 estimates the angle θ1 of the tilt of the object being imaged in the X axis with respect to a plane perpendicular to the imaging direction (Z axis direction) (hatched plane in the figure), as shown in Figure 8(a), and the angle θ2 of the tilt of the object being imaged in the Y axis with respect to a plane perpendicular to the imaging direction (Z axis direction) (hatched plane in the figure), as shown in Figure 8(b).

[0091] Figure 8 shows information indicating the focal position at the time of focus estimated from each estimated target image 91 (specifically, the difference between the focal position when the image was captured and the focal position at the time of focus) (values ​​such as +2, 0, -2, +3, 0, and -3). Figure 8(a) shows an example where the object being imaged is tilted in the X-axis direction, specifically when the right side is lower. Figure 8(b) shows an example where the object being imaged is tilted in the Y-axis direction, specifically when the front side is lower.

[0092] The inclination estimation unit 33 calculates angles θ1 and θ2 using the following formulas.

number

[0093] x and y are determined based on the position of the estimated target image as shown in FIG. 7. For example, x is the X-axis direction distance between two preset estimated target images P a , P b that are separated from each other and used for estimating the tilt (for example, the center coordinates P center ) of the estimated target image. That is, x = |P b - P a | (only the X-axis component). y is the Y-axis direction distance between two preset estimated target images P a , P c that are separated from each other and used for estimating the tilt (for example, the center coordinates P center ) of the estimated target image. That is, y = |P c - P a | (only the X-axis component). When the position of the estimated target image in the image is preset, the above x and y are constant values and may be stored in the tilt estimation unit 33 in advance.

[0094] z1 and z2 are calculated based on the focus positions at the time of focusing corresponding to each of the estimated target images as shown in FIG. 8. For example, z1 is the difference between the focus positions Z a , Z b at the time of focusing corresponding to the above two estimated target images P a , Z b . That is, z1 = |Z b - Z a |. z2 is the difference between the focus positions Z a , Z c at the time of focusing corresponding to the above two estimated target images P a , Z c . That is, z2 = |Z c - Z a| is.

[0095] The target image used to estimate the tilt of the object being imaged may be other than those described above. For example, the target image used to estimate the tilt of the object being imaged may be as follows, and the tilt may be estimated in this way: The image containing the object being imaged is divided into multiple regions, each containing multiple target images. For example, the image is divided into 3x3 rectangular regions. Region (P a Multiple estimated target images (P) for each ) a1 ,P a2 ,P a3 ,…) The corresponding focal position when in focus (Z a1 ,Z a2 ,Z a3 Remove outliers from ,…). The median of the focal positions at the time of focus after removing outliers (Z median ) is taken. The median (Z median ) and the median (Z median The estimated target image (P) corresponding to ) amedian ) to the region (P a The focal position at the time of focus and the tilt are estimated as representative of the target image.

[0096] The tilt estimation unit 33 may estimate the tilt of the object being imaged in the image using methods other than those described above, as long as it estimates the tilt of the object being imaged from the focal position at the time of focus corresponding to each of the multiple estimated target images estimated by the focal position estimation unit 32. The tilt estimation unit 33 may also estimate angles other than those described above, θ1 and θ2, as the tilt of the object being imaged. The tilt estimation unit 33 outputs information indicating the estimated tilt of the object being imaged to the control unit 34. The tilt estimation unit 33 may also output information indicating the focal position at the time of focus corresponding to the estimated target image used to estimate the tilt of the object being imaged to the control unit 34.

[0097] The control unit 34 is a control means that controls the tilt of the object being imaged during imaging based on the tilt of the object being imaged estimated by the tilt estimation unit 33. The control unit 34 may also control the focal position of the object being imaged during imaging based on the focal position estimated by the focal position estimation unit 32.

[0098] The control unit 34 receives information indicating the tilt of the object to be imaged from the tilt estimation unit 33. The control unit 34 may also receive information indicating the focal position at the time of focusing corresponding to the estimated target image from the tilt estimation unit 33. The control unit 34 controls the inspection device 40 or observation device 50 so that the tilt of the object to be imaged, as indicated by the input information, is eliminated during imaging. Specifically, the control unit 34 controls the inspection device 40 or observation device 50 to tilt the object to be imaged in the opposite direction to the tilt of the object to be imaged, as indicated by the input information. The controlled inspection device 40 or observation device 50 adjusts the tilt of the object to be imaged during imaging, for example, by operating the mounting unit. In this way, the control unit 34 controls the tilt correction in the inspection device 40 or observation device 50. As a result, the image captured by the inspection device 40 or observation device 50 will be an image of the object to be imaged with the tilt in an appropriate state.

[0099] Furthermore, the control unit 34 controls the inspection device 40 or observation device 50 so that the focal position during imaging becomes the focal position at the time of focus indicated by the input information. For example, the control unit 34 controls the focal position to the time of focus corresponding to a pre-set estimated target image from among a plurality of estimated target images. The inspection device 40 or observation device 50, under this control, adjusts the focal position during imaging by, for example, operating the stage 46. As a result, the image captured by the inspection device 40 or observation device 50 becomes a focused image. In this way, the control unit 34 controls the autofocus of the inspection device 40 or observation device 50. The above describes the configuration of the tilt estimation system 30.

[0100] Next, the processes performed by the computer 10 according to this embodiment (the methods of operation performed by the computer 10) will be explained using the flowcharts in Figures 9 and 10. First, the process performed when generating the feature output model and the focus position estimation model, that is, the feature output model generation method which is performed by the feature output model generation system 20 according to this embodiment, will be explained using the flowchart in Figure 9.

[0101] In this process, first, the training image acquisition unit 21 acquires multiple training images to which focal position information relating to the focal position at the time of imaging is associated (S01, training image acquisition step). The training image acquisition unit 21 also acquires focal position information relating to the focal position at the time of focus corresponding to each training image. Next, the feature output model generation unit 22 generates a feature output model from the training images using machine learning (S02, feature output model generation step). At this time, the features of two different training images 71 are compared according to the focal position information associated with each of the two training images, and machine learning is performed based on the comparison result. Subsequently, the focal position estimation model generation unit 23 generates a focal position estimation model from the focal position information using machine learning (S03, focal position estimation model generation step).

[0102] The generated feature output model and focal position estimation model are output from the feature output model generation system 20 to the slope estimation system 30. The slope estimation system 30 stores the feature output model and focal position estimation model and uses them in the following processes. The above describes the feature output model generation method, which is the process executed by the feature output model generation system 20 according to this embodiment.

[0103] Next, using the flowchart in Figure 10, we will explain the process performed when estimating the focal position at the time of focus corresponding to the target image, that is, the tilt estimation method which is performed by the tilt estimation system 30 according to this embodiment.

[0104] In this process, first, the estimation target image acquisition unit 31 acquires an image of the object to be imaged, and from this image, multiple partial images, which are estimation target images, are acquired (S11, estimation target image acquisition step). The estimation target images are based on images obtained by imaging for tilt estimation by the inspection device 40 or observation device 50. Next, the focus position estimation unit 32 uses a feature output model to output feature quantities from each of the multiple estimation target images. Subsequently, the focus position estimation unit 32 uses a focus position estimation model to estimate the focal position at the time of focus corresponding to each of the multiple estimation target images from the feature quantities of each of the multiple estimation target images (S12, focus position estimation step).

[0105] Next, the tilt estimation unit 33 estimates the tilt of the object being imaged in the image from the focal position at the time of focus corresponding to each of the multiple estimation target images (S13, tilt estimation step). Subsequently, the control unit 34 controls the tilt of the object being imaged when the inspection device 40 or observation device 50 takes a new image based on the estimated tilt of the object being imaged (S14, control step). As a result, the image taken by the inspection device 40 or observation device 50 will be an image of the object being imaged with the tilt appropriate. At this time, the control unit 34 may also control the focal position when the inspection device 40 or observation device 50 takes an image of the object being imaged based on the estimated focal position. As a result, the image taken by the inspection device 40 or observation device 50 will be a focused image. The above is the tilt estimation method, which is the process performed by the tilt estimation system 30 according to this embodiment.

[0106] In this embodiment, a feature output model that outputs image features is generated by machine learning. In this process, the features of two different training images are compared according to the focal position information associated with those two training images, and machine learning is performed based on the comparison result. With this generation method, for example, a feature output model can be generated in a short time by using an existing trained model that takes image-based information as input. In other words, according to this embodiment, a feature output model, which is a trained model used for image-based estimation such as focal position estimation, can be generated in a short amount of time through training.

[0107] However, as mentioned above, it is not always necessary to use an existing pre-trained model to generate a feature output model. Even in that case, it is possible to generate a feature output model that can output appropriate features according to the focal point.

[0108] Furthermore, as described above, machine learning for generating a feature output model may be performed such that, when two different training images relate to the same focal position, the difference in features between the two training images is small, and when two different training images relate to different focal positions, the difference in features between the two training images is large. This configuration ensures that a feature output model can be generated reliably and appropriately. However, machine learning does not necessarily have to be performed in the manner described above; it can be performed based on the comparison results of the features of the two training images.

[0109] Furthermore, as mentioned above, the training images and target images may be images in which radiation from the object being photographed is detected, images in which light from the object being photographed is detected when light is shone on the object being photographed, or images in which the electrical properties of the object being photographed are detected when light is shone on the object being photographed. Moreover, the training images and target images may be images in which light of a specific wavelength is shone on the object being photographed. With these configurations, it is possible to generate an appropriate feature output model according to the type of image used, and to utilize the feature output model. However, the training images and target images are not limited to those described above, and any image corresponding to the focal position is acceptable.

[0110] Furthermore, as in this embodiment, the feature output model generation system 20 may further include a focus position estimation model generation unit 23 that generates a focus position estimation model. With this configuration, it is possible to generate a focus position estimation model that estimates the focal position when the image is in focus from the image. That is, with this configuration, together with the feature output model, it is possible to generate a trained model used for estimating the focal position based on the image in a short amount of time.

[0111] However, the feature output model generation system 20 does not necessarily have to include a focal position estimation model generation unit 23. That is, the feature output model generation system 20 may be configured to generate only a feature output model. Furthermore, the generated feature output model may be used for purposes other than estimating the focal position at the time of focus.

[0112] In the tilt estimation system 30 according to the present invention, the focal position at the time of focus corresponding to each of the multiple target images is estimated from each of the multiple target images, which are partial images of the image, and the tilt of the object being photographed is estimated. Therefore, if an image of the object being photographed is obtained, the tilt of the object being photographed can be estimated in a short time. The configuration (e.g., lens turret) and space required to install a special optical system within the device, which was necessary when estimating the tilt of the object being photographed using a special optical system, is not required. Furthermore, the cost of preparing a special optical system is not required. In addition, processes such as turret switching, laser scanning, and alignment are not required. Compared to that case, the time required to estimate the tilt of the object being photographed can be significantly reduced.

[0113] Furthermore, the feature output model used for estimating the focal position can output features appropriate for estimating the focal position, and by using these, the tilt of the object being imaged can be appropriately estimated. Therefore, the tilt estimation system according to the present invention can estimate the tilt of the object being imaged with a simple configuration and in a short amount of time.

[0114] In the tilt estimation system 30 according to this embodiment, the feature output model described above is used to estimate the focal position at the time of focus. Therefore, according to this embodiment, the focal position at the time of focus based on the image can be estimated with a short preparation time. Furthermore, according to the tilt estimation system 30 according to this embodiment, the focal position at the time of focus can be estimated with a single image taken for tilt estimation. Therefore, compared to the case where the focal position at the time of focus is searched by taking multiple images while changing the focal position, the focal position at the time of focus can be estimated more quickly. As a result, the tilt of the object being imaged can be estimated quickly.

[0115] Furthermore, the aforementioned focal position estimation model may be used to estimate the focal position at the time of focus. With this configuration, the focal position at the time of focus can be estimated reliably and appropriately. As a result, the tilt of the object being imaged can be estimated reliably and appropriately. However, it is not necessary to use the aforementioned focal position estimation model to estimate the focal position at the time of focus; estimation can be performed using the features output from the feature output model.

[0116] Furthermore, as in this embodiment, the tilt estimation system 30 may further include a control unit 34 that controls the tilt of the object being imaged in the inspection device 40 or observation device 50 during imaging, based on the estimated tilt of the object being imaged. With this configuration, the inspection device 40 or observation device 50 can image the object being imaged at an appropriate tilt. However, the tilt estimation system 30 does not have to include the control unit 34. That is, the tilt estimation system 30 only needs to estimate the tilt of the object being imaged.

[0117] Furthermore, the tilt estimation system 30 according to this embodiment can be configured as a series of systems including the inspection device 40 or observation device 50 described above. That is, the semiconductor inspection system according to this embodiment may be a system including the tilt estimation system 30 and the inspection device 40. Also, the biological observation system according to this embodiment may be a system including the tilt estimation system 30 and the observation device 50.

[0118] In this embodiment, the computer 10 includes a feature output model generation system 20 and a slope estimation system 30, but the feature output model generation system 20 and the slope estimation system 30 may be implemented independently.

[0119] Next, a feature output model generation program and a slope estimation program for executing the processing performed by the series of feature output model generation systems 20 and slope estimation systems 30 described above will be explained. As shown in Figure 11, the feature output model generation program 200 is stored in a program storage area 211 formed on a computer-readable recording medium 210 that is inserted into and accessed by a computer, or is provided by a computer. The recording medium 210 may be a non-temporary recording medium.

[0120] The feature output model generation program 200 comprises a training image acquisition module 201, a feature output model generation module 202, and a focus position estimation model generation module 203. The functions realized by executing the training image acquisition module 201, the feature output model generation module 202, and the focus position estimation model generation module 203 are the same as the functions of the training image acquisition unit 21, the feature output model generation unit 22, and the focus position estimation model generation unit 23 of the feature output model generation system 20 described above.

[0121] As shown in Figure 12, the slope estimation program 300 is stored in a program storage area 311 formed on a computer-readable recording medium 310 that is inserted into and accessed by a computer, or is provided by a computer. The recording medium 310 may be a non-temporary recording medium. The recording medium 310 may also be the same as the recording medium 210.

[0122] The tilt estimation program 300 comprises an image acquisition module 301, a focal position estimation module 302, a tilt estimation module 303, and a control module 304. The functions realized by executing the focal position estimation module 302, the tilt estimation module 303, and the control module 304 are the same as the functions of the image acquisition unit 31, the focal position estimation unit 32, the tilt estimation unit 33, and the control unit 34 of the tilt estimation system 30 described above.

[0123] Furthermore, the feature output model generation program 200 and the slope estimation program 300 may be configured such that part or all of them are transmitted via a transmission medium such as a communication line, received by other devices, and recorded (including installation). Also, each module of the feature output model generation program 200 and the slope estimation program 300 may be installed on any of several computers, not just one. In that case, the series of processes described above will be performed by a computer system consisting of these multiple computers.

[0124] The slope estimation system, slope estimation method, slope estimation program, semiconductor inspection system, and biological observation system of this disclosure have the following configurations. [1] A tilt estimation system for estimating the tilt of an object captured in an image, An estimated target image acquisition means that acquires an image of the object to be imaged and obtains multiple estimated target images, which are partial images, from the said image, A feature output model that takes image-based information as input and outputs the feature quantities of the said image, and a focus position estimation means that outputs the feature quantities of each of the multiple target images acquired by the estimation target image acquisition means, and estimates the focal position at the time of focus corresponding to each of the multiple target images from the output feature quantities, The system includes a tilt estimation means that estimates the tilt of an object captured in an image from the focal position at the time of focus corresponding to each of a plurality of estimated target images estimated by the focal position estimation means, The aforementioned feature output model is generated by machine learning from multiple training images to which focal position information relating to the focal position at the time of imaging is associated. The feature quantities of two training images are compared according to the focal position information associated with two different training images, and machine learning is performed based on the comparison result. [2] The focal position estimation means takes the feature quantities output from the feature quantity output model as input and estimates the focal position at the time of focus corresponding to each of the multiple target images using a focal position estimation model that estimates the focal position at the time of focus corresponding to the image relating to the feature quantities, The aforementioned focal position estimation model is a tilt estimation system described in [1] that is generated by machine learning from focal position information relating to the focal position at the time of focus corresponding to each of the training images. [3] The tilt estimation system according to [1] or [2], further comprising control means for controlling the tilt of the object to be imaged during imaging based on the tilt of the object to be imaged estimated by the tilt estimation means. [4] A slope estimation system described in any of [1]~[3], A mounting section on which a semiconductor device is placed as the object to be imaged in the tilt estimation system, An inspection unit for inspecting the aforementioned semiconductor device, A semiconductor inspection system equipped with the following features. [5] A slope estimation system described in any of [1]~[3], The tilt estimation system includes a mounting section on which a biological sample is placed as the object to be imaged, An observation unit for observing the aforementioned biological sample, A biological observation system equipped with the following features. [6] A tilt estimation method for estimating the tilt of an object captured in an image, An estimated target image acquisition step involves acquiring an image of the object to be imaged and obtaining multiple estimated target images, which are partial images, from that image. A feature output model that takes image-based information as input and outputs the feature quantities of the said image is used to output the feature quantities of each of the multiple target images acquired in the estimation target image acquisition step, and a focus position estimation step is performed to estimate the focal position at the time of focus corresponding to each of the multiple target images from the output feature quantities. The method includes a tilt estimation step which estimates the tilt of an object captured in an image from the focal position at the time of focus corresponding to each of the multiple estimated target images estimated in the focal position estimation step, The aforementioned feature output model is generated by machine learning from multiple training images to which focal position information relating to the focal position at the time of imaging is associated. The slope estimation method compares the features of two training images according to the focal position information associated with two different training images, and machine learning is performed based on the comparison result. [7] In the focal position estimation step, the focal position at the time of focus corresponding to each of the multiple target images is estimated using a focal position estimation model that takes the feature quantities output from the feature quantity output model as input and estimates the focal position at the time of focus corresponding to the image relating to the feature quantities, The aforementioned focal position estimation model is generated by machine learning from the focal position information relating to the focal position at the time of focus corresponding to each of the training images [6], using the tilt estimation method described above. [8] The tilt estimation method according to [6] or [7], further comprising a control step of controlling the tilt of the object being imaged at the time of imaging based on the tilt of the object being imaged estimated in the tilt estimation step. [9] A tilt estimation program that causes a computer to operate as a tilt estimation system for estimating the tilt of an object captured in an image, The computer in question, An estimated target image acquisition means that acquires an image of the object to be imaged and obtains multiple estimated target images, which are partial images, from the said image, A feature output model that takes image-based information as input and outputs the feature quantities of the said image, and a focus position estimation means that outputs the feature quantities of each of the multiple target images acquired by the estimation target image acquisition means, and estimates the focal position at the time of focus corresponding to each of the multiple target images from the output feature quantities, The tilt estimation means functions to estimate the tilt of the imaged object captured in the image from the focal position at the time of focus corresponding to each of the multiple estimated target images estimated by the focal position estimation means. The aforementioned feature output model is a slope estimation program that is generated by machine learning from multiple training images to which focal position information related to the focal position at the time of imaging is associated, and in which the features of two different training images are compared according to the focal position information associated with those two training images, and machine learning is performed based on the comparison result.

[10] The focal position estimation means takes the feature quantities output from the feature quantity output model as input and estimates the focal position at the time of focus corresponding to each of the multiple target images using a focal position estimation model that estimates the focal position at the time of focus corresponding to the image relating to the feature quantities, The focal position estimation model is a tilt estimation program described in [9] that is generated by machine learning from focal position information relating to the focal position at the time of focus corresponding to each of the training images.

[11] The computer is, The tilt estimation program according to [9] or

[10] also functions as a control means for controlling the tilt of the object being imaged during imaging, based on the tilt of the object being imaged estimated by the tilt estimation means. [Explanation of symbols]

[0125] 10...Computer, 20...Feature output model generation system, 21...Training image acquisition unit, 22...Feature output model generation unit, 23...Focus position estimation model generation unit, 30...Slope estimation system, 31...Estimation target image acquisition unit, 32...Focus position estimation unit, 33...Slope estimation unit, 34...Control unit, 40...Inspection device, 41...Camera, 42...Mounting unit, 43...Light source, 44...Optical system, 45...Objective lens, 46...Stage, 50...Observation device, 200...Feature output model generation program, 201...Training image acquisition module, 202...Feature output model generation module, 203...Focus position estimation model generation module, 210...Recording medium, 211...Program storage area, 300...Slope estimation program, 301...Estimation target image acquisition module, 302...Focus position estimation module, 303...Slope estimation module, 304...Control module, 310...Recording medium, 311...Program storage area.

Claims

1. A tilt estimation system for estimating a tilt of an object captured in an image, comprising: an estimation target image acquisition means for acquiring an image including a captured object and acquiring estimation target images, which are a plurality of partial images, from the image; a focus position estimation means for acquiring a feature amount of each of the plurality of estimation target images based on each of the plurality of estimation target images acquired by the estimation target image acquisition means, and estimating a focus position at the time of focusing corresponding to each of the plurality of estimation target images from the acquired feature amount; an inclination estimation means for estimating an inclination of an image capture target object captured in the images from a focus position at the time of focusing corresponding to each of a plurality of estimation target images estimated by the focus position estimation means; A tilt estimation system comprising:

2. The tilt estimation system according to claim 1 , wherein the feature amount reflects a focal position when an image related to the feature amount was captured.

3. The tilt estimation system according to claim 1 , wherein the feature amount is a vector having a preset number of dimensions.

4. The tilt estimation system according to any one of claims 1 to 3, wherein the feature is acquired by inputting information based on an image relating to the feature to a feature output model generated by machine learning.

5. the focus position estimation means estimates focus positions at the time of focusing corresponding to each of the plurality of estimation target images from the feature amounts of each of the plurality of estimation target images using a focus position estimation model that estimates a focus position at the time of focusing corresponding to an image related to a feature amount; The tilt estimation system according to any one of claims 1 to 4, wherein the focus position estimation model is generated by machine learning from a plurality of learning images and focus position information relating to the focus positions at the time of focusing corresponding to each of the learning images.

6. The tilt estimation system according to any one of claims 1 to 5, further comprising a control means for controlling a tilt of the object to be imaged at the time of imaging, based on the tilt of the object to be imaged estimated by the tilt estimation means.

7. A tilt estimation system according to any one of claims 1 to 6, a mounting portion on which a semiconductor device is mounted as an image capture target for the tilt estimation system; an inspection unit that inspects the semiconductor device; A semiconductor inspection system comprising:

8. A tilt estimation system according to any one of claims 1 to 6, a placement unit on which a biological sample is placed as an imaging target for the tilt estimation system; an observation unit for observing the biological sample; A biological observation system comprising:

9. A tilt estimation method for estimating a tilt of an object captured in an image, comprising the steps of: an estimation target image acquisition step of acquiring an image including a captured object and acquiring estimation target images, which are a plurality of partial images, from the image; a focus position estimating step of acquiring feature amounts of each of the plurality of estimation target images based on each of the plurality of estimation target images acquired in the estimation target image acquiring step, and estimating focus positions at the time of focusing corresponding to each of the plurality of estimation target images from the acquired feature amounts; an inclination estimation step of estimating an inclination of an image capture target object captured in the images from focus positions at the time of focusing corresponding to each of the plurality of estimation target images estimated in the focus position estimation step; The method of estimating the slope includes:

10. A tilt estimation program that causes a computer to operate as a tilt estimation system that estimates the tilt of an object captured in an image, comprising: The computer, an estimation target image acquisition means for acquiring an image including a captured object and acquiring estimation target images, which are a plurality of partial images, from the image; a focus position estimation means for acquiring a feature amount of each of the plurality of estimation target images based on each of the plurality of estimation target images acquired by the estimation target image acquisition means, and estimating a focus position at the time of focusing corresponding to each of the plurality of estimation target images from the acquired feature amount; an inclination estimation means for estimating an inclination of an image capture target object captured in the images from a focus position at the time of focusing corresponding to each of a plurality of estimation target images estimated by the focus position estimation means; A tilt estimation program that acts as a.

Images