Microscope image acquisition control device, microscope image acquisition control program

Abstract

To perform efficient and highly accurate observation suitable for each magnification with the field of view of an image taken at a relatively low specific magnification and an image taken at a high specific magnification.SOLUTION: When a SEM image with the same field of view as a low-magnification SEM image is obtained, and because it is not possible to take a SEM image (relatively high magnification image) with the same field of view as the SEM image in one shot, the field of view of the SEM image, which is a low-magnification image, is divided and photographed, and then the fields of view of the SEM image are combined, but a deviation may occur at the boundary of the connection. Therefore, the concept of "guide image" is focused on, and image acquisition control is established to obtain a SEM image with a visual field that matches the low-magnification SEM image (e.g. brightness information matches) and no deviation. The guide image is obtained and the acquisition range of the high magnification image is set. The process of photographing all the divided acquisition areas Xn and combining the high-magnification images is repeated until the photographing of all the divided acquisition areas Xn is completed.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a microscope image acquisition control device and a microscope image acquisition control program for performing an analysis using a microscope image measured at a relatively low magnification and a microscope image continued at a relatively high magnification of a sample.

Background Art

[0002] SEM (Scanning Electron Microscorpe) acquires a relatively high magnification photographed image in order to observe the shape of a sample.

[0003] By the way, when acquiring a high magnification photographed image within a visual field range equivalent to that of a microscope image measured at a low magnification in the same sample, it is necessary to divide, measure, and combine the visual field range.

[0004] At this time, displacement may occur at the boundary portion of the combination. Also, there may be a case where the visual field ranges of the high magnification photographed image generated by combining and the low magnification photographed image do not match. This may be due to the case depending on the accuracy of the stage for positioning the SEM sample or factors such as image distortion.

[0005] Patent Document 1 discloses an image display device including a low magnification image input means for acquiring an image obtained by imaging an observation target at a predetermined magnification, a high magnification image input means for acquiring an image obtained by imaging the observation target at a magnification higher than that of the low magnification image input means, a high magnification image position and orientation processing means for obtaining the relative magnification, position, and orientation of both images, and an image overlapping means for overlapping an index indicating the image range obtained by the high magnification image input means on the image obtained by the low magnification image input means using the processing result of the high magnification image position and orientation processing means, and an image output means for displaying the output image of the image overlapping means.

[0006] Furthermore, Patent Document 2 describes a system comprising: a low-magnification imaging means for imaging the entire specimen at a low magnification, which is a first magnification; a region division information storage means for storing the positional information of each sub-region when the entire specimen is divided into a plurality of sub-regions with partially overlapping regions as region division information; a high-magnification imaging means for sequentially imaging regions substantially identical to each divided region at a second magnification, which is a higher magnification than the first magnification, according to the region division information; a positional displacement detection means for detecting a positional displacement of the captured high-magnification image based on the low-magnification specimen image captured by the low-magnification imaging means; a positional displacement correction means for correcting the position of each high-magnification image based on the detected positional displacement; and an image stitching means for sequentially stitching (combining) each high-magnification image to create a high-magnification image of the entire specimen. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2014-224929 [Patent Document 2] Japanese Patent Publication No. 2004-343222 [Overview of the project] [Problems that the invention aims to solve]

[0008] However, Patent Document 1 describes a system that simultaneously acquires low-magnification and high-magnification images and displays in real time which part of the low-magnification image corresponds to the high-magnification image. It is not intended to acquire high-magnification images over a wide area, and there is no process to combine the high-magnification images, so it is not possible to acquire high-magnification images of the field of view of the low-magnification image.

[0009] Furthermore, since SEM requires scanning pixels to acquire a single image, it is not possible to acquire low-magnification and high-magnification images simultaneously.

[0010] On the other hand, Patent Document 2 attempts to obtain the image by stitching together a low-magnification image and a high-magnification image covering the same area, but it has the following problems: (1) there is a possibility that the shooting and stitching will fail, (2) the final high-resolution image may not match the field of view of the low-magnification image, and (3) it may not be possible to properly stitch together the images when there is a large difference between the low-magnification and high-magnification images.

[0011] In (1), the shooting location is defined as "the positional information of each sub-region when the shooting location is divided into multiple sub-regions that have some overlapping areas." However, there may be places where it is difficult to focus during shooting, or where matching is difficult due to the image pattern. As a result, some images may be blurred and not stitched together properly, or there may be cases where the images do not have a distinctive image pattern and cannot be stitched together.

[0012] (2) describes a function that corrects the shooting location using the difference between low-magnification and high-magnification images. However, this involves actually detecting the positional and angular shifts in advance and correcting the initial shooting location accordingly. If the estimation of the shift amount is incorrect, there is a possibility that the shooting range may be missed.

[0013] In (3), the magnification of the intermediate image is any magnification between the low magnification and the high magnification. However, if the magnification between these two images is large, the low-magnification image may become very blurry, or the high-magnification image may be drastically reduced, resulting in loss of information and potentially causing the images to not stitch together properly.

[0014] For example, if you photograph an overview of a sample at 100x magnification (an example of low magnification) and a part of the sample at 10,000x magnification (an example of high magnification), the width of the field of view of the high-magnification image will be 1 / 100th of that of the low-magnification image. In such cases, it becomes difficult to perform accurate positioning simply by scaling the image.

[0015] The objective of this invention is to provide a microscope image acquisition control device and a microscope image acquisition control program that can efficiently and accurately perform observations suitable for each magnification, while matching the field of view of images taken at a relatively low magnification and images taken at a specific high magnification. [Means for solving the problem]

[0016] The first embodiment of the microscope image acquisition control device includes: a setting unit that sets a high-magnification image range to be photographed at a higher magnification than the specific magnification suitable for other image analyses, within a microscope image taken of a sample at a specific magnification suitable for one image analysis; a division unit that divides the high-magnification image range into a plurality of shooting ranges; an imaging unit that photographs each of the plurality of shooting ranges; and a merging unit that combines the plurality of high-magnification microscope images taken by the imaging unit. The merging unit generates a guide image by photographing the high-magnification image range at a magnification that can be photographed, and arranges each of the plurality of high-magnification microscope images taken by the imaging unit so that the image patterns on the guide image match within a predetermined tolerance range. Furthermore, based on multiple data sets that associate the positional accuracy error corresponding to the difference between the target position and the actual position of the stage used to position the sample, machine learning is used to predict the error between the imaging range of the high-magnification microscope image and the imaging range of the guide image, and the imaging range of the high-magnification microscope image is corrected. It is characterized by doing so. Furthermore, the microscope image acquisition control device includes a setting unit that sets a high-magnification image range to be captured at a higher magnification than the specific magnification suitable for other image analyses, within a microscope image captured at a specific magnification suitable for one image analysis of a sample; a division unit that divides the high-magnification image range into a plurality of shooting ranges; an shooting unit that captures a high-magnification microscope image for each of the plurality of shooting ranges; and a combining unit that combines the plurality of high-magnification microscope images captured by the shooting unit. The combining unit generates a guide image by shooting the high-magnification image range at a magnification that can be captured, and arranges each of the plurality of high-magnification microscope images captured by the shooting unit so that the image patterns on the guide image match within a predetermined tolerance range. If there is a magnification difference of a predetermined amount or more between the magnification of the guide image and the required high-magnification microscope image, it first generates an auxiliary guide image with a magnification intermediate between the magnification of the guide image and the required high-magnification microscope image, and then acquires an intermediate high-magnification microscope image based on the temporarily generated auxiliary guide image before acquiring the required high-magnification microscope image.

[0017] According to the present invention, the setting unit sets a high-magnification image range, which is captured at a higher magnification than the specific magnification suitable for other image analyses, within a microscope image taken at a specific magnification suitable for one image analysis of a sample. In the division unit, if the high-magnification image range cannot be captured in one shot due to the shooting specifications, the high-magnification image range is divided into multiple shooting ranges, and the shooting unit captures high-magnification microscope images of each of the multiple shooting ranges.

[0018] At the combining unit, a plurality of high-magnification microscope images captured by the imaging unit are combined. In this combining unit, a guide image is generated by capturing the high-magnification image range at a magnification that can be captured at least once according to the imaging specifications. Each of the plurality of high-magnification microscope images captured by the imaging unit is arranged at a location where the image patterns on the guide image match. Note that "the image patterns match within a predetermined allowable range" means, for example, that the combination of brightness information of the images in the field-of-view range matches. When determining the match, it may be in terms of pixels in the field-of-view range or for each predetermined area. Also, the allowable range may be determined according to subsequent image observation.

[0019] As a result, it is possible to efficiently and accurately perform observations suitable for each magnification in a state where the field-of-view ranges of an image captured at a relatively low magnification of a specific magnification and an image captured at a high magnification of the specific magnification are aligned.

[0020] The microscope image acquisition control device according to the second aspect extracts in advance, from the guide image, elements that may cause the image to be blurred in the imaging of the plurality of imaging ranges and feature points that serve as indicators for combining the plurality of imaging ranges. The imaging range is corrected so that at least one of shifting the area used for focusing in the plurality of imaging ranges from the elements and having a predetermined number of the feature points present in the plurality of imaging ranges is satisfied.

[0021] The microscope image acquisition control device according to the third aspect is characterized in that, in the aspect of the first or second aspect, a microscope image captured at the specific magnification is applied as the guide image.

[0023] The microscope image acquisition control device according to the fifth aspect compares the imaging range of the combined high-magnification microscope image with the imaging range of the guide image in any one of the first to fourth aspects, identifies a range in the imaging range of the guide image where the high-magnification microscope image does not exist, and supplements the imaging range of the high-magnification microscope image according to the identified portion.

[0025] The microscope image acquisition control program of the present invention causes a computer to operate as any one of the microscope image acquisition control devices of the first to sixth aspects.

Advantages of the Invention

[0026] As described above, in the present invention, in a state where the visual field ranges of an image taken at a relatively low magnification specific magnification and an image taken at a high magnification of the specific magnification are aligned, observations suitable for each magnification can be efficiently and accurately performed.

Brief Description of the Drawings

[0027] [Figure 1] (A) is a schematic configuration diagram of an operation electron microscope (SEM "Scanning Electron Microscope") 10 according to the first embodiment, and (B) is an enlarged perspective view of the stage. [Figure 2] It is a control block diagram of a control device according to the first embodiment. [Figure 3] It is a functional block diagram for executing image acquisition control in the SEM control unit according to the first embodiment. [Figure 4] It is a flowchart showing the main routine of high magnification acquisition control executed in the SEM control unit according to the first embodiment. [Figure 5] It is a control flowchart showing the setting process subroutine of the shooting location in step 104 of FIG. 4. [Figure 6] It is a front view of an image for explaining the transition of an SEM image (low magnification), a guide image, and a high magnification image. [Figure 7] It is a front view of an image for explaining the transition from shooting to combining of a high magnification image. [Figure 8] It is a front view of an image showing the visual field range for obtaining high magnification images of different magnifications. [Figure 9] It is a front view of an image showing the process of correcting candidate points and setting a divided acquisition area. [Figure 10](A) is a characteristic diagram showing the amount of error deviation in the stage, and (B) is a front view showing the setting state of the segmented acquisition area for the guide image, which is set based on the magnitude of the error. [Figure 11] This is a functional block diagram for executing image acquisition control in the SEM control unit according to the second embodiment. [Figure 12] This is a functional block diagram for executing image acquisition control in the SEM control unit according to the third embodiment. [Figure 13] This is a flowchart showing the main routine for high-magnification acquisition control executed in the SEM control unit according to the third embodiment. [Figure 14] This is a front view of an image showing the process of correcting the shooting location in the third embodiment. [Figure 15] This is a flowchart showing the main routine for high-magnification acquisition control executed in the SEM control unit according to a modified example of the third embodiment. [Figure 16] This is a front view of an image showing the process of performing additional imaging in the third embodiment. [Figure 17] This is a front view of an image illustrating the process of modeling the accuracy of the stage position based on the discrepancy between the actual stage position and the target position, and then performing additional imaging. [Figure 18] This is a flowchart showing the main routine for high-magnification acquisition control executed in the SEM control unit according to the fourth embodiment. [Figure 19] Figure 18 is a control flowchart showing the subroutine for determining whether an intermediate guide is necessary in step 118. [Figure 20] (A) is a front view of an image showing the process of acquiring a high-magnification image relative to a guide image in one step, and (B) is a front view of an image showing the process of acquiring a high-magnification image relative to a guide image in two steps. [Modes for carrying out the invention]

[0028] [First Embodiment] Figure 1(A) is a schematic diagram of the operating electron microscope (SEM "Scanning Electron Microscope") 10 according to the first embodiment.

[0029] The SEM10 is a device that uses an electron beam to observe the surface of a sample, and it can be equipped with an Energy Dispersive X-ray Spectroscopy (EDS)12 to perform elemental analysis.

[0030] The SEM10 comprises a microscope body 14, a sample chamber 16, and a control device 18.

[0031] The microscope body 14 is composed of an electron gun 20, a condenser lens 22, a deflection coil 24, and an objective lens 26.

[0032] Furthermore, the sample chamber 16 is equipped with a stage 28 and a vacuum pump 30, and a secondary electron detector 32 and an EDS 12 are mounted on the wall of the sample chamber 16.

[0033] The control device 18 includes an SEM control unit 34, an electron gun drive unit 36, a stage drive unit 38, an input device 40, and an output device 42.

[0034] The electron gun 20 is driven by an electron gun drive unit 36 ​​controlled by the SEM control unit 34, which generates and accelerates an electron beam from an electron source. The acceleration voltage of the electron beam is generally, for example, several hundred volts to about 30 kV. The electron beam generation command and input of various adjustment values ​​can be operated by the input device 40.

[0035] Furthermore, as shown in Figure 1(B), the stage 28 is movable in the x direction (horizontal lateral direction), y direction (horizontal vertical direction), z direction (up and down direction), θ direction (z-axis rotation direction), and s direction (y-axis rotation (tilting) direction) by a stage drive unit 38 controlled by the SEM control unit 34.

[0036] When an electron beam is generated from the electron gun 20, the condenser lens 22 and the objective lens 26 focus the accelerated electron beam as an electron spot on the sample 44 positioned on the stage 28. The deflection coil 24 moves the electron spot over the sample 44 as a probe (electron beam scanning).

[0037] The secondary electron detector 32 detects signal electrons generated from the electron beam irradiation point of the sample and sends them to the SEM control unit 34. The SEM control unit 34 converts the amount of signal electrons sent from the secondary electron detector 32 into brightness values ​​for each point and outputs them as an SEM image using the output device 42 (for example, displayed on a monitor). That is, since the amount of signal electrons generated varies depending on the surface topography, the SEM image reflects the surface morphology of the sample (hereinafter sometimes referred to as the SEM image).

[0038] Furthermore, for electrons generated from the electron source to reach the sample without colliding with gas molecules, 10 -2 ~10 -3 A vacuum of Pa is required. Therefore, the inside of the sample chamber 16 must be kept under vacuum by the vacuum pump 30, and the sample to be observed must be pre-treated, such as removing moisture, so that it does not break down under vacuum. In addition, pre-treatment to make the sample surface conductive is also necessary to prevent it from being charged by electron beam irradiation.

[0039] On the other hand, the EDS12 detects characteristic X-rays emitted from atoms contained in the sample 44, and detects the type and content of elements contained in the sample 44 from their energy level and intensity. The detected values ​​are sent to the SEM control unit 34 and output by the output device 42 (for example, displayed on a monitor).

[0040] Figure 2 is a control block diagram of the control device 18. The SEM control unit 34 consists of a microcomputer 46 comprising a CPU 46A, RAM 46B, ROM 46C, input / output (I / O) 46D, and buses 46E such as a data bus and control bus connecting these, and a large storage device 48 such as an HDD (Hard Disk Drive) or SSD (Solid State Drive).

[0041] The I / O46D is connected to the stage drive unit 38, the electron gun drive unit 36, the secondary electron detector 32, the EDS 12, the input device 40, and the output device 42.

[0042] In the scanning electron microscope 10 according to the first embodiment, it is not possible to capture an SEM image with the same field of view as the relatively low-magnification SEM image (a relatively high-magnification image) in a single shot. Therefore, the field of view of the low-magnification SEM image is divided and captured, and then the images are combined. In the first embodiment, "combining" is synonymous with "bonding" as described in, for example, Patent Document 2.

[0043] In this case, misalignment may occur at the boundary of the joined images. Furthermore, the field of view of the high-magnification image and the low-magnification image generated by joining may not match within a predetermined tolerance range. This may depend on the accuracy of the stage 28 used to position the SEM sample, or may be due to image distortion, etc. Whether the field of view matches within a predetermined tolerance range can be determined by checking whether the combination of brightness information within the field of view matches. The combination may be at the pixel level or for predetermined regions. The tolerance range should be determined in accordance with subsequent image observation.

[0044] Therefore, in the first embodiment, we focused on the concept of a "guide image" and established image acquisition control to obtain an SEM image that matches the field of view of the low-magnification SEM image and is free from misalignment.

[0045] Figure 3 is a functional block diagram for image acquisition control in the SEM control unit 34. Note that each block is classified by function and does not limit the hardware configuration of the SEM control unit 34. Furthermore, some or all of the functions of each block may be executed by the CPU 46A as an image acquisition control program.

[0046] The secondary electron detector 32 is connected to the guide image acquisition unit 50.

[0047] The guide image acquisition unit 50 acquires a low-magnification SEM image using the two-dimensional electron detector 32 and sends it to the high-magnification image acquisition range setting unit 52.

[0048] The high-magnification image acquisition range setting unit 52 is connected to the input device 40 and sets the high-magnification image acquisition range based on operation instructions from the input device 40.

[0049] In other words, as shown in Figure 6, the SEM image 54 (low magnification image) is used as a guide image 56, and the range for which a high magnification image is required is set from the guide image 56.

[0050] Figure 6(A) shows the case where a high-magnification image with the same field of view as the guide image 56 is acquired; Figure 6(B) shows the case where a high-magnification image of a field of view obtained by cropping a portion of the guide image, specifically one location; and Figure 6(C) shows the case where a high-magnification image 58 is acquired by cropping a portion of the guide image 56, specifically multiple locations.

[0051] In Figures 6(A) and (B), multiple magnifications may be set as the magnification for the high-magnification image. Also, in Figure 6(C), the magnifications of each cropped area may be the same or different, and the cropped areas may be on the same side or of different sizes. Furthermore, the cropped areas may overlap.

[0052] In the following explanation, we will use the example of acquiring a high-magnification image 58 with a field of view equivalent to the guide image 56 shown in Figure 6(A).

[0053] As shown in Figure 3, the high-magnification image acquisition range setting unit 52 is connected to the shooting location setting unit 60. The set high-magnification image acquisition range is divided into multiple sections (see Figure 7) to create a divided acquisition area Xn (where n is the number of divisions; in Figure 7, there are 9 sections), and the set divided acquisition area information is sent to the high-magnification shooting instruction unit 62.

[0054] The high-magnification imaging instruction unit 62 sets candidate points for each divided acquisition area Xn (basically, the intersection of diagonals, which is the center position of the acquisition area) in order to image the divided acquisition area Xn, and controls the electron gun drive unit 36 ​​and the stage drive unit 38 to generate an electron beam from the electron gun 20 and scan the target imaging area of ​​the sample 44 (see Figure 7(A)).

[0055] Furthermore, the high-magnification image acquisition unit 64, which receives instructions from the high-magnification shooting instruction unit 62, is instructed to acquire the shooting information of each divided acquisition area Xn detected by the secondary electron detector 32.

[0056] The high-magnification image acquisition unit 64 is connected to the merging unit 66, and each time it acquires imaging information for each segmented acquisition area, it repeatedly sends the imaging information to the merging unit 66 (see Figure 7(B)).

[0057] In the merging unit 66, the imaging information for each segmented acquisition area received from the high-magnification image acquisition unit 64 is merged based on candidate points, and overlapping portions are processed to generate a single high-magnification image with a field of view equivalent to that of the guide image 56 (see Figure 7(C)). During the merging process, the combination of brightness information for each pixel of the guide image 56 allows for accurate merging to match the guide image 56.

[0058] The high-magnification image combined in the coupling unit 66 is sent to the output unit 68 and output from the output device 42. For example, if the output device 42 is a monitor, the high-magnification image is displayed on the monitor.

[0059] The operation of the first embodiment will be described below with reference to the flowcharts in Figures 4 and 5.

[0060] Figure 4 is a flowchart showing the main routine for high-magnification acquisition control executed in the SEM control unit 34 according to the first embodiment.

[0061] In step 100, a guide image is acquired, and then the process moves to step 102 to set the acquisition range for the high-magnification image. In the first embodiment, the acquisition range is set to a field of view equivalent to the field of view of the guide image (see Figure 6(A)). In the next step 104, the shooting location is set.

[0062] (Reasons why setting the shooting location is necessary) When using autofocus for SEM image acquisition, the focus is set to match the candidate point. However, if the candidate point and a depression overlap, the focus will be set to the depression, potentially blurring the surface structure of the sample to be analyzed around the depression. Therefore, the image features of the guide image are analyzed to pre-extract areas where autofocus is difficult, and the imaging location is set so that these areas do not become candidate points in the imaging range.

[0063] Areas that are difficult to focus can be identified by extracting pixels with low brightness from the image, labeling them, and designating areas larger than a certain size as difficult to focus. Alternatively, the difficulty in focusing can be determined for partial images obtained by dividing the image into small regions, using a classifier acquired in advance through machine learning.

[0064] Based on the above, Figure 5 illustrates the flow of the control flowchart showing the subroutine for setting the shooting location in step 104 of Figure 4.

[0065] As shown in Figure 5, in the process of setting the shooting location, first, multiple candidate points are generated in step 200 (see Figure 7(A)), and then the process proceeds to step 202. In Figure 7(A), the candidate points of the divided acquisition area Xn are the center positions of each acquisition area.

[0066] In step 202, unsuitable candidate points for imaging are removed. As shown in Figure 9(A), if a candidate point has a depression 70, the focus may be shifted. Also, as shown in Figure 9(A), if a candidate point is a flat surface 72 with few (less than a predetermined number) feature points for focusing, the focus may be shifted. These depressions 70 and flat surfaces 72 correspond to elements that may cause the image to blur, and can be considered unsuitable candidate points.

[0067] Therefore, as shown in Figure 9(B), for example, when the candidate points (black dots in Figure 9(B)) obtained by equal division overlap with the recess 70 or flat surface 72, the shooting location that is easy to focus on is set by shifting the candidate points, as shown by the white dots in Figure 9(B). In this case, the number of shots may increase. Also, although there is a shooting area that protrudes from the periphery of the guide image 56, this does not hinder shooting. Furthermore, if there are a predetermined number or more feature points for focusing, it may be possible to focus without shifting the candidate points.

[0068] In the next step, 204, after processing in step 202, a list of candidate points, i.e., the divided acquisition area Xn, is generated, and the process proceeds to step 206.

[0069] In step 206, it is determined whether the area to be captured in the candidate point list covers the guide image. If the result in step 206 is negative, the process returns to step 204 to regenerate the candidate point list and proceed to step 206.

[0070] Furthermore, if the result in step 206 is positive, the process proceeds to step 208 to calculate the overlap rate, and then to step 210 to determine whether the calculated overlap rate is smaller than the minimum overlap rate. If the result in step 210 is negative, the process returns to step 204 to generate the candidate point list again and proceeds to step 206.

[0071] Furthermore, if a positive result is obtained in step 210, the process proceeds to step 212 to update the minimum overlap rate, and then proceeds to step 214 to return to step 106 in Figure 4 with the finally generated candidate point list as the shooting location list for the divided acquisition area Xn.

[0072] As shown in Figure 4, step 106 determines whether or not all segmented acquisition areas Xn have been captured. If the determination is negative, including not having captured any areas, the process proceeds to step 108 to instruct the capture of high-magnification images, then proceeds to step 110 to perform high-magnification image merging (from the second time onward), and returns to step 106. Steps 106, 108, and 110 are repeated until all segmented acquisition areas Xn have been captured (see Figures 7(B) and (C)).

[0073] If a positive result is obtained in step 106, it is determined that the capture and merging of all divided acquisition areas Xn has been completed, and this routine terminates.

[0074] (Settings of shooting locations due to the merging process) In high-magnification image merging, feature point information is used to merge each high-magnification image. However, if there are areas with a uniform surface texture and few feature points, the merging process becomes difficult.

[0075] Therefore, in the guide image 56, feature points are extracted in advance, and areas of a certain size or larger with few feature points are extracted. Then, the shooting location is set so that the shooting range is not set only in those areas.

[0076] The high-magnification images, captured in segments and then joined, are generated with a field of view equivalent to the field of view set based on the guide image 56. Furthermore, they are free from accuracy errors in stage 28 and image misalignment due to joining, thus avoiding any adverse effects on subsequent analysis (such as deterioration of analysis accuracy).

[0077] The combined high-magnification image will be displayed on a monitor or similar device. However, by displaying a low-magnification image alongside it, for example, it becomes possible to compare and analyze the images within the same field of view, thereby improving work efficiency.

[0078] In the first embodiment, a high-magnification image with a field of view equivalent to that of the guide image 56 was generated (see Figure 6(A)). However, the high-magnification image does not need to have a field of view equivalent to that of the guide image 56, and the following embodiments are possible.

[0079] (1) As shown in Figure 6(B), a high-magnification image is generated of a region obtained by cropping a portion of the field of view of the guide image 56.

[0080] (2) As shown in Figure 6(C), a high-magnification image is generated of a region obtained by cropping multiple locations within the field of view of the guide image 56.

[0081] In Figures 6(A) to 6(C), two types of high-magnification images 58 may be obtained as a common variation. In this case, the field of view may be different (see Figure 8(A)) or a common field of view (see Figure 8(B)).

[0082] In other words, as shown in Figure 8(A), one high-magnification image 58A is a 3k-magnification image compared to the 1k-magnification guide image 56, and is a suitable magnification for observing the surface topography of the particles. The other high-magnification image 58B is a 10k-magnification image compared to the 1k-magnification guide image 56, and is a suitable magnification for observing even finer particles attached to the particles.

[0083] Furthermore, as shown in Figure 8(B), another approach is to obtain high-magnification images of the same field of view as the guide image 56, but to change the magnification of the high-magnification images according to the EDS images of samples of different elements. For example, it is possible to set a magnification that allows observation of the overall shape relatively well (e.g., 3kx) and a magnification that allows observation of finer details (e.g., 10kx), and obtain different magnifications within the same field of view.

[0084] (Variations in setting the shooting location) As shown in Figure 10, past data on stage position deviations can be used to determine how much deviation should be considered when setting the shooting location.

[0085] As shown in Figure 10(A), if a displacement of stage 28 (see Figure 1) in the x-direction is observed to be larger than the displacement in the y-direction, the maximum displacement observed so far is taken into consideration, and the shooting location is set with a larger margin in the x-direction relative to the y-direction around the high-magnification image acquisition range (see Figure 10(B)). This allows for shooting without any gaps in the guide image.

[0086] The positional shift of stage 28 can be determined from the difference between the combined position on the guide image 56 and the planned shooting position when the high-magnification images, which were captured in segments, are combined.

[0087] [Second Embodiment]

[0088] A second embodiment of the present invention will be described below. Note that components identical to those in the first embodiment are denoted by the same reference numerals, and their descriptions are omitted.

[0089] The second embodiment is characterized by creating big data by creating a database of positional accuracy errors corresponding to the difference between the target position and the actual position in past Stage 28, and using machine learning to predict the error between the shooting range of the high-magnification image and the corresponding range of the guide image. This determines the optimal amount of shift, and the number of shots can be reduced compared to the basic procedure.

[0090] Figure 11 is a functional block diagram of the SEM control unit 34 according to the second embodiment, which is used to perform image acquisition control and includes a function to learn the correction amount and a function to correct the shooting location based on the learned model.

[0091] The high-magnification image acquisition unit 64 and the merging unit 66 are each connected to the image information collection unit 74. The image information collection unit 74 collects each image of the captured segmented acquisition area Xn and the combined high-magnification image, and stores them as big data in, for example, a large-scale storage device 48.

[0092] The correction amount prediction model learning unit 76 uses machine learning to predict the error between the shooting range of the high-magnification image and the corresponding range of the guide image, based on the big data collected by the image information collection unit 74. The prediction results are stored in the learning result storage unit 78 (a portion of the large-scale storage device 48 can be used). The execution of these functions constitutes the phase of learning the correction amount.

[0093] In other words, the image information collection unit 74 acquires and stores the following information to generate big data.

[0094] (1) Amount of movement of the stage 28 obtained from the stage drive unit 38 (2) The correction amount calculated by the joining unit 66 during image joining (the difference between the planned shooting position and the position joined on the guide image) (3) Guide image of the area around the high-magnification image used by the joining section 66 for image joining.

[0095] The correction quantity prediction model learning unit 76 learns a correction quantity prediction model based on the collected data.

[0096] The correction amount prediction model takes the stage movement amount and a partial guide image around the high-magnification image as input and outputs the correction amount for the junction 66. For example, a multimodal deep neural network is constructed based on ResNet, which is used in fields such as image classification, and trained to minimize the error between the correction amount in the dataset and the correction amount output by the model.

[0097] The learning result storage unit 78 is connected to the shooting location correction unit 80. When a shooting location is set in the shooting location setting unit 60, the shooting location correction unit 80 determines the optimal amount of shift from the prediction results and corrects the shooting location set in the shooting location setting unit 60. The execution of this function constitutes the phase in which the shooting location is corrected based on the learned model.

[0098] Furthermore, the big data, which includes multiple datasets relating the positional accuracy error corresponding to the difference between the target position and the actual position in previous Stage 28 stages, may be created in the SEM control unit 34, or separately created data may be used. In addition, a machine learning model may be stored in the SEM control unit 34, or a program executable from machine learning may be installed in the SEM control unit 34.

[0099] In other words, the following processing is performed in the shooting location correction unit 80.

[0100] (1) For each shooting location, calculate the correction amount based on the correction amount prediction model.

[0101] (2) If the absolute value of the correction amount is greater than the threshold, the corresponding shooting area is excluded. This is because areas with a large correction amount are unsuitable for high-magnification image capture (difficult to focus, few feature points, etc.).

[0102] (3) Apply the correction amount to each shooting location and create a list of corrected shooting locations.

[0103] (4) Determine whether the corrected shooting location list covers the guide image (see Figure 4). If it does not, return to the shooting location setting unit 60.

[0104] In summary, by performing a phase to learn the correction amount and a phase to correct the shooting location based on the learned model, the optimal amount of shift can be set.

[0105] [Third Embodiment] A third embodiment of the present invention will be described below. Note that components identical to those in the first embodiment will be denoted by the same reference numerals, and their descriptions will be omitted.

[0106] The third embodiment is characterized by comparing the combined high-magnification image 58 with the guide image 56, identifying the area in the guide image 56 where the high-magnification image 58 does not exist, and supplementing the shooting location according to the identified area. Supplementation means adding a new shooting location.

[0107] Figure 12 is a functional block diagram of the SEM control unit 34 according to the third embodiment for performing image acquisition control.

[0108] In the third embodiment, a shooting location correction unit 82 is provided between the coupling unit 66 and the output unit 68. This shooting location correction unit 82 compares the formed high-magnification image 58 with the guide image 56 and determines the existence of a range in the guide image 56 where the high-magnification image 58 does not exist.

[0109] Furthermore, the shooting location correction unit 82 identifies the range in the guide image 56 where no high-magnification image 58 exists, and sends the identified information to the high-magnification shooting instruction unit 62.

[0110] The high-magnification shooting instruction unit 62 performs shooting based on the identified information.

[0111] Figure 13 is a flowchart showing the main routine for high-magnification acquisition control executed in the SEM control unit 34 according to the third embodiment. Note that the same steps are used for the same processes as in the flowchart shown in Figure 4 (operation of the first embodiment).

[0112] In step 100, a guide image is acquired, and then the process moves to step 102 to set the acquisition range for the high-magnification image. In the third embodiment, the acquisition range is set to a field of view equivalent to the field of view of the guide image (see Figure 6(A)). In the next step 104, the shooting location is set. The setting of the shooting location is the same as the process in Figure 5, so a detailed explanation is omitted here.

[0113] In the next step, 106, it is determined whether or not all segmented acquisition areas Xn have been captured. If the determination is negative, including if they have not been captured, the process proceeds to step 108 to instruct the capture of high-magnification images, then proceeds to step 110 to perform the merging of high-magnification images (from the second time onward), and returns to step 106. Steps 106, 108, and 110 are repeated until all segmented acquisition areas Xn have been captured (see Figures 7(B) and (C)).

[0114] Here, if a positive determination is made in step 106, it is determined that the imaging and merging of all divided acquisition areas Xn have been completed. However, there may be areas in the guide image 56 where the high-magnification image 58 does not exist. Therefore, if a positive determination is made in step 106, the process proceeds to step 112, where the formed high-magnification image 58 and the guide image 56 are compared. Then, the process proceeds to step 114, and if, as a result of the comparison in step 112, it is determined that a high-magnification image 58 exists in the guide image 56 and correction of the imaging area is necessary (positive determination), the process proceeds to step 116, where the area is identified, and the identified information is sent to the high-magnification imaging instruction unit 62. Based on the identified information, additional imaging areas are set, and the process proceeds to step 106. If a negative determination is made in step 114, this routine ends.

[0115] The correction of the shooting location will be explained in detail according to Figure 14.

[0116] Due to the influence of the stage 28 and the stage drive unit 38, it is difficult to move and position the stage 28 to the specified position without displacement, and a displacement of, for example, several hundred nanometers to 1 micrometer may occur relative to the target position. In a high-magnification image of 10kx, this translates to a displacement of several tens to several hundred pixels.

[0117] As shown in Figure 14(A), the final combined high-magnification image 58 is generated with a shift relative to the shooting range specified in the guide image 56, resulting in uncaptured areas within the shooting range.

[0118] Therefore, as shown in Figure 14(B), the shooting area correction unit 82 sets new shooting areas (divided acquisition areas) to fill in the unshooted areas (gap), captures high-magnification images, and combines them. This makes it possible to obtain a high-magnification image without gaps in the guide image 56.

[0119] In the third embodiment described above, the correction of the shooting location was performed after all of the initially set shooting locations had been photographed. However, the correction may also be performed after each high-magnification image of a divided acquisition area has been photographed, either once or at predetermined intervals of multiple times. Alternatively, the shooting location may be corrected when the difference between the planned shooting range (part of the guide image 56) and the combined high-magnification image 58 generated at that time exceeds a certain level.

[0120] Figure 15 is a flowchart showing the main routine for high-magnification acquisition control executed in the SEM control unit 34 according to a modified example of the third embodiment. The processing flow is the same as the flowchart in Figure 13 (third embodiment), with steps 112, 114, and 116 inserted after steps 108 and 110 shown in Figure 13. This allows the additional imaging process to be corrected each time high-magnification images of a divided acquisition area are acquired, either once or at predetermined intervals.

[0121] The additional imaging process will be explained in detail with reference to Figure 16.

[0122] Figure 16(A) shows the areas where imaging has been completed (guide image 56) and the high-magnification image 58 generated at that time. At this point, the area A1 enclosed by a triangle in the plan has not been imaged.

[0123] Figure 16(B) shows the corrected shooting locations to compensate for the unphotographed area A1. In Figure 16(B), a new segmented acquisition area has been added to the upper right portion of the deviation from the plan, where there is no overlap with the area to be photographed.

[0124] In Figure 16, the initially created segmented acquisition area can be left unchanged, and a new segmented acquisition area can be added to G, which has been corrected. The overall segmented acquisition area can then be set (corrected) to match the size and shape of the deviation from the plan, including the initially created segmented acquisition area.

[0125] An example of setting (correcting) a new overall divided acquisition area will be explained according to Figure 17.

[0126] As shown in Figure 16(A) above, if the shooting end position is shifted due to an offset in a specific direction, then, as shown in Figure 16(B), even if the shooting location is corrected, it will be necessary to correct the shooting location again midway through the process.

[0127] To avoid this, as shown in Figure 17(A), the accuracy of the stage position is modeled based on the discrepancy between the actual position of stage 28 and the target position. In Figure 17(A), a model is estimated in which the planned shooting direction is along the dotted line B, but the shooting direction is along the solid line B2. Therefore, the segmented acquisition area is corrected to match this model.

[0128] In other words, as shown in Figure 17(B), the stage position is shifted upward, so by shifting the specified segmented acquisition area downward, recorrection can be avoided.

[0129] [Fourth Embodiment]

[0130] A fourth embodiment of the present invention will be described below. Note that components identical to those in the first embodiment are denoted by the same reference numerals, and their descriptions are omitted.

[0131] The fourth embodiment addresses the problem that if there is a magnification difference greater than a predetermined amount between the guide image 56 and the high-magnification image 58 to be acquired, it may not be possible to match the feature points during the joining process and thus the joining may not be possible. In this embodiment, when there is a magnification difference greater than a predetermined amount, an auxiliary guide image 56A (see Figure 20) with a magnification intermediate between the magnification of the guide image 56 and the magnification of the high-magnification image 58 to be acquired is first generated, and the high-magnification image 58 with the desired magnification is acquired based on the generated auxiliary guide image 56A.

[0132] Figure 18 is a flowchart showing the main routine for high-magnification acquisition control executed in the SEM control unit 34 according to the fourth embodiment. Note that the same steps are used for the same processes as in the flowchart shown in Figure 4 (operation of the first embodiment).

[0133] In step 100, a guide image is acquired, and then the process moves to step 102 to set the acquisition range for the high-magnification image. In the fourth embodiment, the acquisition range is set to a field of view equivalent to the field of view of the guide image (see Figure 6(A)).

[0134] In the next step 120, a process is executed to determine whether an intermediate guide is necessary. This step 120 determines whether the auxiliary guide image 56A is necessary, and the details will be described later according to the flowchart (intermediate guide necessity determination process routine) in Figure 19(A).

[0135] In step 122, if the intermediate guide requirement determination performed in step 120 determines that an intermediate guide is necessary (positive determination), the process proceeds to step 122, the magnification is changed to the set magnification (magnification for generating the auxiliary guide image 56A), and the process proceeds to step 104. If the determination in step 120 is negative, the process proceeds to step 104.

[0136] Step 104 involves setting the shooting location. The process for setting the shooting location is the same as in Figure 5, so a detailed explanation is omitted here.

[0137] In the next step, 106, it is determined whether or not all segmented acquisition areas Xn have been captured. If the determination is negative, including if they have not been captured, the process proceeds to step 108 to instruct the capture of high-magnification images, then proceeds to step 110 to perform the merging of high-magnification images (from the second time onward), and returns to step 106. Steps 106, 108, and 110 are repeated until all segmented acquisition areas Xn have been captured (see Figures 7(B) and (C)).

[0138] Here, if a positive determination is made in step 106, it is determined that the imaging and merging of all divided acquisition areas Xn have been completed. However, there may be areas in the guide image 56 where the high-magnification image 58 does not exist. Therefore, if a positive determination is made in step 106, the process proceeds to step 112, where the formed high-magnification image 58 and the guide image 56 are compared. Then, the process proceeds to step 114, and if, as a result of the comparison in step 112, it is determined that a high-magnification image 58 exists in the guide image 56 and correction of the imaging area is necessary (positive determination), the process proceeds to step 116, where the area is identified, and the identified information is sent to the high-magnification imaging instruction unit 62. Based on the identified information, additional imaging areas are set, and the process proceeds to step 106. If a negative determination is made in step 114, this routine ends.

[0139] Figure 19(A) is a control flowchart showing the intermediate guide requirement determination processing routine in step 118 of Figure 18.

[0140] In step 250, feature points from the base guide image 56 are extracted, and then the process moves to step 252 to determine whether processing has been completed for all feature points. If a negative result is obtained in step 252 (always negative on the first attempt), the process moves to step 254 to measure the radius r of the range where a predetermined number of feature points exist, and then returns to step 252. The processing in steps 250, 252, and 254 is as shown in Figure 19(B).

[0141] If it is determined in step 252 that processing at the feature point of the step is complete (positive determination), the process proceeds to step 256 to calculate the average value ra of the measured radius R, and then proceeds to step 258.

[0142] In step 258, the field of view (ws × hs [μm]) for the desired magnification S is obtained, and the process proceeds to step 260.

[0143] In step 260, we compare twice the average radius (2ra) with the minimum ws and hs (min(ws,hs)).

[0144] If it is determined in step 260 that 2ra > min(ws,hs), the process proceeds to step 262 to determine if a magnification change is necessary, then proceeds to step 264 to set the shooting magnification to one that is larger than 2ra, and returns to step 120 in Figure 18.

[0145] Furthermore, if it is determined in step 260 that 2ra ≤ min(ws,hs), the process returns to step 120 in Figure 18.

[0146] The identification of the feature point range in steps 250, 252, and 254 will be explained in detail with reference to Figure 19(B).

[0147] Feature points are extracted from guide image 56. Next, for all extracted feature points, the range (radius r) around the feature point of interest where a predetermined number of N feature points exist is measured.

[0148] For all feature points, when r “μm” is measured, the average value (ra) is obtained. When feature points exist densely, the average value ra becomes small, and when feature points are sparse, the average value ra becomes large.

[0149] Next, information on the field-of-view range ws×hs (ws is the horizontal (x-direction) field-of-view range, and hs is the vertical (y-direction) field-of-view range) of a single high-magnification image 58 when taken at the magnification S to be acquired next is obtained. When 2ra is larger than min(ws, hs), there is a high possibility that there are relatively few feature points available for collation within the field-of-view range. In this case, the intermediate guide image (auxiliary guide image 56A) is used. Here, the magnification M of the auxiliary guide image 56A may be set to the maximum magnification such that 2ra < min(hs, ws).

[0150] FIG. 20(A) shows the process of directly (in one go) generating the high-magnification image 58 from the guide image 56 as described in the first to third embodiments. In contrast, as shown in FIG. 20(B), in the fourth embodiment, a high-magnification image 58 (magnification 5k times) is once generated from the guide image 56 (magnification 1k times), and this generated high-magnification image 58 (magnification 5k times) is used as the auxiliary guide image 56A to generate the high-magnification image 58 (magnification 10k times) again.

[0151] Table 1 classifies, for reference, Patent Document 2 in the background art section as a comparative example with respect to the present embodiment, and summarizes the differences, implemented technologies, and problems of the comparative example into three items, presenting the differences from the configuration of the present embodiment in tabular form.

[0152]

Table 1

[0153] (Appendix 1) The invention described in Appendix 1 is a microscope image acquisition control device comprising: a setting unit that sets a high-magnification image range to be photographed at a higher magnification than the specific magnification suitable for other image analyses, within a microscope image taken at a specific magnification suitable for one image analysis of a sample; a division unit that divides the high-magnification image range into a plurality of shooting ranges; an imaging unit that photographs each of the plurality of shooting ranges; and a combining unit that combines the plurality of high-magnification microscope images taken by the imaging unit, wherein the combining unit generates a guide image by photographing the high-magnification image range at a magnification that can be photographed, and arranges each of the plurality of high-magnification microscope images taken by the imaging unit so that the image pattern on the guide image matches within a predetermined tolerance range.

[0154] (Note 2) The invention described in Appendix 2 is the invention described in Appendix 1, wherein elements that may cause image blurring when photographing the plurality of shooting ranges and feature points that serve as indicators for the joining of the plurality of shooting ranges are extracted in advance from the guide image, and the shooting range is corrected so that at least one of the following is satisfied: the area used to focus in the plurality of shooting ranges is shifted from the elements, and a predetermined number of the feature points exist within the plurality of shooting ranges.

[0155] (Note 3) The invention described in Appendix 3 applies, in the invention described in Appendix 1 or Appendix 2, a microscope image taken at the specified magnification as the guide image.

[0156] (Note 4) The invention described in Appendix 4, in the invention described in any one of Appendix 1 to 3, uses machine learning to predict the error between the shooting range of the high-magnification microscope image and the shooting range of the guide image, based on a plurality of data associated with the relationship of positional accuracy error corresponding to the difference between the target position and the actual position of the stage that positions the sample, and corrects the shooting range of the high-magnification microscope image.

[0157] (Note 5) The invention described in Appendix 5 is an invention described in any one of Appendix 1 to 4, wherein the imaging range of the combined high-magnification microscope image is compared with the imaging range of the guide image, a region within the imaging range of the guide image in which the high-magnification microscope image does not exist is identified, and the imaging range of the high-magnification microscope image is supplemented according to the identified region.

[0158] (Note 6) The invention described in Appendix 6 is an invention described in any one of Appendix 1 to 5, in which, when there is a predetermined or greater magnification difference between the magnification of the guide image and the required high-magnification microscope image, an auxiliary guide image is first generated which has an intermediate magnification between the magnification of the guide image and the required high-magnification microscope image, and then an intermediate high-magnification microscope image is obtained based on the first generated auxiliary guide image, and then the required high-magnification microscope image is obtained.

[0159] (Note 7) The invention described in Appendix 7 is a microscope image acquisition control program that causes a computer to operate as a microscope image acquisition control device described in any one of Appendix 1 to Appendix 6. [Explanation of symbols]

[0160] 10. Scanning Electron Microscope (SEM) 12 Distributed X-ray spectrometer (EDS) 14. Mirror body 16 Sample Room 18 Control device 20 Electron gun 22 Condenser Lens 24 Deflection coils 26 Objective lens 28 stages 30 Vacuum pumps 32 Secondary electron detector 34 SEM Control Unit 36. Electron gun drive unit 38 Stage drive unit 40 Input Devices 42 Output Devices 44 samples 46A CPU 46B RAM 46C ROM 46D Input / output section (I / O) 46E Bus 46 Microcomputers 48 Mass storage 50 Guide image acquisition unit 52. High-magnification image acquisition range setting section (setting section) 54 SEM image (low magnification) 56 Guide Images 58 High-magnification images 60 Shooting location setting section (divided section) 62. High-magnification shooting instruction unit (shooting unit) 64 High-magnification image acquisition unit 66 Joint part (joint part) 68 Output section

Claims

1. A setting unit that sets a high-magnification image range within a microscope image taken at a specific magnification suitable for one image analysis, and which is taken at a higher magnification than the specific magnification suitable for other image analyses, A division unit that divides the aforementioned high-magnification image range into multiple shooting ranges, A shooting unit that captures high-magnification microscope images of each of the aforementioned multiple shooting ranges, It has a coupling unit that combines multiple high-magnification microscope images captured by the imaging unit, The aforementioned joint portion, A guide image is generated by capturing the aforementioned high-magnification image range at a magnification that allows for such capture, and each of the multiple high-magnification microscope images captured by the capturing unit is positioned such that the image patterns on the guide image match within a predetermined tolerance range. Based on multiple data sets that associate the positional accuracy error corresponding to the difference between the target position and the actual position of the stage used to position the sample, machine learning is used to predict the error between the imaging range of the high-magnification microscope image and the imaging range of the guide image, and to correct the imaging range of the high-magnification microscope image. Microscope image acquisition and control device.

2. A setting unit that sets a high-magnification image range in which a microscope image taken at a specific magnification suitable for one image analysis of a sample is taken at a higher magnification than the specific magnification suitable for another image analysis, A division unit that divides the aforementioned high-magnification image range into multiple shooting ranges, A shooting unit that captures high-magnification microscope images of each of the aforementioned multiple shooting ranges, It has a coupling unit that combines multiple high-magnification microscope images captured by the imaging unit, The aforementioned joint portion, A guide image is generated by capturing the aforementioned high-magnification image range at a magnification that allows for such capture, and each of the multiple high-magnification microscope images captured by the capturing unit is positioned such that the image patterns on the guide image match within a predetermined tolerance range. If there is a predetermined or greater magnification difference between the magnification of the guide image and the required high-magnification microscope image, an auxiliary guide image with an intermediate magnification between the magnification of the guide image and the required high-magnification microscope image is first generated, and an intermediate high-magnification microscope image is obtained based on the generated auxiliary guide image, after which the required high-magnification microscope image is obtained. Microscope image acquisition and control device.

3. The microscope image acquisition control device according to Claim 1 or Claim 2, wherein elements that may cause image blurring when photographing the plurality of shooting ranges and feature points that serve as indicators for the joining of the plurality of shooting ranges are extracted in advance from the guide image, and the shooting range is corrected so that at least one of the following is satisfied: the area used to focus in the plurality of shooting ranges is shifted from the elements, and a predetermined number of the feature points are present in the plurality of shooting ranges.

4. The microscope image acquisition control device according to claim 1 or claim 2, wherein the microscope image taken at the specified magnification is applied as the guide image.

5. A microscope image acquisition control device according to claim 1 or 2, comprising: comparing the shooting range of the combined high-magnification microscope image with the shooting range of the guide image; identifying a range within the shooting range of the guide image where the high-magnification microscope image does not exist; and supplementing the shooting range of the high-magnification microscope image according to the identified range.

6. A computer, To be operated as a microscope image acquisition control device according to claim 1 or claim 2, Microscope image acquisition control program.

Images