Method implemented by a data processing device and charged particle beam device for detecting a sample using such a method
By setting desired image quality parameters and image modification techniques, image quality can be improved or degraded to a suitable level, thus solving the problem of image segmentation quality degradation caused by unstable imaging conditions and achieving efficient image processing in production environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FEI CO
- Filing Date
- 2021-04-19
- Publication Date
- 2026-08-04
AI Technical Summary
Existing charged particle microscopy image processing methods rely on the stability of imaging parameters when imaging conditions are unstable, which leads to a decline in image segmentation quality. Furthermore, retraining neural networks in production environments is costly and impractical.
By receiving images, setting desired image quality parameters, comparing current image quality parameters, and using image modification techniques to improve or degrade images as necessary to make them meet medium quality standards, the neural network is trained to perform analysis based on this, avoiding retraining.
It simplifies the image processing workflow, reduces reliance on high-quality images, lowers training costs, and improves the stability and efficiency of image processing.
Smart Images
Figure CN113538329B_ABST
Abstract
Description
[0001] This invention relates to a method implemented by a data processing device. It also relates to a charged particle beam apparatus for examining samples using such a method.
[0002] Charged particle microscopy, particularly in the form of electron microscopy, is a well-known and increasingly important imaging technique for microscopic objects. Historically, the basic class of electron microscopes has evolved into many well-known apparatus types, such as transmission electron microscopes (TEM), scanning electron microscopes (SEM), and scanning transmission electron microscopes (STEM), and has also evolved into various sub-species, such as so-called "dual-beam" apparatuses (e.g., FIB-SEM), which additionally employ "processing" focused ion beams (FIB) to allow for supporting activities such as, for example, ion beam milling or ion beam-induced deposition (IBID). Technicians will be familiar with different types of charged particle microscopy.
[0003] In SEM, the irradiation of a sample by a scanning electron beam promotes the emission of "auxiliary" radiation from the sample in the form of secondary electrons, backscattered electrons, X-rays, and cathodoluminescence (infrared, visible, and / or ultraviolet photons). One or more components of this emitted radiation can be detected and used for sample analysis.
[0004] In TEM, an electron beam is transmitted through a sample to form an image through the interaction of electrons with the sample as the beam passes through. The image is then magnified and focused onto an imaging device, such as a fluorescent screen, a photographic film layer, or a sensor, such as a scintillator connected to a charge-coupled device (CCD). The scintillator converts the electrons in the microscope into photons, which the CCD can then detect.
[0005] Charged particle microscopy produces images of the sample under study. These images typically require processing. This processing may include analyzing and / or manipulating the acquired images. For example, in SEM images of cell membranes in brain tissue, segmentation techniques are desired. Artificial neural networks (ANNs) and / or convolutional neural networks (CNNs) can be used for this task. While relatively robust under varying imaging conditions, segmentation quality can still degrade if there is significant data noise, poor focus, or other adverse instrument conditions. This is undesirable, as it can lead to subsequent errors, including flawed medical decisions.
[0006] One way to overcome the degradation of analysis and / or manipulation results is to retrain the network to address these changes by using new conditions. However, this poses a serious problem if the network is in a production (customer) environment, where network retraining is generally not an efficient option because it is costly due to computation time and often requires specialized input for product labeling.
[0007] Therefore, one of the biggest challenges for ANNs and CNNs used in image processing remains the fact that they are highly dependent on the stability of imaging conditions and / or imaging parameters.
[0008] Therefore, the object of the present invention is to provide a method that can be used to solve one or more drawbacks of using ANN and / or CNN in image processing.
[0009] To this end, a method implemented by a data processing device is provided. The method defined herein includes the following steps: receiving an image; providing a setpoint for desired image quality parameters of the image; and processing the image using image analysis techniques to determine current image quality parameters of the image. The current image quality parameters are then compared with the desired setpoint. In other words, an image is provided and analyzed to determine image-related parameters, and then the image-related parameters are examined to see if they meet the desired quality. Based on the comparison result, a modified image is generated using image modification techniques. This image modification technique may include the use of ANNs and / or CNNs.
[0010] As defined herein, the step of generating the modified image includes the following steps: improving the image based on the image quality parameter if the current image quality parameter is lower than the set point; and degrading the image based on the image quality parameter if the current image quality parameter exceeds the set point. In other words, either of the two actions is possible based on the comparison results. In the first case, the image quality is lower than the expected value. In this case, the image is improved by using image modification techniques to ensure that the image quality of the modified image is improved. In the second case, the image quality is higher than the expected value. In this case, the image is intentionally degraded by using image modification techniques to ensure that the image quality of the modified image is actually reduced. As defined herein, the modified image is then output and analyzed by an ANN and / or a CNN. As can be seen above, the corresponding ANN and / or CNN for analysis is trained on an image set, wherein each image in the image set generally has the desired image quality parameter.
[0011] As an example, a method as defined herein may include the step of providing a substantially noise-free image, and a method as defined herein may include the step of degrading the image by adding noise to the image. Generally, one or more of the following operations can be performed on content that can be considered a high-quality image: reducing resolution, reducing color depth, reducing dynamic range, weakening focus, reducing sharpness, increasing directional blur, reducing contrast, and adjusting white balance. Similarly, one or more of the following operations can be performed on content that can be considered a low-quality image: increasing resolution, increasing color depth, increasing dynamic range, strengthening focus, increasing sharpness, eliminating directional blur, increasing contrast, and adjusting white balance. In this sense, a high-quality image is downgraded to a medium-quality image, and a low-quality image is upgraded to a medium-quality image. The medium-quality image can then be analyzed using an ANN and / or a CNN, wherein the ANN and / or CNN are first trained substantially on the medium-quality image.
[0012] The method can be performed on multiple images. An incoming image can be transformed into an image with predefined target properties, where the target properties correspond to a more suitable image quality setting. Incoming images of different qualities, such as images obtained from different settings, are transformed into images with similar, more suitable properties. This involves steps of improving some images and significantly degrading others. The final modified image set has comparable image quality parameters and can be processed in a simpler and more efficient manner. In this way, the objective is achieved.
[0013] As defined herein, the modified images are further analyzed using an ANN and / or a CNN. The method defined herein allows all images to be transformed into “medium-quality” images on which the ANN and / or CNN are trained. Therefore, by providing medium-quality images, the variation in input data to the ANN and / or CNN is reduced. The inventors realized that rather than always improving the received images (e.g., by reducing noise, improving focus, etc.), it is actually advantageous to introduce a step that “makes the images worse,” as this significantly simplifies the operation compared to improving image quality. This has the beneficial consequence that the training data does not necessarily consist of the best possible images. Alternatively, the network can be trained with “medium-quality” images, and the method defined herein can be used to transform any subsequently acquired images to the known conditions on which the original network was trained. As mentioned above, this involves improving or reducing image quality before feeding the images to the ANN and / or CNN. In this way, the proper functioning of the ANN and / or CNN is guaranteed without retraining the main NN.
[0014] The method defined herein eliminates the need to retrain neural networks in the art. Instead, the neural network can be trained on a type of image, which may contain images with moderate image quality attributes rather than those with good image quality attributes. The method described above can then be used to transform the incoming image into a modified image suitable for the neural network, which can process these images in the desired manner. As an additional benefit, if it is found that the neural network does not adequately process the transformed or modified image, the image manipulation technique used in the step of generating the modified image can be changed. Modifying the image manipulation technique to ensure that the modified image can be processed by the neural network is relatively easy and effective, and this alleviates the need to retrain the neural network. Therefore, existing ANNs and / or CNNs can still be used by transforming the images input to the ANN and / or CNN, rather than retraining the network. This is a significant advantage because no new training data or labels need to be collected, and the image receiving NN, which may be very complex and deployed in embedded systems that may be difficult to retrain, can thus remain unchanged.
[0015] It should be noted that in the methods defined herein, transformation includes both enhancement and degradation of the incoming image. Enhancement and degradation refer to one or more image parameters, which may include resolution, color depth, dynamic range, focus, sharpness, directional blur, contrast, white balance, and noise. Other image quality parameters are also conceivable. For one or more of the above parameters, the image modification techniques used in the methods can enhance and degrade the received image. Those skilled in the art will be familiar with suitable parameters and algorithms used in these image modification techniques.
[0016] As described above, the desired setpoint for the desired image quality parameter is expected to correspond to a medium image quality parameter value.
[0017] The method may include an additional step of analyzing the modified image. This analysis may include steps using artificial neural networks (ANNs) and / or convolutional neural networks (CNNs).
[0018] The analysis may include identifying one or more objects in the image.
[0019] Images can be provided to the data processing device in various ways. Images can be retrieved from non-transitory computer-readable media. Images can be retrieved from cloud computing networks. Images can be acquired via a camera device connected to the data processing device.
[0020] In this embodiment, the image is obtained using a microscope, specifically a charged particle microscope. The charged particle microscope can be an electron microscope.
[0021] According to one aspect, a non-transitory computer-readable medium is provided, wherein the non-transitory computer-readable medium stores software instructions thereon, the software instructions causing the data processing device to perform methods as defined herein when executed by a data processing device.
[0022] According to one aspect, a charged particle beam device for examining a sample is provided, comprising: - A sample holder for holding the sample in place; - A source used to generate beams of charged particles; - An illuminator for focusing the charged particle beam onto the sample; - A detector for detecting the radiation flux emitted from the sample in response to the irradiation by the charged particle beam; and - Data processing equipment.
[0023] As defined herein, the charged particle beam device is arranged to perform the methods defined herein.
[0024] The data processing device can be directly connected to the detector and receive data and / or images directly from the detector. An intermediate connection, for example, via an additional device between the detector and the processing device is also possible. It is conceivable, for example, that the charged particle beam device includes a controller arranged for operating at least a portion of the charged particle beam device. This controller can be connected to, or at least connected to, the detector and the data processing device, and can be arranged to forward data (containing images) from the detector to the processing device. The controller can be arranged to process data emitted from the detector, or can be arranged to forward raw data to the data processing device. Once the data is received, the data processing device will be able to perform methods as defined herein. In an embodiment, the controller includes the data processing device.
[0025] According to one aspect, a data processing device is provided that is arranged to perform methods as defined herein.
[0026] The invention will now be explained in more detail based on exemplary embodiments and the accompanying schematic diagrams, wherein: Figure 1 - A longitudinal cross-sectional view of a charged particle microscope according to a first embodiment of the present invention is shown; Figure 2 - A longitudinal cross-sectional view of a charged particle microscope according to a second embodiment of the present invention is shown; Figure 3 - A flowchart of the first embodiment of the method as defined herein; Figure 4 - A flowchart of another embodiment of the method defined herein; Figure 5 - Illustrations of possible methods for analyzing and processing images using the methods defined in this article.
[0027] Figure 1 (Not to scale) A highly schematic depiction of an embodiment of a charged particle microscope M according to an embodiment of the invention. More specifically, it shows an embodiment of a transmission microscope M, which in this case is TEM / STEM (although, in the context of the invention, it may simply be effectively SEM (see...). Figure 2 (or, for example, ion-based microscopy). In Figure 1 In the vacuum chamber 2, electron source 4 generates electron beam B, electrons... Beam B The electron optical axis B' propagates and passes through the electron optical illuminator 6, thereby guiding / focusing electrons onto selected portions of the sample S (e.g., for (local) thinning / planarization). A deflector 8 is also depicted, which ( especially It can be used to achieve the scanning motion of beam B.
[0028] The sample S is held in a sample holder H, which can be positioned in multiple degrees of freedom by a positioning device / stage A that moves a bracket A' in which the holder H is (removably) attached; for example, the sample holder H may include (in particular) fingers movable in the XY plane (see the depicted Cartesian coordinate system; typically, movement parallel to Z and tilting about X / Y are also possible). Such movement allows different portions of the sample S to be illuminated / imaged / examined (in the Z direction) by an electron beam B traveling along axis B' (and / or allows scanning motion to be performed as an alternative to beam scanning). If desired, an optional cooling device (not depicted) may be brought into close thermal contact with the sample holder H to maintain it (and the sample S thereon) at a low temperature, for example.
[0029] The electron beam B will interact with the sample S in a manner that causes various types of "stimulated" radiation to be emitted from the sample S, including, for example, secondary electrons, backscattered electrons, X-rays, and optical radiation (catholuminescence). If desired, one or more of these radiation types can be detected by means of an analytical device 22, which may be, for example, a combination of scintillator / photomultiplier tube or EDX or EDS (energy-dispersive X-ray spectrometer) modules; in this case, an image can be constructed using essentially the same principles as in SEM. However, alternatively or additionally, electrons that pass through the sample S, exit from it, and continue to propagate along axis B' (essentially, although usually with some deflection / scattering) can be studied. This transmitted electron flux enters the imaging system (projection lens) 24, which typically includes various electrostatic / magnetic lenses, deflectors, correctors (such as astigmatism reducers), etc. In normal (non-scanning) TEM mode, this imaging system 24 focuses the transmitted electron flux onto a fluorescent screen 26, which, if desired, can be retracted / withdrawn (as schematically shown by arrow 26') away from axis B'. An image (or diffraction pattern) of a portion of the sample S is formed by the imaging system 24 on screen 26, and this can be viewed through an observation port 28 located in a suitable portion of the wall of housing 2. The retraction mechanism of screen 26 may be, for example, inherently mechanical and / or electrical, and is not depicted herein.
[0030] Instead of observing the image on screen 26, the fact that the focusing depth of the electron flux exiting imaging system 24 is typically large (e.g., about 1 meter) can be utilized. Therefore, various other types of analytical devices can be used downstream of screen 26, such as: - TEM camera 30. At camera 30, the electron flux can form a still image (or diffraction pattern), which can be processed by controller / processor 20 and displayed on display device 14, such as a flat panel display. When not needed, camera 30 can be retracted / withdrawn (as schematically indicated by arrow 30') to remove it from axis B'.
[0031] - STEM camera 32. The output from camera 32 can be recorded as varying with the (X, Y) scan position of beam B on sample S, and can be constructed as a “mapped” image of the output from camera 32 varying with X, Y. Camera 32 may include single pixels with a diameter of, for example, 20 mm, as opposed to the pixel matrix typically present in camera 30, but camera 32 may also be an electron microscope pixel array detector (EMPAD). Furthermore, the acquisition rate of camera 32 (e.g., 10 pixels per second) 6 (a few points) will typically be more than a camera 30 (e.g., 10 per second). 2The acquisition rate is much higher (per image). Again, when not needed, camera 32 can be retracted / withdrawn (as schematically shown by arrow 32') to remove it from axis B' (but such retraction is not necessary in the case of, for example, a donut-shaped annular dark field camera 32; in such cameras, the central aperture will allow the flux to pass through when the camera is not in use).
[0032] - As an alternative to imaging using camera 30 or 32, beam splitter device 34 may also be invoked, which may be, for example, an EELS module.
[0033] It should be noted that the order / position of items 30, 32, and 34 is not strict, and many possible variations can be considered. For example, the beam splitter device 34 can also be integrated into the imaging system 24.
[0034] In the illustrated embodiment, the microscope M also includes a retractable X-ray computed tomography (CT) module, typically indicated by reference numeral 40. In computed tomography (also known as computed tomography imaging), a source and (radially opposed) detectors are used to observe the sample along different lines of sight in order to obtain penetrating observations of the sample from a variety of angles.
[0035] Note that the controller (computer processor) 20 is connected to various illustrated components via control lines (buses) 20'. This controller 20 can provide various functions, such as synchronizing actions, providing setpoints, processing signals, performing calculations, and displaying messages / information on a display device (not depicted). Needless to say, the controller 20 (schematically depicted) may be (partially) inside or outside the housing 2, and may have an integral or modular structure as needed. As shown in this embodiment, the controller includes a data processing device P arranged to perform the methods defined herein.
[0036] Those skilled in the art will understand that the interior of housing 2 need not be maintained under a strict vacuum; for example, in so-called “environmental TEM / STEM,” a background atmosphere of a given gas is intentionally introduced / maintained within housing 2. Those skilled in the art will also understand that, in practice, it may be advantageous to limit the volume of housing 2 such that, where possible, it is substantially around axis B', thus taking the form of a small tube through which the electron beam used passes (e.g., approximately 1 cm in diameter), but widened to accommodate structures such as source 4, sample holder H, screen 26, camera 30, camera 32, spectroscopic device 34, etc.
[0037] Now for reference Figure 2 This illustrates another embodiment of the device according to the invention. Figure 2(Not to scale) A highly schematic depiction of a charged particle microscope M according to the invention; more precisely, it shows an embodiment of a non-transmissive microscope M, which in this case is a SEM (although, in the context of the invention, it may simply be, for example, an ion-based microscope). In the figure, with Figure 1 The corresponding parts of the items in the diagram are indicated using the same reference numerals and will not be discussed separately here. (Attached to...) Figure 1 The following components are (especially): - 2a: Vacuum port, which can be opened to introduce articles (components, samples) into / remove articles (components, samples) from the interior of vacuum chamber 2, or, for example, to mount auxiliary equipment / modules. If desired, microscope M may include multiple such ports 2a; - 10a, 10b: Schematic depiction of the lens / optical element in illuminator 6; - 12: Voltage source, if necessary, to allow the sample holder H or at least the sample S to be biased (floated) to a potential relative to ground; - 14: Monitors, such as FPDs or CRTs; - 22a, 22b: Segmented electron detector 22a, which includes multiple independent detection segments (e.g., quadrants) arranged around a central aperture 22b (allowing beam B to pass through). Such detectors can be used, for example, to study the angular dependence of the output (secondary or backscattered) flux of electrons emitted from sample S.
[0038] This document also includes a controller 20. The controller is connected to a display 14, and the display 14 can be connected to a data processing device P arranged to perform the methods defined herein. In the illustrated embodiment, the data processing device P is a separate structure that does not constitute part of the controller, or even part of the microscope P. The data processing device P can be local or cloud-based, and is not limited to any particular location in principle.
[0039] Turn now Figure 3 A flowchart of method 100 as defined herein is shown. The method, implemented by data processing device P, includes the following steps: - Receive 101 images; - Provides 111 setpoints for the desired image quality parameters of the image; - The image 102 is processed using image analysis techniques to determine the current image quality parameters of the image 103; - Compare the current image quality parameters with the desired setpoint 111 103; and Based on the comparison, a modified image of 104 is generated using image modification techniques, wherein generating 104 includes the following steps: - If the current image quality parameter is lower than the set point, improve the image according to the image quality parameter 104a; and - If the current image quality parameter exceeds the set point, degrade the image by 104b according to the image quality parameter; and - Output the modified image described in 105.
[0040] The step of generating the 104 modified image may include using an artificial neural network (ANN) and / or a convolutional neural network (CNN). Other image modification techniques may also be used.
[0041] Figure 4 Another embodiment of the method as defined herein is shown. This embodiment is similar to... Figure 3 The embodiment shown includes an additional step of analyzing the modified image 106. This analysis can be performed using ANNs and / or CNNs and may include, for example, segmenting the modified image and / or identifying one or more objects within the modified image. The analysis may also include image reconstruction techniques.
[0042] like Figure 1 or Figure 2 As shown, the image received by the data processing device P can be provided by the charged particle microscope M. Other methods for providing images to the data processing device P are also conceivable.
[0043] In the example, Figure 5An example of how the method as defined herein operates is illustrated. Three input images 201-203 are shown here. The left image 201 has low image quality (displayed by low contrast, low sharpness, and low detail), the middle image 202 has medium image quality (medium contrast, medium sharpness, and medium detail), and the right image 203 has high image quality (high contrast, high sharpness, and high detail). The method as defined herein determines one or more image parameters from the input images 201-203 and then compares these one or more image parameters with desired target image quality parameters. In the method as defined herein, the desired image quality parameters correspond to the target, more moderate image quality parameter values. Each of the images 201-203 is processed by a data processing device and compared with the desired quality, and then image modification techniques are applied to generate an image, for example, with the target image quality. In the illustrated embodiment, the method is arranged to transform the left input image 201 into a medium quality image 211 by improving the quality with respect to contrast, sharpness, and detail. The method is also arranged to transform the input image 203 on the right into a medium-quality image 211 by degrading the quality with respect to contrast, sharpness, and detail. For the intermediate image 202, it is conceivable that image transformation techniques may not be applied if the determined quality parameters are unlikely to deviate from the desired quality parameters. Therefore, in an embodiment, the method includes the step of maintaining the input image 202 as the output image 211 if the determined image quality parameters are equal to or within a finite threshold of the desired image quality parameters. In other embodiments, the intermediate image 202 may still be transformed with respect to the quality parameters. In any case, the input images 201-203 will be processed and may be improved, degraded, and / or passed on to ultimately produce a (nearly) identical image 211.
[0044] Once the output image 211 is formed, it can be further analyzed, for example, using ANN and / or CNN. Figure 5 In this process, ANN and / or CNN can be used to identify particles 231-234 and their corresponding region boundaries 241-244. It should be noted that this can be done for each of the three input images 201-203. The resulting output image 211 should not be considered as the average of the three input images 201-203.
[0045] It should be noted that the methods defined herein are described with reference to images. The methods defined herein are applicable in principle to any 2D or 3D representation. In one embodiment, an image as defined herein may refer to an image obtainable by charged particle microscopy, including EM images, BSE images, spectral images such as EELS, etc.
[0046] The method has been described above with the aid of several non-limiting examples. The desired protection is determined by the appended claims.
Claims
1. A method implemented by a data processing device, comprising: - Receive images obtained from a microscope; - Provides a setpoint for the desired image quality parameters for the image; - The image is processed using image analysis techniques to determine the current image quality parameters; - Compare the current image quality parameters with the set point, and generate a modified image based on the comparison using image modification techniques, wherein the generation includes the following steps: - Improve the image based on the image quality parameter if the current image quality parameter is lower than the set point; and - Degrade the image according to the image quality parameter if the current image quality parameter exceeds the set point; and - Output and analyze the modified image, wherein the modified image is comparable to the set point for its image quality parameters, the analysis including the step of using an artificial neural network (ANN) and / or a convolutional neural network (CNN) on the modified image, the neural network having been trained with images whose respective image quality parameters are comparable to the set point, and wherein the image quality parameters include one or more parameters selected from the group consisting of: resolution, color depth, dynamic range, focus, sharpness, directional blur, contrast, white balance, and noise.
2. The method of claim 1, wherein the image modification technique includes the step of using an artificial neural network (ANN) and / or a convolutional neural network (CNN).
3. The method according to claim 1 or 2, wherein the set point for the desired image quality parameter corresponds to a medium image quality parameter value.
4. The method according to claim 1 or 2, wherein the setpoint corresponds to medium quality of the image quality parameter.
5. The method according to claim 1 or 2, wherein the analysis step includes identifying one or more objects in the image.
6. The method according to claim 1 or 2, wherein the image is obtained by a microscope, particularly a charged particle microscope.
7. The method according to claim 1 or 2, wherein the image quality parameter comprises one or more parameters selected from the group consisting of: Image resolution, image focus, image noise.
8. A non-transitory computer-readable medium having software instructions stored thereon, the software instructions causing the data processing device, when executed by the data processing device, to perform the method according to one or more of claims 1 to 7.
9. A charged particle beam device for examining a sample, comprising: - A sample holder for holding the sample in place; - A source used to generate beams of charged particles; - An illuminator for focusing the charged particle beam onto the sample; - A detector for detecting the radiation flux emitted from the sample in response to irradiation by the charged particle beam; and - Data processing equipment; The charged particle beam device is characterized in that it is arranged to perform the method as defined in one or more of the preceding claims 1 to 7.
10. A non-transitory computer-readable medium, wherein the non-transitory computer-readable medium stores software instructions thereon, the software instructions causing the data processing device to perform the method as defined in one or more of claims 1 to 7 when executed by the data processing device.