Method implemented by a data processing device and charged particle beam device for inspecting a specimen with such a method - Patents.com

Abstract

To provide a data processing method and a data processing device.SOLUTION: The invention relating to a method implemented by a data processing device includes the steps of: receiving an image; providing a desired set value of an image quality parameter of the image; and processing the image using an image analysis technique for determining the current image quality parameter of the image. In this method, the current image quality parameter is compared with the desired set value. Based on this comparison, a modified image is generated by using an image modification technique. The generation includes a step of improving the image with respect to the image quality parameter when the current image quality parameter is lower than the set value, and a step of degrading the image with respect to the image quality parameter when the current image quality parameter exceeds the set value. Next, the modified image is output.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The invention relates to a method implemented by a data processing apparatus The invention further relates to a charged particle beam device for inspecting a specimen using such a method.

[0002] Charged particle microscopy is a well-known and increasingly important technique for imaging small objects, especially in the form of electron microscopy.So far, the basic type of electron microscope has evolved into several well-known instrumental types, such as transmission electron microscope (TEM), scanning electron microscope (SEM) and scanning transmission electron microscope (STEM), and also into various auxiliary types, such as so-called "dual beam" instruments (e.g., FIB-SEM), which additionally employ a "machining" focused ion beam (FIB) to enable auxiliary operations such as ion beam milling or ion beam induced deposition (IBID).Those skilled in the art will be familiar with the different types of charged particle microscopy.

[0003] In an SEM, irradiation of a sample by a scanning electron beam stimulates emission of "auxiliary" radiation from the sample in the form of secondary electrons, backscattered electrons, X-rays, and cathodoluminescence (infrared, visible, and / or ultraviolet photons). One or more components of this emitted radiation can be detected and used for sample analysis.

[0004] In a TEM, an electron beam passes through the sample to form an image due to the interaction of the electrons with the sample as the beam passes through it. The image is then magnified and focused onto an imaging device, such as a fluorescent screen, a layer of photographic film, or a sensor such as a scintillator attached to a charge-coupled device (CCD). The scintillator converts the primary electrons in the microscope into photons that the CCD can detect.

[0005] Charged particle microscopy may generate images of the sample under investigation. In many cases, the obtained images need to be processed. The processing may include analyzing and / or manipulating the acquired images. For example, in SEM images of cell membranes of brain tissue, it is desirable to perform segmentation techniques on the images. For this task, artificial neural networks (ANNs) and / or convolutional neural networks (CNNs) may be used. Although reasonably robust to various imaging conditions, the quality of the segmentation still deteriorates if the data is noisy, out of focus, or in any other way affected by inappropriate instrument conditions. Naturally, this is undesirable as it may lead to subsequent errors, including incorrect medical decisions.

[0006] One way to overcome this degradation of analytical and / or operational results is to retrain the network using new conditions that account for these variations. However, this poses significant problems in that retraining the network is often not a valid option when the network is in a production (customer) environment, as retraining is expensive due to computation time and often requires expert input for proper labels.

[0007] Therefore, one of the biggest challenges of ANNs and CNNs for use in image processing remains that it is highly dependent on the stability of the imaging conditions and / or imaging parameters.

[0008] It is therefore an object of the present invention to provide a method that can be used to overcome one or more drawbacks with respect to the use of ANNs and / or CNNs in processing images.

[0009] To this end, a method implemented by a data processing device is provided. The method defined herein comprises the steps of receiving an image, providing settings of desired image quality parameters of the image, and processing the image using image analysis techniques to determine current image quality parameters of the image. The current image quality parameters are then compared to the desired settings. In other words, an image is provided and analyzed to determine image-related parameters and then ascertained whether they match the desired quality. Based on the results of the comparison, a modified image is generated by using image modification techniques. The image modification techniques may include the use of ANN and / or CNN.

[0010] As defined herein, the step of generating a modified image includes the steps of improving the image with respect to the image quality parameter if the current image quality parameter is lower than the set value, and degrading the image with respect to the image quality parameter if the current image quality parameter exceeds the set value. In other words, based on the result of the comparison, one of two actions is possible: in the first case, the image quality is lower than the desired value. In this case, image correction techniques are used to improve the image, ensuring that the image quality of the modified image is improved. In the second case, the image quality is higher than the desired value. In this case, image correction techniques are used to intentionally degrade the image, ensuring that the image quality of the modified image is actually reduced. The modified image is then output and analyzed by an ANN and / or a CNN, as defined herein. Following the above, the corresponding ANN and / or CNN used for the analysis is trained on a set of images, such that each image in the set of images has substantially the desired image quality parameter.

[0011] As an example, the method defined herein may include providing a substantially noise-free image, and the method defined herein may include degrading the image by adding noise to the image. In general, one or more of the following may be performed on what may be considered a high-quality image: reduce the resolution, reduce the color depth, reduce the dynamic range, degrade the focus, reduce the sharpness, add directional blur, reduce the contrast, and adjust the white balance. Similarly, one or more of the following may be performed on what may be considered a low-quality image: increase the resolution, increase the color depth, increase the dynamic range, improve the focus, increase the sharpness, remove directional blur, increase the contrast, and adjust the white balance. In this sense, the high-quality image is degraded to a medium-quality image, and the low-quality image is also improved to a medium-quality image. The medium-quality image may then be analyzed by an ANN and / or a CNN, where the ANN and / or the CNN have been substantially trained on the medium-quality image first.

[0012] The method may be performed on multiple images. An input image may be transformed into an image with target characteristics, which may be predefined, corresponding to a more moderate image quality setting. For example, input images that differ in quality, because they were created with different settings, are transformed into images with similar, more moderate characteristics. This includes improving some images and explicitly degrading other images. The final set of modified images has comparable image quality parameters and can be processed in a simpler and more effective way. The objective is then achieved.

[0013] The modified images are further analyzed using an ANN and / or a CNN, as defined herein. The methods defined herein allow for converting all images to the "medium quality" images on which the ANN and / or CNN were trained. Providing images of medium quality therefore reduces the variance of different input data to the ANN and / or CNN. The inventors have realised that rather than always improving the received images (e.g. by enhancing them in terms of reducing noise, improving focus, etc.), it is in fact advantageous to introduce a step of "worsening" the images, as the operations used are significantly simpler compared to improving the image quality. This has the beneficial consequence that the training data does not need to consist of the best possible images. Instead, the network can be trained on "medium quality" images, and the methods defined herein can be used to convert subsequently obtained images to the known conditions on which the original network was trained. As mentioned above, this involves either improving or degrading the image quality before providing the images to the ANN and / or CNN. In this way, proper operation of the ANN and / or CNN is ensured without the need for retraining the primary NN.

[0014] The method defined herein eliminates the need to retrain the neural network in the field. Instead, the neural network may be trained with a particular type of image, which may include images with more moderate image quality characteristics, instead of images with excellent image quality characteristics. The above method may then be used to convert the input images into modified images suitable for the neural network, which may process these images in the desired manner. As an added advantage, the image manipulation techniques used in the step of generating the modified images may be changed if it turns out that the transformed or modified images are not processed satisfactorily by the neural network. It is relatively simple and effective to modify the image manipulation techniques to ensure that the modified images are processed by the neural network, which alleviates the need to retrain the neural network. Thus, existing ANNs and / or CNNs can continue to be used by transforming the images input to the ANNs and / or CNNs instead of retraining the networks. This is a major advantage since no new training data or labels need to be collected and the NNs receiving the images may remain unchanged, while being deployed in embedded systems that may be very complex and difficult to retrain.

[0015] It should be noted that in the method defined herein, the transformation includes both enhancing and degrading the input image. The enhancing and degrading are with respect to one or more image parameters, which may include resolution, color depth, dynamic range, focus, sharpness, directional blur, contrast, white balance, and noise. Other image quality parameters are also contemplated. The image enhancement techniques used in the method may enhance and degrade the received image with respect to one or more of the above parameters. Those skilled in the art will be familiar with the appropriate parameters and algorithms used in these image enhancement techniques, etc.

[0016] As already mentioned above, the desired image quality parameter setting preferably corresponds to a medium image quality parameter value.

[0017] The method may include the further step of analysing the modified image, which may include using an artificial neural network (ANN) and / or a convolutional neural network (CNN).

[0018] Analyzing may include identifying one or more objects within the image.

[0019] The images may be provided to the data processing device in a number of ways: The images may be obtained from a non-transitory computer readable medium; The images may be obtained from a cloud computing network; The images may be obtained by a camera device connected to the data processing device.

[0020] In one embodiment, the image is obtained by a microscope, in particular a charged particle microscope, which may be an electron microscope.

[0021] According to one aspect, a non-transitory computer readable medium is provided having stored thereon software instructions which, when executed by a data processing apparatus, cause the data processing apparatus to perform the methods defined herein.

[0022] According to one aspect, there is provided a charged particle beam device for inspection of a specimen, comprising: a sample holder for holding a sample; a source for generating a beam of charged particles; - an illuminator for focusing the charged particle beam onto the sample; a detector for detecting a radiation flux emitted from said sample in response to said irradiation by said charged particle beam; -A data processing device.

[0023] As defined herein, a charged particle beam device is configured to carry out the methods defined herein.

[0024] The data processing device may be directly connected to the detector and receive data and / or images from the detector in a direct manner. An intermediate connection by an additional device between the detector and the processing device is also possible. For example, it is conceivable that the charged particle beam device comprises a controller configured to operate at least a part of the charged particle beam device. This controller may be connected or at least connectable to the detector and the data processing device and may be configured to transfer data (including images) from the detector to the processing device. The controller may be configured to process data emitted from the detector or may be configured to transfer raw data to the data processing device. Once the data is received, the data processing device can execute the method defined herein. In an embodiment, the controller includes the data processing device.

[0025] According to one aspect, there is provided a data processing apparatus configured to perform the methods defined herein. [Brief description of the drawings]

[0026] The invention will now be explained in more detail on the basis of exemplary embodiments and the accompanying schematic drawings. [Figure 1] 1 shows a longitudinal section of a charged particle microscope according to a first embodiment of the invention; [Diagram 2] 2 shows a longitudinal section of a charged particle microscope according to a second embodiment of the invention; [Diagram 3] 1 is a flow chart of a first embodiment of the method defined herein; [Figure 4] 4 is a flow chart of a further embodiment of the method defined herein. [Diagram 5] 1 is an illustration of a possible way of analysing and processing an image in the manner defined herein;

[0027] FIG. 1 (not to scale) is a highly schematic illustration of an embodiment of a charged particle microscope M according to an embodiment of the present invention. More specifically, an embodiment of a transmission microscope M, in this case a TEM / STEM, is shown (although in the context of the present invention an SEM (see FIG. 2) or, for example, an ion-based microscope may be equally valid). In FIG. 1, within a vacuum enclosure 2, an electron source 4 generates an electron beam B propagating along an electron optical axis B′ and traversing an electron optical illuminator 6, which serves to direct / focus the electrons onto a selected portion of a sample S (which may, for example, be (locally) thinned / flattened). A deflector 8 is also shown, which can be used to (among other things) effect a scanning movement of the beam B.

[0028] The sample S is held in a sample holder H that can be positioned in multiple degrees of freedom by a positioning device / stage A, which moves a cradle A' to which the holder H is (removably) fixed; for example, the sample holder H can include fingers that can move (among other things) in the XY plane (see the Cartesian coordinate system shown; typically, movements parallel to Z and tilts about X / Y are also possible). Such movements allow different parts of the sample S to be illuminated / imaged / inspected by the electron beam B traveling along an axis B' (in the Z direction) (and / or allow a scanning movement to be performed, as an alternative to beam scanning). If desired, an optional cooling device (not shown) can be in intimate thermal contact with the sample holder H to maintain the sample holder H (and the sample S thereon) at, for example, a cryogenic temperature.

[0029] The electron beam B interacts with the sample S in such a way as to cause it to emit various types of "stimulated" radiation, including (for example) secondary electrons, backscattered electrons, X-rays, and light radiation (cathodoluminescence). If desired, one or more of these types of radiation can be detected with the aid of an analysis device 22, which can be, for example, a combination of a scintillator / photomultiplier tube or an EDX or EDS (energy dispersive X-ray spectroscopy) module, and in such a case an image can be constructed using essentially the same principles as in an SEM. However, alternatively or supplementarily, electrons that traverse (pass) the sample S, emerge / emit from the sample S, and continue to propagate (substantially, but generally deflected / scattered to some extent) along the axis B' can be studied. Such a transmitted electron flux is incident on an imaging system (projection lens) 24, which generally includes various electrostatic / magnetic lenses, deflectors, correctors (stigmators, etc.), etc. In normal (non-scanning) TEM mode, this imaging system 24 can focus the transmitted electron beam onto a phosphor screen 26 which can be retracted / withdrawn, if desired, out of axis B'. An image (or diffractogram) of (a part of) the sample S is formed on the screen 26 by the imaging system 24, and this image can be viewed through a viewing port 28 located in a suitable part of the wall of the housing 2. The retraction mechanism of the screen 26 can be, for example, mechanical and / or electrical in nature, but is not shown in the drawings.

[0030] As an alternative to viewing an image on screen 26, one can instead take advantage of the fact that the focal depth of the electron flux leaving imaging system 24 is typically quite deep (e.g., about one meter). As a result, various other types of analytical devices (such as those listed below) can be used downstream of screen 26. -TEM camera 30. At the position of camera 30 the electron flux can form a still image (or diffractogram) which can be processed by controller / processor 20 and displayed on a display device 14, such as a flat panel display. When not required, camera 30 can be retracted / retracted (as shown diagrammatically by arrow 30') to move the camera off axis B'. - STEM camera 32. The output from camera 32 can be recorded as a function of the (X,Y) scanning position of beam B on sample S, and an image can be constructed that is a "map" of the output from camera 32 as a function of X,Y. Camera 32 can also be an Electron Microscope Pixel Array Detector (EMPAD), but can comprise a single pixel, e.g., 20 mm in diameter, as opposed to the pixel matrix typically present in camera 30. Furthermore, camera 32 generally can be a 10 mm pixel array, e.g., ... 2 Much higher acquisition rates (e.g., 10 images / sec) 6 points / sec). Again, when not needed, camera 32 can be retracted / withdrawn (as shown diagrammatically by arrow 32') to move the camera off axis B' (such retraction is not required, for example, in the case of a doughnut-shaped annular dark field camera 32, where a central hole allows the passage of the electron beam when the camera is not in use). As an alternative to imaging using the cameras 30 or 32, a spectroscopic device 34 can also be invoked, which can be, for example, an EELS module.

[0031] It should be noted that the order / position of components 30, 32, and 34 is not strict and many possible variations are possible. For example, spectroscopic device 34 could be integrated with imaging system 24.

[0032] In the embodiment shown, the microscope M further includes a retractable X-ray computed tomography (CT) module, generally indicated by reference 40. In computed tomography (also referred to as tomographic imaging), a source and a detector (at opposite poles) are used to interrogate a sample along different lines of sight to obtain transmitted views of the sample from various viewpoints.

[0033] It is noted that a controller (computer processor) 20 is connected to the various components shown via control lines (bus) 20'. This controller 20 may provide various functions such as synchronizing actions, providing set points, processing signals, performing calculations and displaying messages / information on a display device (not shown). Needless to say, the controller 20 (shown diagrammatically) may be (partially) inside or outside the housing 2 and may have a unitary or composite structure as desired. The controller comprises, as shown in this embodiment, a data processing device P arranged to carry out the method defined herein.

[0034] Those skilled in the art will understand that the interior of the housing 2 does not have to be kept in a strict vacuum. For example, in so-called "environmental TEM / STEM", a background atmosphere of a given gas is intentionally introduced / maintained in the housing 2. Those skilled in the art will also understand that in practice it may be advantageous to limit the volume of the housing 2, so that, if possible, the housing 2 essentially encloses the axis B' and takes the form of a small diameter tube (e.g., about 1 cm in diameter) through which the electron beam employed passes but which expands to accommodate structures such as the source 4, the sample holder H, the screen 26, the camera 30, the camera 32, the spectrometer 34, etc.

[0035] Referring now to Figure 2, another embodiment of the device according to the invention is shown. Figure 2 (not to scale) is a very schematic representation of a charged particle microscope M according to the invention, more specifically showing an embodiment of a non-transmission microscope M, in this case an SEM (although in the context of the invention, for example, an ion-based microscope may be equally useful). In the figure, parts corresponding to items in Figure 1 are indicated using the same reference numbers and are not described separately here. The following parts (among others) are added to Figure 1: -2a: a vacuum port that can be opened to introduce / remove items (components, samples) into / from the interior of the vacuum chamber 2 or onto which, for example, auxiliary devices / modules can be mounted. The microscope M can be equipped with multiple such ports 2a, if necessary. - 10a, 10b: lenses / optical elements within the illuminator 6 shown diagrammatically. -12: Optionally, a voltage source that allows the sample holder H or at least the sample S to be biased (floated) to a certain potential with respect to ground. -14: Display such as FPD or CRT. 22a, 22b: a segmented electron detector 22a, comprising a number of independent detection segments (e.g., quadrants) arranged around a central aperture 22b (allowing the passage of the beam B). Such a detector can be used, for example, to investigate the (angular dependence of) the output (secondary or backscattered) electron flux emanating from the sample S.

[0036] Here too, there is a controller 20. The controller is connected to a display 14, which in turn may be connectable to a data processing device P arranged to carry out the methods defined herein. In the embodiment shown, the data processing device P is a separate structure that does not form part of the controller or even part of the microscope P. The data processing device P may be local or cloud-based and is in principle not limited in location.

[0037] Turning now to Figure 3, there is shown a flow chart of a method 100 defined herein. The method, implemented by a data processing device P, comprises: a step 101 of receiving an image; - a step 111 of providing settings for desired quality parameters of said image; - processing 102 the image using image analysis techniques to determine 103 current image quality parameters of said image; a step 103 of comparing the current image quality parameters with the desired settings 111; - generating 104 a modified image by using an image modification technique based on said comparison, - a step 104a of improving the image with respect to the image quality parameter if the current image quality parameter is lower than the set value; - a step 104b of degrading the image with respect to the image quality parameter if the current image quality parameter exceeds the set value; - outputting 105 said modified image.

[0038] The step of generating a modified image 104 may include using an artificial neural network (ANN) and / or a convolutional neural network (CNN). Other image modification techniques may be used as well.

[0039] Figure 4 shows a further embodiment of the method defined herein. This embodiment is similar to the embodiment shown in Figure 3, but includes the further step 106 of analyzing the modified image. The analyzing can be performed using ANN and / or CNN and can include, for example, segmenting the modified image and / or identifying one or more objects in the modified image. The analysis can also include image reconstruction techniques.

[0040] The image received by the data processing device P may be provided by a charged particle microscope M, as shown in Figure 1 or Figure 2. Other ways of providing the image to the data processing device P are also conceivable.

[0041] FIG. 5 shows, by way of example, an example of how the method defined herein works. Here, three input images 201-203 are shown. The left image 201 is of low quality (shown with low contrast, low sharpness, low detail), the center image 202 is of medium quality (medium contrast, medium sharpness, medium detail) and the right image 203 is of high quality (high contrast, high sharpness, high detail). The method defined herein can determine one or more image parameters from the input images 201-203 and then compare these one or more image parameters with a desired target image quality parameter. In the method defined herein, the desired image quality parameter corresponds to a target, more medium image quality parameter value. Each of the images 201-203 is processed by a data processing device and compared with the desired quality and then image correction techniques are applied to generate, for example, an image with the target image quality. In the embodiment shown, the method is configured to convert the left input image 201 into a medium quality image 211 by improving its quality in terms of contrast, sharpness and detail. The method is also configured to transform the right input image 203 into a medium quality image 211 by degrading its quality with respect to contrast, sharpness and details. For the central image 202, whose determined quality parameters may not deviate significantly from the desired quality parameters, it is conceivable that no image transformation techniques have been applied. Thus, in one embodiment, the method comprises a step of keeping the input image 202 as the output image 211 if the determined image quality parameters are equal to or within a limited threshold of said desired image quality parameters. In other embodiments, the central image 202 may nevertheless be transformed with respect to said quality parameters. In any case, the input images 201-203 may be processed, improved, degraded and / or approved, ultimately resulting in a (substantially) identical image 211.

[0042] Once the output image 211 is formed, further analysis may be performed on the output image 211, for example, using an ANN and / or a CNN. In FIG. 5, an ANN and / or a CNN may be used to identify the particles 231-234 and corresponding region boundaries 241-244. Note that this may be done for each of the three input images 201-203. The resulting output image 211 should not be considered to be an averaged image 211 of the three input images 201-203.

[0043] It should be noted that the methods defined herein are described with reference to images. The methods defined herein are in principle applicable to any 2D or 3D representation. The images defined herein may in one embodiment relate to images obtainable by charged particle microscopy, including EM images, BSE images, spectral images such as EELS, etc.

[0044] The method has thus been described by means of some non-limiting examples. The desired protection is determined by the appended claims.

Claims

1. 1. A method implemented by a data processing apparatus, comprising: - receiving an image; - providing desired quality parameter settings for said image; - processing said image using image analysis techniques to determine current image quality parameters of said image; - comparing said current image quality parameters with settings of said desired image quality parameters and generating a modified image based on said comparison by using image modification techniques, - improving said image with respect to said image quality parameter if said current image quality parameter is lower than said set value; - degrading the image with respect to the image quality parameter if the current image quality parameter exceeds the set value; - outputting and analysing the modified image, said analysing comprising using an artificial neural network (ANN) and / or a convolutional neural network (CNN) on the modified image; A method, wherein the image quality parameters comprise one or more parameters selected from the group consisting of image focus, image noise.

2. The method of claim 1 , wherein the image correction technique includes using an artificial neural network (ANN) and / or a convolutional neural network (CNN).

3. The method of claim 1 or 2, wherein the setting of the desired image quality parameter corresponds to an intermediate image quality parameter value.

4. The method of any one of claims 1 to 3, wherein the image quality parameters comprise one or more parameters selected from the group consisting of resolution, color depth, dynamic range, focus, sharpness, directional blur, contrast, white balance, and noise.

5. The method according to any one of claims 1 to 4, wherein said set value corresponds to a medium quality of said image quality parameter.

6. The method of any one of claims 1 to 5, wherein the analyzing step comprises identifying one or more objects within the image.

7. The method according to any one of claims 1 to 6, wherein the image is obtained by microscopy.

8. A method according to any one of claims 1 to 6, wherein the image is obtained by a charged particle microscope.

9. A non-transitory computer readable medium having stored thereon software instructions which, when executed by a data processing apparatus, cause said data processing apparatus to perform the method according to one or more of claims 1 to 8.

10. 1. A charged particle beam device for inspecting a specimen, comprising: a sample holder for holding a sample, a source for generating a beam of charged particles; an illuminator for focusing said charged particle beam on said sample; a detector for detecting a radiation flux emitted from said sample in response to irradiation by said charged particle beam; a data processing device, Charged particle beam device, characterized in that it is adapted to carry out the method defined in one or more of claims 1 to 8.

11. Data processing device, characterized in that it is arranged to carry out the method defined in one or more of claims 1 to 8.

Images