Systems and methods for sample analysis

Abstract

A new method and apparatus for improving the (physical or chemical) analysis of an object is described. An electronic file of a digital twin constituting a sample to be analyzed is generated using photogrammetry and computer-readable fiducial markers. This contributes to communicating and discussing where the sample is analyzed, and allows a variety of analysis techniques to be applied using a common coordinate system, thereby facilitating correlation microscopy studies. Furthermore, we propose a method and software, referred to as PARS (Portable Analytical Registration Software), allowing points defined on one computer-operated imaging instrument to be easily found on another computer-operated imaging instrument without accessing or altering the software running each instrument. The method enables correlating images from a variety of surface imaging techniques to provide unprecedented levels of surface detail that cannot be provided alone by a single technique, possibly reaching nanoscale.

Description

Field of the Invention

[0001] The present disclosure relates generally to analysis of samples, particularly chemical and physical analysis of samples, including imaging and spectroscopy. Background Art

[0002] In analytical laboratories, it is often necessary to describe a small sample, specimen, or artifact in order to communicate a location on that sample (e.g., to discuss chemical analysis results via e-mail). Examples of samples include, but are not limited to: a small piece of catalyst material that will be subjected to electron microscopy or microanalysis, a piece of meteorite material studied by microscopy and x-ray analysis, and a tooth from an extinct human species found in a cave. In this document, the terms "specimen" and "sample" will be used interchangeably. Similarly, the terms "analytical instrument" and "microscope" will be used interchangeably.

[0003] In science, there are different kinds of analytical instruments, such as microscopes. Some microscopes use light to form an image, some use electrons, some may use X-rays to form an image, and so on. Some microscopes only output an image (i.e., a micrograph), while others (such as imaging X-ray photoelectron spectroscopy) output a chemical image of the sample or a spectrum of surface chemical features. These analytical instruments are usually very expensive, but have been developed over the years to be able to provide micrographs quickly. Some of these microscopes operate in air, while others require a vacuum or ultra-high vacuum (UHV). Some can accommodate very large samples, while others can only accommodate very small samples. Typically, modern analytical instruments are run by desktop computers, and the operating system is often Less often Examples of such microscopes are Figures 5 to 7 and Figures 11 to 15 Each microscope operates using different physical methods and therefore produces images with very different contrast mechanisms. Figure 5 , 6 As shown in the analysis instrument diagrams in , 7, 11, 12, 13, 14 and 15, today spatially resolved analysis (i.e., analysis of a specific point or area) is usually performed using a microscope or other instrument connected to and operated by a computer. Fig.16 and Fig.17 The computer screens for operating a scanning electron microscope (SEM) and an X-ray photoelectron spectroscopy (XPS) analysis instrument are shown, respectively. In both images, an image of the sample surface is shown ( Fig.16 Large particle panels in Fig.17 ). In both cases, the operator selects a point in this sample view, clicks the mouse, and acquires its spectrum (energy dispersive X-ray (EDS) and XPS, respectively) using that location.

[0004] Thus, analytical techniques that may be beneficially applied to the present invention include, but are not limited to, AES - Auger electron spectroscopy, AFM - atomic force microscopy, ARPES - angle resolved photoelectron spectroscopy, ARUPS - angle resolved ultraviolet photoelectron spectroscopy, CARS - coherent anti-Stokes Raman scattering spectroscopy, CET - cryo-electron tomography, Cryo-EM - cryo-electron microscopy, Cryo-SEM - cryo-scanning electron microscopy, EBIC - electron beam induced current, EBSD - electron backscatter diffraction, EDAX - energy dispersive X-ray analysis, EDS or EDX - energy dispersive X-ray spectroscopy, EELS - electron energy loss spectroscopy, ESCA - electron spectroscopy chemical analysis ( See XPS), ESEM – environmental scanning electron microscopy, ESTM – electrochemical scanning tunneling microscopy, EXAFS – extended X-ray absorption fine structure, FCS – fluorescence correlation spectroscopy, FCCS – fluorescence cross-correlation spectroscopy, FEM – field emission microscopy, FIB – focused ion beam microscopy, FLIM – fluorescence lifetime imaging, fluorescence microscopy, FRET – fluorescence resonance energy transfer, GIXR – grazing incidence X-ray reflectivity, HAS – helium atomic scattering, HREELS – high resolution electron energy loss spectroscopy, HREM – high resolution electron microscopy, HRTEM – high resolution transmission electron microscopy, HE-PIXE – high energy proton induced X-ray emission, IAES – ion induced Auger electron spectroscopy, ISS – ion scattering spectroscopy, LEED – low energy electron diffraction, LEEM – low energy electron microscopy, LEIS – low energy ion scattering, MALDI – matrix assisted laser desorption ionization, MEIS – intermediate energy ion scattering, MFM – magnetic force microscopy, MRFM – magnetic resonance force microscopy, Micro-CT – micro-computed tomography, MRI – magnetic resonance imaging, NEXAFS – near edge X-ray absorption fine structure, NSOM – near field optical microscopy, PhD – photoelectron diffraction, PED – photoelectron diffraction, PEELS – parallel electron energy loss spectroscopy, PEEM – photoelectron emission microscopy (or photoelectron emission microscopy), PES – photoelectron spectroscopy, PIXE – particle (or proton) induced X-ray spectroscopy, RBS – Rutherford backscattering spectroscopy, RHEED – reflection high-energy electron diffraction, SAXS – small-angle X-ray scattering, SCANIIR – surface composition by analysis of neutral particle and ion impact radiation, SCEM – scanning confocal electron microscope, SE – spectroscopic ellipsometry, SEM – scanning electron microscope, SERS – surface enhanced Raman spectroscopy, SERRS – surface enhanced resonance Raman spectroscopy, SIMS – secondary ion mass spectrometry, SNMS – sputtered neutral particle mass spectrometry, SNOM – scanning near-field optical microscopy, SPM – scanning probe microscopy,STEM - Scanning Transmission Electron Microscope, STM - Scanning Tunneling Microscope, STS - Scanning Tunneling Spectroscopy, SXRD - Surface X-ray Diffraction, TEM - Transmission Electron Microscope, TOF-MS - Time-of-Flight Mass Spectrometry, Two-Photon Excitation Microscope, TXRF - Total Reflection X-ray Fluorescence Analysis, UPS - Ultraviolet Photoelectron Spectroscopy, XAES - X-ray Induced Auger Electron Spectroscopy, XANES - X-ray Near Edge Absorption Fine Structure (synonymous with NEXAFS), XAS - X-ray Absorption Spectroscopy, XPEEM - X-ray Photoelectron Emission Microscope, XPS - X-ray Photoelectron Spectroscopy and XRF - X-ray Fluorescence Analysis.

[0005] Increasingly, large and expensive analytical or processing instruments are applied to samples, both at universities and in industry. These universities or industrial institutions are often called "central facilities" and are usually staffed with permanent personnel to run and operate these analytical instruments. For example, there may be 20 different analytical instruments in a central facility, each costing between $1 million and $10 million. Each university or industrial institution may not be able to afford to purchase all of these analytical tools - there are usually only two or three such "central facilities" in the entire country, each with hundreds of users, each of which sends their catalyst samples, meteorites, teeth, or other samples to the facility for analysis or modification. Examples of instruments in these facilities include electron microscopes, X-ray photoelectron spectrometers, imaging mass spectrometers, and some other instruments in the above list.

[0006] This disclosure does not concern instruments used to analyze homogenous or liquid samples, such as nuclear magnetic resonance (NMR) analysis of most solutions, or analysis of blood samples, or analysis of contaminated river water samples. These types of homogenous samples are usually best managed using a "LIMS" or laboratory information management system, in which each sample is assigned a unique identifier.

[0007] We are concerned with those sample types where it is necessary to know where in the sample the analysis is being performed, rather than just a unique identifier for the entire sample, and with the instruments used to perform these spatially resolved analyses. One example is the scanning electron microscope. Many scanning electron microscopes will image secondary electrons (providing the familiar monochromatic high resolution images of surface topography), but also allow the X-rays produced at the sample surface by the incident electrons to be analyzed, giving "energy dispersive X-ray analysis" or EDS. Chemical elements can be identified from their characteristic X-rays. The point of the sample being imaged at any particular moment is the same point from which the X-rays were emitted, so the chemical image and the secondary electron image can be superimposed, providing both topographic and analytical information. Another example is X-ray photoelectron spectroscopy, an example of an analytical purpose such as Fig.10As shown, we can see that two different analysis positions (P1 and P2) on the same sample produce different spectra and therefore different analysis results in terms of percent composition.

[0008] Often, more than one type of microscope is needed to resolve most problems. For example, some features may be visible in a light microscope (e.g. Figure 6 ), but then the sample needs to be taken to an electron microscope (as Fig.12 In another embodiment, the sample may first be imaged using a chemically specific microscope (e.g., imaging X-ray photoelectron spectroscopy, XPS, system, such as Fig.15 In these cases, researchers currently have two options. The first is to buy or build a specialized combined microscope capable of performing both techniques on a single sample; the second is to transfer the sample between two or more microscopes while keeping track of the region of interest so that images from multiple techniques can be compared with certainty.

[0009] The first option is often expensive, with the combined microscope costing more than the individual microscopes. In some cases, the throughput and performance of the combined microscope is often compromised, and combined microscopes tend to be more difficult to use and may require individuals to be trained in both techniques. Nonetheless, the first option can still be useful in certain situations. Fig.21 Schematically shows A combined optical / SEM instrument from the company Delft, the Netherlands. Specifically, Fig.21 Schematic showing how an optical microscope and an electron microscope can be combined into a single instrument, allowing the user to image a portion of a sample in both the optical microscope and the electron microscope almost simultaneously with good image registration. Such instruments are typically expensive to build, and when one part (e.g., the electron microscope) fails, it often renders the other part unusable. The present invention provides an alternative to this. As another example of a combined, multi-technique instrument, The company (Münster, Germany) provides the M6 The in-situ combination of MRI and SPM uses a precise piezoelectric stage to move the sample between each analytical position. and The instruments are much more expensive.

[0010] The second option is generally cheaper, taking advantage of existing investments in separate microscopes and training, but the problem with such correlative microscopy is ensuring that the same area is analyzed in both microscopes, and identifying specific points that need to be analyzed when those points are sharp in the first analytical instrument but not in the second. It is this second option, and the problem of registering one image with another, that is of interest here and to which the present disclosure greatly aids.

[0011] A particularly common scenario for the second option is when two techniques are applied to a sample, one of which gives a good image (e.g., optical microscopy or electron microscopy), and the researcher then wants to perform a chemical analysis of a specific feature within that field of view[^2]. For example, an atomic force microscope (AFM) image might show particles on the surface of a sample, and the researcher might want to analyze the chemical composition of one of those particles because it has a shape that is different from the others. At this point, it might be necessary to take the sample to an instrument with chemical analysis capabilities to see what that particle is made of. In these cases, one or more “point analyses” are typically performed at points of interest (POIs), perhaps in a Raman spectrometer or using energy dispersive X-ray analysis in a scanning electron microscope (SEM). The problem here is how to identify points (or regions) of interest, once they have been identified by imaging in instrument A, within the accessible area of ​​instrument B when the coordinate systems of the two instruments (i.e., the x, y, and z of their samples) are completely different.

[0012] As an example of a specific application, consider the following situation. Suppose a researcher sees a lesion on the tooth of a human ancestor and sends the tooth off for analysis to find the chemical composition of the trace material within the lesion. Alternatively, a failed catalyst may show spots of unknown brown material - at this point, finding out the chemical nature of the brown material becomes important to correct the problem that caused the catalyst to fail. So we have a situation where many users need to perform spatially resolved analysis on a sample. Furthermore, the instrumentation to perform the analysis is usually very expensive and therefore located in a central facility and often not at the same location as the researcher. Furthermore, analyzing every single point on the surface of the sample is slow and expensive, so only the points or regions (POIs or ROIs, respectively) that the researcher is interested in should be analyzed. Therefore, the researcher must bring (or send) the sample to the staff at the central facility and describe where on the sample they want the analysis to be performed.

[0013] Such descriptions are often difficult at the resolution required. A researcher who excavates a human tooth might look at it under a light microscope and identify a spot of interest for analysis (such as an ancient lesion). The researcher then sends the tooth to a central facility along with a photograph taken under the microscope, marking the spot to be analyzed on the photo, like an "X" on a pirate's treasure map.

[0014] Another difficulty that arises is that samples look different in an electron microscope than in an optical microscope. Orientation and scale may be different, of course. More importantly, the sources of image contrast are completely different. Sometimes it is easy to find the location to be analyzed, sometimes it is extremely difficult even with a photograph. Even tooth enamel that looks uniformly white in an optical microscope may show chemical variations in a scanning electron microscope (SEM) image, leading to disorientation. Conversely, even if the chemical composition of a meteorite surface is fairly uniform, there may be differences in its color, so that marks visible to the naked eye are not visible at all in the SEM or are obscured by the topographic contrast. Even the researchers themselves may have difficulty navigating on the sample to the location they want to analyze. As a result, time with expensive instruments is wasted. Despite the application of high technology, researchers often have to resort to primitive and inaccurate methods to locate the point to be analyzed. For example, making a cross on the sample with a needle, or determining the location based on nearby random features that look similar in both optical and electron microscopes (even though they may only look similar by chance, potentially leading to errors). For example, "2 mm up from something that looks like a fish, 1 mm to the left from something that looks a bit like a map of Australia".

[0015] A fine annotated grid, perhaps made of copper, could be placed over the sample, with each square hole of the grid marked with tiny letters. The grid itself is easily visible in both optical and electron microscopes. But it can get in the way, can alter the surface it touches, and is difficult to hold in place without glue (which also changes the surface chemistry). So a dynamic system is needed that can address the above issues. There are approaches that can help, or may help in the future, but they have serious problems of their own. One solution for some situations in the future (although I have not yet seen it in practice) is to use a “digital twin” for manufactured components. A digital twin is a virtual representation that serves as a real-time digital counterpart to a physical object or process. In design and manufacturing, a digital twin is a digital representation that is similar to (or contains) a computer-aided design (CAD) file for a component. Take a stainless steel bolt, for example. If a brown spot of corrosion is seen on a bolt made in the factory, the xyz coordinate system of the digital twin can be referenced to communicate the location on the bolt. The coordinates of that brown spot can be added to the digital twin file before the bolt itself is sent to a central facility. Even without having to match an optical photograph, staff at a central facility could (at least in theory) use the digital twin information to point the analytical instrument directly to the spot with the brown stain. In practice, there is currently no software infrastructure to make this possible, even for small manufactured components. In summary, most samples submitted for analysis are either natural products (e.g. meteorites) for which no digital twin has ever existed, or are cut or broken pieces from larger items (e.g. a few square centimeters of building cladding material that needs to be chemically analyzed to assess its fire resistance), for which digital twins either do not exist or cannot be digitally sampled in the same way as the real sample was created. To date, to my knowledge, these issues have not been addressed anywhere using CAD-generated digital twins, although they may be in a few cases in the future. A slight exception is the special case of semiconductor chips, where “fiducial marks” such as crosses are built into the design along with the transistors on the chip, designed to be visible under optical and other microscopes. The analysis points can be defined on this flat chip surface (perhaps using electron microscopy and X-ray methods) with reference to the chip’s CAD layout. These points can be found based on coordinates given by the fiducial marks[3] seen under the microscope. But all of this is in two dimensions. This is fine for semiconductor chips, but most samples are three-dimensional objects with a large number of topography and three-dimensional shapes.

[0016] Existing methods

[0017] Often, incidental features visible in both microscopes (e.g., a specific random dust pattern on a surface) can be used to identify regions common to both microscopes, and one of the micrographs can then be zoomed, translated, and reoriented to match the region seen in the other micrograph, potentially allowing the images to be superimposed. Sometimes, fiducial marks can be added to provide points that can be used for subsequent registration (e.g., a small cross carefully made with a sharp scalpel). Sometimes, annoyingly, even when these fiducial marks are present in both images, they can look very different. This is due to the different contrast mechanisms that occur in the different imaging modalities. Thus, in the worst cases, registration becomes quite subjective, and co-registration can be time-consuming to get right, or to be repeated or reproduced in another lab. Even when the registration problem is clear, there is often no way to record spectra or other information associated with specific locations on the specimen, other than marking the locations on the photographs (e.g., #1, #2, etc.) and correlating these numbers with the spectral file names in a lab notebook. This makes it very difficult to discuss this with a client or collaborator, for example, over a remote conference.

[0018] Many methods have been developed for the registration problem. They are either difficult to implement, time consuming, or are proprietary methods developed by a particular instrument manufacturer that “lock you in” to using their hardware. A single instrument manufacturer may produce multiple types of instruments, such as an SEM and an AFM. They may serve their customers by providing an easy way to transfer samples between the SEM and AFM, perhaps using a universal sample carrier design with specially designed fiducial markers and / or special proprietary software running on a monitoring PC. However, many users inevitably have multiple instruments from a variety of instrument manufacturers, and a proprietary solution from one company (such as Company P) is often completely different and incompatible with the method sold by another company (such as Company Q). Furthermore, neither Company P nor Company Q may produce a Raman spectrometer (for example), so none of these proprietary solutions to the registration problem will help the researcher when he or she needs to perform a point analysis on the sample surface.

[0019] We (myself and a former PhD student in the UK) have recently published a research paper describing a method for image registration using computer-readable fiducial markers[^13], specifically including information about how these markers were made, for example by focused ion beam (FIB) milling. This allows software running on a computer to identify the QR code-like fiducial markers in different microscope images and then overlay them. The computer-readable fiducial markers used are called [^4], originally developed for robotic computer vision (although QR codes and other markers can also be used). To our knowledge, we are the first applied to the problem of microscopy image registration. In an effort to make this approach more useful and more widely applicable, I was inspired by the "screen annotation software" available on personal computers.

[0020] Some existing technologies

[0021] There are many publications on the problem of correlative microscopy research on the registration problem [12,13,21,22]. The Thermo Scientific “CISA” workflow

[23] achieves correlative registration between SEM and XPS by mechanical registration based on a large proprietary sample holder in both instruments. It locks the coordinates of the sample stage between the two instruments and is limited to two dimensions (2D). It does not use CRFM (described below) and is therefore limited to analytical tools from a single manufacturer and its proprietary software system. It is unlikely that a single manufacturer will produce optimal analytical instruments for all the techniques a user requires, so if the manufacturer’s tools are beyond their range, the user can only mark an “X” on the image to find the point of interest (POI) in the next instrument. Since out-of-plane CRFM (such as Figure 1 As shown in Figure 2, the Correlated Imaging and Surface Analysis (CISA) workflow

[23] is limited to two-dimensional sample navigation. The CISA workflow has no equivalent to a DAT scanner and therefore requires the sample to be loaded into an expensive XPS instrument before the work can begin.

[0022] The concept of computer-readable fiducial markers and their advantages have been recently published

[13] . Many other types of fiducial markers have been tried over the years

[21] , but they often require manual identification and registration. They often overcome the problem of different contrast mechanisms in different types of microscopes through special choices of fiducial marker materials (e.g., fluorescent nanoparticles that can be easily identified in both light and electron microscopes

[24] ), but even if they are clearly visible in all microscopes, manual registration is still required.

[0023] Screen Annotation Software

[0024] There are various software packages (some free, some commercial) that allow the user to write on anything on the PC screen - in effect providing the opportunity for annotation. This works even when the application window being displayed is "live", i.e., running on that PC. The effect is like writing on a piece of cellophane in front of the computer screen running the software - for any software. Such screen annotation software is very useful for teachers and lecturers, as they can write on the screen. Imagine, for example, explaining a piece of computer-aided design (CAD) software; being able to circle specific buttons, draw arrows to specific parts of the design, etc., without affecting the CAD software itself. SUMMARY OF THE INVENTION

[0025] A scanning system, method, and computer program product focused on sample analysis are provided herein.

[0026] In one aspect, a scanning system for sample analysis is disclosed. The scanning system includes a processor and a memory connected to the processor in communication, wherein the memory stores a plurality of processor executable instructions, which, when executed by the processor, cause the processor to control a plurality of image capture devices to capture a first set of images of a sample and at least one fiducial marker, wherein the sample is placed on a sample holder, and at least one fiducial marker is associated with at least one of the sample or the sample holder. The scanning system is further configured to generate a three-dimensional model (3D model) of the sample based on applying one or more photogrammetric techniques to the captured first set of images, wherein the one or more photogrammetric techniques capture information of a first coordinate system associated with the generated 3D model. The scanning system is also configured to generate a data structure associated with the sample based on the generated 3D model. The scanning system is further configured to output the generated data structure, which includes a first coordinate system associated with the generated 3D model, a second coordinate system associated with the sample, and a correspondence between the first coordinate system and the second coordinate system.

[0027] In other embodiments, the scanning system is configured to control the movement of the sample holder to rotate from the first position to the second position. The scanning system is further configured to control the plurality of image capture devices to capture a second set of images of the sample and at least one fiducial marker when the sample holder is in the second position. The scanning system is further configured to generate a 3D model of the sample based on applying one or more photogrammetric techniques to the captured first set of images and the second set of images.

[0028] In other embodiments, the generated data structure corresponds to an Extensible Markup Language (XML) file.

[0029] In other embodiments, the at least one fiducial marker corresponds to one of: a Quick Response (QR) code, a barcode, ARtag or ArUco marker.

[0030] In other embodiments, the scanning system is configured to generate a second coordinate system associated with the sample based on the at least one fiducial marker.

[0031] In other embodiments, the generated data structure includes information related to a first region of interest (RoI) of the sample for analysis using a first analytical instrument integrated into the scanning system or using a second analytical instrument having a different coordinate system than the first analytical instrument.

[0032] In other embodiments, the scanning system is configured to receive a first user input related to marking at least one point of interest on the generated 3D model. The scanning system is further configured to store information related to the marking of the at least one point of interest in a generated data structure based on the received first user input, and output the generated data structure.

[0033] In other embodiments, at least one point of interest is marked for analysis under one or more analytical instruments.

[0034] In other embodiments, the sample corresponds to a heterogeneous sample.

[0035] In one aspect, a method for analyzing a sample is provided. The method includes rendering a first image of a region of interest (RoI) of the sample on a first analytical instrument, wherein the rendered first image includes at least one fiducial marker and is captured by the first analytical instrument. The method also includes receiving a second user input from the first analytical instrument associated with selecting a point of interest within the rendered first image. The method also includes determining position information associated with the selected point of interest based on the received second user input, wherein the determined position information includes the position of the selected point of interest relative to the at least one fiducial marker. The method also includes storing the determined position information in a data structure. The method also includes receiving a third user input associated with rendering a second image of the region of interest on a second analytical instrument, wherein the second analytical instrument is different from the first analytical instrument. The method also includes controlling the second analytical instrument to scan the stored data structure to determine the position of at least one fiducial marker in the second image based on the received third user input. The method also includes applying at least one transformation technique to the position information stored in the data structure based on the scanning. The method also includes rendering the first image of the selected point of interest on the second analytical instrument based on the application of the at least one transformation technique.

[0036] In other method embodiments, a first image of the selected point of interest is captured by a second analysis instrument.

[0037] In other method embodiments, at least one transformation technique applied comprises a projective geometry transformation technique.

[0038] In other method embodiments, the method includes scanning the rendered first image to determine a first position of at least one fiducial marker in the rendered first image, and determining position information associated with the selected point of interest based on the scanning of the first image.

[0039] In other method embodiments, the method includes receiving a fourth user input related to the selected point of interest, wherein the received fourth user input includes a first tag and first information related to the selected point of interest, and based on the received fourth user input, storing the first tag and the first information related to the selected point of interest in a data structure.

[0040] In other method embodiments, the data structure corresponds to a digital analytical twin (DAT) data structure associated with the sample and is a digital replica of the sample.

[0041] In other method embodiments, the sample corresponds to a heterogeneous sample.

[0042] In other method embodiments, the fiducial marker corresponds to a quick response (QR) code, a barcode, One of the ARtag or ArUco tags.

[0043] In one aspect, a method for analyzing a sample is provided. The method includes controlling a plurality of image capture devices to capture a first set of images of a sample and at least one fiducial marker, wherein the sample is placed on a sample holder, and at least one fiducial marker is associated with at least one of the sample or the sample holder. The method further includes generating a three-dimensional model (3D model) of the sample based on applying one or more photogrammetric techniques to the captured first set of images, wherein the one or more photogrammetric techniques capture information of a first coordinate system associated with the generated 3D model. The method also includes generating a data structure associated with the sample based on the generated 3D model. The method further includes generating a data structure associated with the sample based on the generated 3D model.

[0044] In other method embodiments, the method includes controlling the movement of a sample holder to rotate from a first position to a second position, wherein the sample is placed on the sample holder. The method further includes controlling a plurality of image capture devices to capture a second set of images of the sample and at least one fiducial marker when the sample holder is in the second position. The method further includes generating a 3D model of the sample based on applying one or more photogrammetric techniques to the captured first set of images and the second set of images.

[0045] In other method embodiments, the fiducial marker corresponds to one of: a quick response (QR) code, a barcode, ARtag or ArUco marker.

[0046] The present invention includes a Portable Analytical Registration Software (PARS) system, which is a third party system that runs in parallel with (but not as part of) the software running the analytical instrument and preferably records information into a Digital Analytical Twin (DAT) data structure for the sample in question. The Digital Analytical Twin (DAT) data structure is acquired via a DAT scanner, a hardware device used to create a DAT data structure from a physical sample.

[0047] PARS Software System

[0048] As mentioned above, the PARS system may be a third-party system that runs in parallel with (but not as part of) the software running the analytical instrument. An important feature of a PARS system is how the PARS software interacts with the existing software used to acquire spectra or images from the analytical instrument. This will be explained with an example.

[0049] Let's consider a situation where we have two analytical instruments, a first analytical instrument "J" and a second analytical instrument "K". The first analytical instrument "J" might be manufactured by a first company "JJ" and might be operated by a first software package "JJJ". Similarly, the second analytical instrument "K" might be manufactured by a second company "KK" and might be operated by a second software package "KKK". The first analytical instrument "J" and the second analytical instrument "K" could, for example, be taken from Figures 5 to 7 or Figures 11 to 15 The instrument type shown, or an instrument in the above technical list. In one embodiment, the first company "JJ" and the second company "KK" may be the same, but even so, the first software package "JJJ" and the second software package "KKK" may still be incompatible for different microscopes and provide nothing in terms of interchangeability of image registration. The present disclosure includes a third software package "N" that is capable of running on an acquisition computer operating the first analytical instrument "J" and the second analytical instrument "K", or at least a single computer capable of reading and displaying files from the first analytical instrument "J" and the second analytical instrument "K". The third software package "N" can first be run on a computer operating the first analytical instrument "J" and then subsequently run on a computer operating the second analytical instrument "K".

[0050] 1. A third software package "N" runs in the background (i.e., has few visible features on the screen) on the same computer running the first software package "JJJ". It runs in parallel with the first software package "JJJ" but does not interact directly with it. The first software package "JJJ" controls the operation of the first analytical instrument "J" and displays images from the first analytical instrument "J".

[0051] 2. A first software package "JJJ" displays a view of the sample surface obtained by a first analytical instrument "J". The image includes computer-readable fiducial markers (e.g., ) of the sample, as presented by the first analytical instrument "J".

[0052] 3. When the user wishes to mark a point on the sample, the third software package "N" is alerted by pressing a "hot key" or clicking a button on the third software package "N"'s small graphical user interface (GUI) (if displayed).

[0053] 4. The user then clicks on the location to be marked in the image displayed by the first software package "JJJ". The third software package "N" records this screen location and scans the screen image for any computer-readable fiducial markers (in another embodiment, this may be done repeatedly so that when the user clicks on the location, the scan is already complete). The location of the selected point relative to the computer-readable fiducial marker is then stored (e.g., in a file on a USB flash drive, perhaps a DAT data structure).

[0054] 5. Optionally, the third software package "N" then prompts the user to enter a tag or other information associated with the stored point. This information is then stored in a file along with the point coordinates. The stored data, including the point location relative to the fiducial marker and other information about the point, is stored as data "D". The data "D" may be a DAT data structure.

[0055] 6. Later, when the same sample is viewed under a second analytical instrument "K", possibly operated by a different computer in a different location running a second software package "KKK", the user calls up a third software package "N" to mark the location of the previously stored point (possibly by using a hot key).

[0056] 7. A third software package "N" scans the screen, searching for computer-readable fiducial markers. It then finds the corresponding marker in its stored data file (probably on a USB drive), and the points previously defined relative to said visible marker.

[0057] 8. The third software package "N" then indicates the requested point at the correct location on the image from the second analytical instrument "K". This may be done by moving the mouse pointer to the pixel in the image corresponding to that point. It may also indicate by superimposing on the display provided by the second software package "KKK" a marker of those points (perhaps an arrow or circle), together with any labels associated with those points (taken from the stored data "D"), in a manner similar to the "screen annotation software" described above.

[0058] 9. The user may then choose to have one or more of these points on the sample analysed using the second analytical instrument "K" via the second software package "KKK". Typically this means that the user clicks on a point marked by the third software package "N" and the second software package "KKK" is sensitive to the mouse position selected by the user. In fact, the mouse may also be moved to that position so that if the second software package "KKK" is ready to acquire a point analysis, the mouse is already set in the correct position to operate.

[0059] Some embodiments of the present invention may facilitate the transfer of a third software package "N" and data "D" from a first analytical instrument "J" to a second analytical instrument "K". This may be accomplished, for example, by providing the third software package "N" on a network storage accessible to both, where the data "D" may also be stored. In another embodiment, the third software package "N" may be stored on a portable drive (perhaps a USB drive) that the user may take from the first analytical instrument "J" to the second analytical instrument "K". When inserted into the first analytical instrument "J", the data "D" may be stored on the drive and subsequently read when the third software package "N" is run on a computer associated with the second analytical instrument "K". One embodiment is to present the data "D" as a digital analytical twin (DAT) described below.

[0060] Note that in step 8 above, a transformation of the coordinate system is required to find the position of the pixel on the screen corresponding to the defined point. This is typically an affine transformation, unless there are other non-projective transformations (such as spherical aberration) in the optical system of the first analyzer "J" or the second analyzer "K". It can be expected that this may be rare in well-made commercial instruments. In Euclidean geometry, an affine transformation can be a geometric transformation that preserves straight lines and parallelism (but not necessarily distances and angles). The equations providing this affine transformation can be conveniently expressed in augmented matrix form. Distance (at least in terms of screen pixels) is not preserved between the first analyzer "J" and the second analyzer "K" because, for example, these may be analyzers with different magnifications. Angles may not be preserved either, because the angle of the normal direction of the observation surface may be different in "J" and "K", so a right angle observed in one may be greater than (or less than) 90 degrees in the other.

[0061] One advantage of the present disclosure is that even if the user does not have access to the first software package "JJJ" and the second software package "KKK" and cannot modify or rewrite them so that they convey coordinate information sufficient to find the sample point of interest on both instruments, correlative microscopy analysis can still be performed. Such a modification would be difficult and expensive and rely on the cooperation of the instrument manufacturer, which is generally not available.

[0062] Now, let us turn from the PARS system (or the third software package "N") to a discussion of the Digital Analytical Twin (DAT) structure, under which the data acquired through the use of the PARS system can be utilized very effectively.

[0063] A method and a scanning system (DAT scanner) are disclosed herein. The function of the scanning system is to generate a "digital twin" from a sample using one or more photogrammetric techniques1. Generally speaking, photogrammetry is a mathematical technique that can generate three-dimensional coordinates of identified points from multiple images of the same object acquired at different angles. In the context of analysis (rather than in manufacturing as the term is currently used), a digital twin can be a virtual representation of a physical object or process that serves as its real-time digital counterpart and aids in the systematic analysis of the physical object by including spatial and / or analytical (compositional) information. In an analytical environment (for the reasons described in the previous section), a digital twin must typically be generated through measurement rather than computer-aided design (CAD), and one or more photogrammetric techniques can be a quick and effective starting point. One or more software related to these photogrammetric techniques is now widely available and known in the industry, such as 2.

[0064] In addition, a new method is disclosed for registering all analytical techniques applied to a sample by using computer-readable fiducial markers fixed to the physical sample and recorded in the digital twin. The fiducial markers are fixed to the sample or to a sample holder that can be attached to the sample (preferably fiducial markers that can be automatically read by a computer from the image). Examples of common sample holders in microscopes are Fig.18 As shown. Computer readable fiducial markers are then securely fixed to the sample holder, or printed on its surface, or written (e.g. by inkjet printing or laser engraving). The sample can be securely fixed to the sample holder (as is currently done). This means that the sample now has a fixed position and orientation relative to the computer readable fiducial markers (CRFM). These fiducial markers are then used to define a coordinate system, preferably an xyz Cartesian coordinate system, for use in the 3D model of the digital twin after the photogrammetry provides a data file (e.g. a 3D representation in ".STL" format).

[0065] The methods for specific samples are as follows ("sample" and "specimen" are used interchangeably here):

[0066] 1. The sample is securely fixed to a sample stage with one or more fiducial marks, or the fiducial marks are securely fixed to the sample itself (e.g. by printing, gluing or focused ion beam milling).

[0067] a. Optionally, these fiducial markers can be automatically identified and read by computer software (e.g., [7,8]).

[0068] b. Sufficient fiducial markers will be used to define a coordinate system appropriate to the dimensions of the sample. Thus, for a two-dimensional sample (such as the surface of a silicon chip), the fiducial markers will define three or more points in space. For a three-dimensional sample, these fiducial markers will define at least four points.

[0069] c. As a specific example, place two Can be used to define eight points in space ( The cube is firmly fixed to the sample so that the two are all visible.

[0070] d. The sample carrier can be pre-fabricated and display the fiducial markings. The sample then only needs to be firmly fixed on the sample carrier as is commonly done in many types of sample analysis.

[0071] 2. Acquire a first set of images from multiple image capture devices, typically nearly simultaneously. These first set of images are conveniently captured by multiple image capture devices (or cameras), providing views from multiple angles around the sample. Multiple image capture devices may be controlled to capture a first set of images of the sample. Typically, the first set of images includes at least five images of the sample.

[0072] 3. Apply one or more photogrammetric techniques to generate a 3D model of the sample (e.g. in the form of a “.STL” file). Crucially, these photogrammetric techniques, in addition to capturing the morphology, also capture fiducial markers in the image and use them to define the xyz coordinate system of points on the 3D model of the sample surface. This makes the “digital analytical twin” scanner or scanning system more advanced than traditional 3D photogrammetric scanners used for rapid prototyping. The fiducial markers can also be used to generate a 3D (or 2D for flat samples) coordinate system, which can then be used to identify points on the sample using other microscopes that can image the same fiducial markers but cannot generate any 3D model. This coordinate system and its (fixed) relationship to the 3D model coordinate system are recorded in the digital twin file.

[0073] 4. Optionally, the user can add points or regions on the digital analytical twin 3D model, possibly with text labels, for subsequent analysis.

[0074] 5. The digital analytical twin may be delivered to the analytical facility along with the sample itself (e.g., a USB drive included with the sample, or via email, file transfer server, or any other digital file transfer method). In one embodiment, the digital analytical twin file may be stored in a laboratory information management system (LIMS) that organizes and / or schedules analytical laboratory work (e.g., Agilent’s [9] or Thermo Scientific LIMS

[10] ).

[0075] 6. Optionally, the digital analytical twin can be used to discuss analytical strategy (i.e., which points of the sample to analyze next and in what order). This can be conveniently done by viewing a rendered 3D model on a computer, or (if the parties are in different locations) using a or and other Internet conferencing and screen sharing software.

[0076] There is a key point to note in step 3 above. Here, one or more photogrammetric techniques are used to determine the coordinates of many points on the surface of the sample, thereby generating a "3D model". However, this 3D model also contains a coordinate system described by computer-readable fiducial markers. This means that later when the sample is brought to an analytical instrument without photogrammetric capabilities, the previously recorded surface points that make up the 3D model, and the coordinate axes defined relative to the computer-readable fiducial markers, can all be used to identify points on the surface using only the computer-readable fiducial markers that are visible in the image of the sample taken by the analytical instrument.

[0077] When the analytical expert at the central facility views the sample using a computer connected to his analytical tool (e.g. an electron microscope), the points of interest defined in step 4 are automatically superimposed on the image by software in that computer (e.g. the PARS system described above), optionally with the original user comments or other information (e.g. the presence of elements). If the fiducial markers are computer-readable, this can be done automatically, i.e. the software identifies one or more such fiducial markers in the displayed analytical instrument image and then finds by 3D triangulation where the point of interest appears in the current image (its coordinates are recorded in the digital twin file). Examples of computer-readable fiducial markers include ARTags

[11] , ArUco markers, etc. Importantly, this does not rely at all on being able to modify the instrument manufacturer's software. Instead, screen annotation software can be used to superimpose the markers "on top" of the image displayed by the instrument manufacturer's software, i.e. displayed on the screen but not by the analytical instrument manufacturer's software. The common xyz coordinate system provided by the "digital analytical twin" file allows for easier correlative microscopy analysis, i.e. being able to superimpose the results of one technique on another, correlating different microscopy techniques from often very different types of microscopes or analytical instruments.

[0078] Step 2 and optionally step 3 are performed by a scanning system also called a "Digital Analytical Twin Scanner" (DATS). This constitutes an apparatus / system of the present invention.

[0079] Figure 1The key components of a DAT scanner (DATS) instrument are schematically shown. The sample (130) can be fixed on a sample holder (150), and computer-readable reference marks (110 and 120) are firmly fixed to the sample (130) and / or the sample holder (150). Figure 1 The fiducial marks shown are A first set of images of the sample and the fiducial markers are acquired by a plurality of image capture devices (cameras) (100), and the first set of images are further transmitted to a computer (140) for numerical processing by photogrammetry. V4K Camera( Company, Sunnyvale, California, USA Fig.19 shown) and a common USB microscope camera (e.g., from Hong Kong, China Instrument Co., Ltd. 2 megapixel models, such as Fig. 20 ) is used for smaller samples, although most digital cameras will work well in this application. Although the schematic is two-dimensional, it is intended to show multiple image capture devices (110) pointed at the sample (130) from a wide range of stereoscopic angles in three-dimensional space to minimize ambiguity in photogrammetry. In one embodiment, the scanning system can be configured to control the movement of the sample holder (150) so that it rotates from a first position to a second position. The scanning system can also control the multiple image capture devices to capture a second set of images of the sample and at least one fiducial marker when the sample holder is in the second position. The scanning system can further generate a three-dimensional model of the sample based on applying one or more photogrammetric techniques to the captured first set of images and the second set of images. It should be noted that the sample holder (150) can be rotated about at least one axis so that multiple images from each camera can be recorded, improving the quality and accuracy of the photogrammetry.

[0080] To ensure good lighting conditions and minimize shadows, the sample can be placed in a diffusely reflecting "integrating sphere", or a structure similar to an integrating sphere, with an integrated set of light-emitting diodes (LEDs). In some embodiments, the integrated LED set can also be a component of the scanning system or DATS scanner. Fig.24 and Fig.25 The device shown achieves this.

[0081] Several examples of using computer-readable fiducial markers have been published

[13] . In short, on any analytical instrument connected to a computer, the image displayed on the computer monitor through the microscope can be imaged with fiducial markers, and there are at least a minimum number of such fiducial markers in the field of view, points previously defined by the user can be highlighted on the screen. This is possible even if the software provided by the instrument manufacturer is not accessible, as long as the operating system has access to the image currently displayed on the monitor.

[0082] In some cases, a user-defined point or region of interest may be outside the microscope's field of view when the fiducial markers are in the field of view. Figure 2 This is illustrated schematically in FIG. Figure 2 In the example, the reference mark (200) is a Located within the initial field of view (210) of the microscope. The four corners of the fiducial marker have known locations in digital twin information accessible by a computer operating the microscope. Using this location information, the software can determine that the point of interest (220) is not visible in this initial field of view, although an indication of its direction can be given (e.g., by an on-screen arrow annotation). In this case, the software can automatically follow the translation of the microscopic image on the screen as it moves under operator control from the initial field of view (FOV) (230) to the next FOV (240), to the next FOV (250), and finally to the final FOV (260), so that when the point of interest (210) enters the field of view, it is indicated (e.g., by an on-screen annotation at that location). When such a translation occurs, the computer's correlation of successive images (230, then 240, then 250, then 260) can accurately locate the point even though the fiducial marker is no longer in the field of view.

[0083] This requires sufficient contrast and occasional features within these fields of view (FOV) Figure 2 230, 240, 250 and 260) so that each field of view can be accurately positioned relative to the previous field of view. For real samples, such contrast and features are almost always present.

[0084] Analytical result information can be added to the digital twin file, correlating it to the analysis location on the real sample. Thus, over time, results from multiple different analytical methods can be accumulated in this digital analytical twin for review by center staff and analytical customers / users. Therefore, the specific information of each analytical technique can be fully utilized to gain a comprehensive understanding of the sample.

[0085] Practical verification: mathematical modeling and solutions

[0086] First consider the coordinate system of the analysis instrument for a square mark patterned on the sample surface. For simplicity, we assume that the size of the square mark in each direction is unit length. The actual size of the square may be, for example, 10μm or 50nm. Assume that the side length of the square mark is twice the distance across the sample surface.

[0087] For square fiducial marks, such as or QR code, can form an image of the location of the center of the mark and its four corners (A, B, C and D), such as Figure 3 As shown. For QR code positioning, there may be multiple software packages available. These software typically provide at least the four corners of a square marker and the location of its center in the pixel array. In some cases, there may be more information available from the software that automatically recognizes these fiducial markers, such as transformation matrices that can be used for image co-registration. These can be used. In some embodiments, these additional matrices may not be available without additional calibration, or may be the least reliable part of the output of these software because they are not used frequently and therefore have not been tested in depth. Therefore, the corners and centers of the fiducial markers can be used for image co-registration.

[0088] The corners of the fiducial markers (labeled here A, B, C, and D) can be distinguished from each other based on the internal pattern of the AprilTag or QR Code. Thus, for example, corner A can never be confused with corner C because the pattern is not symmetric enough to cause such confusion. The x, y plane of the image consists of a large number of pixels in a grid, so, for example, corner A of the AprilTag or QR Code is reported in the image as a pixel with position x1' and y1' ( Figure 2 ). Usually these coordinates are integers, but in some cases the software recognizing the tags can give more precise locations as floating point numbers, such as when corners are located between pixels.

[0089] There are two categories of two-dimensional linear transformations - projective and affine. In most transformations related to microscopy, the affine description is sufficient, but the projective model was chosen here because it handles oblique views of certain surfaces better. A projective transformation can be represented by the following matrix:

[0090] Equation (1):

[0091]

[0092] in:

[0093] Equation (2):

[0094]

[0095] is the rotation matrix. This matrix defines the type of transformation that will be performed: scaling, rotation, etc.

[0096] Equation (3):

[0097]

[0098] is the translation vector. It simply moves the center of the square, and

[0099] Equation (4):

[0100] (c1c2)

[0101] is the projection vector. In most applications in microscopy, the elements of this projection vector are likely to be small, however in the perspective case of macrophotography, for example, they can be significant.

[0102] If x and y are the coordinates of a point, the transformation can be done by simple multiplication:

[0103] Equation (5):

[0104]

[0105] Here, x′x′x′ and y′y′y′ are the coordinates of the transformed point.

[0106] Let A be the transformation matrix:

[0107] Equation (6):

[0108]

[0109] There are five points (the four corners and the center of the square marker) with known positions in the x,y plane of the focused ion beam (FIB): (x,y) = (-1,-1), (1,-1), (1,1), (-1,1), and (0,0), the last point being the center of the marker.

[0110] Equation (7):

[0111]

[0112] The values ​​of these five pairs of x′x'x′ and y′y'y′ are known from the identification of the landmarks in the image. First, A must be found for this image so that we can map the pixels of this image to the original defined plane (x,y). There may be too many measurements; in fact, this system of equations is overdetermined and can be solved in a least squares sense using the Moore-Penrose generalized inverse of the (3×5) matrix of (x,y) values ​​in the original defined plane. In fact, since this is always the same, the Moore-Penrose generalized inverse can be given in one go as:

[0113] Equation (8):

[0114]

[0115] Therefore, to find the coordinates of a pixel (x′,y′)(x',y')(x′,y′) in the original plane in any image acquired by any different technique, just multiply these coordinates by the generalized inverse:

[0116] Equation (9):

[0117]

[0118] In three dimensions, this is extended by using computer-readable fiducial markers that extend into three dimensions, e.g. Figure 1 The two shown These marks are not in the two-dimensional plane of the sample and can therefore be used to define a three-dimensional xyz coordinate system. When two or more computer-readable fiducial marks are present in this manner, the photogrammetrically generated array of sample surface points and all coordinate systems defined by the CRFMs (except one) are referenced to a single coordinate system of one of the CRFMs by a linear transformation:

[0119] Equation (10):

[0120] x=x(x′,y′,z′),y=y(x ′ ,y ′ ,z ′ ),z=z(x ′ ,y ′ ,z ′ )

[0121] Practice Validation II: Prototyping Software Components

[0122] A system ("N") has been prototyped, as described above, using components from a number of sources:

[0123] (a) In C++ Identification software (e.g., reference 14) used to identify the

[0124] (b)

[15] , for implementing a small GUI (step 3 etc.) to ask for the names of the samples and points, to save screen captures to files in steps 4 and 7, and to run components (a), (c) and (d) on the correct files in the correct order.

[0125] (c)

[16] , used to convert between .png and .jpg file formats used in (a) and (b). It is also used for contrast enhancement or inversion when required due to limitations in the microscope contrast mechanism. I have chosen the "portable 32-bit" version because this facilitates use on portable USB drives and some instrument operating software still uses 32-bit

[0126] (d)

[17] , for screen annotation.

[0127] BRIEF DESCRIPTION OF THE DRAWINGS

[0128] Figure 1 A scanning system (also referred to as a digital analysis twin scanner) for analyzing a sample (130) is shown, comprising one or more (five in this figure) cameras (100), a sample (130), a sample holder (150), computer readable fiducial markers (CRFMs) (110 and 120), and a computer (140) for operating the system;

[0129] Figure 2 It is shown how PARS indicates a point (210) outside the initial field of view (230); intermediate fields of view (240 and 250) are used by correlating random features of the sample surface within these fields of view (not shown) so that the position and orientation of the field of view (260) containing the point of interest is known relative to a computer readable fiducial marker (CRFM) (200);

[0130] Figure 3 A square fiducial marker is schematically shown on the x,y coordinate plane;

[0131] Figure 4 represents square reference marks in different directions in the x', y' plane, i.e. Figure 3 the possible appearance of the said marking in different instruments having an x', y' coordinate system;

[0132] Figure 5 A typical transmission electron microscope (TEM) is shown, including an electron column (500) and a computer (510) for operating the microscope;

[0133] Figure 6 A typical wide-field optical inspection microscope optical system (600) is shown having a built-in display (610) containing a computer for operating the microscope for optical microscopic observation of a sample (620);

[0134] Figure 7A typical confocal optical microscope with very high spatial resolution is shown, including a microscope column (700), a light source (usually a laser) (720), and a computer (710) for operating the microscope;

[0135] Figure 8 A typical energy dispersive (X-ray) spectrum (EDS) is shown, including multiple peaks representing the presence of various elements within the sample, including iron (800);

[0136] Fig. 9 An X-ray photoelectron spectroscopy (XPS) spectrum is shown, including multiple peaks representing the presence of several elements within the sample surface, including carbon (900) and oxygen (910);

[0137] Fig.10 shows a typical screen as seen by an XPS instrument operator, with two important points of analytical interest on the surface (P1 and P2), and the spectra obtained from these points (1000);

[0138] Fig.11 An atomic force microscope (AFM) (1110) and a computer (1100) for operating it are shown;

[0139] Fig.12 A conventional scanning electron microscope (SEM) is shown, including a sample entry chamber (1200) for sample entry and a computer (1210) for operating the SEM;

[0140] Fig.13 A state-of-the-art helium ion microscope (HIM) is shown, including a helium ion column (1300), a sample injection chamber (1310), and a computer (1320) for operating the HIM;

[0141] Fig.14 A typical "benchtop" scanning electron microscope (SEM) is shown, including a vacuum chamber (1410) and a computer (1400) for operating the SEM;

[0142] Fig.15 A typical X-ray photoelectron spectrometer (XPS) is shown, having a hemispherical analyzer (1520) and an analysis chamber (1500), with an operator sitting nearby and planning the XPS analysis using software running on a computer (1510) that operates the XPS instrument;

[0143] Fig.16 Demonstrated the operation of a scanning electron microscope or SEM (actually Fig.14 Computer screen of a benchtop SEM (shown), including an optical “plan view” of the sample;

[0144] Fig.17A computer screen showing the operation of an X-ray photoelectron spectrometer, including an optical “flat view” of the sample;

[0145] Fig.18 A variety of different types of sample holders used in analytical instruments such as scanning electron microscopes are shown, including the "needle holder" (1800) used in some of the experimental research I have conducted;

[0146] Fig.19 Shown by The digital camera manufactured by the company (model V4K) has been successfully used in Figure 1 A portion of a scanning system (or DAT scanner) is shown, including a CMOS optical camera sensor (1900), an adjustable stand (1910), and a USB cable (1920) through which images are transferred to a computer;

[0147] Fig. 20 A general-purpose, inexpensive digital microscope of the type widely available and successfully used in Figure 1 A portion of a scanning system (or DAT scanner) is shown, including a stand (2020), a focus knob (2010), and a USB cable (2030) through which images are transferred to a computer;

[0148] Fig.21 Schematically showing how an optical microscope and an electron microscope can be combined into a single instrument, including an optical camera (2120), a mirror (2110), and a secondary electron detector (2100);

[0149] Fig. 22 schematically showing a computer screen operating a scanning electron microscope, including computer-readable fiducial marks etched on a sample surface using a focused ion beam (2310), a point on the sample surface defined by a user (labeled P1) (2320), and an energy dispersive X-ray spectrum (EDS) from that point (2330);

[0150] Fig.23 Schematic diagram of a computer screen for operating an X-ray photoelectron spectrometer (XPS), including a computer readable fiducial marker (CRFM) (2210) attached to a sample holder, a Fig. 22 FIG. 2 shows a SEM (SEM) in which a focused ion beam is used to etch different CRFMs (2220) on the sample surface, and position markers (2230) superimposed on the screen by the PARS software to instruct the user on the position of the CRFMs (2220). Fig. 22 The points marked in the SEM shown;

[0151] Fig.24shows the arrangement of computer readable fiducial markers (CRFMs) (3210, 3220 and 3230) used in the experiment on a sample holder, viewed from three different angles (a), (b) and (c), and the sample of interest is an electronic device (3200) firmly fixed on the sample holder;

[0152] Fig.25 The experimental setup is shown from different angles, including a set of CRFMs fixed on a sample holder (3300), and another distinguishable array of CRFMs (3310) attached to a motorized rotation stage (3330). The camera (3320) type is as follows Fig. 20 As shown, this is Figure 1 One embodiment of the system shown;

[0153] All of these are consistent with one embodiment of the present disclosure.

[0154] Detailed Description of the Invention

[0155] Let's assume that in an analytical facility, perhaps affiliated with a university or a commercial company, there may be 100 different analytical instruments, about 100 staff members, and about 3,000 users. The users may be university researchers, corporate researchers, or others. Each user typically needs to use 3 or 4 of the 100 instruments. Let's assume that a particular user needs to perform scanning electron microscopy (SEM), energy dispersive X-ray analysis (EDS), and X-ray photoelectron spectroscopy (XPS) analysis on a set of 10 samples from different catalyst studies. This is a very typical situation.

[0156] For the purposes of this example embodiment, let us assume that the user is working some distance from a central facility where the analytical work will be performed, perhaps in Leeds, UK. Optionally, the user first mounts each sample on a separate sample holder, each holder having a unique computer readable fiducial marker, such as a QR code, some barcode marking or No special care is required to position or orient the sample relative to the fiducial marks on these brackets. Instead, alternatively, use Figure 1The digital analysis twin scanner (DATS) shown is used to scan each sample and create a digital analysis twin (DAT) file for each sample, which is a computer data structure that contains captured images of the sample, as well as information about the orientation and position of these images relative to computer-readable fiducial markers. In fact, these fiducial markers will be visible in at least one image. The DAT data structure can be conveniently stored on a USB memory or portable USB drive, or equivalently stored in a specific location on a network drive or cloud storage. Optionally, this storage device (such as a USB memory) also contains the PARS software in executable form (sometimes called a "portable application"). All of this can be done in the user's own laboratory in Leeds, which may be far away from the central analysis facility that the user intends to use (possibly in London). The scanning system (or DAT scanner) used by the user in his own laboratory may be much cheaper than the instruments and microscopes at the central facility. For a scanning system of typical sample size, it may be a desktop-sized instrument that is easy to place in a laboratory or even an office. However, in some cases, if the sample is larger, the scanning system may be much larger.

[0157] At this point, the user can review the image, selecting points or regions that he or she wishes to investigate by SEM, EDS, or XPS. These points of interest (POIs) or regions of interest (ROIs) are recorded by the PARS software in the Digital Analysis Twin (DAT) data structure (the PARS software performs the coordinate transformation described in Equations 1-9 above to achieve this). When the user brings the USB memory stick to the SEM instrument (perhaps at a central facility, such as London, miles from his or her own laboratory), the user looks for one or more computer readable fiducial markers (CRFMs) on the screen. The PARS software (running in parallel with the SEM acquisition software, but accessing only the screen displayed to the user) automatically recognizes these CRFMs as they appear on the computer screen and adds annotations to the screen where the POIs or ROIs were previously marked, making use of the photogrammetrically generated model and the coordinate transformation of Equations 1-9 in the process. The user can see these POIs and ROIs marked and annotated on the screen (even though the computer running the SEM has no knowledge of this). Now that the SEM is displaying an image of the surface, the user can add more POIs or ROIs to the DAT data structure. This can be done, for example, by the user clicking the mouse at a specific location. The PARS software will capture this information, calculate the 3D position of these points relative to the CRFM, and record it in the DAT data structure. These images can be captured by PARS and stored in the DAT data structure. The user can use the SEM controls to navigate the field of view of the SEM instrument to the POI or ROI that appears as an annotation on the screen.

[0158] The user then performs an EDS analysis at some point of interest (POI). The EDS spectrum is recorded and the spectrum file itself or a link (e.g. a unique file name) is stored in the DAT, associated with that POI. Typically, EDS can be performed in a SEM instrument.

[0159] Knowing that he is about to take the sample to a different instrument, the user uses a focused ion beam (FIB) to mark a new CRFM at a specific location near the POI that will be visible in the next instrument to be used. The view that the user sees on the computer screen operating the SEM is as follows: Fig. 22 shown.

[0160] The user then takes the sample and the USB stick containing its DAT to a completely different instrument - an X-ray Photoelectron Spectrometer (XPS). This could be in Newcastle, for example. The user uses XPS mapping to chemically image an area containing some POIs and FIBCRFM. Because this CRFM is smaller in scale and closer to the POIs, it enables the user to define the POIs with greater accuracy and repeatability. The user then records XPS spectra from some of the POIs and these spectral files are stored in (or linked to) the DAT on the USB stick, associated with the coordinates of the positions of those POIs in the sample. Fig.23 Schematic representation of the screen that the user will see when using the XPS instrument computer. Note the FIB etched CRFM (2220) and the position marker (2230) superimposed on the screen by the PARS software, indicating the point previously defined by the user in the SEM. The PARS software calculates where to place this annotation and label "P1" based on the CRFM (2220) identified directly from the computer screen.

[0161] Later (perhaps months or even years later), the user can review the analysis results of the sample using a USB memory stick and his or her own personal computer (without any analytical instrumentation connected). When writing a paper or industry report on the results, the user can browse the required images and quantitatively analyze the spectra while knowing the coordinates on the sample where these spectra originated. The user can generate a 2D or 3D digital rendering of the sample based on imaging from DATS and / or SEM, EDS, XPS, possibly with false color imaging overlays to emphasize specific chemicals present.

[0162] Among the advantages to users of PARS, DAT, and DATS for this work are that preparation can be done inexpensively in the user's own lab using a DAT scanner (DATS) without expensive time commitment on instruments such as SEM or XPS; potential errors in delineating POIs or ROIs can be avoided; analytical data are stored in a one-to-one correspondence with the sample, making it easy for the user to generate a view of the sample with analytical data overlaid that makes sense to any expert on the sample, even if they are not an expert in the analytical technique; the DAT data structure can be standardized and recognized by the analytical facility or more broadly, whereas handwritten photo annotations in lab notebooks cannot; and the ability to share data over time in remote conferences or Displaying DAT information in different ways during meetings can facilitate remote collaboration without taking up time on expensive SEMs, XPSs, or other instruments to achieve this.

[0163] Compared with the prior art, the present invention has many new features, including (but not limited to) the following:

[0164] (a) The PARS software operates solely through images displayed on the screen, finding, identifying and locating computer readable fiducial markers (CRFMs) within those images, and can therefore be run as third-party software (not necessarily from the company that sells the software that operates the analytical instrument).

[0165] (b) A DAT scanner is a new device whose task is to generate a DAT model of a sample by taking photogrammetric measurements of the sample (and attached computer-readable fiducial markers) from multiple (usually several) angles. It is not just a photogrammetric scanner (there are several good prior art examples in the literature12), it adds automatic recognition of CRFMs and creates an xyz coordinate system defined by these CRFMs, in which the points defining the sample surface (determined by photogrammetry) are placed.

[0166] (c) A digital analytical twin (DAT), as described above, differs from the digital twins currently in use3 because a DAT is not derived from a CAD or CAM model of a manufactured object or building, but is instead created empirically through photogrammetry of a sample that may be manufactured or found (e.g., an archaeological artifact or a natural object such as a meteorite). A digital twin can, and often does, exist before the physical entity exists, but a DAT does not. Using a digital twin during the creation phase allows the entire lifecycle of the intended entity to be modeled and simulated3. A DAT does not exist at the time of the object’s creation; it must be created through photogrammetry or other measurement techniques. A DAT starts with the physical object and allows the analytical lifecycle of the object to be modeled and simulated, including data corruption due to certain techniques previously applied.

[0167] The DAT data structure can be implemented in a number of different ways, one example being an Extensible Markup Language (XML) file4. This allows tools that have been developed for parsing and inspecting XML files to be applied to DAT. In this structure, spectra, images, or other analytical data about a sample can be represented using a specific (proprietary or open) analytical data format.

[0168]

[0169]

[0170]

[0171] Where "s12345.vms" is probably the file name of a linked XPS spectrum stored in ISO14976 format, "E98765.eds" is the file name of a linked EDS spectrum stored in a proprietary (SEM manufacturer's) format. The file "s31987.stl" contains the sample morphology determined by photogrammetry in a DAT scanner.

[0172] References

[0173] [1] Céline Loussert Fonta and Bruno M Humbel Correlative microscopy, Archives of Biochemistry and Biophysics, Volume 581, 1 September 2015, Pages 98-110.

[0174] [2] Jeffrey Caplan, Marc Niethammer, Russell M Taylor and Kirk JCzymmek, The power of correlative microscopy: multi-modal, multi-scale, multi-dimensional, Current Opinion in Structural Biology, Volume 21, Issue 5, October 2011, Pages 686-693.

[0175] [3]F Bergamasco et al,Image-Space Marker Detection and RecognitionUsing Projective Invariants,IEEE Conference Paper·June 2011,DOI:10.1109 / 3DIMPVT.2011.55

[0176] [4]E.Olson,“AprilTag:a robust and flexible visual fiducial system,”Robotics and Automation(ICRA),2011 IEEE International Conference on.IEEE,2011;M.Fiala,“ARTag,a fiducial marker system using digital techniques,”Computer Vision and Pattern Recognition,2005.CVPR 2005.IEEE Computer SocietyConference on.Vol.2.IEEE,2005.

[0177] [5]Karl Kraus,“Photogrammetry:Geometry from Images and Laser Scans:2nd Edition”(De Gruyter,Berlin,2007)

[0178] [6]https: / / www.pix4d.com / product / pix4dmapper-photogrammetry-software

[0179] [7]AprilTag 2:Efficient and robust fiducial detection,Proceedings ofthe IEEE / RSJ International Conference on Intelligent Robots and Systems(IROS),John Wang and Edwin Olson,2016

[0180] [8]AprilTag:A robust and flexible visual fiducial system,Proceedingsof the IEEE International Conference on Robotics and Automation(ICRA),EdwinOlson

[0181] [9]https: / / www.agilent.com / en / service / laboratory-services / lab-operations-management / sample-processing-management

[0182]

[10] https: / / www.thermofisher.com / order / catalog / product / INF-11000# / INF-11000

[0183]

[11] ARTag Revision 1.A Fiducial Marker System Using DigitalTechniques-Fiala,M.,November 2004:https: / / www.cs.cmu.edu / afs / cs / project / skinnerbots / Wiki / AprilTags / NRC-47419.pdf

[0184]

[12] “Correlative microscopy”by Céline Loussert Fonta and Bruno MHumbel,Archives of Biochemistry and Biophysics,Volume 581,1 September 2015,Pages 98-110

[0185]

[13] Computer-readable Image Markers for Automated Registration inCorrelative Microscopy–“autoCRIM”,J Sheriff,I W Fletcher,P J Cumpson,Ultramicroscopy,113322(2021).

[0186]

[14] https: / / april.eecs.umich.edu / software / apriltag;also https: / / april.eecs.umich.edu / software /

[0187]

[15] https: / / www.autoitscript.com / site / autoit /

[0188]

[16] https: / / imagemagick.org / script / download.php

[0189]

[17] https: / / github.com / geovens / gInk

[0190]

[18] https: / / www.ibm.com / topics / what-is-a-digital-twin

[0191]

[19] Grieves,Michael(October 5,2015)."Can the digital twin transformmanufacturing".World Economic Forum Emerging Technologies.Retrieved November19,2022.

[0192]

[20] Elliotte Harold,XML in a nutshell,3rd Edn(O’Reilly,NorthSebastopol,CA,USA 2004).

[0193]

[21] Jeffrey Caplan,Marc Niethammer,Russell M Taylor II,Kirk JCzymmek,The power of correlative microscopy:multi-modal,multi-scale,multi-dimensional,Current Opinion in Structural Biology 2011,21:686–693

[0194]

[22] The 2018 correlative microscopy techniques roadmap,Toshio Ando etal2018 J.Phys.D:Appl.Phys.51 443001.

[0195]

[23] “Multi-modal analysis of 2D materials with the XPS-SEM CISAworkflow”,Thermo Scientific Applications Note

[0196]

[24] J.J.H.A.Van Hest et al,Towards robust and versatile singlenanoparticle fiducial markers for correlative light and electron microscopy,Journal of Microscopy,Vol.274,Issue 1 2019,pp.13–22 doi:10.1111 / jmi.12778

[0197]

[25] B,Schmelzle S,Blüthgen N,Heethoff M(2018)An automateddevice for the digitization and 3D modelling of insects,combining extended-depth-of-field and all-side multi-view imaging.ZooKeys 759:1–27.https: / / doi.org / 10.3897 / zookeys.759.24584

[0198]

[26] Plum F,Labonte D.2021.scAnt—an open-source platform for thecreation of 3D models of arthropods(and other small objects).PeerJ 9:e11155DOI 10.7717 / peerj.11155。

Claims

1. A scanning system, comprising: processor; a memory communicatively coupled to the processor, wherein the memory stores a plurality of processor-executable instructions that, when executed by the processor, cause the processor to: controlling at least one image capture device to capture a first plurality of images of a sample and at least one fiducial marker, wherein the sample is disposed on a sample carrier and the at least one fiducial marker is associated with at least one of the sample or the sample carrier; generating a three-dimensional model (3D model) of the sample based on applying one or more photogrammetric techniques to the captured first plurality of images, wherein the one or more photogrammetric techniques capture information of a first coordinate system associated with the generated 3D model; generating a data structure associated with the sample based on the generated 3D model; as well as The generated data structure is output, wherein the data structure includes a first coordinate system associated with the generated 3D model, a second coordinate system associated with the sample, and a corresponding relationship between the first coordinate system and the second coordinate system.

2. The scanning system of claim 1, wherein the processor is further configured to: Controlling the movement of the sample holder so that it rotates from a first position to a second position; controlling the at least one image capture device to capture a second set of images of the sample and the at least one fiducial marker when the sample holder is in a second position; and A 3D model of the sample is further generated based on applying the one or more photogrammetric techniques to the captured first and second plurality of images.

3. The scanning system of claim 1, wherein the generated data structure corresponds to an Extensible Markup Language (XML) file.

4. The scanning system of claim 1, wherein the at least one fiducial marker corresponds to one of: a Quick Response (QR) code, a barcode, an AprilTag, an ARTag, or an ArUco tag.

5. The scanning system of claim 1, wherein the processor is further configured to generate a second coordinate system associated with the sample based on the at least one reference marker.

6. A scanning system according to claim 5, wherein the generated data structure includes information related to a first region of interest (RoI) analyzed using a first analytical instrument integrated in the scanning system or using a second analytical instrument having a different coordinate system from the first analytical instrument.

7. The scanning system of claim 1, wherein the processor is further configured to: receiving a first user input related to marking at least one point of interest on the generated 3D model; Based on the received first user input, storing information related to the marker of the at least one point of interest in the generated data structure; and Output the generated data structure.

8. The scanning system of claim 7, wherein the at least one point of interest is marked for analysis under one or more analytical instruments.

9. The scanning system of claim 1, wherein the sample corresponds to a non-homogeneous sample.

10. A method comprising: rendering a first image of a region of interest (RoI) of the sample on a first analytical instrument, wherein the rendered first image includes at least one fiducial marker and is captured by the first analytical instrument; receiving a second user input from the first analytical instrument related to selecting a point of interest within the rendered first image; determining, based on the received second user input, position information associated with the selected point of interest, wherein the determined position information includes a position of the point of interest relative to the at least one fiducial marker; storing the determined position information in a data structure; receiving a third user input related to rendering a second image of the region of interest on a second analytical instrument, wherein the second analytical instrument is different from the first analytical instrument; controlling the second analytical instrument to scan the stored data structure to determine a location of the at least one fiducial marker in the second image based on the received third user input; applying at least one transformation technique to the position information stored in the data structure based on the scanning; as well as Based on the applied at least one transformation technique, a first image of the selected point of interest is rendered on the second analytical instrument.

11. The method of claim 10, wherein the first image of the selected point of interest is captured by the second analysis instrument.

12. The method of claim 10, wherein at least one transformation technique applied comprises an affine transformation technique.

13. The method according to claim 10, further comprising: scanning the rendered first image to determine a first position of the at least one fiducial marker in the first image; as well as Based on the scanning of the first image, position information associated with the selected point of interest is determined.

14. The method according to claim 10, further comprising: receiving a fourth user input related to the selected point of interest, wherein the received fourth user input includes the first tag and first information related to the selected point of interest; as well as Based on the received fourth user input, the first tag and the first information for the selected point of interest are stored in the data structure.

15. The method of claim 10, wherein the data structure corresponds to a digital analytical twin (DAT) data structure associated with the sample and is a digital replica of the sample.

16. The method according to claim 10, wherein the sample corresponds to a heterogeneous sample.

17. The method of claim 10, wherein the fiducial marker corresponds to one of the following: a Quick Response (QR) code, a barcode, an AprilTag, an ARTag, or an ArUco tag.

18. A method comprising: controlling at least one image capture device to capture a first plurality of images of a sample and at least one fiducial marker, wherein the sample is disposed on a sample carrier and the at least one fiducial marker is associated with at least one of the sample or the sample carrier; generating a three-dimensional model (3D model) of the sample based on applying one or more photogrammetric techniques to the captured first plurality of images, wherein the one or more photogrammetric techniques capture information of a first coordinate system associated with the generated 3D model; generating a data structure associated with the sample based on the generated 3D model; as well as The generated data structure is output, wherein the data structure includes a first coordinate system associated with the generated 3D model, a second coordinate system associated with the sample, and a corresponding relationship between the first coordinate system and the second coordinate system.

19. The method according to claim 18, further comprising: controlling the movement of the sample holder to rotate it from a first position to a second position, wherein the sample is placed on the sample holder; controlling the at least one image capture device to capture a second plurality of images of the sample and at least one fiducial marker when the sample holder is in a second position; as well as A 3D model of the sample is further generated based on applying the one or more photogrammetric techniques to the captured first and second plurality of images.

20. The method of claim 18, wherein the fiducial marker corresponds to one of the following: a Quick Response (QR) code, a barcode, an AprilTag, an ARTag, or an ArUco tag.

Images