Systems and methods for specimen analysis

Abstract

New methods and apparatus are described for improving the analysis (physical or chemical) of objects. Using photogrammetry and computer-readable fiducial markers, an electronic file is generated that constitutes a digital twin of a specimen to be analyzed. This facilitates communication and discussion about where to analyze the specimen and promotes correlative microscopy because multiple analysis techniques can be applied using a common coordinate system. Further, using a method and software called PARS (Portable Analytical Registration Software), points defined on one computer-controlled imaging device can be easily found on another computer-controlled imaging device without the need to access or modify the software operating each device. This methodology enables correlating images from various surface imaging techniques and providing unprecedent levels of surface detail at the nanometer scale that are not available with one technique alone.

Description

Technical Field

[0001] The present disclosure relates to the analysis of specimens, and in particular, to the chemical and physical analysis of specimens including imaging and spectroscopy.

Background Art

[0002] In analytical laboratories, there is often a need to describe small samples, specimens, or artifacts in order to communicate information about their location (e.g., when exchanging chemical analysis results via email). Examples of specimens include small chunks of catalyst material on which electron microscopy or microanalysis is performed, parts of meteorites being studied by microscopy or X-ray analysis, and teeth of extinct humans found in caves. In this document, the terms "specimen" and "sample" are used synonymously. Similarly, the terms "analytical device" and "microscope" are also used synonymously.

[0003] In a scientific context, different types of analytical instruments, such as microscopes, are used. Some microscopes form images using light, while others use electrons, X-rays, etc. Some microscopes only output images (i.e., micrographs), while others output chemical images or chemical micrographs (e.g., imaging X-ray photoelectron spectroscopy) or spectra showing the characteristics of surface chemistry. These analytical instruments are often expensive but have evolved over the years to be able to provide micrographs rapidly. Some microscopes operate in air, while others require a vacuum or ultra-high vacuum (UHV). Some can accommodate very large samples, while others can only accommodate very small samples. Usually, modern analytical instruments are operated with a desktop computer, and an operating system such as Microsoft® Windows® (or, less commonly, Linux®) is often running. Examples of such microscopes are shown in FIGS. 5 to 7 and 11 to 15. Each microscope operates using a different physical technique and accordingly presents images with very different contrast mechanisms. As can be seen from the drawings of the analytical instruments shown in FIGS. 5, 6, 7, 11, 12, 13, 14, 15, spatially resolved analysis (i.e., analysis of specific points or regions) is generally performed on a microscope or other instrument and is connected and operated via a computer. FIGS. 16 and 17 show the screens of computers operating a scanning electron microscope (SEM) and an X-ray photoelectron spectroscopy (XPS) analyzer. In both figures, an image of the specimen surface (the large granular panel in FIG. 16 and the upper central panel in FIG. 17) is displayed. In both cases, the operator selects a point within this specimen view and clicks the mouse to acquire spectra (energy-dispersive X-rays (EDS and XPS) respectively) from that position.

[0004] Therefore, analytical techniques that can be usefully applied in combination with this invention include AES - Auger electron spectroscopy, AFM - atomic force microscopy, ARPES - angle - resolved photoemission spectroscopy, ARUPS - angle - resolved ultraviolet photoemission spectroscopy, CARS - coherent anti - Stokes Raman spectroscopy, CET - cryo - electron tomography, Cryo - EM - cryo - electron microscopy, Cryo - SEM - cryo - scanning electron microscopy, EBIC - electron - beam - induced current, EBSD - electron backscatter diffraction, EDAX - X - ray energy - dispersive analysis, EDS or EDX - energy - dispersive X - ray spectroscopy, EELS - electron energy - loss spectroscopy, ESCA - electron spectroscopy for chemical analysis (see XPS), ESEM - environmental scanning electron microscopy, ESTM - electrochemical scanning tunneling microscopy, EXAFS - extended X - ray absorption fine structure, FCS - fluorescence correlation spectroscopy, FCCS - fluorescence cross - correlation spectroscopy, FEM - field - emission microscope, FIB - focused ion beam microscope, FLIM - fluorescence lifetime imaging, fluorescence microscope, FRET - fluorescence resonance energy transfer, GIXR - grazing - incidence X - ray reflectivity, HAS - helium atom scattering, HREELS - high - resolution electron energy - loss spectroscopy, HREM - high - resolution electron microscopy, HRTEM - high - resolution transmission electron microscopy, HE - PIXE - high - energy proton - induced X - ray emission, IAES - ion - induced Auger electron spectroscopy, ISS - ion - scattering spectroscopy, LEED - low - energy electron diffraction, LEEM - low - energy electron microscopy, LEIS - low - energy ion - scattering, MALDI - matrix - assisted laser desorption ionization, MEIS - medium - energy ion - scattering, MFM - magnetic force microscopy, MRFM - magnetic resonance force microscopy, Micro - CT - micro - computed tomography, MRI - magnetic resonance imaging, NEXAFS - near - edge X - ray absorption fine structure, NSOM - near - field optical microscope, PhD - photoelectron diffraction, PED - photoelectron diffraction, PEELS - parallel electron energy - loss spectroscopy, PEEM - photoemission electron microscopy, PES - photoelectron spectroscopy, PIXE - particle (or proton) - induced X - ray spectroscopy, RBS - Rutherford backscattering spectroscopy, RHEED - reflection high - energy electron diffraction, SAXS -Small-angle X-ray scattering, surface composition analysis by SCANIIR - neutral species and ion impact radiation, SCEM - scanning confocal electron microscope, SE - spectroscopic ellipsometry, SEM - scanning electron microscope, SERS - surface-enhanced Raman spectroscopy, SERRS - surface-enhanced resonance Raman spectroscopy, SIMS - secondary ion mass spectrometry, SNMS - sputter neutral mass spectrometry, SNOM - scanning near-field optical microscope, SPM - scanning probe microscope, STEM - scanning transmission electron microscope, STM - scanning tunneling microscope, STS - scanning tunneling spectroscopy, SXRD - surface X-ray diffraction, TEM - transmission electron microscope, TOF-MS - time-of-flight mass spectrometry, two-photon excitation microscopy, TXRF - total reflection X-ray fluorescence analysis, UPS - ultraviolet photoelectron spectroscopy, XAES - X-ray induced Auger electron spectroscopy, XANES - XANES, synonymous with NEXAFS (near-edge X-ray absorption fine structure), XAS - X-ray absorption spectroscopy, XPEEM - X-ray photoelectron emission microscope, XPS - X-ray photoelectron spectroscopy, and XRF - X-ray fluorescence analysis.

[0005] Moreover, in universities and industries, large-scale and expensive analytical and processing equipment is increasingly being used for specimens. These universities and industries are often called "central facilities," and usually, full-time staff for operating the analytical equipment are deployed. For example, a central facility may have 20 different types of analytical equipment, and the cost of each may be between $1 million and $10 million. Each university and industry cannot purchase all the analytical equipment. There are usually only 2 to 3 "central facilities" in the country, and each facility has hundreds of users. Each user sends specimens such as catalyst specimens, meteorites, and teeth to the facility for analysis and improvement. Examples of the equipment in these facilities include electron microscopes, X-ray photoelectron spectrometers, imaging mass spectrometers, etc., and other equipment is also selected from the aforementioned list.

[0006] Examples of devices not related to the present disclosure include devices for analyzing homogeneous specimens or liquid samples. For example, NMR (nuclear magnetic resonance) analysis of solutions, analysis of blood samples or water samples from polluted rivers, etc. Such homogeneous samples are usually managed using a "LIMS" or laboratory information management system, and a unique identifier is given to each sample.

[0007] What we are interested in is not the unique identifier of the entire specimen, but the specimens for which it is important to know where the analysis is performed, and the devices used to perform these spatially resolved analyses. For example, a scanning electron microscope is an example. Many scanning electron microscopes can image secondary electrons (providing a well-known monochrome high-resolution image of the surface) and at the same time analyze the X-rays generated by the collision of electrons with the specimen surface, performing "energy-dispersive X-ray analysis" or EDS. Chemical elements are identified from their characteristic X-rays. The point on the specimen being imaged at a particular instant is the same place where the X-rays are being emitted, and by overlaying the chemical image and the secondary electron image, information about both the shape and the analysis can be obtained. Another example is X-ray photoelectron spectroscopy, and an example of the purpose of such an analysis is shown in FIG. 10. Here, different analysis positions (P1 and P2) on the same specimen show different spectra, and thus it can be seen that different analysis results are obtained regarding the proportion of the elemental composition.

[0008] To solve many problems, multiple types of microscopes are often required. For example, a certain feature can be confirmed with an optical microscope (such as the one shown in FIG. 6), but then it may be necessary to transfer the sample to an electron microscope (such as the one shown in FIG. 12) to visualize features much smaller than the wavelength of light [1]. In other embodiments, the sample may first be observed with one type of chemically specific microscope (e.g., an imaging X-ray photoelectron spectroscopy (XPS) system, such as the one shown in FIG. 15), and then with another microscope (e.g., a Raman microscope) to examine chemical aspects that the first microscope is not sensitive to. In such cases, researchers currently have two options. The first option is to purchase or construct a special compound microscope that can perform both techniques on the same sample, and the second option is to move the sample between multiple microscopes while tracking the area of interest to ensure that images obtained from multiple techniques can be compared.

[0009] The first option is usually expensive, and a compound microscope is more costly than individual microscopes. Depending on the scenario, the processing power and performance of the compound microscope are often compromised. The compound microscope is difficult to operate and may require an individual proficient in both technologies. Still, the first option may be useful in some cases. FIG. 21 schematically shows an optical / SEM combination device of Delmic (registered trademark), Delft, the Netherlands. In particular, FIG. 21 schematically shows how an optical microscope and an electron microscope are integrated into one device, enabling a user to image a part of a specimen almost simultaneously with both the optical microscope and the electron microscope and allowing for good alignment of the images. Constructing such a device is often expensive, and if one part fails (e.g., the electron microscope), the other is often rendered inoperable. The present invention provides an alternative means. As another example of a combined, multi-technology device, Iontof (registered trademark), Münster, Germany, provides a system that combines M6 ToFSIMS (registered trademark) and SPM in situ and uses an accurate piezo stage to move the specimen between each analysis position. However, this is much more expensive than individual devices for ToFSIMS (registered trademark) and SPM (registered trademark).

[0010] The second option is generally inexpensive and leverages the investment in existing individual microscopes and training. The problem with this type of correlative microscope is to confirm that the same area is being analyzed with both microscopes and to identify specific points that are clearly visible in the first analyzer but not in the second. This second option, and the problem of aligning one image with respect to another, are the focus here and are greatly assisted by the present disclosure.

[0011] A particularly common case of the second option is when two techniques are applied to a specimen, and one of them provides excellent images (for example, this could potentially be an optical microscope or an electron microscope). Subsequently, the researcher wishes to perform a chemical analysis of specific features within that field of view [2]. For example, an atomic force microscope (AFM) image shows particles on the specimen surface, and the researcher may want to analyze the chemical composition of particles with different shapes. In this case, it is necessary to move the sample to an instrument with chemical analysis capabilities to confirm what those particles are made of. In such cases, usually, "point analysis" with a Raman spectrometer or energy-dispersive X-ray analysis with a scanning electron microscope (SEM) is used to perform the analysis at the point of interest (POI). The problem at this time is how to identify the point (or region) within the accessible range of instrument B after identifying the point of interest (or region) by imaging with instrument A, given that the coordinate systems of both instruments (that is, the x, y, z coordinates with respect to the sample) are completely different.

[0012] As an example of a specific application, consider the following. Suppose a researcher discovers a lesion in a tooth of a human ancestor, sends the tooth for analysis, and wants to examine the chemical composition of trace substances within the lesion. Also, in a failed catalyst, there may appear spots of an unknown brown substance, and it becomes important to identify the chemical properties of that brown substance to help solve the problem that led to the failure of the catalyst. Thus, there are situations where many users require spatially resolved analysis of specimens. Additionally, analytical instruments are usually very expensive and are located in central facilities, often in a different location from the researcher's site. Also, analyzing all points on the specimen surface is time-consuming and costly, so the researcher should only analyze the points or regions of interest (POI or ROI, respectively). Therefore, the researcher needs to bring (or send) the specimen to the staff at the central facility and explain which part of the specimen should be analyzed.

[0013] Such explanations are often difficult at the required resolution. Researchers may observe excavated human teeth under an optical microscope and identify interesting points of analysis (e.g., ancient lesions). Subsequently, the researchers mark the location to be analyzed with an "X" on the microscope-taken photo and send it to the central facility. This is like a pirate's treasure map.

[0014] Another difficulty is that specimens may look different under an optical microscope and an electron microscope. Certainly, the orientation and scale may vary. More importantly, the origin of the image contrast is completely different. Sometimes, it may be easy to identify the location to be analyzed, but it can also be very difficult even using photos. For example, even though tooth enamel may appear uniformly white under an optical microscope, chemical variations may appear in SEM images, causing confusion. Conversely, even when color differences are visible on the surface of a meteorite, it may be chemically uniform, and marks visible to the naked eye may not appear at all in SEM or may be hidden by topographical contrast. Even the researchers themselves may have difficulty navigating the specimen to the location they want to analyze. As a result, expensive equipment may be used in vain. Despite the application of advanced technology, researchers often have to rely on primitive and inaccurate means to identify the points of analysis. For example, scratching crosses with pins on the specimen or identifying points based on random surrounding features that look similar under both an optical microscope and an electron microscope (however, these features may accidentally look similar and can cause errors). For example, "2 mm above something that looks like a fish and 1 mm to the left of something that looks like a map of Australia."

[0015] A fine-mesh grid made of copper can be placed on the specimen. Small letters are labeled in each square hole of this grid and can be easily seen with an optical microscope or an electron microscope. However, the grid is obstructive and may change the contacting surface, making it difficult to fix without an adhesive (the adhesive may also change the chemical properties of the surface). Therefore, a dynamic system that can solve the above problems is needed. Some approaches are useful or may be useful in the future, but they have their own significant problems. In some cases, the use of "digital twins" may be a solution in the future (although in fact, it has never been seen before). A digital twin is a virtual representation that functions as a real-time digital counterpart of a physical object or process. In design and manufacturing, a digital twin is a digital representation and is similar to (or includes) the computer-aided design (CAD) file of a component. Take a stainless-steel bolt as an example. If brown corrosion spots are found on a bolt manufactured in a factory, their positions can be conveyed based on the xyz coordinate system of the digital twin. Before sending the bolt itself to a central facility, the coordinates of the corrosion spots are added to the digital twin file. Even without the need to match an optical photograph, the staff at the central facility can (at least theoretically) use the digital twin information to directly guide an analytical device to the position of the brown stain. In reality, the software infrastructure to achieve this does not yet exist even for small manufactured parts. In any case, most of the specimens presented for analysis are natural objects like meteorites, for which there is no digital twin, or are cut or broken from larger items (for example, a few square centimeters of the exterior of a building where chemical analysis is required to evaluate fire safety), for which there is no digital twin or digital sampling cannot be done in the same way as the actual specimen was created. So far, there is no knowledge that these problems have been solved anywhere using digital twins generated by CAD, but it may happen in some cases in the future.Somewhat exceptional is the special case of semiconductor chips, where fiducial markers (such as crosses) are incorporated into the design along with the transistors on the chip and are designed to be visible optically and by other microscopy methods. For analysis, points on this planar chip surface can be defined based on the chip's CAD layout (e.g., by electron microscopy or X-ray methods). These points can be found using coordinates based on fiducial markers visible under the microscope. However, all of this is about the two-dimensional world. There is no problem with semiconductor chips, but most samples are 3D shapes with many topographical features in three dimensions.

[0016] Existing approaches Often, accidental features (e.g., a particular random dust pattern on the surface) are used to identify common areas in both microscopes, scale, shift, and match one microscope image to the other. And if possible, the images can be overlaid. Sometimes, fiducial markers such as small crosses carefully made with a sharp scalpel blade are added so that they can be used as registration markers later. Sometimes it is very difficult, and even if these fiducial markers are present in both images, they may appear very different. This is because the contrast mechanisms that occur in different imaging modes are different. As a result, in the most difficult cases, it becomes subjective, takes time to perform accurate co-registration, and may be difficult to repeat or reproduce in another laboratory. Even when the registration problem is clear, the method of recording spectra and other information associated with specific locations on the specimen is mostly limited to marking the position (e.g., #1, #2, etc.) on the photograph and recording the file name of the spectrum corresponding to that number in the lab notebook. This method makes it very difficult to discuss with customers and co-researchers in remote locations, for example, via a video conference.

[0017] Many approaches to the registration problem have been developed, but all are either difficult, time-consuming, or are proprietary methods developed by specific equipment manufacturers, often requiring continued use of that manufacturer's hardware. Equipment manufacturers may produce multiple types of equipment, such as SEMs and AFM. These manufacturers may, for example, provide customers with a way to facilitate the transfer of samples from an SEM to an AFM by using a specially designed fiducial marker or a common sample holder equipped with proprietary software. However, many users use a variety of equipment from a variety of equipment manufacturers, and the proprietary solutions of one company (e.g., Company P) are generally completely different from and incompatible with the methods sold by other companies (e.g., Company Q). Furthermore, if neither Company P nor Company Q manufactures a Raman spectrometer (as an example), even if a researcher needs to perform point analysis on the specimen surface, neither company's proprietary solution to the registration problem will help the researcher.

[0018] I (and my former Ph.D. students in the UK) recently published a research paper describing a method of registering images using computer-readable fiducial markers

[13] . In particular, it also includes information on how these markers can be fabricated (e.g., by focused ion beam (FIB) machining). This enables software operating on a computer to identify fiducial markers such as QR codes within images from different microscopes and overlay them. These computer-readable fiducial markers are known as "Apriltags (registered trademark)" [4] and were developed for use in computer vision for robotics (although QR codes and other markers also function). As far as we know, we are the first to apply Apriltags (registered trademark) to the registration problem of microscope images. While searching for ways to make this approach more useful and widely applicable, I was inspired by a type of "screen annotation software" available for personal computers.

[0019] Some prior arts There are many publications on correlative microscopy [12, 13, 21, 22] regarding registration problems. The "CISA" workflow by Thermo Scientific

[23] realizes correlative registration using a proprietary holder for large samples based on mechanical registration between SEM and XPS. In this method, the stage coordinates are locked from one device to another and are only operated in two dimensions (2D). Since CRFM (described below) is not used, it is limited to specific manufacturer's analysis tools and their proprietary software systems. Since it is almost impossible for a single manufacturer to produce the best analytical instruments for all the technologies that users desire, when going outside of this manufacturer's tool, one has to rely on means such as marking an X on the image to find the POI with the next device. Since out-of-plane CRFM as shown in FIG. 1 is not used, the CISA workflow

[23] is limited to 2D sample navigation. There is no equivalent to a DAT scanner in the CISA workflow, and it is necessary to load the specimen into an expensive XPS device before starting work.

[0020] The idea of computer-readable fiducial markers and their advantages has been recently published

[13] . Various types of fiducial markers have been tried so far

[21] , but usually manual identification and registration were required. In order to overcome the problem that different contrast mechanisms occur in different types of microscopes, the material of the fiducial marker is often specially selected (for example, fluorescent nanoparticles

[24] that can be easily identified with both optical microscopes and electron microscopes, etc.). However, even if it can be clearly seen with all microscopes, manual registration is required.

[0021] Screen annotation software Some software packages, both free and commercial, can write on what is displayed on a PC screen, thereby allowing one to effectively annotate it. This is possible even if the displayed application window is "live," i.e., running on that PC. The effect is like writing on something like cellophane in front of the screen of the computer running the software, and it is possible with any software. Such screen annotation software is very convenient for teachers and lecturers because one can write on what is displayed on the screen. For example, when giving a lecture on computer-aided design (CAD) software, one can circle specific buttons or draw arrows pointing to specific parts of the design without affecting the CAD software at all.

Summary of the Invention

[0022] A scanning system, method, and computer program product focused on the analysis of specimens are provided herein.

[0023] In one aspect, a scanning system for specimen analysis is disclosed. The scanning system includes a processor and a memory communicatively coupled to the processor. The memory stores a plurality of processor-executable instructions that, when executed by the processor, control a plurality of image capture devices to capture a plurality of initial images of a specimen and at least one fiducial marker. The specimen is disposed in a specimen holder, and the at least one fiducial marker is associated with either the specimen or the specimen holder. The scanning system is configured to apply one or more photogrammetry techniques to the plurality of captured initial images to generate a three-dimensional model (3D model) of the specimen. The one or more photogrammetry techniques capture information related to an initial coordinate system associated with the generated 3D model. The scanning system is further configured to generate a data structure related to the specimen based on the generated 3D model. The scanning system is further configured to output a data structure that includes an initial coordinate system associated with the generated 3D model, a second coordinate system related to the specimen, and a correspondence between the initial coordinate system and the second coordinate system.

[0024] In an additional embodiment, the scanning system is configured to control the movement of the specimen holder and rotate it from a first position to a second position. The scanning system is further configured to control the plurality of image capture devices to capture a second plurality of images of the specimen and the at least one fiducial marker, where the specimen holder is in the second position. Further, the scanning system is configured to generate a 3D model of the specimen based on applying one or more photogrammetry techniques to the plurality of captured initial images and the plurality of captured second images.

[0025] In an additional embodiment, the generated data structure corresponds to an Extensible Markup Language (XML) file.

[0026] In additional embodiments, at least one fiducial marker corresponds to any one of a Quick Response (QR) code, a barcode, an AprilTag (registered trademark), an ARtag, or an ArUco marker.

[0027] In additional embodiments, the scanning system is configured to generate a second coordinate system related to the specimen based on at least one fiducial marker.

[0028] In additional embodiments, the generated data structure includes information related to a first region of interest (RoI) of the specimen that is analyzed using a first analyzer integrated with the scanning system or using a second analyzer having a coordinate system different from that of the first analyzer.

[0029] In additional embodiments, the scanning system is configured to receive a first user input related to marking at least one point of interest in the generated 3D model. Further, the scanning system is configured to store information related to the marking of at least one point of interest in the generated data structure based on the reception of the first user input and output the generated data structure.

[0030] In additional embodiments, at least one point of interest is marked for analysis under one or more analyzers.

[0031] In additional embodiments, the specimen corresponds to a heterogeneous specimen.

[0032] In one aspect, a method for analyzing a specimen is provided. The method includes rendering a first image of a region of interest (RoI) of the specimen on a first analyzer, the rendered first image including at least one fiducial marker, which is captured by the first analyzer. Further, the method includes receiving, from the first analyzer, a second user input related to the selection of a point of interest within the rendered first image. The method also includes determining position information related to the point of interest selected based on the received second user input, the determined position information including the position of the selected point of interest relative to at least one fiducial marker. Further, the method includes storing the determined position information in a data structure. The method also includes receiving a third user input related to rendering a second image of the region of interest on a second analyzer, which is different from the first analyzer. Further, the method includes scanning the stored data structure based on the received third user input and controlling the second analyzer to determine the position of at least one fiducial marker within the second image. The method further includes applying at least one transformation technique to the position information stored in the data structure based on the scan. The method further includes rendering, on the second analyzer, the first image of the selected point of interest based on the application of the at least one transformation technique.

[0033] In an additional method embodiment, the first image of the selected point of interest is captured by the second analyzer.

[0034] In an additional method embodiment, the at least one transformation technique applied includes a projective geometric transformation technique.

[0035] In additional method embodiments, the method includes scanning the rendered first image to determine an initial position of at least one fiducial marker therein and determining position information associated with a selected point of interest based on the scan of the first image.

[0036] In additional method embodiments, the method includes receiving a fourth user input associated with the selected point of interest, the received fourth user input including a first label and first information associated with the selected point of interest, and storing the first label and the first information in a data structure based on the fourth user input.

[0037] In additional method embodiments, the data structure corresponds to a digital analysis twin (DAT) data structure associated with the specimen and is a digital replica of the specimen.

[0038] In additional method embodiments, the specimen corresponds to a heterogeneous specimen.

[0039] In additional method embodiments, the fiducial marker corresponds to any one of a quick response (QR) code, a barcode, an AprilTag (registered trademark), an ARtag, or an ArUco marker.

[0040] In one aspect, a method for analyzing a specimen is provided. The method includes controlling a plurality of image capture devices to capture a plurality of initial images of the specimen and at least one fiducial marker, the specimen being disposed in a specimen holder and the at least one fiducial marker being associated with either the specimen or the specimen holder. Further, the method includes generating a three-dimensional model (3D model) of the specimen based on applying one or more photogrammetry techniques to the captured plurality of initial images, the one or more photogrammetry techniques capturing information associated with an initial coordinate system associated with the generated 3D model. The method further includes generating a data structure associated with the specimen based on the generated 3D model.

[0041] In additional method embodiments, controlling movement of the specimen holder to rotate it from a first position to a second position while the specimen is disposed in the specimen holder. Also, controlling a plurality of image capture devices to capture a second plurality of images of the specimen and at least one fiducial marker, wherein the specimen holder is in the second position at this time. Further, the method includes further generating a 3D model of the specimen based on applying one or more photogrammetry techniques to the first plurality of captured images and the second plurality of captured images.

[0042] In additional method embodiments, the fiducial marker corresponds to any one of a quick response (QR) code, bar code, AprilTag (registered trademark), ARtag, or ArUco marker.

[0043] The invention includes a portable analysis registration software (PARS) system, a third-party system (but not part of the analyzer software) that operates in parallel with the software running the analyzer, and recording information about the specimen in a digital analysis twin (DAT) data structure. This digital analysis twin (DAT) data structure is created from a physical specimen using a hardware device called a DAT scanner.

[0044] PARS software system As described above, the PARS system may be a third-party system that operates in parallel with, rather than being part of, the software operating the analyzer. An important feature of the PARS system lies in how the PARS software interacts with the existing software used to acquire spectra and images from the analyzer. This will be explained using examples.

[0045] Take as an example two analytical devices, a first analytical device "J" and a second analytical device "K". The first analytical device "J" may be manufactured by a first company "JJ" and operated by a first software package "JJJ". Similarly, the second analytical device "K" may be manufactured by a second company "KK" and operated by a second software package "KKK". The first analytical device "J" and the second analytical device "K" may be selected from devices of the type shown in FIGS. 5-7 or FIGS. 11-15, or may be selected from the list of the aforementioned technologies. In one embodiment, the first company "JJ" and the second company "KK" may be the same, but the first software package "JJJ" and the second software package "KKK" are not compatible with different microscopes and may not provide anything regarding image registration. The present disclosure includes a third software package "N" that operates on the acquisition computers of both the first analytical device "J" and the second analytical device "K", or at least operates on a single computer that can read and display files from the first analytical device "J" and the second analytical device "K". This third software package "N" is first executed on the computer that operates the first analytical device "J", and then executed on the computer that operates the second analytical device "K". 1. The third software package "N" operates in the background on the same computer that is running the first software package "JJJ" (i.e., with few functions displayed on the screen). This operates on the same computer in parallel with the first software package "JJJ" but does not interact directly. The first software package "JJJ" controls the operation of the first analytical device "J" and displays images from the first analytical device "J". 2. The first software package "JJJ" displays an image of the surface of the specimen acquired by the first analytical device "J". This image includes a portion of the sample marked with a fiducial marker (e.g., Apriltag (registered trademark)) that can be read by a computer. 3. The user notifies the third software package "N" of the desire to mark a point on the sample (by pressing a "hot key" or, if displayed, pressing a button on the small graphical user interface (GUI) of the third software package "N"). 4. The user clicks on the location to be marked within the image displayed by the first software package "JJJ" using a mouse or pointer. The third software package "N" records this position on the screen and scans the image of the screen to search for a fiducial marker that can be read by the computer (in another embodiment, this can be done repeatedly so that the scan is completed by the time the user clicks on this position). The position of the selected point relative to the fiducial marker is saved (e.g., saved as a file in a DAT data structure on a USB memory, etc.). 5. Optionally, the third software package "N" prompts the user to enter labels and other information related to the saved points. This information is saved in a file together with the coordinates of the points. This saved data includes both the position of the point relative to the fiducial marker and other information about that point, and is saved as data "D". This data "D" may be in a DAT data structure. 6. Later, when the same sample is being observed by the second analyzer "K", the user calls the third software package "N" and marks the previously saved point positions (e.g., using a hot key). 7. The third software package "N" scans the screen to search for a fiducial marker that can be read by the computer. Then, it finds the corresponding marker in the saved data file (e.g., on a USB drive) and finds the previously defined points relative to that visualized marker. 8. The third software package "N" indicates the requested point at its exact position on the image from the second analyzer "K". This may be shown by moving the mouse pointer to the pixel corresponding to that point, and optionally, as an overlay on the display of the second software package "KKK", a marker (such as an arrow or a circle) of the point may be displayed together with a label. 9. Thereafter, the user can use the second analyzer "K" to analyze one or more of those points on the sample. Often, this means clicking on one of the points marked by the third software package "N", and the second software package "KKK" may be sensitive to the mouse position selected by the user. In fact, the mouse has already been moved to that position, and if the second software package "KKK" is ready to obtain point analysis, the mouse is already set to that point.

[0046] In some embodiments of the present invention, the third software package "N" and the data "D" can be easily transferred from the first analyzer "J" to the second analyzer "K". For example, the third software package "N" may be on a network storage accessible to both analyzers, and the data "D" may also be stored there. In another embodiment, the third software package "N" may be stored on a portable drive (such as a USB drive) that the user can carry from the first analyzer "J" to the second analyzer "K". When connected to the first analyzer "J", the data "D" is stored on this drive and later read by the third software package "N" executed on a computer associated with the second analyzer "K". As one embodiment, the data "D" may be a digital analysis twin (DAT) as described later.

[0047] It should be noted that in the above step 8, coordinate system conversion is required to find the position of the pixels on the screen corresponding to the defined points. This is generally an affine transformation, and the affine transformation is applied as long as there is no non-projective transformation (such as spherical aberration) in the optical system of the first analyzer "J" or the second analyzer "K". This is expected to be rare in well-manufactured commercial equipment. In Euclidean geometry, an affine transformation is a geometric transformation that preserves lines and parallelism (not necessarily distances and angles). The equation providing this affine transformation can be expressed in the form of an augmented matrix. In the first analyzer "J" and the second analyzer "K", since they are analyzers with different magnifications, distances (at least from the perspective of the pixels on the screen) are not preserved. Angles may not be preserved either. Since the angles with respect to the surface normal are different between "J" and "K", something that appears perpendicular in one may appear greater than or less than 90 degrees in the other.

[0048] One advantage of the present disclosure is that even if the user cannot access the first software package "JJJ" and the second software package "KKK", modify them, or reprogram them to transmit sufficient coordinate information to find the points of interest of the sample on both devices, correlation microscopy can still be performed. Such modifications are difficult and expensive and often require the cooperation of the device manufacturer, which is often not easily obtained.

[0049] Digital Analysis Twin and "DAT" Scanner Here, leaving the PARS system (or the third software package "N"), a Digital Analysis Twin (DAT) structure that can very effectively utilize the data obtained from the use of the PARS system will be described.

[0050] A method and a scanning system (DAT scanner) are disclosed. The function of this scanning system is to generate a "digital twin" from a specimen using one or more photogrammetry techniques. Generally, photogrammetry is a mathematical technique that generates the three-dimensional coordinates of points identified from multiple images of the same object taken from different angles. In the context of analysis (not manufacturing, where this term is currently used), a digital twin is a virtual representation that functions as a real-time digital counterpart of a physical object or process, and supports the systematic analysis of that physical object by including spatial and / or analytical (composition) information. In the context of analysis (for the reasons explained in the previous section), a digital twin typically needs to be generated by measurement rather than computer-aided design (CAD), and one or more photogrammetry techniques can be initiated in a quick and effective manner. Software related to one or more photogrammetry techniques is currently widely available and known in the technical field. For example, there is Pix3D® [6].

[0051] Furthermore, a new approach is disclosed for registering all analysis techniques applied to a specimen using fiducial markers that are computer-readable and fixed to a physical specimen. The fiducial markers are fixed to the specimen or fixed to a specimen holder that is fixed to the specimen (preferably a fiducial marker that is automatically readable by a computer from an image). An example of a specimen holder commonly used in a microscope is shown in FIG. 18. The computer-readable fiducial markers are firmly fixed to the specimen holder, printed on the surface, or written (e.g., by inkjet printing or laser engraving). The specimen is firmly fixed to the specimen holder (in accordance with existing practice). As a result, the specimen comes to have a fixed position and orientation with respect to the computer-readable fiducial marker (CRFM). These fiducial markers are used to define a coordinate system, preferably an xyz Cartesian coordinate system, and after a data file is provided by photogrammetry, a 3D representation can be obtained, for example, in the ".STL" format.

[0052] The method for a specific specimen is as follows (here, "sample" and "specimen" are used synonymously). 1. The specimen is firmly attached to a specimen holder equipped with one or more fiducial markers, or the fiducial markers are firmly attached to the specimen itself (e.g., by printing, adhesion, or focused ion beam machining). a. Optionally, these fiducial markers are automatically identifiable and readable by computer software (e.g., "AprilTags (registered trademark)" [7,8]). b. The fiducial markers are used to define a coordinate system suitable for the dimensionality of the specimen. Thus, for a 2D specimen (e.g., the surface of a silicon chip), three or more points in space are defined by the fiducial markers. For a 3D specimen, at least four points are defined by the fiducial markers. c. As a specific example, two AprilTags (registered trademark) on two sides of a cube are used to define eight points (the corners of the AprilTags (registered trademark)) in space. This cube is firmly fixed to the specimen and both AprilTags (registered trademark) are visible. d. The specimen holder displaying the fiducial markers may be pre-manufactured. In this case, the specimen only needs to be firmly fixed to the specimen holder. This is a common method in many types of specimen analysis. 2. The first plurality of images are captured from a plurality of image capture devices, usually almost simultaneously. These first plurality of images are conveniently captured by a plurality of image capture devices (or cameras) that provide views from a plurality of angles around the specimen. The plurality of image capture devices are controlled to capture the first plurality of images of the specimen. Usually, the first plurality of images are at least five images of the specimen. 3. One or more photogrammetry techniques can be applied to generate a 3D model of the specimen. For example, it can be generated as a ".STL" file. Importantly, one or more photogrammetry techniques capture fiducial markers within the image as well as the shape, and use them to define the xyz coordinate system in which the points of the 3D model of the specimen surface are defined. This enables "digital analysis twin" scanners and scanning systems to be more functional than conventional 3D photogrammetry scanners (e.g., the types used for rapid prototyping). Fiducial markers can later be imaged with the same fiducial markers on other microscopes, and are also used to generate the 3D (or 2D in the case of a planar specimen) coordinate system for identifying points on the specimen, even when a 3D model cannot be generated. The (fixed) relationship between this coordinate system and the 3D model coordinate system is recorded in the digital twin file. 4. Optionally, the user can also add points or regions to the 3D model of the digital analysis twin and label them with text for later analysis. 5. The digital analysis twin is taken to the analysis facility together with the specimen itself (e.g., saved on a USB drive and sent with the specimen attached, or transmitted via email or a file transfer server, using any method for transferring digital files). In one embodiment, the digital analysis twin file may be stored in a laboratory information management system (LIMS) that organizes and / or schedules the operations of the analysis laboratory (e.g., Agilent's iLab® [9] or Thermo Scientific's SampleManager® LIMS

[10] ). 6. Optionally, the digital analysis twin may be used to discuss the analysis strategy (i.e., which points of which specimens to analyze next and in what order). This can be done easily by visually inspecting the 3D model rendered on the computer screen, or (if the stakeholders are in different locations) using Internet conferencing and screen sharing software such as "Microsoft® Teams®" or "Zoom®".

[0053] The important point is the problem that occurs in step 3 above. Here, one or more photogrammetry techniques are used to determine the coordinates of many points on the surface of the specimen and generate a "3D model". However, this 3D model also incorporates a coordinate system defined by fiducial markers that can be read by a computer. This allows, when the specimen is later brought into an analytical device without photogrammetry capabilities, the points on the surface that make up the previously recorded 3D model to be identified using the coordinate axes defined based on the fiducial markers that can be read by a computer. For this, it is only necessary to use the fiducial markers that can be read by a computer and are visible in the image of the specimen obtained by the analytical device.

[0054] When the analysis specialist staff at the central facility observes a specimen using a computer connected to their analysis tool (e.g., an electron microscope), the points of interest defined in step 4 are automatically overlaid on the image using the software within that computer. For example, using the aforementioned PARS system, it is also possible to optionally annotate with the original user's comments and other information such as existing elements. This can be done automatically if fiducial markers are computer-readable. That is, the software identifies one or more fiducial markers within the image of the analyzer being displayed, and uses 3D trigonometry to find the position of the point of interest in the current image (the coordinates recorded in the digital twin file). Examples of computer-readable fiducial markers include AprilTags (registered trademark), ARTags

[11] , Arco markers, etc. Importantly, this does not require any access to or modification of the equipment manufacturer's software. Instead, annotation software on the screen can be used to overlay markers "on top of" the image being displayed by the equipment manufacturer's software. That is, it is displayed on the screen but not by the equipment manufacturer's software. The "digital analysis twin" file that provides a common xyz coordinate system makes it easier to correlate different microscopy techniques from different types of microscopes and analyzers. That is, it becomes possible to overlay the results of one technique on the results of another technique.

[0055] Steps 2 and the optional step 3 are performed by a scanning system called a "digital analysis twin scanner" or DATS. This constitutes the apparatus / system of the present invention.

[0056] Figure 1 schematically shows the main components of a DAT scanner (DATS) device. The specimen (130) may be fixed to a specimen holder (150), and fiducial markers (110 and 120) readable by a computer are firmly fixed to the specimen (130) and / or the specimen holder (150). In Figure 1, Apriltags (registered trademark) are shown as fiducial markers. The first plurality of images of the specimen and the fiducial markers are acquired by a plurality of image capture devices (cameras) (100), and the first plurality of images are further transmitted to a computer (140) for photogrammetric numerical processing. In experiments, an Ipevo (registered trademark) V4K camera (Ipevo (registered trademark) Inc., Sunnyvale, CA, USA, shown in Figure 19) and a general USB microscope camera (e.g., Colemeter (registered trademark) 2 megapixel type, Colemeter (registered trademark) Instrument Co. Ltd, Hong Kong, shown in Figure 20) were used for small specimens, but most digital cameras are suitable for this purpose. Although the figure is drawn in 2D, the plurality of image capture devices (110) are directed at the specimen (130) from a wide solid angle in 3D, intended to minimize ambiguity in photogrammetry. In an embodiment, the scanning system is configured to control the movement of the specimen holder (150) to rotate it from a first position to a second position. Further, the plurality of image capture devices are controlled to capture a second plurality of images of the specimen and at least one fiducial marker in the second position. The scanning system can further generate a 3D model of the specimen based on applying one or more photogrammetry techniques to the captured first plurality of images and the captured second plurality of images. Also, it should be noted that the specimen holder (150) is set to be rotatable along at least one axis, and by recording a plurality of images from each camera, the quality and accuracy of photogrammetry can be improved.

[0057] To ensure good lighting conditions and minimize shadows, the specimen is housed in a diffusely reflecting "integrating sphere" or a structure similar thereto, and a set of LEDs (light emitting diodes) may be integrated. In some embodiments, the integrated LED set may also be part of the components of the scanning system or the DATS scanner. During the experiment, the devices shown in FIGS. 24 and 25 were used individually to achieve this purpose.

[0058] Several examples of using fiducial markers readable by a computer have been published

[13] . Briefly, an analyzer that displays an image through a microscope on a computer monitor is connected to the computer, the analyzer can image the fiducial markers, and when at least the required number of fiducial markers are within the field of view, the points previously defined by the user are highlighted on the screen. This is possible even without access to the software provided by the device manufacturer, as long as the operating system can access the image currently being displayed on the monitor.

[0059] In some cases, the points of interest or regions of interest defined by the user may be outside the microscope's field of view when the fiducial marker is within the field of view. This case is schematically shown in Figure 2. In Figure 2, the fiducial marker (200) is an Apriltag (registered trademark) and is within the initial field of view (210) of the microscope. The four corners of the fiducial marker are at known positions within the digital twin information accessible to the computer operating the microscope. Using this position information, the software can determine that the point of interest (220) is not visible within this initial field of view, but can indicate in which direction it is (e.g., by displaying an arrow annotation on the screen). In this case, the software can automatically follow the panning of the microscope image on the screen when the operator manipulates to move the field of view. By moving from the initial field of view (FOV) (230) to the next FOV (240), the next FOV (250), and the last FOV (260), when the point of interest (210) enters the field of view, an annotation is displayed on the screen at that position. When panning is performed, by using the computer correlation of consecutive images (230, then 240, then 250, and then 260), this point can be accurately identified even when the fiducial marker is not within the field of view.

[0060] This is limited to the case where there is sufficient contrast and characteristic elements (not shown in Figure 2 but assumed to be present within 230, 240, 250, and 260) that can be accurately positioned relative to the previous FOV within these FOVs. In an actual specimen, it is almost always the case that such contrast exists and characteristic elements are available.

[0061] It is also possible to add the information of the analysis results to the digital twin file and associate it with the analysis position of the actual specimen. Thus, over time, the results from different analysis methods are accumulated in this digital analysis twin and reviewed by the center staff and analysis customers / users. This will utilize the special information of each analysis technique and deepen the understanding of the entire specimen.

[0062] Application to Practice: Mathematical Modeling and Solutions First, consider the coordinate system of an analyzer that patterns square tags on the sample surface. For simplicity, assume that the size in each direction of the square tag is taken as the unit. The actual size of the square may be, for example, 10 μm or 50 nm. Assume that the side of the square tag is twice the distance unit on the sample surface. In the case of a square fiducial marker, such as an AprilTag (registered trademark) or a QR code, it is possible to image the center position of the tag and the positions of its four corners (A, B, C, D), as shown in Figure 3. Multiple software packages are available for locating AprilTags (registered trademarks) and QR codes, and these typically provide at least the positions of the four corners of the square tag and the position of its center within a pixel array. In some cases, additional information such as a transformation matrix to assist in co - registration of the images may be obtained from software that automatically identifies these fiducial markers. These can be used. However, these additional matrices may not be obtainable without additional calibration or may be the least reliable elements in the software output, because they are not used frequently and have not been thoroughly tested. Therefore, the corners and center of the fiducial marker may be used for co - registration of the images. The corners (here labeled A, B, C, D) can be distinguished from each other based on the internal pattern of the AprilTag or QR code. For example, corner A will not be confused with corner C because the pattern is not symmetric enough for such confusion to occur. Since the x,y plane of the image is composed of a large number of grid - like pixels, for example, corner A of an AprilTag or QR code within the image is reported as a pixel having positions x1' and y1' (Figure 2). Usually, these coordinates are integers, but depending on the software that identifies the tag, more accurate positions may be provided as floating - point numbers, allowing the corner located between pixels to be found. There are two classes of 2D linear transformations - projective transformations and affine transformations. Although affine transformations are sufficient for most microscope-related transformations, a projective model is selected here. This is because it is more suitable for an oblique view of the surface. A projective transformation can be represented by the following matrix.

[0063] JPEG2025519265000002.jpg28102

[0064] Here, JPEG2025519265000003.jpg20102 is the rotation matrix. This matrix defines what kind of transformations are performed, such as scaling and rotation.

[0065] JPEG2025519265000004.jpg21101 is the translation vector. This simply moves the center of the square,

[0066] JPEG2025519265000005.jpg16102

[0067] is the projection vector. In many applications in microscopy, the elements of this projection vector are often small, but in the case of perspective in macrophotography, for example, they can be important elements. If x and y are the coordinates of a point, the transformation can be performed by simple multiplication.

[0068] JPEG2025519265000006.jpg26141

[0069] Here, x' and y' are the coordinates of the point after transformation. Define matrix A as follows: JPEG2025519265000007.jpg27139

[0070] The five points of the four corners and the center of the square tag are at known positions in the x,y plane of the FIB: (x,y) = (-1,-1), (1,-1), (1,1), (-1,1) and (0, 0). This last point is the center of the tag.

[0071] JPEG2025519265000008.jpg18134

[0072] The values of the five pairs of x' and y' are known from the tag identification within the image. First, find the matrix A of that image so that the pixels of the image can be mapped to the original defined plane (x, y). Often there are more than enough measurements, and this system of equations is overdetermined and can be solved by the least squares method using the Moore-Penrose pseudoinverse matrix. In fact, this pseudoinverse matrix is calculated only once as follows and can be used repeatedly.

[0073] JPEG2025519265000009.jpg41143

[0074] To find the coordinates of the original plane at the pixels (x', y') of any image obtained by different techniques, it is only necessary to multiply these coordinates by the pseudoinverse matrix.

[0075] JPEG2025519265000010.jpg26147

[0076] In 3D, as shown in Figure 1, it is extended by using fiducial markers that can be read by a computer extended to 3D, for example, two Apriltag (registered trademark). These are not for the 2D plane of the sample but can be used to define the xyz coordinate system in 3D. When there are fiducial markers that can be read by two or more computers like this, all but one of the point arrays on the sample surface generated by photogrammetry and one of the coordinate systems defined by CRFM are referred to one coordinate system by a linear transformation.

[0077] JPEG2025519265000011.jpg14162

[0078] Application to Practice II: Prototype Software Component The system "N" is prototyped using components from several sources, as described above: (a) Apriltag (registered trademark) recognition software (implemented in C++) [Example: 14] (for the recognition of Apriltag (registered trademark) in steps 4 and 7 above) (b) Autoit (registered trademark)

[15] (implementation of a small GUI for asking for sample names and point names in step 3 and other scenarios, capturing the screen to a file in steps 4 and 7, and for running components (a), (c), (d) in the correct order on the correct files) (c) Imagemagick (registered trademark)

[16] (for converting the file format from.png files to.jpg files, used in (a) and (b). Also used for contrast enhancement and inversion when necessary due to the limitations of the microscope's contrast mechanism. I selected the "portable 32-bit" version, which facilitates use on a portable USB drive as some device operation software still uses 32-bit Microsoft (registered trademark) Windows (registered trademark)) (d) gInk (registered trademark)

[17] (for screen annotation).

Brief Description of the Drawings

[0079]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

[0080] All are in accordance with embodiments of the present invention.

[0081] Detailed Description of the Invention For example, in an analytical facility affiliated with a university or a company, there may be 100 different analytical instruments, about 100 staff members, and perhaps about 3,000 users. The users may be university researchers, corporate research staff, and other people. Each user usually needs access to 3 or 4 different analytical instruments out of the 100 available. For example, assume a particular user needs to perform scanning electron microscopy (SEM), energy-dispersive X-ray analysis (EDS), and X-ray photoelectron spectroscopy (XPS) on 10 samples obtained from various catalyst studies. This is a very common case.

[0082] In this embodiment, it is assumed that the user is working at a location far from the central facility where the analysis work is being carried out. For example, assume the user is in Leeds, UK. As an option, the first thing the user does is attach each sample to an individual sample holder. These holders are attached with computer-readable fiducial markers such as QR codes, barcode markers, or AprilTag (registered trademark). There is no particular need to accurately position or orient the sample with respect to these fiducial markers. Instead, as an option, each sample is scanned using a Digital Analysis Twin Scanner (DATS) as shown in FIG. 1, and a Digital Analysis Twin (DAT) file of each sample, i.e., an image of the sample, is created as a computer data structure that is saved along with information on the orientation and position of the image with respect to the computer-readable fiducial marker. In fact, these fiducial markers are displayed in at least one image. This DAT data structure can be conveniently saved on a USB memory stick, a portable USB drive, or similarly, a specific location on a network drive or cloud storage. As an option, this storage (e.g., a USB memory stick) may contain the PARS software in an executable form (as a so-called "portable application"). All of these can be done in the user's own Leeds laboratory, far away from the central analysis facility (e.g., in London) that the user plans to use. The scanning system (or DAT scanner) that the user uses in their laboratory can potentially be much cheaper than the equipment and microscopes at the central facility. For typical sample sizes, the scanning system is likely to be a desktop-sized device that can easily fit in a laboratory or office, but if the sample is large, the scanning system may also be larger accordingly.

[0083] At this point, the user can view the image and select the points or areas they want to investigate with SEM, EDS, or XPS. These points of interest (POIs) or regions of interest (ROIs) are recorded in the digital analysis twin (DAT) data structure by the PARS software (this operation is performed by executing the coordinate transformation as described in the above equations 1 - 9). When the user brings a USB memory stick into the SEM device (e.g., to a central facility miles away from their own laboratory in London), the user searches for fiducial markers (CRFMs) that can be read by a computer on the screen. The PARS software (which runs in parallel with the SEM acquisition software and has access only to the screen displayed to the user) automatically recognizes the CRFM when it is displayed on the computer screen and adds annotations to the locations where previously created POIs or ROIs are present. At this time, the PARS software uses a photogrammetry-based model and coordinate transformation (equations 1 - 9 above). The user can see these POIs and ROIs marked and annotated on the screen (even if the computer operating the SEM does not have information about them). The user can add further POIs or ROIs to the DAT data structure once the SEM displays an image of the surface. This is done, for example, by the user clicking a computer mouse at a specific location. The PARS software captures this and calculates the 3D positions of these points relative to the CRFM and records them in the DAT data structure. These images are captured by PARS and saved in the DAT data structure. The user can move the SEM device's field of view to the POIs and ROIs that are displayed as annotations on the screen using the SEM controls.

[0084] Next, the user performs EDS on several POIs. The EDS spectra are recorded and the spectrum file itself, or a link to that spectrum file (e.g., a unique file name), is associated with the POI and saved in the DAT. EDS can usually be performed on the SEM device.

[0085] If the user knows that they will next bring the specimen to another device, a focused ion beam (FIB) is used to mark a new CRFM near the POI so that it can be visually recognized by the device to be used next. The view that the user sees on the screen of the computer operating the SEM is schematically shown in FIG. 22.

[0086] Thereafter, the user takes the specimen and the USB memory stick on which its DAT is stored to another device, such as an X-ray photoelectron spectrometer (XPS). This may be, for example, in Newcastle upon Tyne. The user uses XPS mapping to obtain a chemical image of the region containing some POIs and the FIB CRFM. This CRFM enables the POI to be defined more accurately and reproducibly because it is near the POI and has a small scale. Next, XPS spectra are recorded from some of the POIs, and these spectral files are saved or saved as links associated with the position coordinates of the POIs in the DAT for the specimen on the USB memory stick. FIG. 23 schematically shows the screen that the user sees when using the computer of the XPS device. The CRFM (2220) processed by FIB and the overlay annotation (2230) generated by the PARS software indicating the points previously defined by the user with the SEM are displayed. The PARS software recognizes the CRFM (2220) directly from the computer screen and calculates where to place the annotation and the label "P1".

[0087] At a later time (in some cases, months or even years later), the user can use the USB stick to view the analysis results of the specimen on their personal computer (without the need to connect to the analyzer). When summarizing the results for papers or industry reports, the user can navigate through the required images and quantitatively analyze the spectra while identifying the coordinates on the specimen from which the spectra were obtained. The user can also create 2D or 3D digital renderings that overlay false-color images highlighting specific chemical substances based on images from DATS, SEM, EDS, and XPS.

[0088] Some of the advantages for the user to perform this task using PARS, DAT, and DATS are that instead of spending time on very expensive equipment such as SEM and XPS, it can be inexpensively prepared in the user's own laboratory using a DAT scanner (DATS), potential errors in the description of POI and ROI can be avoided, since the analysis data is stored in a one-to-one correspondence with the specimen, the user can easily generate a view of the specimen with analysis data that is easy to understand for experts related to the specimen, even if the expert is not an expert in the analysis technique, the DAT data structure is standardized and may be agreed upon on a wider scale including analysis facilities (this cannot be done with experimental notes with handwritten annotations), and remote collaboration is promoted, so that in remote meetings or "Zoom (registered trademark)" meetings, DAT information can be displayed in various ways, saving the time spent using expensive SEM, XPS, and other equipment.

[0089] The present invention has several new features compared to the prior art, including but not limited to the following points. (a) The PARS software operates only through the images displayed on the screen, finds, recognizes, and locates fiducial markers that can be read by a computer within the image. Therefore, it can operate as third-party software rather than being provided by the company that sells the software for operating the analyzer. (b) The DAT scanner is a new device that creates a DAT model of a specimen by photogrammetry from multiple angles of the specimen (and fiducial markers attached and readable by a computer). Different from conventional photogrammetry scanners (there are several excellent examples in the literature [25,26]), it adds automatic recognition of CRFMs, creates an xyz coordinate system defined by these CRFMs, and places points that define the surface of the specimen (determined by photogrammetry) therein. (c) The digital analysis twin (DAT) described as above is different from the digital twins currently in use

[19] . DAT does not start from the CAD or CAM model of a manufactured item or building, but is empirically created by photogrammetry from a manufactured item or a found specimen (such as a natural object like an archaeological relic or a meteorite). Digital twins may be created before a physical entity exists and often do exist, but DAT does not. By being used in the creation stage of digital twins, the entire life cycle of an object can be modeled and simulated

[19] . DAT does not exist at the time of creation of an object and must be created by photogrammetry and other measurement techniques. DAT starts from a physical object, models and simulates the analysis life cycle of that object, and enables analysis including damage to data obtained as a result of previously applied techniques.

[0090] The DAT data structure can be realized in various ways. As an example, there is an extensible markup language (XML) file

[20] . This allows applying tools already developed for parsing and checking XML files to DAT. Within this structure, a specific (proprietary or open) analysis data format may be used to represent other analysis data regarding spectra, images, or samples.

[0091] JPEG2025519265000012.jpg243127

[0092] 「s12345.vms」 is the file name of a linked XPS spectrum in ISO14976 format, and 「E98765.eds」 is the file name of a linked EDS spectrum in proprietary format (the format of the SEM manufacturer). The file 「s31987.stl」 contains the topographic information of the specimen determined by photogrammetry with a DAT scanner.

[0093] Citation list JPEG2025519265000013.jpg208163JPEG2025519265000014.jpg236170JPEG2025519265000015.jpg244167

Claims

**Claim 1** A scanning system, the scanning system comprising: a processor; a memory communicatively coupled to the processor, the memory storing a plurality of processor-executable instructions that, when executed by the processor, cause the processor to perform the following operations: controlling at least one image capture device to capture a first plurality of images of a specimen and at least one fiducial marker. The specimen is disposed in a specimen holder, and the at least one fiducial marker is associated with at least one of the specimen or the specimen holder; generating a three-dimensional model (3D model) of the specimen based on applying one or more photogrammetry techniques to the captured first plurality of images. The one or more photogrammetry techniques capture information related to a first coordinate system associated with the generated 3D model; generating a data structure related to the specimen based on the generated 3D model; outputting the generated data structure, including a first coordinate system associated with the generated 3D model, a second coordinate system associated with the specimen, and a correspondence between the first coordinate system and the second coordinate system. **Claim 2** The scanning system according to claim 1, wherein the processor is further configured to: control the movement of the specimen holder to rotate the specimen holder from a first position to a second position; control at least one image capture device to capture a second plurality of images of the specimen and at least one fiducial marker. The specimen holder is in the second position; generating a further 3D model of the specimen based on applying one or more photogrammetry techniques to the captured first plurality of images and the captured second plurality of images. **Claim 3** The scanning system according to claim 1, wherein the generated data structure corresponds to an Extensible Markup Language (XML) file. **Claim 4** The scanning system according to claim 1, wherein the at least one fiducial marker corresponds to any one of a QR code, a barcode, an AprilTag, an ARtag, or an ArUco marker. **Claim 5** The scanning system according to claim 1, wherein the processor is configured to generate a second coordinate system related to the specimen based on at least one fiducial marker.

6. The scanning system according to claim 5, wherein the generated data structure includes information related to a first region of interest (RoI) of a specimen that is analyzed using a first analytical instrument integrated with the scanning system or using a second analytical instrument having a coordinate system different from that of the first analytical instrument.

7. The scanning system according to claim 1, wherein the processor is further configured to: receive a first user input related to marking at least one point of interest on the generated 3D model; save information related to marking the at least one point of interest in the generated data structure based on the reception of the first user input; and output the generated data structure.

8. The scanning system according to claim 7, wherein at least one point of interest is marked for analysis by one or more analytical instruments.

9. The scanning system according to claim 1, wherein the specimen is a non-uniform specimen.

10. A method, comprising: rendering a first image of a region of interest (RoI) of a specimen on a first analytical instrument, the rendered first image including at least one fiducial marker and being captured by the first analytical instrument; receiving, from the first analytical instrument, a second user input related to selection of a point of interest in the rendered first image; determining, based on the reception of the second user input, position information related to the selected point of interest, the determined position information including the position of the selected point of interest relative to at least one fiducial marker; saving the determined position information in a data structure; receiving a third user input related to rendering a second image of the region of interest on a second analytical instrument, the second analytical instrument being different from the first analytical instrument; Control a second analytical instrument to scan a data structure stored for determining the position of at least one fiducial marker in a second image based on a received third user input; Apply at least one transformation technique to the position information stored in the data structure based on the scan; Render a first image of a point of interest selected by the second analytical instrument based on the application of the at least one transformation technique.

11. The method according to claim 10, wherein the first image of the selected point of interest is captured by a second analytical instrument.

12. The method according to claim 10, wherein the at least one transformation technique applied includes an affine transformation technique.

13. The method according to claim 10, further comprising scanning the rendered first image to determine a first position of at least one fiducial marker in the rendered first image and determining position information related to the selected point of interest based on the scan of the first image.

14. The method according to claim 10, further comprising receiving a fourth user input related to the selected point of interest, the received fourth user input including an initial label and initial information related to the selected point of interest, and storing the initial label and initial information related to the selected point of interest in a data structure based on the received fourth user input.

15. The method according to claim 10, wherein the data structure corresponds to a digital analysis twin (DAT) data structure related to a specimen and is a digital replica of the specimen.

16. The method according to claim 10, wherein the specimen is a non-uniform specimen.

17. The method according to claim 10, wherein the fiducial marker corresponds to any one of a QR code, a barcode, an AprilTag, an ARtag, or an ArUco marker.

18. A method, comprising the following: Control at least one image capture device to capture a first plurality of images of a specimen and at least one fiducial marker. The specimen is disposed in a specimen holder, and the at least one fiducial marker is associated with at least one of the specimen or the specimen holder; Generate a three-dimensional model (3D model) of the specimen based on applying one or more photogrammetry techniques to the first plurality of captured images. The one or more photogrammetry techniques capture information related to a first coordinate system associated with the generated 3D model; Generate a data structure related to the specimen based on the generated 3D model; Output the generated data structure, including a first coordinate system associated with the generated 3D model, a second coordinate system associated with the specimen, and a correspondence between the first coordinate system and the second coordinate system.

19. The method according to claim 18, further comprising: Control the specimen holder to control a movement of rotating from a first position to a second position, and the specimen is disposed in the specimen holder; Control at least one image capture device to capture a second plurality of images of the specimen and at least one fiducial marker, and the specimen holder is in the second position; A method characterized by further generating a 3D model of the specimen based on applying one or more photogrammetry techniques to the first plurality of captured images and the second plurality of captured images.

20. The method according to claim 18, wherein the fiducial marker corresponds to any one of a QR code, a bar code, an AprilTag, an ARtag, or an ArUco marker.

Images