Method and device for processing multidimensional data to map atomic vibrations and / or atomic temperature

Abstract

Method and device for multidimensional data processing to map atomic vibrations and / or atomic temperature The present invention relates to a method for multidimensional microscopy data processing to map atomic vibrations and / or atomic temperature in a sample of at least one material, comprising, after acquisition (40) of a time sequence of input data blocks, for each input data block of said sequence, an application (44) of a super-resolution homogeneous component block transformation of an input data block, to obtain a super-resolved droplet block, comprising droplets representative of an atomic structure of a part of the observed sample; a time alignment (80) of the super-resolved droplet blocks of said time sequence;an extraction of centroids (84) from said droplet tiles obtained after time alignment, and a calculation of a time average (86) of said centroids allowing to obtain an atomic temperature map and / or projection (88) of displacement vectors of said centroids allowing to obtain an atomic vibration map. Figure for the abstract: Figure 2;

Description

Title of the invention: Method and device for processing multidimensional data to map atomic vibrations and / or atomic temperature

[0001] The present invention relates to a method and device for processing multidimensional microscopy data for mapping atomic vibrations and / or atomic temperature in a sample, comprising acquiring at least two images forming an input data block, each image of said block being representative of a part of the observed sample, said input data block being represented in an N-dimensional space, N being greater than or equal to two.

[0002] The invention lies in the field of processing multidimensional data, obtained by observation of samples composed of one or more materials, for the analysis of their physical properties and structures.

[0003] More particularly, the invention finds applications in the characterization of phonons and in the measurement of local temperature in materials, for example, applied in quality inspection in a production line for materials and devices, during the calibration or technological optimization phase of various devices using the materials. Physically, a phonon corresponds to a collective excitation in an arrangement of atoms constituting the structure. At the level of an individual atom, the greater the amplitude of vibration of an atom around its equilibrium position, the higher its temperature; therefore, measuring the amplitude of atomic vibrations makes it possible to probe the temperature locally.

[0004] The wave-particle duality stipulated by quantum mechanics highlights the fundamental importance of vibrations, and more generally of waves, for probing the properties of matter. It is therefore useful to have powerful tools for analyzing atomic vibrations, and vibrational spectroscopy is classically used in optics to map the nature of chemical bonds and the presence of defects, for example by infrared, UV-visible, or Raman spectroscopy. Indeed, atomic and molecular vibrations are characteristic of the arrangement of atomic bonds and structural dynamics, which precisely reflect the signature of a particular local chemical environment. However, the spatial resolution of optical methods is limited by the diffraction limit, which is particularly severe in the usual wavelength range of 10 to 100 microns.Despite numerous efforts to improve spatial resolution, no optical technique currently allows for the mapping of vibrations. atomic with atomic spatial resolution.

[0005] The analysis of physical properties of materials by processing data from spectra (1D) and images (2D) obtained by microscopy has been developed for the inspection of materials.

[0006] Images and spectra are obtained, for example, by high-resolution transmission electron microscopy (HRTEM and HRSTEM and 4D-STEM), X-ray fluorescence microscopy (HRSTEM-EDX), electron energy loss microscopy (HRSTEM-EELS and HRSTEM-VEELS), energy-filtered transmission electron microscopy (EF-HRTEM, EF-HRSTEM), atomic force microscopy (AFM), scanning tunneling microscopy (STM), atom probe tomography, or electron tomography. The microscopy images obtained are atomic-resolution images, that is, images with a spatial resolution of 2 nanometers or less.

[0007] Sets of spectra and representative images, each representing at least a portion of the observed sample, are obtained by microscopy, forming an N-dimensional data cube (or multidimensional data), also called a "datacube," where N is an integer greater than or equal to 2. This data cube contains droplets, also called spots, spikes, or blobs, that stand out against a homogeneous background (for example, light droplets or spots on a dark homogeneous background). These droplets are representative of structural features of the observed sample material, for example, an alignment of atoms along the direction of observation. The data cube acquired by microscopy may also contain noise.

[0008] For example, when observing crystals, drops arranged in a regular pattern represent the crystal lattice.

[0009] The simplest atomic model is a hard sphere model, which represents the particulate nature of an atom. The most powerful microscopes allow us to visualize atoms, which generally appear as droplets. In two dimensions, droplets are also called spots. In one dimension, droplets are points.

[0010] Mathematically, a droplet is defined as a simply connected component of a discrete topological space, in the sense of general topology. This means that any loop traced in a droplet can be reduced by homotopy to a point. Physically, a droplet is defined as the electrical signal produced by the pixels of a matrix detector following the impact of a particle (electron, photon, ion, fermion, boson, etc.). The particle is always much smaller than the pixel, so the droplet can always be reduced to a point by homotopy, in accordance with the mathematical definition of a droplet. There is therefore a precise agreement between the definition Physics and the mathematical definition of a droplet.

[0011] In the state of the art, the panorama of the spatial resolution of the different vibrational spectroscopies (EELS, NSOM, IXS, Raman, INS, FTIR) is summarized in the publication "Imaging vibrational excitations in the electron microscope", V. Kumar, J. Camden, J. Phys. Chem. C 2022, 126, 16919-16927. The best current method for measuring phonons and temperature of materials observed by microscopy is the electron energy loss technique (M. Lagos and PE Batson, “Thermometry with Subnanometer Resolution in the Electron Microscope Using the Principle of Detailed Balancing”, Nano Lett. (2018), 7, 4556-4563), which has been proposed to map phonons and absolute temperature with subnanometer spatial resolution (200 pm), with inevitable delocalization over several nanometers.A potential drawback of this technique is the potentially lengthy data acquisition time, which could damage the most fragile materials through electron bombardment. Furthermore, this method lacks sufficient spatial resolution to measure the atomic vibrations of an elementary atom in the sample with a spatial resolution <20 pm.

[0012] The evolution of the most advanced technological nodes of the ITRS (“International Technology Roadmap for Semiconductors”) generates the need to localize atomic vibrations and temperature on an increasingly smaller scale, while preserving the observed sample.

[0013] There is therefore a need to develop a method for measuring temperature and vibrations at the scale of a single atom.

[0014] To this end, the invention proposes, according to one aspect, a method for processing multidimensional microscopy data to map atomic vibrations and / or atomic temperature in a sample of at least one material, comprising acquiring at least two images of said sample forming an input data pad, each image of said input data pad being representative of a portion of the observed sample, said input data pad being represented in an N-dimensional space, N being greater than or equal to two, each data point of said pad corresponding to a point in the N-dimensional space. This method comprises the steps of:

[0015] a) acquisition of a temporal succession of input data blocks,

[0016] b) for each input data block of said sequence, for at least one data block extracted from said input data block, referred to as a component block, comprising homogeneous data according to a homogeneity criterion relating to at least one material of the observed sample, application of a super-resolution transformation to said component block to obtain a block, referred to as a super-resolved droplet block, comprising droplets representative of an atomic structure of a part of the sample observed,

[0017] c) temporal alignment of the super-resolved droplet tiles of said temporal succession,

[0018] d) extraction of centroids from said droplet blocks obtained after temporal alignment,

[0019] e) calculation of a time average of said centroids allowing to obtain an atomic temperature map and / or projection of displacement vectors of said centroids allowing to obtain an atomic vibration map.

[0020] Advantageously the applied treatment makes it possible to map atomic vibrations and / or atomic temperature with picometric spatial resolution.

[0021] The method for processing multidimensional microscopy data to map atomic vibrations and / or atomic temperature in a sample of at least one material according to the invention may also have one or more of the following characteristics, taken independently or according to all technically feasible combinations.

[0022] The application of the super-resolution transformation involves a convolution by an HBSG filter with a square kernel of size less than or equal to the median of the distances between neighboring drops.

[0023] The application of a super-resolution transformation further entails an increase in the number of points of the N-dimensional tile obtained by said super-resolution transformation.

[0024] The application of a super-resolution transformation further includes another application of a square kernel HBSG filter of size less than or equal to the median of the distances between neighboring drops after increasing the number of points and a local normalization step.

[0025] The application of a super-resolution transformation to obtain a super-resolved droplet tile comprising drops representative of an atomic structure of a part of the observed sample implements a segmentation step using a machine learning algorithm, trained to classify the data of the super-resolved droplet tile into two classes, respectively a "background" class and a "droplet" class, thus obtaining a segmented droplet tile.

[0026] The application of a super-resolution transformation to obtain a super-resolved droplet tile comprising drops representative of an atomic structure of a part of the observed sample further implements a special spectral filtering applied to the segmented droplet tile.

[0027] The special spectral filtering comprises applying a Fourier transform to the segmented droplet patch to obtain a diffractogram of the droplet patch, applying a specific spectral filtering mask to said diffractogram, and a application of an inverse Fourier transform.

[0028] The method includes a calculation of said specific spectral filtering mask comprising: a calculation of a normalized adjusted theoretical diffractogram between 0 and 1 and a machine learning segmentation of the diffractogram of the droplet paving allowing to obtain a diffractogram of the segmented droplet paving, then a normalization between 0 and 1 of the diffractogram of the segmented droplet paving, the specific spectral filtering mask being formed from the maximum of the normalized adjusted theoretical diffractogram and said normalized segmented droplet paving diffractogram.

[0029] The method further comprises, after the time alignment step, a time super-resolution step (82), comprising a calculation of one or more intermediate droplet blocks.

[0030] Extracting centroids from the droplet blocks obtained after temporal alignment involves calculating, for each of said droplet blocks, and for each droplet in the droplet block, a centroid of said droplet, and tracking the temporal position of the centroids between successive droplet blocks.

[0031] The invention also relates to a computer program comprising software instructions which, when executed by a computer, implement a method for processing multidimensional microscopy data to map atomic vibrations and / or atomic temperature in a sample of at least one material as defined above.

[0032] The invention also relates to a multidimensional microscopy data processing device for mapping atomic vibrations and / or atomic temperature in a sample of at least one material, implementing the acquisition of at least two images of said sample forming an input data pad, each image of said input data pad being representative of a portion of the observed sample, said input data pad being represented in an N-dimensional space, N being greater than or equal to two, each data point of said pad corresponding to a point in the N-dimensional space. This device comprises:

[0033] - a module for obtaining a temporal sequence of input data blocks,

[0034] - for each input data block of said succession, for at least one block data extracted from said input data block, called component block, comprising homogeneous data according to a homogeneity criterion relating to at least one material of the observed sample, a super-resolution transformation module applied to the component block to obtain a block, called the super-resolved droplet block, comprising droplets representative of an atomic structure of a part of the observed sample,

[0035] - a time alignment module for the super-resolved droplet tiles of said temporal succession,

[0036] - a centroid extraction module from said droplet blocks obtained after temporal alignment,

[0037] -a calculation module for a time average of said centroids allowing to obtain an atomic temperature map and / or a projection module for displacement vectors of said centroids allowing to obtain an atomic vibration map.

[0038] The invention will become clearer upon reading the following description, given solely by way of non-limiting example, and made with reference to the drawings in which:

[0039] [Fig-1] [Fig.1] is a block diagram of a system for mapping vibrations atomic and / or an atomic temperature in a sample comprising a multidimensional microscopy data processing device according to an embodiment;

[0040] [Fig.2] [Fig.2] is a synoptic diagram of a multidimensional microscopy data processing method according to one embodiment;

[0041] [Fig.3] [Fig.3] is a synoptic diagram of the steps of a super-re-solution transformation according to one embodiment;

[0042] [Fig.4] [Fig.4] is a synoptic diagram of the steps of a specific spectral filtering according to an embodiment;

[0043] [Fig.5] [Fig.5] is a two-dimensional example of an input data tile and corresponding super-resolution tile obtained by applying a super-resolution transformation;

[0044] [Fig.6] [Fig.6] illustrates a two-dimensional example of centroid positioning;

[0045] [Fig.7] [Fig.7] represents an example of a time average of a droplet with the trace of all its movements integrated over the data acquisition period.

[0046] Fig. 1 schematically illustrates a system 2 for mapping atomic vibrations and / or atomic temperature in a sample of one or more materials from multidimensional data representative of the sample, acquired by a characterization machine 4.

[0047] Any microscopy technique suitable for the characterization of such a sample is applicable.

[0048] In one embodiment, the characterization machine 4 is a transmission electron microscope (TEM) which allows the acquisition of images of a sample composed of one or more materials, including, for example, crystalline materials.

[0049] The electron microscope 4 allows simultaneous acquisition of images of different electron fraction, EELS spectra (for "Electron Energy Loss Spectroscopy"), EDX spectra (for "Energy Dispersive X-Ray Analysis"), signals from various sensors (BF for "Bright field", DF for "Dark field", DPC for "Differential phase contrast", or any other suitable or customized sensor), for each point of the sample in scanning microscopy mode called STEM mode (for "Scanning Transmission Electron Microscope"). Another possible acquisition mode is TEM mode, which allows obtaining an overall image without having to scan the electron beam.

[0050] In another embodiment, the characterization machine 4 is a probe microscope such as the atomic force microscope or the scanning tunneling microscope or other variants such as the Kelvin probe microscope for example, in a measurement mode enabling the acquisition of atomic spatial resolution images.

[0051] Each acquired spectrum or image is represented as a digital image, comprising points or pixels, each image being representative of at least a part of the observed sample.

[0052] The set of acquired spectra and images forms an N-dimensional data cube, where N is a natural number greater than or equal to 2. Such a data cube is also called a "datacube".

[0053] To obtain the trace of atomic movements, several spectra or images are acquired over time, during a data acquisition period, showing an evolution of the sample during the analysis. Time is therefore a dimension of the data set, which allows the temporal evolution of the material, i.e. its dynamics, to be analyzed.

[0054] As an optional complement, another dimension of the data block is the microscope focus, the electron energy, an angle (of the sample, of electron collection, of electron convergence), or any other microscope setting parameter that may vary in a controlled manner during the measurement.

[0055] Incomplete data may be extrapolated from neighboring values ​​where appropriate, for example, to correct any imperfections encountered during data acquisition. As a general rule, measurements at the atomic scale are very often affected by defects because simple acoustic or electronic noise can sometimes interfere with them.

[0056] In a multidimensional data cube of a crystalline sample, the unit cell usually forms a regular network of droplets representing the arrangement of atoms. Simulations make it possible to predict which N-dimensional data cubes will be expected for a given material and for a particular microscope setting.

[0057] As already indicated above, mathematically, a drop is a component simply connected to a discrete topological space, in the sense of general topology.

[0058] The pixel values ​​belonging to a droplet are distinguished from an image background. This background is defined by an intensity reduced to zero after segmentation.

[0059] A multidimensional data block of an observed sample is transmitted to a multidimensional microscopy data processing device 6 to map atomic vibrations and / or atomic temperature in a sample.

[0060] For example, the transmission is carried out by a wired link or by a wireless link (optical, radio, or other).

[0061] The processing device 6 is, in one embodiment, a programmable electronic device, e.g. a computer.

[0062] The device 6 includes a processor 8 (CPU or GPU) associated with an electronic memory 10. Optionally, the device 6 includes a human-machine interface 12, notably comprising a data display screen. In addition, the device 6 includes or is connected to a storage memory 14. The elements 8, 10, 12, and 14 of the device 6 are adapted to communicate via a communication bus 16.

[0063] The processor 8 (CPU or GPU) is configured to execute modules 18, 20, 22, 24, 26, 28, 30 and 32 stored in electronic memory 10, to implement a multidimensional data processing method representative of the observed sample to map atomic vibrations and / or atomic temperature in the sample.

[0064] Module 18 is a module for obtaining multidimensional data blocks to be processed, configured to obtain a data block comprising at least two input images, and for example a plurality of input images over a data acquisition period, from a data block obtained by the characterization machine 4.

[0065] For example, the acquisition module 18 is configured to obtain the data blocks to be processed (or input data block) from an electronic memory, where this data has been stored after acquisition.

[0066] Module 20 is a partitioning module which divides the data block into several blocks (or sub-blocks) each containing homogeneous data according to a chosen homogeneity criterion.

[0067] Module 22 is a super-spatial resolution transformation module of the input pad into a pad of drops representative of the atomic structure of the observed sample.

[0068] Module 24 is a temporal alignment module, implementing a correction of possible spatial derivatives of the sample during acquisition.

[0069] Module 26 is a super-temporal resolution module, configured to increase temporal resolution. The super-temporal resolution module 26 is implemented optionally.

[0070] The centroid extraction module 28 implements centroid extraction from drops, enabling the detection of possible displacements relative to an average equilibrium position.

[0071] The atomic temperature calculation module 30 implements a time average calculation of centroids, allowing an atomic temperature map 34 to be obtained.

[0072] The module 32 for calculating a map 36 of atomic vibrations implements a projection of displacement vectors, as described in more detail below.

[0073] In one embodiment, the modules for obtaining 18 a multidimensional data block, partitioning 20, spatial super-resolution transformation 22, temporal alignment 24, temporal super-resolution 26, centroid extraction 28, atomic temperature calculation 30 and atomic vibration mapping 32 are each implemented in the form of software, and form a computer program, comprising software instructions which, when executed by a computer, implement a method for processing multidimensional microscopy data to map atomic vibrations and / or atomic temperature in a sample of at least one material as described in more detail below.

[0074] In an alternative not shown, the modules for obtaining 18 a multidimensional data block, partitioning 20, spatial super-resolution transformation 22, temporal alignment 24, temporal super-resolution 26, centroid extraction 28, atomic temperature calculation 30, and atomic vibration mapping 32 are each implemented as a programmable logic component, such as an FPGA (Field Programmable Gate Array), a GPU (graphics processing unit), or a GPGPU (General-purpose processing on graphics processing unit), or as a dedicated integrated circuit, such as an ASIC (Application-Specific Integrated Circuit). The alternative configurations obviously include developments known to those skilled in the art in the field of computer science and, more generally, automated information processing.

[0075] The multidimensional microscopy data processing software for mapping atomic vibrations and / or atomic temperature in a sample of at least one material is further capable of being recorded, in the form of a computer program comprising software instructions, on a computer-readable medium, not shown. The computer-readable medium is, for example, a medium capable of storing electronic instructions and being coupled to a bus of a computer system. By way of example, the readable medium is an optical disc, a magneto-optical disc, a ROM, a RAM, any type of non-volatile memory (e.g. EPROM, EEPROM, FLASH, NVRAM, RRAM, PCRAM, 2D NAND, 3D NAND, SLC NAND, MLC NAND, TLC NAND, V-NAND, QLC), a magnetic card or an optical card, an SSD, and any variant known to those skilled in the art in the field of digital data storage.

[0076] An embodiment of the method for processing multidimensional microscopy data to map atomic vibrations and / or atomic temperature in a sample of at least one material will be described below with reference to Figures 2, 3 and 4.

[0077] Fig. 2 is a synoptic diagram of the main steps of a multidimensional microscopy data processing method for mapping atomic vibrations and / or temperature in a sample of at least one material.

[0078] The method includes a step 40 of obtaining (or acquiring) a block of input data to be processed, comprising at least one image, representing the sample to be inspected.

[0079] Step 40 is repeated so as to acquire a temporal succession of blocks of input data to be processed over a chosen data acquisition period.

[0080] The data block comprises droplets detached from a homogeneous background, for example, light droplets on a dark background in HRSTEM-HAADF microscopy, or HRTEM possibly with energy filtering, representative of the atomic structure of the observed sample. The relationship between the position of the atoms and the position of the droplets is obtained by simulations that take into account, in particular, the microscope settings.

[0081] In general, the input data pad is an N-dimensional pad, N greater than or equal to 2.

[0082] The data in the tile are numerical values, each value being associated with a point in N-dimensional space. Such a point is also called a pixel when N=2. Each point has an associated coordinate in each dimension, generally represented by an index.

[0083] In one embodiment, N=2, the input data block is then an LxW matrix (i.e. L rows and W columns), composed of pixel values, each pixel having respective coordinates (x,y), x being for example a row index and y a column index.

[0084] Optionally, if the material is not homogeneous, step 42 of spatial partitioning or cutting the input data block into sub-blocks of input data is implemented.

[0085] By definition, each sub-block of input data is a discrete connected space in the sense of general topology and comprises homogeneous data according to a criterion relating to at least one material of the observed sample.

[0086] The homogeneity criterion is structural homogeneity, referring to the arrangement of atoms in a given material. Partitioning is performed, for example, by machine learning, e.g., by supervised segmentation, semi-supervised segmentation, or by autonomous classification after training (e.g., deep learning). Classically, the structure is quantified, for example, by means of a digital diffractogram. A sub-tile of drops is considered representative of the structure if it contains more than 3 to the power of n, where n represents the size of the tile. For example, for an image n=2, a sub-tile is considered representative of the local structure if it contains at least 9 neighboring drops.

[0087] Structural homogeneity is obtained when the digital diffractograms of all representative droplet sub-blocks are similar, i.e., when their relative distances are less than a chosen threshold. Numerous methods for quantifying similarity exist in the literature, which can be used to quantify structural homogeneity via the similarity of Fourier transforms. The Fourier transform provides an image of the structural order, similar to the crystalline order revealed by an electron diffraction pattern. Depending on the variant, the homogeneity criterion is a criterion of homogeneity of data intensity values ​​and / or dielectric permittivity and / or electromagnetic properties, for example, measured in-situ during the acquisition of the data block. VEELS measurements, in particular, allow for obtaining the local dielectric permittivity, as demonstrated in the literature.This criterion of structural homogeneity or physical properties therefore makes it possible to partition the input data block into areas of interest which are homogeneous sub-blocks of input data.

[0088] Step 42 is followed by a transformation step 44 called a super-resolution transformation of said input data block to obtain a droplet block called a super-resolution droplet block, comprising droplets representative of an atomic structure of a part of the observed sample or of the observed sample.

[0089] Step 44 of the super-resolution transformation is performed on the homogeneous input data pad or on each sub-pad of homogeneous input data.

[0090] When several homogeneous input data sub-blocks have been obtained at the end of step 42, step 44 is followed by a spatial departitioning or reassembly step 75, consisting of joining the super-resolved drop sub-blocks into a super-resolved drop block representative of the sample.

[0091] An embodiment of the super-resolution transformation step 44 is described with reference to [Fig.3].

[0092] Step 44 implements a convolutional filtering (step 46) with an HBSG kernel (for "Half Bail Savitzky-Golay") as described in the application of French patent FR3 118 256 ("IMAGE PROCESSING FILTER FOR DROPPER LOCATION IN A MULTIDIMENSIONAL IMAGE AND ASSOCIATED IMAGE PROCESSING METHOD"). This filtering method, for example, uses a square HBSG kernel with a size corresponding to the median distance between neighboring drops (e.g., 21 pixels), with 2nd or 3rd order smoothing. The smaller this kernel, the more it preserves the finest details; therefore, the size is chosen to match the desired level of detail. Conversely, a kernel that is too small eliminates less noise.

[0093] The convolution filtering step 46 is followed by an optional step 48 of increasing the number of points (or pixels).

[0094] In one embodiment, step 48 implements interpolation using a reconstruction technique or a super-resolution technique such as the total variation (TV) method. This method involves regulating the creation of new points based on the total variation of contrasts in the original image. This approach preserves the overall contrast variations while maintaining the edges of the image droplets. The total variation method is described in particular in the publication by L.I. Rudin, S. Osher, and E. Fatemi, "Nonlinear total variation based noise removal algorithms," Physica D, 60 (1), 259-268 (1992).

[0095] The optional step 48 of increasing the number of points is followed by a second convolution filtering 50, identical to the convolution filtering 46 with an HBSG kernel, with the objective of correcting any artifacts introduced by the step 48 of increasing the number of points.

[0096] Optionally, step 44 also includes a local normalization step 52 consisting of dividing the values ​​of the processed data block by the local variance, obtained by calculating the square root of the convolution with a Gaussian filter of the squares of the values ​​of the processed data block. Preferably, the dimension of the Gaussian filter used to calculate the local variance represents half the median of the distances between neighboring drops.

[0097] The super-resolution transformation step 44 then includes a super-resolution droplet segmentation step 54, in order to replace the droplet neighborhood with a uniformly black background.

[0098] According to a first variant, the segmentation 54 implements an adaptive Bemsen thresholding, known in the field of image processing, with a radius close to half the median of the distances between neighboring drops (for example 10 pixels).

[0099] According to a second variant, segmentation 54 implements machine learning as described in more detail below. The machine learning algorithm makes it possible to distinguish two classes, respectively the class of "drops" (or "spots") (in English) and the "background" class.

[0100] For example, machine learning is performed with a fast random forest type algorithm, described in the article by Leo Breiman (2001). “Random Forests”, published in Machine Learning. 45(1):5-32.

[0101] Alternatively, other machine learning classification algorithms can be implemented, for example Bayesian, function-based, rule-based, or a combination of such algorithms, or based on the entire range of tools used in machine learning classification, such as deep learning. Machine learning is performed with hole-free training regions because the goal is to approximate the theoretical shape of an ideal droplet, which by definition is hole-free.

[0102] Typically, droplets represent less than 20% of the image, and their size is less than 1 nm, after calibration against the actual sizes in the sample. Simulations of theoretical images allow us to determine the expected distribution of droplets in the image in order to verify that the segmentation is correct. Segmentation is correct if the number and position of the droplets approach the expected theoretical values ​​beyond a given uncertainty threshold.

[0103] The super-resolution transformation step 44 then includes a spectral filtering step 56, which implements a specific spectral filtering mask whose construction will be described below with reference to [Fig. 4]. The spectral filtering 56 implements a Fourier transform, processing steps described below, and an inverse Fourier transform.

[0104] Spectral filtering step 56 is optionally followed by artifact removal step 58, which consists of eliminating all areas exhibiting residual imperfections, particularly the edges of the image. Indeed, Fourier transforms always generate artifacts around the perimeter of the data frame, which must therefore be masked. Atomistic simulation of perfect theoretical images makes it possible to precisely determine which regions of the data frame exhibit such local digital aberrations that must be eliminated by selective masking.

[0105] Finally, the super-resolution transformation step 44 includes a droplet segmentation step 60, in order to obtain droplets that stand out against a homogeneous background, for example a black background, completely free of noise. For example, step 60 implements a segmentation analogous to the segmentation 54 described above.

[0106] As illustrated with reference to [Fig.4], in one embodiment the spectral filtering step 56 first implements a Fourier transform applied in step 62 on the droplet patch from the segmentation step 54.

[0107] The expression "Fourier transform" refers to the Fourier transform and all its variants and generalizations to hyperspaces, including the Hartley transform, the discrete Hartley transform (DHT), the fast Fourier transform, the discrete Fourier transform, the generalized discrete Fourier transform, the short-time Fourier transform (STFT), the fractional Fourier transform (FRFT), the Chirplet transform, the Hankel transform, the Bros-Lagolnitzer Fourier transform, the canonical linear transform, the discrete-time Fourier transforms (DTFT), the discrete-space Fourier transform (DSFT), the Z-transform, the modified discrete cosine transform (MDCT), and the Fourier transform of finite groups.

[0108] At the end of step 62 of application of the Fourier transform a spectral tile TF1 is obtained, which is a diffractogram of the droplet tile after segmentation 54.

[0109] The process then includes a step 64 of segmenting the spectral tile TF1 to obtain a segmented spectral tile (or diffractogram) TF1*.

[0110] For example, segmentation step 64 implements a machine learning segmentation method similar to step 54 which allows two classes to be distinguished, respectively the class of "drops" (or "spots" in English) and the class of "background".

[0111] For example, machine learning is performed with a fast random forest type algorithm, described in the article by Leo Breiman (2001). “Random Forests”, published in Machine Learning. 45(1):5-32.

[0112] The method also includes a calculation 66 of a theoretical diffractogram DT fitted to the segmented diffractogram TF1* of the paving of drops.

[0113] Two scenarios must be considered.

[0114] In a first scenario, the nature of the sample material and its structure are known. In this first scenario, the theoretical diffractogram DT is also known, and it suffices to perform slight distortions by affine transformation to superimpose it on the segmented diffractogram TF1*.

[0115] In a second scenario, the nature of the sample material and its structure are unknown. In this second scenario, the theoretical diffractogram is calculated from the segmented diffractogram TF1*: periodic peaks are detected in the segmented diffractogram TF1*, and the observed periodicity is extended in all directions to fill the N-dimensional tile to the edges. This yields a theoretical diffractogram DT extrapolated onto the existing droplets.

[0116] The method then includes an adjustment step 68, consisting of adjusting the theoretical diffractogram DT as a function of the diffractogram or tile The segmented spectral diffractogram TF1* was calculated in step 64. The adjustment 68 of the theory with respect to the experiment consists of varying the theoretical parameters of lengths (a,b,c) and angles (a,[3,y]) defining the crystal unit cell in order to minimize the root mean square deviation between the experimental segmented diffractogram TF1* and the theoretical diffractogram DT. A fitted theoretical diffractogram, DT*, is then obtained.

[0117] The process then includes a step 70 of calculating a specific spectral filtering mask MFS. The calculation step 70 includes a normalization between 0 and 1 of the theoretical adjusted diffractogram DT* and experimental TF1*, and then a combination of these diffractograms, the specific spectral filtering mask MFS being formed from the maximum of the theoretical adjusted diffractogram DT* normalized between 0 and 1 and the experimental segmented diffractogram TF1* normalized between 0 and 1.

[0118] The specific spectral filtering mask MFS thus obtained is applied in step 72 to the Fourier transform of the segmented drops obtained after step 54.

[0119] Mathematically, applying the mask involves multiplying each term of the Fourier transform of the segmented drops obtained after step 54 by its counterpart in the spectral filtering mask MFS, to retain only the spectral information contained in the non-zero regions of MFS. This filtering therefore preserves only the frequencies selected by MFS. An optional binarization then improves the contrast of the diffractogram and reduces its size. The 32-bit or 64-bit data is then converted into 8-bit data, which is easier to handle in the case of very large data volumes.

[0120] The process then includes an application 74 of the inverse Fourier transform, thus allowing a filtered droplet tile to be obtained in the spatial domain after correction by spectral filtering.

[0121] Returning to [Fig.2], the super-resolution transformation step 44 is followed by a departitioning step 75 already mentioned.

[0122] Steps 40 to 75 are repeated for a plurality of data block acquisitions, so as to form a time sequence of filtered droplet blocks, which are stored in an electronic memory of the processing device.

[0123] The process then includes a time alignment step 80. This time alignment step 80 implements a time series realignment (or registration), also known as "drift correction." For example, the NanoJ software described in "NanoJ: a high-performance open-source super-resolution microscopy toolbox" by Romain F. Laine et al., published in Journal of Physics D: Applied Physics; 52, 2019, is implemented.

[0124] The method then optionally includes a super-temporal resolution step 82, consisting of increasing the temporal resolution if necessary by calculating intermediate droplet tiles (for example, images when N=2 or 3). The step 82 implements any known classical algorithm, for example bilinear, bicubic or B-spline interpolation; or a method based on generative artificial intelligence or any other classical or AI variant known to the person skilled in the art.

[0125] The process then includes a step 84 of extracting centroids from the drops, from each processed drop image.

[0126] The centroid is classically the center of gravity of a drop. A single image is therefore sufficient to determine the centroid of each drop in that image.

[0127] It should be noted that each drop represents an atom or an alignment of atoms, so the proposed process can have a resolution down to the individual atom.

[0128] Step 84 allows the center of gravity of the drops to be found automatically, which will subsequently allow the temporal vibrations of the drops to be characterized.

[0129] For example, step 84 implements a method for tracking the temporal position of centroids between successive droplet blocks, which in one embodiment is a particle tracking method, for example described in the publication N. Chenouard et al., “Objective comparison of particle tracking methods”, Nature Methods, 11 (3), 281-289 (2014).

[0130] The process then includes a step 86 of calculating time average and / or a step 88 of projecting displacement vectors.

[0131] Step 86, which calculates the time average of the centroids, calculates an average deviation from the equilibrium position over a temporal succession of droplet clusters during the data acquisition period. The average position deviation is one-to-one with the droplet temperature, thus step 86 provides an atomic temperature map. The more the trace of successive positions is spread out in space, the higher the temperature.

[0132] Step 88 of displacement vector projection performs a projection of displacement vectors between centroids associated with the same droplet between successive blocks of drops, for each time step. If M; is the position of the centroid at time i, then the displacement vector is the vector M;Mi+i. Step 88 of displacement vector projection is therefore simply the sum of all the vectors M;Mi+i when the indices i sweep through all the time acquisition steps, from the first image (i= 1 ) to the last image (i=hnax).

[0133] According to embodiments, steps 86 and 88 are both implemented, or either of the steps is implemented to obtain either the atomic temperature map 34 or the atomic vibration map 36.

[0134] Fig. 5 illustrates, in dimension N=2, a homogeneous silicon data block 47 and a corresponding droplet block 51 obtained at the end of the super-resolution transformation step 44.

[0135] Fig. 6 represents, in dimension N=2, a portion of a block of drops 63, as obtained at the end of step 84 of centroid extraction, comprising segmented drops 65 and respective centroids 67, each centroid being indicated by a '+'.

[0136] Figure 7 represents in an image 71 the time average of the drops with at the center the trace of their movements over time. The temporal evolution of the position of the centroids provides a measure of the atomic vibrations (phonons) projected onto the direction of observation.

[0137] Advantageously, the proposed method makes it possible to measure the amplitude and direction of atomic vibrations, for example to probe for the presence of possible defects, and also finds applications in the field of plasmonics and quantum devices, for example used in quantum computers. Indeed, in this type of device, the physical effects obtained are generally related to subnanometer-sized species. Picothermometry also finds important applications in fields such as catalysis, energy, advanced electronic devices, and nanophotonics, for example.

Claims

Demands

1. A method for processing multidimensional microscopy data for mapping atomic vibrations and / or atomic temperature in a sample of at least one material, comprising acquiring at least two images of said sample forming an input data block, each image of said input data block being representative of a portion of the observed sample, said input data block being represented in an N-dimensional space, N being greater than or equal to two, each data point of said block corresponding to a point in the N-dimensional space, the method being characterized in that it comprises the steps of: a) acquiring (40) a temporal succession of input data blocks, b) for each input data block of said succession, for at least one data block extracted from said input data block, referred to as a component block,comprising homogeneous data according to a homogeneity criterion relating to at least one material of the observed sample, application (44) of a super-resolution transformation to said component tile to obtain a tile, called a super-resolved droplet tile, comprising droplets representative of an atomic structure of a part of the observed sample, c) temporal alignment (80) of the super-resolved droplet tiles of said temporal sequence, d) extraction of centroids (84) from said droplet tiles obtained after temporal alignment, e) calculation of a temporal average (86) of said centroids allowing to obtain an atomic temperature map and / or projection (88) of displacement vectors of said centroids allowing to obtain an atomic vibration map.

2. A method according to claim 1, wherein the application (44) of the super-resolution transformation (44) comprises a convolution (46) by a square-kernel HBSG filter of size less than or equal to the median of the distances between neighboring drops.

3. A method according to claim 2, wherein the application of a super-resolution transformation (44) further comprises an increase (48) in the number of points of the N-dimensional tile obtained by said super-resolution transformation.

4. A method according to claim 3, wherein the application of a super-resolution transformation (44) further comprises another application of a square kernel HBSG filter of size less than or equal to the median of the distances between neighboring drops after increasing (48) the number of points and a local normalization step (52).

5. A method according to any one of claims 1 to 4, wherein the application of a super-resolution transformation (44) to obtain a super-resolved droplet tile comprising drops representative of an atomic structure of a part of the observed sample implements a segmentation step (54) using a machine learning algorithm, trained to classify the data of the super-resolved droplet tile into two classes, respectively a "background" class and a "droplet" class, thus obtaining a segmented droplet tile.

6. A method according to any one of claims 1 to 5, wherein the application of a super-resolution transformation (44) to obtain a super-resolution droplet pad comprising drops representative of an atomic structure of a portion of the observed sample further implements a special spectral filtering (56) applied to the segmented droplet pad.

7. A method according to claim 6, wherein the special spectral filtering (56) comprises an application (62) of a Fourier transform of the segmented droplet patch to obtain a diffractogram of the droplet patch (TF1), an application (72) of a specific spectral filtering mask (MFS) on said diffractogram and an application (74) of an inverse Fourier transform.

8. A method according to claim 7, comprising a calculation (70) of said specific spectral filtering mask (SFM) comprising: a calculation of a normalized theoretical adjusted diffractogram (DT*) between 0 and 1 and a machine learning segmentation of the diffractogram of the droplet patch (TF1) allowing to obtain a segmented diffractogram of the droplet patch (TF1*), then a normalization between 0 and 1 of the segmented diffractogram of the droplet patch (TF1*), the specific spectral filtering mask being formed from the maximum of the normalized theoretical adjusted diffractogram (DT*) and said normalized segmented diffractogram of the droplet patch (TF1*).

9. A method according to any one of claims 1 to 8, further comprising, after the time alignment step (80), a super- temporal resolution (82), including a calculation of one or more intermediate droplet blocks.

10. A method according to any one of claims 1 to 9, wherein the extraction (84) of centroids from the droplet blocks obtained after time alignment implements a calculation, for each of said droplet blocks, and for each drop of the droplet block, of a centroid of said drop, and a tracking of a time position of the centroids between successive droplet blocks.

11. A computer program comprising software instructions which, when executed by a programmable electronic device, implement a method for processing multidimensional microscopy data according to claims 1 to 10.

12. A multidimensional microscopy data processing device for mapping atomic vibrations and / or atomic temperature in a sample of at least one material, implementing the acquisition of at least two images of said sample forming an input data block, each image of said input data block being representative of a portion of the observed sample, said input data block being represented in an N-dimensional space, N being greater than or equal to two, each data point of said block corresponding to a point in the N-dimensional space, the device being characterized in that it comprises: - a module for obtaining a temporal sequence of input data blocks, - for each input data block of said sequence, for at least one data block extracted from said input data block, called a component block,comprising homogeneous data according to a homogeneity criterion relating to at least one material of the observed sample, a super-resolution transformation module (22) applied to a component tile to obtain a tile, called a super-resolved droplet tile, comprising drops representative of an atomic structure of a part of the observed sample, - a temporal alignment module (24) of the super-resolved droplet tiles of said temporal succession, - a centroid extraction module (28) from said droplet tiles obtained after temporal alignment, - a module (30) for calculating a temporal average of said centroids allowing to obtain an atomic temperature map (34) and / or or a module (32) for projecting displacement vectors of said centroids allowing to obtain a map (36) of atomic vibrations.

Images