Method and device for processing multidimensional microscopy data for the localization of deformations and structural orientations in a sample

Abstract

Method and device for processing multidimensional microscopy data for locating deformations and structural orientations in a sample This method for processing multidimensional microscopy data for mapping deformations and structural orientations in a sample comprises an acquisition of at least one microscopy image of said sample, and: - for at least one component tile extracted from said input data tile, application of a super-resolution transformation to obtain a super-resolved drop tile, comprising drops representative of an atomic structure of a part of the observed sample, spectral filtering of said super-resolved drop tile, comprising an application of a direct spectral transformation, of a predefined mask to select two chosen crystal planes, and of an inverse spectral transformation of said direct spectral transformation,spectral filtering to obtain a filtered tile, application of a Helmholtz analysis on the filtered tile to obtain for each point of said filtered tile an interplanar distance value and a polar angle value between the chosen crystal planes. Figure for the abstract: Figure 2,

Description

Title of the invention: Method and device for processing multidimensional microscopy data for the localization of deformations and structural orientations in a sample

[0001] The present invention relates to a method for processing multidimensional microscopy data to map deformations and structural orientations in a sample of at least one material.

[0002] The invention also relates to a device for processing multidimensional microscopy data for mapping deformations and structural orientations in a sample of at least one associated material, as well as an associated computer program.

[0003] The invention lies in the field of processing multidimensional data, obtained by observation of samples composed of one or more materials, for the analysis of their physical properties and their structures.

[0004] More particularly, the invention finds applications in the localization of structural deformations in a sample, through the estimation of interplanar distances and polar angles between crystallographic planes. The invention can for example be applied in quality inspection in a production line of materials and devices.

[0005] As is known, crystals have by definition a discrete diffractogram, that is to say composed of spots or diffraction balls, in the sense of the IUCR (International Union of Crystallography).

[0006] In practice, observed materials comprising one or more crystalline and / or amorphous species may exhibit defects, i.e. deformations or irregularities, which may induce stresses or malfunctions in devices using such materials, for example leakage currents in transistors.

[0007] It is useful to have high-performance tools for carrying out the inspection of materials, and detecting the possible presence of deformations or irregularities, in the calibration phase or in the technological optimization phase of various devices using the materials.

[0008] The analysis of physical properties of materials by processing spectral data (1D) and images (2D) obtained by microscopy has been developed for this purpose.

[0009] The images and spectra are for example obtained by high-resolution transmission electron microscopy (HRTEM and HRSTEM and 4D-STEM), X-ray fluorescence microscopy (HRSTEM-EDX), electron energy loss microscopy (HRSTEM-EELS and HRSTEM-VEELS), energy-filtered transmission electron microscopy (EF-HRTEM, EF-HRSTEM), atomic force microscopy (AFM), scanning tunneling microscopy (STM), atom probe tomography or electron tomography for example. The microscopy images obtained are atomic resolution images, i.e. with a spatial resolution of less than or equal to 2 nanometers.

[0010] Spectral and image data sets, each representative of at least a portion of the observed sample, are obtained by microscopy, forming an N-dimensional data block (or multidimensional data), also called a "datacube", N being an integer greater than or equal to 2. This data block comprises drops, also called spots, or "spikes" or "blobs" in English, which stand out against a homogeneous background (for example, light drops or spots on a dark homogeneous background). These drops are representative of structural characteristics of the material of the observed sample, for example, an alignment of atoms along the observation direction. The data block acquired by microscopy may also contain noise.

[0011] For example, when observing crystals, drops arranged in a regular pattern represents the crystal mesh.

[0012] The simplest atomic model is a hard sphere model, which represents the particulate nature of an atom. The most powerful microscopes can visualize atoms, which usually appear as a droplet. In two dimensions, drops are also called spots. In one dimension, drops are points.

[0013] Mathematically, a drop is defined in this patent as a simply connected component of a discrete topological space, in the sense of general topology. This means that any loop traced in a drop can be reduced by homotopy to a point. Physically, a drop is defined as the electrical signal produced by the pixels of a matrix detector following the impact of a particle (electron, photon, ion, fermions, boson, etc.). The particle is always much smaller than the pixel, so the drop can always be reduced to a point by homotopy, in accordance with the mathematical definition of the drop. There is therefore a precise agreement between the physical definition and the mathematical definition of the drop.

[0014] In the state of the art, methods are known for measuring deformations in materials observed by microscopy. In particular, a technique called precession electron diffraction (PED) was proposed in the article by R. Vincent and PA Midgley “Double conical beam-rocking System for measurement of integrated electron diffraction intensities”, published in Ultramicroscopy, Vol. 53, No. 3, 1994. This technique makes it possible to obtain a spatial resolution of the order of 2 nm and an accuracy of 0.02%. A potential drawback of this technique is the potentially long data acquisition which is likely to damage the most fragile materials by electron bombardment effect. This method does not have sufficient spatial resolution to measure the individual deformations and rotations of each crystal plane in the sample.

[0015] The evolution of the most advanced technological nodes of the ITRS (International Technology Roadmap for Semiconductors) generates the need to localize local deformations and rotations at the scale of crystalline planes, while preserving the observed sample.

[0016] The aim of the invention is then to propose a method and device for locating deformations and structural orientations in a sample with improved precision while preserving the observed sample.

[0017] For this purpose, the subject of the invention is a method for processing multidimensional microscopy data to map deformations and structural orientations in a sample of at least one material, comprising an acquisition of at least one microscopy image of said sample forming an input data block, the or each image of said input data block being representative of a part of the observed sample, said input data block being represented in an N-dimensional space, N being greater than or equal to two, each data item of said block corresponding to a point in the N-dimensional space. This method comprises steps of: - for at least one data block extracted from said input data block, called a component block, comprising homogeneous data according to a homogeneity criterion relating to at least one material of the observed sample, applying a super-resolution transformation to said component block to obtain a block, called a super-resolved drop block, comprising drops representative of an atomic structure of a part of the observed sample, - spectral filtering of said super-resolved drop block, comprising an application of a direct spectral transformation, of a predefined mask to select two chosen crystal planes, and of an inverse spectral transformation of said direct spectral transformation, the spectral filtering making it possible to obtain a filtered block,

[0018] - application of a Helmholtz analysis on the filtered block allowing to obtain for each point of said filtered block an interplanar distance value and a polar angle value between the chosen crystal planes, the interplanar distance values ​​forming a map of the structure deformations according to the chosen crystal planes and the polar angle values ​​forming a map of the associated structure orientations.

[0019] Advantageously, the treatments of applying a superresolution transformation and spectral filtering according to chosen crystal planes, then an analysis Helmholtz, allow the calculation of interplanar distances and associated polar angles, representative of local deformations and rotations of the crystal planes chosen by digital processing, with very good precision, down to the individual atom, with fast data acquisition techniques. Indeed, a single full-field image, even noisy, is enough to obtain the result, unlike the PED technique which requires the acquisition of a diffraction image per point on the sample, i.e. more than 33 million acquisition points if the mapping is in UHD 8K format.

[0020] The method of processing multidimensional microscopy data for mapping deformations and structural orientations in a sample according to the invention may also have one or more of the characteristics below, taken independently or in any technically conceivable combination.

[0021] The application of a super-resolution transformation involves a convolution by a HBSG filter with a square kernel of size less than or equal to the median of the distances between neighboring drops.

[0022] The application of a super-resolution transformation further comprises an increase in the number of points of the N-dimensional tile obtained by said super-resolution transformation.

[0023] The method further comprises a local normalization step.

[0024] Applying a super-resolution transformation to obtain a super-resolved droplet tile comprising drops representative of an atomic structure of a portion of the observed sample implements a segmentation step using a machine learning algorithm, trained to classify the data of the super-resolved droplet tile into two classes, respectively a “background” class and a “droplet” class.

[0025] Applying a super-resolution transformation to obtain a super-resolved droplet tile comprising drops representative of an atomic structure of a portion of the observed sample further implements a refining step.

[0026] The refining step implements an application of a refining spectral filtering mask, in which the refining spectral filtering mask is obtained from the maximum of the machine learning segmentation of the Fourier transform of the component tile normalized between 0 and 1 and an adjusted theoretical diffractogram normalized between 0 and 1.

[0027] The refining step implements a calculation of an experimental diffractogram of said droplet block and a masking by a theoretical diffractogram adjusted on said experimental diffractogram.

[0028] In the spectral filtering step, said spectral transformation is a transform of Fourier.

[0029] The method further comprises a step of correcting spectral filtering artifacts.

[0030] The invention also relates to a computer program comprising software instructions which, when executed by a programmable electronic device, implement a method of processing multidimensional microscopy data to map deformations and structural orientations in a sample as defined above.

[0031] The invention also relates to a device for processing multidimensional microscopy data for mapping deformations and structural orientations in a sample of at least one material, comprising an acquisition of at least one microscopy image of said sample forming an input data block, the or each image of said input data block being representative of a part of the observed sample, said input data block being represented in an N-dimensional space, N being greater than or equal to two, each data item of said block corresponding to a point in the N-dimensional space. This device comprises a processor configured to implement: - - for at least one data block extracted from said input data block, called a component block, comprising homogeneous data according to a homogeneity criterion relating to at least one material of the observed sample, a module for applying a super-resolution transformation to the component block to obtain a block, called a super-resolved drop block, comprising drops representative of an atomic structure of a part of the observed sample, - a spectral filtering module for said super-resolved droplet tile, configured to apply a direct spectral transformation, a predefined mask for selecting two chosen crystal planes, and an inverse spectral transformation of said direct spectral transformation, the spectral filtering making it possible to obtain a filtered tile, - a module for applying a Helmholtz analysis to the filtered block making it possible to obtain at each point of said filtered block an interplanar distance value and a polar angle value between the chosen crystal planes, the interplanar distance values ​​forming a map of the structure deformations according to the chosen crystal planes and the polar angle values ​​forming a map of the associated structure orientations.

[0032] The invention will appear more clearly on reading the description which follows, given solely by way of non-limiting example, and made with reference to the drawings in which:

[0033] [Fig.l] [Fig.l] is a block diagram of a multidimensional microscopy data inspection system comprising a multidimensional microscopy data processing device according to one embodiment;

[0034] [Fig.2] [Fig.2] is a block diagram of a method for processing multidimensional microscopy data according to one embodiment;

[0035] [Fig.3] [Fig.3] is an example of division into homogeneous components;

[0036] [Fig.4] [Fig.4] is a two-dimensional example of an input data block and corresponding super-resolved tile obtained by applying a super-resolution transformation;

[0037] [Fig.5] [Fig.5] is an example of spectral filtering for the selection of a pair of crystal planes;

[0038] [Fig.6] [Fig.6] is an example of deformation mapping of the input image of [Fig.3] for the crystal planes (0-11) and (01-1);

[0039] [Fig.7] [Fig.7] is an example of orientation mapping obtained for the input image of [Fig.3] for the crystal planes (0-11) and (01-1);

[0040] [Fig.8] [Fig.8] represents profiles of the deformations and orientations of the planes (00 22) and (00 -22) obtained from the input image of [Fig.3].

[0041] [Fig.l] schematically illustrates a system for detecting and measuring the deformations and orientations of the crystalline planes 2 in a sample of one or more materials from multidimensional data representative of the sample, acquired by a characterization machine 4.

[0042] Any microscopy technique suitable for the characterization of such a sample is applicable.

[0043] In one embodiment, the characterization machine 4 is a transmission electron microscope (TEM) which makes it possible to acquire images of a sample composed of one or more materials, for example comprising crystalline materials.

[0044] The electron microscope 4 makes it possible to simultaneously acquire electron diffraction images, EELS spectra (for "Electron Energy Loss Spectroscopy"), EDX spectra (for "Energy Dispersive X-Ray Analysis") of the signals from different sensors (BF for "Bright field", DF for "Dark field", DPC for "Differential phase contrast" or any other suitable or customized sensor), for each point of the sample in scanning microscopy mode called STEM mode (for "Scanning Transmission Electron Microscope"). Another possible acquisition mode is the TEM mode which makes it possible to obtain a global image without having to scan the electron beam.

[0045] In another embodiment, the characterization machine 4 is a probe microscope such as the atomic force microscope or the scanning tunneling microscope or the other variants such as the Kelvin probe microscope for example, in a measurement mode allowing images of atomic spatial resolution to be obtained.

[0046] Each acquired spectrum or image is represented in the form of a digital image, comprising points or pixels, each image being representative of at least part of the observed sample.

[0047] All the acquired spectra and images form an N-dimensional data block, N being a natural integer greater than or equal to 2. Such a data block is also called a “datacube”.

[0048] In one embodiment, multiple spectra or images are acquired over time, showing an evolution of the sample during analysis. In this embodiment, time is an additional dimension of the data block.

[0049] Additionally, another dimension of the data block is the microscope focus, electron energy, an angle (of the sample, electron collection, electron convergence), or any other microscope adjustment parameter that may vary in a controlled manner during the measurement.

[0050] Incomplete data may possibly be extrapolated from neighboring values ​​if necessary, for example to correct any imperfections encountered during data acquisition. As a general rule, measurements at the atomic scale are very often marred by defects because a simple acoustic or electronic noise can sometimes disturb them.

[0051] In a multidimensional data block of a crystalline sample, the mesh usually forms a regular network of drops representative of the arrangement of atoms. Simulations make it possible to predict what the expected N-dimensional blocks (or datacubes) will be for a given material and for a particular microscope setting.

[0052] As already indicated above, mathematically, a drop is a simply connected component of a discrete topological space, in the sense of general topology.

[0053] The values ​​of the pixels belonging to a drop are distinguished from an image background. This background is defined by an intensity brought back to zero after segmentation.

[0054] A block of multidimensional data of an observed sample is transmitted to a device 6 for processing multidimensional microscopy data for mapping the deformations and local orientations of the crystal planes.

[0055] For example, the transmission is carried out by a wired connection or by a wireless connection (optical, radio, or other).

[0056] The processing device 6 is, in one embodiment, a programmable electronic device, e.g. a computer.

[0057] The device 6 comprises a processor 8 (CPU or GPU) associated with an electronic memory 10. Optionally, the device 6 comprises a human-machine interface 12, including in particular a data display screen. In addition, the device 6 includes or is connected to a storage memory 14. The elements 8, 10, 12, 14 of the device 6 are adapted to communicate via a communication bus 16.

[0058] The processor 8 (CPU or GPU) is configured to execute modules 18, 20, 22, 24, 26, stored in the electronic memory 10, to implement a method for processing multidimensional data representative of the observed sample.

[0059] The module 18 is a module for obtaining a block of multidimensional data to be processed, configured to obtain a block of data comprising at least one input image, from a block of data obtained by the characterization machine 4.

[0060] For example, the obtaining module 18 is configured to obtain the data block to be processed (or input data block) from an electronic memory, where this data has been stored after acquisition.

[0061] Module 20 is a partitioning module which divides the data block into several blocks (or sub-blocks) each comprising homogeneous data according to a chosen homogeneity criterion.

[0062] Module 22 is a super-resolution transformation module of the input tile into a tile of drops representative of the atomic structure of the observed sample.

[0063] In one embodiment, the module 22 is configured to implement a convolution, an optional increase in the number of points (or pixels) of the convolved tile, a local normalization then a segmentation either by thresholding the data or by machine learning and a refinement as described in more detail below.

[0064] Module 24 is a spectral filtering module which makes it possible to select the crystal planes in the Fourier transform of the droplet block obtained in 22, and then to obtain a filtered block containing data relating to the selected crystal planes.

[0065] In a resolution mode, the spectral filtering module 24 implements an application of a direct spectral transformation, for example a direct Fourier transformation TF1; an application of a predefined mask to select at least two chosen crystal planes, preferably symmetrical with respect to the center of the reciprocal lattice, and an application of an inverse spectral transformation of said direct spectral transformation, for example the inverse Fourier transformation of TF1.

[0066] The term reciprocal lattice is understood to mean the definition classically given by the IUCR, namely the set of vectors k such that eikr=l for all position vectors r of the Bravais lattice which describes the crystal in the real lattice. The reciprocal lattice makes it possible to describe the crystal in phase space.

[0067] In one embodiment, the module 24 further implements a correction of the Fourier transform artifacts.

[0068] The Helmholtz analysis module 26 implements a Helmholtz analysis making it possible to obtain a mapping 30 of the interplanar distances between the crystal planes selected by the spectral filtering module 24 and a mapping 32 of the polar angles in 2D, and of the hyperspherical angles in dimension N>2, between the crystal planes selected by the spectral filtering module 24.

[0069] The mapping 30 is a mapping making it possible to locate possible structural deformations relating to the interplanar distances (or interreticular distances) for the selected crystalline planes.

[0070] The mapping 32 is a mapping representative of the structure orientations, eg the orientations of the selected crystal planes.

[0071] The parameters of the super-resolution transformation method of the module 22 are stored, for example in the electronic storage memory 14.

[0072] In one embodiment, the modules 18, 20, 22, 24, 26 are each implemented in the form of software, and form a computer program, comprising software instructions which, when executed by a computer, implement a method for processing multidimensional microscopy data to map deformations and structural orientations in a sample of at least one material as described in more detail below.

[0073] In a variant not shown, the modules 18, 20, 22, 24, 26 are each produced in the form of a programmable logic component, such as an FPGA (Field Programmable Gate Array), a GPU (graphics processor) or a GPGPU (General-purpose graphics processing), or in the form of a dedicated integrated circuit, such as an ASIC (Application Specific Integrated Circuit). The variants obviously include developments known to those skilled in the art in the field of computing and more generally of automated information processing.

[0074] The multidimensional microscopy data processing software for mapping deformations and structural orientations in a sample of at least one material is further capable of being recorded, in the form of a computer program comprising software instructions, on a computer-readable medium, not shown. The computer-readable medium is, for example, a medium capable of storing the electronic instructions and of being coupled to a bus of a computer system. By way of example, the readable medium is an optical disk, a magneto-optical disk, a ROM memory, a RAM memory, any type of non-volatile memory (for example EPROM, EEPROM, FLASH, NVRAM, RRAM, PCRAM, 2D NAND, 3D NAND, SLC NAND, MLC NAND, TLC NAND, V-NAND, QLC), a magnetic card or an optical card, an SSD disk, and any variant known to those skilled in the art in the field of digital data storage.

[0075] [Fig.2] is a block diagram of the main steps of a method for processing multidimensional microscopy data to map deformations and structural orientations in a sample of at least one material, in one embodiment.

[0076] It should be noted that the method applies to crystalline materials and / or amorphous materials.

[0077] The method comprises a step 40 of obtaining a block of input data to be processed, comprising at least one image, representing the sample to be inspected.

[0078] The data block includes drops standing out from a homogeneous background, for example light drops on a dark background in HRSTEM-HAADF microscopy, or HRTEM possibly energy filtered, representative of the atomic structure of the observed sample. The link between the position of the atoms and the position of the drops is obtained by simulations which take into account in particular the settings of the microscope.

[0079] Generally, the input data block is an N-dimensional block, N greater than or equal to 2.

[0080] The data in the tile are numeric values, each value being associated with a point in N-dimensional space. Such a point is also called a pixel when N=2. Each point has an associated coordinate in each dimension, usually represented by an index.

[0081] In one embodiment, N=2, the input data block then being an LxW matrix (i.e. L rows and W columns), composed of pixel values, each pixel having respective coordinates (x,y), x being for example a row index and y a column index.

[0082] Optionally, if the material is not homogeneous, step 42 of spatial partitioning or cutting of the input data block into input data sub-blocks is implemented.

[0083] By definition, each sub-block of input data is a discrete connected space in the sense of general topology and comprises homogeneous data according to a criterion relating to at least one material of the observed sample.

[0084] The homogeneity criterion is a homogeneity of structure, with reference to the arrangement of atoms in a given material. The partitioning division is for example carried out by machine learning, for example by supervised segmentation, semi-supervised or by autonomous classification after learning (for example deep). Conventionally, the structure is for example quantified by means of the digital diffractogram. A sub-block of drops is designated as representative of the structure if it contains a number of drops greater than 3 to the power n, where n represents the dimension of the tile. For example, for an image n=2 and a sub-tile is considered representative of the local structure if it contains at least 9 neighboring drops.

[0085] Structural homogeneity is achieved when the digital diffractograms of all representative droplet sub-blocks are similar, i.e., when their relative distances are less than a chosen threshold. In the literature, there are many similarity quantification methods that can be used to quantify structural homogeneity via the similarity of Fourier transforms.

[0086] [Fig. 3] illustrates divisions of an input image 70 (two-dimensional input data block) into three sub-images 72, 74, 76 each comprising homogeneous data according to a structure criterion. Each of the sub-images 72, 74, 76 comprises a homogeneous part in gray on a black background, the homogeneous parts corresponding to partitions of the input image 70.

[0087] Step 42 is followed by a step 44 of transformation called super-resolution transformation of said input data block to obtain a drop block called super-resolved drop block, comprising drops representative of an atomic structure of a part of the observed sample.

[0088] Step 44 is carried out on each sub-block of drops, then if necessary an optional merging 55 of the sub-blocks (i.e. a spatial partitioning or reassembly) is carried out before step 56.

[0089] As already indicated above, the drops are simply connected components of the N-dimensional discrete topological space.

[0090] Step 44 implements convolutional filtering (step 46) with an HBSG (for “Half Bail Savitzky-Golay”) kernel as described in the patent application FR3 118 256 (“IMAGE PROCESSING FILTER FOR LOCALIZING DROPS IN A MULTIDIMENSIONAL IMAGE AND ASSOCIATED IMAGE PROCESSING METHOD”). This filtering, for example, uses a square HBSG kernel of size corresponding to the median of the distances between neighboring drops (for example 21 pixels), with smoothing of order 2 or 3. The smaller this kernel, the more it preserves the finest details, so the size is chosen to adapt to the fineness of the details sought. On the other hand, a size that is too small eliminates less noise.

[0091] Step 46 of convolution filtering is followed by an optional step 48 of increasing the number of points (or pixels).

[0092] In one embodiment, step 48 implements an interpolation, using a reconstruction technique or a super-resolution technique such as the total variation (TV) method. This method consists of regulating the creation of new points on the total variation of the contrasts of the original image. This approach makes it possible to preserve the overall variations of contrast while preserving the edges of the drops of the image. The total variation method is notably described in the publication LI Rudin, S. Osher, and E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D, 60 (1), 259-268 (1992).

[0093] Optionally, step 44 also comprises a local normalization step 50 consisting of dividing the values ​​of the processed data block by the local variance, obtained by calculating the square root of the convolution by a Gaussian filter of the square of the values ​​of the processed data block. Preferably, the dimension of the Gaussian filter used to calculate the local variance represents the median of the distances between neighboring drops. To highlight smaller details, a smaller size can be chosen. On the other hand, the smaller the size, the greater the risk of exacerbating the contribution of the noise.

[0094] Finally, the last transformation of step 44 consists of segmenting (step 52) super-resolved drops, in order to replace the neighborhood of the drops with a uniformly black background.

[0095] According to a first variant, the segmentation 52 implements adaptive Bemsen thresholding, known in the field of image processing, with a radius close to half the median of the distances between neighboring drops (for example 10 pixels).

[0096] According to a second variant, the segmentation 52 implements machine learning as described in more detail below. The machine learning algorithm makes it possible to distinguish two classes, respectively the “drop” (or “spot” in English) class and the “background” class.

[0097] For example, machine learning is performed with a fast random forest algorithm, described in the article by Léo Breiman (2001). “Random Forests”, published in Machine Learning. 45(1):5-32).

[0098] Alternatively, other machine learning classification algorithms may be implemented, for example of the Bayesian type, or based on functions, or based on rules, or a combination of such algorithms, or based on the entire range of tools used in machine learning classification such as deep learning (in English "deep learning" for example).

[0099] Typically, droplets represent less than 20% of the image, and their size is less than 1 nm, after calibration to the actual sizes in the sample. Simulations of the theoretical images allow one to determine the expected distribution of droplets in the image in order to verify that the segmentation is correct. A segmentation is correct if the number and position of the droplets approach the expected theoretical values ​​beyond a given uncertainty threshold.

[0100] Optionally, step 44 further comprises a refining 54 making it possible to get even closer to the theoretical shape of the drops.

[0101] According to a first variant, the refining step 54 comprises convolution filtering to fill any holes inside the drops. For example, an HBSG filter is implemented, and / or a “fill holes” type algorithm (K.-J. Oh, S. Yea, and Y.-S. Ho, “Hole filling method using depth based inpainting for view synthesis in free viewpoint television and 3-D video,” published in Proc. Picture Coding Symp., Chicago, IL, May 2009.). The objective is to approach the ideal theoretical shape, necessarily free of holes.

[0102] A second refinement variant 54 to get even closer to the theoretical distribution of the drops consists of carrying out a theoretical convolution, that is to say a masking of the diffractogram of the drops with a theoretical diffractogram adjusted as best as possible to the experimental diffractogram. This adjustment of the theory with respect to the experiment consists of varying the theoretical parameters of lengths (a,b,c) and angles (a,[3,y) defining the crystal mesh in order to minimize the mean square deviation between the experimental diffractogram and the theoretical diffractogram.

[0103] A third refinement variant 54 consists of segmenting the diffractogram of the drops by machine learning which makes it possible to distinguish two classes, respectively the “diffraction peak” class and the “background” class.

[0104] For example, machine learning is performed with a fast random forest algorithm, described in the article by Léo Breiman (2001). “Random Forests”, published in Machine Learning. 45(1):5-32).

[0105] Alternatively, other machine learning classification algorithms may be implemented, for example of the Bayesian type, or based on functions, or based on rules, or a combination of such algorithms, or based on the entire range of tools used in machine learning classification such as deep learning for example. This machine learning segmentation of the diffractogram of the drops makes it possible to define the spectral filtering mask making it possible to retain only the significant frequency peaks and to selectively eliminate the unwanted information coming from the noise to be eliminated.

[0106] A fourth refinement variant 54 consists of masking the diffractogram of the drops with a mask which is the maximum of the (adjusted theoretical diffractogram normalized between 0 and 1) and the (machine learning segmentation result normalized between 0 and 1 of the Fourier transform of the drops). This particular spectral filtering mask combines the theoretically expected information with the most significant information present in the experimental diffractogram.

[0107] It is the atomistic simulation of the “perfect” theoretical images which makes it possible to verify that the image processing parameters make it possible to converge towards the ideal theoretical form, for each crystal, each orientation and each set of instrumental parameters.

[0108] [Fig.4] illustrates a super-resolved image 78 corresponding to a two-dimensional super-resolved droplet tile obtained by applying step 44 to each sub-region of the input image 70 and merging the sub-regions.

[0109] The method also comprises a step 56 of super-resolved droplet paved spectral filtering resulting from step 55.

[0110] The spectral filtering step makes it possible to select the crystal planes, preferably symmetrical with respect to the center of the reciprocal lattice, and to eliminate the other information by spectral filtering.

[0111] Step 56 comprises an application 58 of a direct spectral transformation TF1, an application 60 of a predefined mask to select at least two chosen crystal planes symmetrical with respect to the center of the reciprocal network, and an inverse spectral transformation 62 of the direct spectral transformation TF1, this spectral filtering 56 making it possible to obtain a filtered spatial tile, to obtain a filtered tile.

[0112] In one embodiment, the spectral transformation is a Fourier type transform.

[0113] The expression "Fourier transform" means the Fourier transform and all its variants and generalizations to hyperspaces, in particular the Hartley transform, the discrete Hartley transform (DHT), the fast Fourier transform, the discrete Fourier transform, the generalized discrete Fourier transform, the short-time Fourier transform (STFT), the fractional Fourier transform (FRFT), the Chirplet transform, the Hankel transform, the Fourier-Bros-Lagolnitzer transform, the canonical linear transform, the discrete-time Fourier transform (DTFT), the discrete-space Fourier transform (DSFT), the Z transform, the modified discrete cosine transform (MDCT),and the Fourier transform of finite groups. ,

[0114] Step 58 makes it possible to obtain the diffractogram of the drops, i.e. the electron diffraction pattern.

[0115] The masking step 60 makes it possible to keep only the regions selected by the upper value of the mask. All the spectral information contained in the zero regions of the mask is therefore eliminated. Mathematically, each term of the spectral block is multiplied by its counterpart in the mask to keep only the information contained in the non-zero regions.

[0116] The mask to be applied is formed according to the theoretical diffractogram of the chosen crystal planes, by pair of crystal planes symmetrical with respect to the center.

[0117] It is understood that the method as described is applicable to several pairs of chosen crystal planes, by iterating steps 56 and 66.

[0118] The determination of a mask corresponding to a pair of crystal planes is determined on tiles representing the theoretical reciprocal lattice adjusted on the real reciprocal lattice during the refining step 54. Each drop of the diffractogram can then be indexed like any diffractogram, by applying the IUCR conventions.

[0119] The spectral filtering step 56 then comprises the inverse Fourier transform step 62 which regenerates a filtered data block, obtained after spectral filtering.

[0120] Preferably, the method also comprises a step 64 of correcting the spectral filtering artifacts consisting of eliminating all the zones which present residual imperfections, in particular the edges of the image.

[0121] Indeed, the convolution step 46 always generates artifacts around the edges of the data block which must therefore be masked. It is the atomistic simulation of the perfect theoretical images which makes it possible to determine precisely which regions of the data block present such local digital aberrations.

[0122] [Fig.5] illustrates image 80 in the spectral domain (or 2D data block) obtained by applying a Fourier transform to the super-resolved image 78 of [Fig.4], and a mask 82 applied to select the crystal planes (0-11) and (01-1). The filtered image 84, representative of the data of the crystal planes (0-11) and (01-1) of the input image 70, is obtained by applying the inverse Fourier transform after spectral masking. The contrast is here pushed to the maximum to better visualize the result in the form of very marked zebra stripes.

[0123] The method then comprises a step 66 of applying a Helmholtz analysis to the filtered block making it possible to obtain at each point of the filtered block an interplanar distance value and a polar angle value between crystal planes, the interplanar distance values ​​forming a map of the deformations and the polar angle values ​​forming a map of the orientations of the crystal planes.

[0124] Helmholtz analysis is based on the Helmholtz equation (after physicist Hermann von Helmholtz) which is an elliptic partial differential equation used to search for harmonic solutions of d'Alembert's wave propagation equation.

[0125] In its application to image processing, Helmholtz analysis, or Helmholtz decomposition, is applied to the analysis of image structures (e.g. textures) comprising multiple streaks similar to waves propagating in a space. Helmholtz analysis consists of solving derivative equations partial in order to locally fit waves on an image of this type, and more generally multidimensional waves on an N-dimensional tile. Solving the Helmholtz equation generally consists of finding the eigenvalues ​​of the Laplacian and in the literature there is a wide variety of methods for solving these types of equations widely used in wave physics. For example, it is possible to use the method of separation of variables or the paraxial approximation.

[0126] These tools allow to calculate the wavelength and the local orientation, and to deduce in the application to the N-dimensional filtered tile, a local interplanar distance and a local orientation, allowing to form respectively the deformation mapping and the orientation mapping. In 2D, the orientation is quantified by the polar angle. In hyperspaces, the orientation is classically obtained by means of the angles in hyperspherical coordinates.

[0127] The term local designates the taking into account of a neighborhood of predetermined size for the calculation of the partial derivatives at each point of the processed image, and more generally of the processed N-dimensional block.

[0128] The method optionally comprises a step 68 of calculating spatial averages in each of the maps to obtain evolution profiles representative of the estimated interplanar distances and orientations.

[0129] [Fig.6] illustrates the interplanar distance mapping 88, or deformation mapping, obtained for the input image 70, for the crystal planes (0-11) and (01-1) by applying the method described above. The grayscale representation extends over a scale from 238 pm (darkest pixels of the mapping 88) to 250 pm (lightest pixels of the mapping 88).

[0130] Image 90 corresponds to the superposition of image 88 of the interplanar distances and image 84 representative of the data of the crystal planes (0-11) and (01-1) of the input image 70 in the form of zebra stripes.

[0131] [Fig.7] illustrates the polar angle mapping 92, or orientation mapping, obtained for the input image 70, for the crystal planes (0-11) and (01-1) by applying the method described above. The grayscale representation extends over a scale from 111 degrees (darkest pixels in the 92 map) to 121 degrees (lightest pixels in the 92 map).

[0132] Image 94 corresponds to the superposition of image 92 and image 84 representative of the data of the crystal planes (0-11) and (01-1) of the input image 70.

[0133] [Fig.8] illustrates graphs 96 and 98 representative of deformation profiles averaged over the entire height of the image estimated from the calculated maps.

[0134] Graph 96 comprises a profile (or curve) of the interplanar distances, averaged by column on a map of the interplanar distances obtained by the method described above, for the GaN (gallium nitride) material and for the planes crystalline (00 22) and (00-22). The profile of graph 96 represents interplanar distances d(00 22) in picometers on the ordinate axis as a function of a distance from an origin point of the observed sample (in nm) on the abscissa axis.

[0135] Graph 98 comprises a profile (or curve) of the polar angles, averaged per column on a map of the polar angles obtained by the method described above, for the GaN (gallium nitride) material and for the crystal planes (00 22) and (00-22). The profile of graph 98 represents the angles q>(00 22) in degrees on the ordinate axis as a function of a distance from an origin point of the observed sample (in nm).

[0136] As can be seen from the illustrated examples, the precisions obtained are very fine, the fluctuations of the interplanar distances between crystal planes being of the order of 0.2 pm around 23.6 pm while the nominal spatial resolution of the microscope used is 80 pm. Advantageously, this record precision is made possible by the steps of increasing the number of points (or pixels) and refining. Advantageously, these 2 steps make it possible to obtain the record precision.

[0137] Thus, advantageously, the proposed method makes it possible to obtain a precision that is largely improved compared to the precision of the observation microscope, which was inconceivable in the prior art.

Claims

Claims

1. Method for processing multidimensional microscopy data to map deformations and structural orientations in a sample of at least one material, comprising an acquisition of at least one microscopy image of said sample forming an input data block, the or each image of said input data block being representative of a part of the observed sample, said input data block being represented in an N-dimensional space, N being greater than or equal to two, each data item of said block corresponding to a point in the N-dimensional space, the method being characterized in that it comprises steps of: - for at least one data block extracted from said input data block, called a component block, comprising homogeneous data according to a homogeneity criterion relating to at least one material of the observed sample,applying (44) a super-resolution transformation to said component tile to obtain a tile, called a super-resolved drop tile, comprising drops representative of an atomic structure of a part of the observed sample, - spectral filtering (56) of said super-resolved drop tile, comprising an application of a direct spectral transformation (58), of a predefined mask (60) for selecting two chosen crystal planes, and of an inverse spectral transformation (62) of said direct spectral transformation, the spectral filtering making it possible to obtain a filtered tile, - applying a Helmholtz analysis (66) to the filtered tile making it possible to obtain for each point of said filtered tile an interplanar distance value and a polar angle value between the chosen crystal planes, the interplanar distance values ​​forming a map (30,88) structural deformations according to the chosen crystal planes and the polar angle values ​​forming a mapping (32,92) of the associated structure orientations.,

2. The method of claim 1, wherein applying a super-resolution transformation (44) comprises a convolution (46) by a square kernel HBSG filter of size less than or equal to the median of the distances between neighboring drops.

3. The method of claim 2, wherein applying a super-resolution transformation (44) further comprises increasing (48) the number of points in the N-dimensional tile obtained by said super-resolution transformation.

4. Method according to claim 2 or 3, further comprising a local normalization step (50).

5. A method according to any one of claims 1 to 4, wherein applying a super-resolution transformation (44) to obtain a super-resolved droplet tile comprising drops representative of an atomic structure of a portion of the observed sample implements a segmentation step (52) using a machine learning algorithm, trained to classify the data of the super-resolved droplet tile into two classes, respectively a “background” class and a “droplet” class.

6. A method according to any one of claims 1 to 5, wherein applying a super-resolution transformation (44) to obtain a super-resolved droplet tile comprising drops representative of an atomic structure of a portion of the observed sample further implements a refining step (54).

7. The method of claim 6 wherein the refining step (54) implements an application of a refining spectral filtering mask, wherein the refining spectral filtering mask is obtained from the maximum of the machine learning segmentation of the Fourier transform of the component tile normalized between 0 and 1 and an adjusted theoretical diffractogram normalized between 0 and 1.

8. Method according to claim 6 in which the refining step (54) implements a calculation of an experimental diffractogram of said droplet block and a masking by a theoretical diffractogram fitted to said experimental diffractogram.

9. A method according to any one of claims 1 to 8, wherein in the spectral filtering step (56), said spectral transformation is a Fourier transform.

10. Method according to any one of claims 1 to 9, further comprising a step of correcting the spectral filtering artifacts (64).

11. A computer program comprising software instructions which, when executed by a programmable electronic device, implement a method for processing multidimensional microscopy data according to claims 1 to 10.

12. Device for processing multidimensional microscopy data for mapping deformations and structural orientations in a sample of at least one material, comprising an acquisition of at least one microscopy image of said sample forming an input data block, the or each image of said input data block being representative of a part of the observed sample, said input data block being represented in an N-dimensional space, N being greater than or equal to two, each data item of said block corresponding to a point in the N-dimensional space, the device comprising a processor characterized in that it is configured to implement: - - for at least one data block extracted from said data block input data, called component block, comprising homogeneous data according to a homogeneity criterion relating to at least one material of the observed sample, a module (22) for applying a super-resolution transformation to the component block to obtain a block, called super-resolved drop block, comprising drops representative of an atomic structure of a part of the observed sample, - a module (24) for spectral filtering of said super-resolved droplet tile, configured to apply a direct spectral transformation, a predefined mask for selecting two chosen crystal planes, and an inverse spectral transformation of said direct spectral transformation, the spectral filtering making it possible to obtain a filtered tile, - a module (26) for applying a Helmholtz analysis to the filtered block making it possible to obtain at each point of said filtered block an interplanar distance value and a polar angle value between the chosen crystal planes, the interplanar distance values ​​forming a map (30, 88) of the structure deformations according to the chosen crystal planes and the polar angle values ​​forming a map (32, 92) of the associated structure orientations.

Images