An atomic-scale image processing method and system

Abstract

Embodiments of the present specification provide an atomic-level image processing method and system, the method comprising: acquiring a plurality of images; for each of the plurality of images, determining an atomic category of atoms in the image; based on the atomic category, determining a correlation relationship between adjacent images in the plurality of images; based on the correlation relationship, determining a moving track of the atoms in the plurality of images.

Description

Technical Field

[0001] This specification relates to the field of image processing technology, and in particular to an atomic-level image processing method and system. Background Technology

[0002] In-situ electron microscopy (TEM) is a technique that uses transmission electron microscopy (TEM) to observe the internal structure and chemical reactions of materials in situ. Specifically, the sample is placed in a high-vacuum environment, irradiated with an electron beam, and the reflected or transmitted signals of electrons are captured to obtain a high-resolution in-situ image of the sample. In particular, the advent of aberration-corrected electron microscopy has improved the resolution of in-situ electron microscopy from the nanometer level to the atomic level and even the sub-angstrom level. For example, scanning transmission electron microscopy (STEM) can acquire atomic-level images, revealing the position, distribution, and arrangement of atoms. Atomic-level in-situ electron microscopy allows for in-situ atomic-level observation of materials under conditions such as high temperature, high pressure, stress, and chemical reactions. For instance, under high-temperature conditions, atoms constantly vibrate and perturb; atomic-level in-situ electron microscopy allows for the observation of these atomic-level changes in the sample. Accurate analysis and study of atomic diffusion and migration processes are crucial for researching material properties and behavior, as well as for material design, preparation, and performance optimization. However, a major challenge in atomic-level in-situ electron microscopy is the sheer number of atoms in a material, their similar shapes, and the difficulty in accurately distinguishing between them, thus hindering the effective and accurate tracking of individual atom movement over time. This presents a significant challenge for real-time monitoring and tracking of atomic motion. Furthermore, even during short-term observations, atoms may undergo dramatic movements and changes, making precise tracking of their positions and movements even more difficult. Therefore, it is necessary to provide an atomic-level image processing method and system that can accurately and effectively distinguish atoms in an image and track their movement trajectories, providing atomic-level support for studying the physical or chemical reaction processes of materials. Summary of the Invention

[0003] This specification provides an atomic-level image processing method, the method comprising: acquiring multiple images; for each of the multiple images, determining the atom category of atoms in the image; determining the correlation between adjacent images in the multiple images based on the atom category; and determining the movement trajectory of atoms in the multiple images based on the correlation.

[0004] In some embodiments, the plurality of images includes a plurality of images ordered in chronological order of acquisition time.

[0005] In some embodiments, for each of the plurality of images, determining the atom category of the atoms in the image includes: for each of the plurality of images, identifying the atom position and atom diameter of the atoms in the image using a recognition model; and determining the atom category of the atoms in the image based on the atom position and the atom diameter using a classification model.

[0006] In some embodiments, determining the atom category of atoms in the image based on the atom position and the atom diameter using a classification model includes: for each atom in the image, determining a feature value based on the atom position and the atom diameter; and performing clustering processing on the atoms in the image based on the feature value to determine the atom category of the atoms in the image.

[0007] In some embodiments, the feature value is determined based on pixel values ​​within a preset neighborhood range of the atom diameter and the atom position.

[0008] In some embodiments, determining the association between adjacent images in the plurality of images based on the atom category includes: for each atom category, constructing an association matrix of the adjacent images, wherein the association matrix reflects the changes in atoms in the adjacent images.

[0009] In some embodiments, for each atom category, constructing the association matrix of the adjacent images includes: determining an initial association matrix; iteratively adjusting the initial association matrix based on a cost function and constraints to determine the association matrix, wherein the cost function is related to the distance between atoms in the adjacent images and the pixel value difference within a preset neighborhood range of the atoms in the adjacent images.

[0010] This specification also provides an atomic-level image processing system, the system comprising: an acquisition module for acquiring multiple images; a category determination module for determining the atom category of atoms in each of the multiple images; an association determination module for determining the association relationship between adjacent images in the multiple images based on the atom category; and a trajectory determination module for determining the movement trajectory of atoms in the multiple images based on the association relationship.

[0011] This specification also provides an atomic-level image processing apparatus, the apparatus comprising: at least one storage medium storing computer instructions; and at least one processor executing the computer instructions to implement the above-described atomic-level image processing method.

[0012] This specification also provides a computer-readable storage medium that stores computer instructions. When a computer reads the computer instructions, the computer executes the above-described atomic-level image processing method.

[0013] In the embodiments of this specification, the atomic categories in atomic-level images (e.g., transmission electron microscope images) are identified by recognition models and / or classification models. This allows for the accurate and efficient identification and differentiation of different categories of atoms in the images. Furthermore, based on the atomic category of each atom, the correlation between adjacent images can be determined, which can reflect the positional changes of atoms in adjacent images. Consequently, the movement trajectory of each atom in multiple images can be accurately and effectively determined, achieving accurate tracking of the movement and changes of atoms. Attached Figure Description

[0014] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0015] Figure 1 This is a schematic diagram illustrating an application scenario of an exemplary atomic-level image processing system according to some embodiments of this specification.

[0016] Figure 2 This is a block diagram of an exemplary atomic-level image processing system according to some embodiments of this specification.

[0017] Figure 3 This is a flowchart illustrating an exemplary atomic-level image processing method according to some embodiments of this specification.

[0018] Figure 4 This is a schematic diagram illustrating the determination of atom categories according to some embodiments of this specification.

[0019] Figure 5A and 5B This is a schematic diagram of atomic neighborhoods according to some embodiments of this specification.

[0020] Figure 6 This is a schematic diagram of an exemplary association matrix shown according to some embodiments of this specification.

[0021] Figure 7 This is a schematic diagram of atomic trajectories shown according to some embodiments of this specification. Detailed Implementation

[0022] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0023] It should be understood that the terms “system,” “device,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0024] As indicated in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0025] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0026] Figure 1 This is a schematic diagram illustrating an application scenario of an exemplary atomic-level image processing system according to some embodiments of this specification. In some embodiments, the atomic-level image processing system 100 can be used for processing atomic-level images, such as atomic-level high-resolution image processing of a transmission electron microscope.

[0027] In some embodiments, such as Figure 1 As shown, the atomic-level image processing system 100 may include a processing device 110, a transmission electron microscope 120, a network 130, and a storage device 140.

[0028] The processing device 110 can be used to process data and / or information from components of the atomic-level image processing system 100 or external data sources (e.g., cloud data centers). For example, the processing device 110 can acquire multiple images (e.g., multiple atomic-level high-resolution electron microscopy images) acquired by the transmission electron microscope 120; for each of the multiple images, determine the atom category of the atoms in the image; based on the atom category, determine the correlation between adjacent images in the multiple images; and based on the correlation, determine the movement trajectory of the atoms in the multiple images.

[0029] In some embodiments, the processing device 110 may include a central processing unit (CPU), a digital signal processor (DSP), a system-on-a-chip (SoC), a microcontroller unit (MCU), a computer, a user console, or any combination thereof. In some embodiments, the processing device 110 may include a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, the processing device 110 may be local or remote. In some embodiments, the processing device 110 may be implemented on a cloud platform. By way of example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-tiered cloud, or any combination thereof.

[0030] A transmission electron microscope 120 can be used to acquire images of samples. In some embodiments, the transmission electron microscope 120 can emit an electron beam, which, upon colliding with the sample, changes direction and produces diffraction, thereby forming images of varying brightness. These images are then magnified and focused to form a final image. In some embodiments, the transmission electron microscope 120 may include a high-resolution transmission electron microscope (HRTEM), a scanning transmission electron microscope (STEM), a high-angle annular dark-field transmission electron microscope (HAADF-STEM), an annular bright-field transmission electron microscope (ABF-STEM), etc. In some embodiments, the transmission electron microscope 120 can acquire high-resolution images (e.g., atomic-level images). For example, the images acquired by the transmission electron microscope 120 may include HAADF images. In some embodiments, the transmission electron microscope 120 may include a vacuum assembly, a power supply assembly, an electron gun, a condenser lens, an objective lens, an intermediate lens, a fluoroscope, an imaging assembly, etc. In some embodiments, the transmission electron microscope 120 may include a display assembly (e.g., a display screen) for displaying the acquired images. In some embodiments, the transmission electron microscope 120 may include a stage for controlling imaging parameters during the imaging process.

[0031] In some embodiments, the transmission electron microscope 120 can communicate with the processing device 110 and / or the storage device 140 via a network 130. For example, the transmission electron microscope 120 can send acquired images to the processing device 110 via the network 130. As another example, the transmission electron microscope 120 can send device status information (e.g., whether it is operating normally, whether data acquisition is normal, etc.) to the processing device 110 via the network 130. As yet another example, the transmission electron microscope 120 can transmit acquired images to the storage device 140 for storage via the network 130.

[0032] Network 130 can facilitate the exchange of information and / or data. In some embodiments, one or more components of the atomic-level image processing system 100 (e.g., processing device 110, transmission electron microscope 120, storage device 140) can send information and / or data to other components of the atomic-level image processing system 100 via network 130.

[0033] In some embodiments, network 130 may include any one or more of wired or wireless networks. In some embodiments, network 130 may include cable networks, fiber optic networks, telecommunications networks, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, ZigBee networks, near field communication (NFC), device bus, device wiring, cable connections, etc., or any combination thereof. In some embodiments, the network connections between the components of the atomic-level image processing system 100 may employ one or more of the above methods. In some embodiments, the network may be a point-to-point, shared, centralized, or other topologies, or a combination of multiple topologies.

[0034] Storage device 140 can be used to store data, instructions, and / or any other information. For example, storage device 140 can store images acquired by transmission electron microscope 120, the types of atoms in the images, the movement trajectories of atoms, the operating status of transmission electron microscope 120, operating parameters, etc. In some embodiments, storage device 140 may include random access memory (RAM), read-only memory (ROM), mass storage, removable memory, volatile read-write memory, etc., or any combination thereof. In some embodiments, storage device 140 may be integrated into or included in one or more other components of atomic-level image processing system 100 (e.g., processing device 110, transmission electron microscope 120).

[0035] It is important to note that the atomic-level image processing system 100 is provided for illustrative purposes only and is not intended to limit the scope of this specification. Various changes and modifications can be made by those skilled in the art based on the description herein. For example, the atomic-level image processing system 100 may also include a database, information source, etc. Furthermore, the atomic-level image processing system 100 may be implemented on other devices to achieve similar or different functions. However, these changes and modifications will not depart from the scope of this specification.

[0036] Figure 2 This is a block diagram of an exemplary atomic-level image processing system according to some embodiments of this specification. In some embodiments, the atomic-level image processing system 200 may be implemented by processing device 110. In some embodiments, such as Figure 2 As shown, the atomic-level image processing system 200 may include an acquisition module 210, a category determination module 220, a correlation determination module 230, and a trajectory determination module 240.

[0037] The acquisition module 210 can acquire multiple images.

[0038] The category determination module 220 can determine the atom category of atoms in each image.

[0039] The association determination module 230 can determine the association between adjacent images in multiple images based on the atom category.

[0040] The trajectory determination module 240 can determine the movement trajectory of atoms in multiple images based on the correlation relationship.

[0041] More details about the above modules can be found here. Figure 3 Its description will not be repeated here.

[0042] It should be noted that the above description of the atomic-level image processing system 200 and its modules is for convenience only and should not be construed as limiting this specification to the scope of the embodiments described. It is understood that those skilled in the art, after understanding the principles of the system, may arbitrarily combine the various modules or construct subsystems connected to other modules without departing from these principles. In some embodiments, Figure 2 The modules disclosed herein can be different modules within a system, or a single module can implement the functions of two or more of the aforementioned modules. For example, modules can share a single storage module, or each module can have its own separate storage module. Such variations are all within the scope of protection of this specification.

[0043] Figure 3This is a flowchart illustrating an exemplary atomic-level image processing method according to some embodiments of this specification. In some embodiments, Figure 3 The illustrated process 300 can be executed by atomic-level image processing system 100 (e.g., processing device 110) or atomic-level image processing system 200. For example, process 300 can be stored in a storage device (e.g., storage device 140) as a program or instructions, and process 300 can be implemented when atomic-level image processing system 100 (e.g., processing device 110) or atomic-level image processing system 200 executes the instructions. In some embodiments, the process can be accomplished using one or more additional operations not described and / or one or more operations not discussed. Additionally, Figure 3 The order of operations shown in the diagram and described below in process 300 is not restrictive.

[0044] In step 310, the processing device 110 (e.g., acquisition module 210) can acquire multiple images.

[0045] In some embodiments, the acquisition module 210 can acquire multiple images collected by the transmission electron microscope 120. In some embodiments, after the transmission electron microscope 120 acquires images, it can transmit and store the images in the storage device 140, and the acquisition module 210 can acquire multiple images from the storage device 140.

[0046] In some embodiments, multiple images may include multiple images ordered in chronological order of acquisition time. For example, the acquisition module 210 may acquire multiple images continuously acquired by the transmission electron microscope 120 within a preset time period (e.g., within 30 seconds, within 1 minute, within 5 minutes).

[0047] In step 320, for each of the multiple images, the processing device 110 (e.g., category determination module 220) can determine the atom category of the atoms in that image.

[0048] In some embodiments, taking an image as an example, the category determination module 220 can identify atoms in the image and classify the atoms to determine the atom category of each atom, based at least on the characteristics of the atoms (e.g., position, size, shape, etc.).

[0049] In some embodiments, the category determination module 220 can identify the atomic positions and diameters of atoms in an image using a recognition model; and determine the atomic category of the atoms in the image based on the atomic positions and diameters of each atom using a classification model. This is merely an example. Figure 4As shown, for each image, the category determination module 220 can identify the position coordinates of atoms in the image (e.g., the coordinates of the center point of the atom in the image coordinate system) and the diameter of the atom using a recognition model. Further, the category determination module 220 can determine the atom category of each atom in the image based on the identified atom position and diameter using a classification model. For example, the category determination module 220 can determine multiple atomic clusters based on the atom position and diameter, each atomic cluster including atoms of one category. Further description is available... Figure 4 The same applies to Figure 5, which will not be elaborated upon here.

[0050] In step 330, the processing device 110 (e.g., the association determination module 230) can determine the association between adjacent images in multiple images based on the atom category.

[0051] In some embodiments, the relationship between adjacent images can reflect the changes in atoms in adjacent images (e.g., changes in the position of atoms, movement of atoms, etc.).

[0052] In some embodiments, for each type of atom (e.g., each atom cluster), the association determination module 230 can construct an association matrix of adjacent images, which can reflect the changes in atoms in adjacent images (e.g., changes in position, movement, etc.).

[0053] In some embodiments, for each atom category, taking two adjacent images t and t+1 as an example, the correlation matrix between the two images can be defined as follows:

[0054]

[0055] Where k represents the atom category, if the i-th atom in image t... With the j-th atom in image t+1 If they are the same atom, then If not, then In some embodiments, the order of atoms in an image can be from left to right and top to bottom, and correspondingly, the i-th atom in the image refers to the i-th atom counted in the order from left to right and top to bottom. It is understood that atoms in an image can also be counted in other orders, and this specification does not limit this. Further, the association matrix... It also includes row 0 and column 0, representing newly appearing atoms (e.g., atoms that have moved into the image acquisition range) and disappearing atoms (e.g., atoms that have moved out of the image acquisition range) in image t+1, respectively. Specifically, if newly appearing atoms in image t+1... but If in image t+1, the original atoms in image t Disappearance, then

[0056] For example, such as Figure 6 As shown, images t and t+1 are two adjacent images. Image t contains 5 atoms, numbered sequentially as atoms 1, 2, 3, 4, and 5. Atoms 1, 2, and 3 belong to category I, while atoms 4 and 5 belong to category II. Image t+1 contains 6 atoms, numbered sequentially as atoms 1, 2, 3, 4, 5, and 6. Atoms 1, 2, and 3 belong to category I, while atoms 4, 5, and 6 belong to category II. Atom 6 is a newly appearing atom in image t+1. Accordingly, based on the atom categories and correspondences in the two images, the correlation matrix between images t and t+1 can be defined as follows: Figure 6 The matrix shown on the right.

[0057] It can be seen that the correlation matrix is ​​an m+1 row and n+1 column matrix, where m is the number of atoms in image t and n is the number of atoms in image t+1.

[0058] In some embodiments, in conjunction with the definition of the association matrix, the association determination module 230 can construct an initial association matrix and iteratively adjust the initial association matrix based on the cost function and constraints to determine the association matrix.

[0059] In some embodiments, the cost function may be related to the distance between atoms in adjacent images, the difference in pixel values ​​within a preset neighborhood range of atoms in adjacent images, etc.

[0060] In some embodiments, the cost function can be expressed as follows:

[0061]

[0062] in, This represents the i-th atom in image t. And the j-th atom in t+1 of the diagram The value of the time, and Representing the i-th atom respectively The x and y coordinates in the image coordinate system and Representing the j-th atom respectively The x and y coordinates in the image coordinate system, where 'c' is the scaling factor (which can be the system default or user-defined). and Representing the i-th atom respectively and the j-th atom The average pixel value within a preset neighborhood range (see [link to description of preset neighborhood range]). Figure 5A and Figure 5B L is a distance parameter (e.g., the maximum distance an atom can move between adjacent images, which can be a system default value or set by the user). It can be seen that when calculating the cost between atoms in adjacent images, not only the distance between atoms is considered, but also the pixel value differences within the atom's neighborhood, enabling more accurate tracking of atoms.

[0063] In some embodiments, the constraints may be related to the atom categories of atoms in adjacent images, the correspondence between atoms, etc.

[0064] In some embodiments, in conjunction with the above, based on the definition of the association matrix, the constraints are: in the association matrix, except for the 0th row, each row has exactly one 1 element and the rest are 0; except for the 0th column, each column has exactly one 1 element and the rest are 0.

[0065] In some embodiments, the association determination module 230 can determine an initial association matrix based on the cost value between atoms. Specifically, for each atom in image t, the cost value between it and each atom in image t+1 is calculated. Elements of the association matrix corresponding to the minimum cost value Let 1 be the initial value and 0 be the remaining elements. This means that we initially assume that atom j and atom i correspond to the same atom, assuming they are the same atom. As mentioned above, one atom in image t corresponds to one row in the association matrix. Accordingly, during the construction of the initial association matrix, we proceed row by row. For an element in a row that is already set to 1, in subsequent rows, all elements in the column containing that element are set to 0 (i.e., the column containing that element is ignored in subsequent processing). This can be understood as follows: since we have initially assumed that atom j and atom i are the same atom, atom j cannot have any further correspondence with other atoms in image t, and therefore can be ignored in subsequent processing. Finally, if there are remaining rows (for example, if the number of atoms in image t is greater than the number of atoms in image t+1, then the extra atoms in image t cannot be found in image t+1), then the 0th column of the remaining row is set to 1, indicating the atoms that disappeared in image t+1; if there are remaining columns (for example, if the number of atoms in image t is less than the number of atoms in image t+1, then after all atoms in image t have found their corresponding atoms in image t+1, there are still extra atoms in image t+1), then the 0th row of the remaining column is set to 1, indicating the atoms that newly appeared in image t+1.

[0066] Furthermore, the association determination module 230 can iteratively adjust the initial association matrix based on the cost function and constraints. In some embodiments, the association determination module 230 can iteratively adjust the 1 or 0 elements in the initial association matrix based on the cost function and constraints to reduce the cost of the association matrix until the cost no longer decreases or approaches constant, at which point the iterative adjustment ends. Specifically, the adjustment operation and the corresponding cost reduction scenarios can include the following:

[0067] 1. Assume before adjustment and If it is 1, then... If adjusted to 1, then according to the constraints, it is necessary to... and Adjusted to 0, and Adjust to 1. In this case, the cost is reduced to

[0068] 2. Assume before adjustment and If it is 1, then... If adjusted to 1, then according to the constraints, it is necessary to... and Adjusted to 0.

[0069] In this case, the cost is reduced to

[0070] 3. Assume before adjustment If it is 1, then... If adjusted to 1, then according to the constraints, it is necessary to... Adjusted to 0, and Adjust to 1. In this case, the cost is reduced to

[0071] 4. Assume before adjustment If it is 1, then... If adjusted to 1, then according to the constraints, it is necessary to... Adjusted to 0, and Adjust to 1. In this case, the cost is reduced to

[0072] For one or more of the above adjustment operations, the relationship determination module 230 can sequentially scan the cost reduction of all elements in the association matrix. If the cost reduction is less than 0, the adjustment operation is considered to be beneficial to cost reduction, and the adjustment operation is executed. This process is repeated until the cost no longer decreases or approaches constant, at which point the iterative adjustment ends, and the final association matrix is ​​determined.

[0073] In step 340, the processing device 110 (e.g., trajectory determination module 240) can determine the movement trajectory of atoms in multiple images based on the correlation.

[0074] In some embodiments, in conjunction with step 330, the correlation matrix can reflect the changes in atoms in adjacent images (e.g., changes in position, movement, etc.). That is, based on the element 1 or 0 in the correlation matrix, the correspondence between atoms in adjacent images can be quickly determined. Accordingly, the trajectory determination module 240 can quickly determine the movement trajectory of atoms between adjacent images in multiple images based on the correspondence between atoms.

[0075] In some embodiments, the movement trajectory of an atom can reflect the movement trajectory of the atom among multiple images. In conjunction with the preceding text, since the multiple images are acquired sequentially in chronological order, the movement trajectory of the atom can reflect the movement path of the atom over time; that is, the movement trajectory of the atom can comprehensively reflect both the temporal and spatial changes of the atom.

[0076] For example only, such as Figure 7 As shown, for a specific atom in category I, the trajectory determination module 240 can quickly determine its position in multiple images based on the corresponding association matrix elements (1 or 0). The lines connecting these positions can then represent the atom's movement trajectory across the multiple images. Since the multiple images are acquired sequentially, this movement trajectory can correspondingly reflect the atom's movement trajectory over time. Similarly, for a specific atom in category II, the trajectory determination module 240 can quickly determine its movement trajectory across multiple images based on the corresponding association matrix elements (1 or 0).

[0077] It should be noted that the above description of process 300 is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to process 300 under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0078] Figure 4 This is a schematic diagram illustrating the determination of atom categories according to some embodiments of this specification.

[0079] In conjunction with the preceding text Figure 3The processing device 110 can acquire multiple images and input them into the recognition model. In some embodiments, taking one image as an example, for each atom in the image, the recognition model can output the atom's position (e.g., position coordinates in the image coordinate system) and the atom diameter. In some embodiments, the recognition model can output the atom's position and the length and width of the rectangular frame surrounding the atom (e.g., a rectangular frame centered on the atom's position with its edge tangents as side lengths), and then determine the atom diameter based on the average length and width of the rectangular frame.

[0080] In some embodiments, the recognition model may include a machine learning model. In some embodiments, the recognition model may include a YOLO model, including a preprocessing layer, a backbone network, a head layer, fully connected layers, convolutional layers, pooling layers, etc. In some embodiments, the recognition model may include a CNN model, an RNN model, etc., or any combination thereof. In some embodiments, the recognition model may be trained based on training samples. In some embodiments, training samples may include images acquired by a transmission electron microscope (e.g., HAADF images) and / or simulated atomic-level images and sample atomic information in the images (e.g., atomic positions and atomic diameters (or the length and width of the rectangles surrounding the atoms)), wherein the atomic information serves as the label for model training. In some embodiments, the label may be manually labeled or automatically labeled using a labeling tool. Further, the training samples may be input into the initial recognition model, and the labels corresponding to the training samples may be used as supervision to iteratively adjust the parameters of the initial recognition model to minimize the loss function until the model training is complete; or training may be stopped after a certain number of iterations.

[0081] After determining the atomic position and diameter of each atom in the image, the processing device 110 can input the atomic position and diameter into the classification model. Taking an image as an example, for each atom in the image, the classification model can determine the atom's feature value based on the atom's position and diameter; furthermore, based on the feature values ​​of each atom, the atoms in the image are clustered, and the atom category of each atom in the image is determined based on the clustering results.

[0082] In some embodiments, the feature values ​​of an atom can be determined based on pixel values ​​within a preset neighborhood of the atom's diameter and position. In some embodiments, the feature values ​​of an atom can be determined based on the following formula:

[0083] D(i)=d i +c·m i (3)

[0084] Where D(i) represents the eigenvalue of atom i, d i m represents the atomic diameter of atom i. iis the average pixel value within the preset neighborhood range of atom i, and c is the scaling factor (which can be the system default value or set by the user).

[0085] In some embodiments, the preset neighborhood range can be set based on image processing requirements. In some embodiments, the preset neighborhood range can be related to image parameters, such as resolution, sharpness, grayscale, or any combination thereof. This is merely an example. Figure 5A and 5B As shown, the preset neighborhood range can be 12 or 20 atoms surrounding the current atom. By determining the feature value based on the diameter of the atom itself and the pixel values ​​of the surrounding neighborhood, the size of the atom itself and the pixel intensity of the image can be comprehensively considered, thereby improving the accuracy of atom classification.

[0086] After determining the feature values ​​of each atom, the classification model can cluster the atoms in the image. Specifically, k atoms can be specified (e.g., randomly specified) as initial cluster centers, and the distance from each atom to the k cluster centers is calculated. Further, each atom is assigned to the cluster with the smallest distance. Even further, the centers of each cluster (e.g., centroids) are recalculated, and the division operation is repeated until the cluster centers no longer move, at which point the clustering process is complete. After the clustering process is complete, the resulting clusters represent different atom categories. For example, as... Figure 4 As shown, clusters I and II represent two different categories of atoms.

[0087] It should be noted that the above description regarding the determination of atom categories is for illustrative purposes only and does not limit the scope of this specification. Various modifications and changes can be made by those skilled in the art under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0088] This specification also provides an atomic-level image processing apparatus in one or more embodiments. The apparatus includes at least one storage medium for storing computer instructions and at least one processor for executing the computer instructions to implement the atomic-level image processing method described in any of the above embodiments.

[0089] This specification also provides a computer-readable storage medium that stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the atomic-level image processing method as described in any of the above embodiments.

[0090] The beneficial effects that the embodiments of this specification may bring include, but are not limited to: (1) Identifying the atomic categories in atomic-level images (e.g., transmission electron microscope images) through recognition models and / or classification models can accurately and efficiently identify and distinguish atoms of different categories in the image. (2) When classifying atoms, determining feature values ​​based on the diameter of the atom itself and the pixel values ​​of its surrounding neighborhood can comprehensively consider the size of the atom itself and the pixel intensity of the image, thereby improving the accuracy of atom classification. (3) Determining the correlation between adjacent images based on the atomic category of each atom can reflect the positional changes of atoms in adjacent images, and can accurately and effectively determine the movement trajectory of each atom in multiple images, achieving accurate tracking of the movement and changes of atoms. (4) When calculating the cost value between atoms in adjacent images, not only the distance between atoms is considered, but also the pixel value difference within the neighborhood of the atom, which can achieve more accurate tracking of atoms.

[0091] It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects may be any one or a combination of the above, or any other possible beneficial effects.

[0092] The basic concepts have been described above. Obviously, for those skilled in the art, the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

[0093] Furthermore, this specification uses specific terms to describe embodiments thereof. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Moreover, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0094] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of some embodiments of the invention that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments described herein. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely using software solutions, such as installing the described system on existing servers or mobile devices.

[0095] Similarly, it should be noted that, in order to simplify the description disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of embodiments in this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0096] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. An atomic-level image processing method, characterized in that, The method includes: Acquire multiple images; For each of the multiple images, the atom category of the atoms in the image is determined, wherein the atom category is determined based on pixel values ​​within a preset neighborhood range of the atom diameter and the atom position; For each atom category, determine the initial association matrix; Based on the cost function and constraints, the initial correlation matrix is ​​iteratively adjusted to determine the correlation matrix between adjacent images in the multiple images. The cost function is related to the distance between atoms in the adjacent images and the pixel value difference within a preset neighborhood range of the atoms in the adjacent images. Based on the correlation matrix, the movement trajectories of atoms in the multiple images are determined.

2. The method as described in claim 1, characterized in that, The multiple images include multiple images ordered in chronological order of acquisition time.

3. The method as described in claim 1, characterized in that, For each of the multiple images, determining the atom category of the atoms in the image includes: For each of the multiple images The atomic positions and diameters of atoms in the image are identified using a recognition model. The atomic category of the atoms in the image is determined using a classification model based on the atom position and the atom diameter.

4. The method as described in claim 3, characterized in that, Using a classification model, based on the atom position and the atom diameter, the atom category in the image is determined, including: For each atom in the image, a feature value is determined based on the atom's position and diameter; Based on the feature values, the atoms in the image are clustered to determine the atom categories of the atoms in the image.

5. The method as described in claim 1, characterized in that, The correlation matrix reflects the changes in atoms in adjacent images.

6. An atomic-level image processing system, characterized in that, The system includes: The acquisition module acquires multiple images; The category determination module determines the atom category of the atoms in each of the multiple images, wherein the atom category is determined based on pixel values ​​within a preset neighborhood range of the atom diameter and the atom position; The association determination module determines an initial association matrix for each atom category; based on a cost function and constraints, iteratively adjusts the initial association matrix to determine the association matrix between adjacent images in the multiple images, wherein the cost function is related to the distance between atoms in the adjacent images and the pixel value difference within a preset neighborhood range of the atoms in the adjacent images; The trajectory determination module determines the movement trajectory of atoms in the multiple images based on the correlation matrix.

7. An atomic-level image processing device, characterized in that, The device includes: At least one storage medium that stores computer instructions; At least one processor executes the computer instructions to implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium storing computer instructions that, when read by a computer, execute the method as described in any one of claims 1 to 5.

Images