A physically directed multi-modal dynamic electron microscope video generation method and system

Abstract

The application provides a kind of physical guided multimodal dynamic electron microscope video generation method and system, belong to computer vision and augmented reality technical field, including obtaining in-situ dynamic low signal-to-noise ratio video data shot by transmission electron microscope, and the video data is normalized, and normalized video sequence is generated.The application can generate dynamic electron microscope synthetic video with high fidelity and physical correctness, through the synergistic effect of asymmetric space-time decomposition and reverse diffusion process, the method uses the atomic displacement field and order degree entropy and other physical quantities generated in the generation process, directly extracts atomic center coordinates and grain boundary position, and generates high-precision atomic trajectory annotation sequence through space-time consistency correction, this integrated generation and labeling mechanism completely gets rid of the dependence on time-consuming and labor-intensive expert manual labeling, and provides the possibility for large-scale data-driven materials science research.

Description

Technical Field

[0001] This invention relates to the field of digital data processing technology, and in particular to a physically guided method and system for generating multimodal dynamic electron microscope videos. Background Technology

[0002] Transmission electron microscopy (TEM) is a core characterization tool in materials science and nanotechnology research, providing structural information at atomic resolution. With the development of in-situ electron microscopy, researchers can capture the dynamic evolution of materials under specific environmental conditions, such as heating or electricity, by recording video sequences. Precise analysis and processing of these dynamic electron microscopy videos are crucial for revealing the microscopic mechanisms of material growth, phase transitions, and failure.

[0003] However, existing technologies have fundamental flaws when processing high temporal resolution dynamic electron microscope videos. First, in order to capture the rapid dynamics of atoms at the millisecond level, the exposure time of a single frame is extremely short, resulting in an extremely low signal-to-noise ratio. Traditional image processing algorithms struggle to effectively separate atomic signals in strong noise backgrounds. Second, existing general-purpose deep learning generative models primarily pursue visual realism during training, lacking inherent constraints on the physical laws of materials science. The generated atomic motion or structural evolution processes often violate the principle of minimum energy or the laws of dynamics, resulting in generated video data that lacks scientific research value. Summary of the Invention

[0004] This invention provides a physics-guided method and system for generating multimodal dynamic electron microscopy videos. It relies on asymmetric spatiotemporal decomposition, back diffusion, and a multi-physical quantity joint constraint architecture, and integrates atomic potential energy gradient, order entropy, and atomic jump frequency to construct physical gating constraints. This ensures that the generated content conforms to the laws of materials physics while reducing noise and preserving details.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, a physically guided method for generating multimodal dynamic electron microscopy video includes: acquiring in-situ dynamic low signal-to-noise ratio video data captured by a transmission electron microscope, and normalizing the video data to generate a normalized video sequence. Asymmetric spatiotemporal decomposition is performed on the normalized video sequence to extract the spatial noise field features of each frame and the temporal noise field features between frames, thereby obtaining asymmetric spatiotemporal noise parameters. Based on the currently generated local atomic configuration, a dynamic physical constraint gating factor is generated through a physical field calculation function that calculates the inter-atomic potential energy gradient, order entropy, and atomic jump frequency in real time. The activation intensity of the feature map channels in the deconvolution layer of the generator network is adjusted by the dynamic physical constraint gating factor, and random Gaussian noise is gradually restored to an electron microscope video frame sequence that conforms to the asymmetric spatiotemporal noise parameters and the dynamic physical constraint gating factor through a reverse asymmetric diffusion process, thereby generating a physically guided dynamic electron microscope synthetic video.

[0006] Optionally, the step of performing asymmetric spatiotemporal decomposition on the normalized video sequence includes: For each frame of the normalized video sequence, an anisotropic diffusion kernel with a diffusion rate slower along the atomic arrangement direction than in the vertical direction is used to generate a spatially anisotropic noise distribution. Based on the atomic displacement between adjacent frames of the normalized video sequence, the denoising step size is adaptively adjusted to generate a time-adaptive noise sequence. The asymmetric spatiotemporal noise parameters are generated by fusing the spatial anisotropic noise distribution with the time-adaptive noise timing.

[0007] Optionally, the step of generating the dynamic physical constraint gating factor includes: The interatomic potential gradient of the local atomic configuration is calculated based on the empirical potential function, the order entropy is calculated based on the degree of lattice regularity, and the atomic hopping frequency is calculated based on the statistical characteristics of atomic hopping. The interatomic potential energy gradient, the order entropy, and the atomic hopping frequency are normalized to generate normalized physical indices. The normalized physical indices are weighted and summed to obtain the dynamic physical constraint gating factor.

[0008] Optionally, the step of adjusting the activation intensity of the feature map channels of the deconvolution layer in the generator network using the dynamic physical constraint gating factor includes: The dynamic physical constraint gating factor is broadcast to the same number of channels as the feature map output from the previous layer to generate a gating tensor. The feature map is multiplied element-wise with the gate tensor to suppress feature channels corresponding to high potential energy gradients, low order entropy, or high atomic jump frequencies, and to enhance feature channels corresponding to low potential energy gradients, high order entropy, or low atomic jump frequencies, thereby generating a physically adjusted feature map.

[0009] Optionally, the steps of the reverse asymmetric diffusion process include: Initialize random Gaussian noise as the initial hidden variable; In each back-diffusion time step, anisotropic denoising is applied in the spatial domain and adaptive denoising is applied in the time domain, based on the asymmetric spatiotemporal noise parameters. The dynamic physical constraint gating factor is applied to the feature map correction of the time step and the initial latent variable is updated until the final physical-guided dynamic electron microscope synthesis video is generated.

[0010] Optionally, after generating the physically guided dynamic electron microscope composite video, the process further includes: Using the atomic displacement field calculated from the dynamic physical constraint gating factor and the order entropy, the atomic center coordinates and grain boundary positions of each frame are automatically extracted; Markov chain spatiotemporal consistency correction is performed on the atomic center coordinates of adjacent frames to remove isolated noise points; By connecting the corrected coordinates of the atom centers, a self-supervised atomic trajectory annotation sequence is generated.

[0011] Optionally, the method also includes the step of training the generative network: Construct a total loss function consisting of noise reconstruction loss, physical field consistency loss, and asymmetric noise residual loss; The weights of the noise reconstruction loss, the physical field consistency loss, and the asymmetric noise residual loss are automatically balanced by a dynamic weight adjuster that adjusts the weights based on the current generated quality and physical conformity. The generator network is trained end-to-end according to the total loss function until the total loss function converges.

[0012] Optionally, the step of training the generative network further includes: The physical field consistency loss is obtained by comparing the difference between the dynamic physical constraint gating factor and the physical statistical distribution extracted from real material data. The asymmetric noise residual loss is obtained by comparing the difference between the asymmetric spatiotemporal noise parameters and the preset anisotropy and adaptive diffusion characteristics.

[0013] Optionally, it also includes a step of predicting temporal evolution based on the physically guided dynamic electron microscopy synthesized video: Input the initial noisy frame into the trained generative network; The initial noise frame is directly extrapolated using the dynamic physical constraint gating factor. Output the atomic motion trajectories for multiple consecutive frames in the future to generate predictive dynamic electron microscopy videos.

[0014] Secondly, a physically guided multimodal dynamic electron microscope video generation system is provided, comprising: a data acquisition module, used to acquire in-situ dynamic low signal-to-noise ratio video data captured by a transmission electron microscope, and to normalize the video data to generate a normalized video sequence. The asymmetric spatiotemporal decomposition module is used to perform asymmetric spatiotemporal decomposition on the normalized video sequence, extracting the spatial noise field features of each frame and the temporal noise field features between frames to obtain asymmetric spatiotemporal noise parameters. The physics adapter is used to generate dynamic physical constraint gating factors based on the currently generated local atomic configuration by calculating the inter-atomic potential energy gradient, order entropy, and atomic hopping frequency in real time. The generation network module contains multiple deconvolutional layers, which are used to adjust the activation intensity of the feature map channels of the deconvolutional layers using the dynamic physical constraint gating factor, and through a reverse asymmetric diffusion process, gradually restore random Gaussian noise to an electron microscope video frame sequence that conforms to the asymmetric spatiotemporal noise parameters and the dynamic physical constraint gating factor, thereby generating a physically guided dynamic electron microscope synthetic video.

[0015] Thirdly, a computer-readable storage medium is provided, comprising: a computer program or instructions; when the computer program or instructions are executed on a computer, the computer causes the computer to perform the physically guided multimodal dynamic electron microscope video generation method and system described in the first aspect.

[0016] In summary, the above methods and systems have the following technical effects: This invention generates dynamic electron microscopy composite videos with both high fidelity and physical accuracy. Through the synergistic effect of asymmetric spatiotemporal decomposition and reverse diffusion processes, this method directly extracts the atomic center coordinates and grain boundary positions using physical quantities such as atomic displacement fields and order entropy generated during the generation process. Furthermore, it generates a high-precision atomic trajectory annotation sequence through spatiotemporal consistency correction. This integrated generation and annotation mechanism completely eliminates the reliance on time-consuming and laborious manual annotation by experts, making large-scale data-driven materials science research possible. By elevating the generation model from a simple data reproduction tool to a scientific simulation platform with predictive capabilities, the model not only learns the visual patterns of the data but also internalizes the physical laws of material evolution because the generation process is constantly guided by dynamic physical constraint gating factors. Therefore, this method can predict atomic motion trajectories over a future period based on the initial state, providing strong technical support for virtual experiments, material performance prediction, and new material design. Attached Figure Description

[0017] Figure 1 A schematic diagram of a control system provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the dynamic physical constraint gating factor calculation process in an embodiment of the present invention; Figure 3 This is a flowchart of the gated modulation process in an embodiment of the present invention; Figure 4 This is a physically adjusted feature map generated in an embodiment of the present invention; Figure 5 This is a flowchart of temporal evolution prediction of physically guided dynamic electron microscope synthesized video in an embodiment of the present invention. Figure 6 This is a flowchart of network training in an embodiment of the present invention; Figure 7 This is a schematic diagram of physical constraint embedding in an embodiment of the present invention; Detailed Implementation

[0018] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0019] In this embodiment of the invention, "instruction" can include direct and indirect instructions, as well as explicit and implicit instructions. The information indicated by a certain piece of information is called the information to be instructed. In specific implementation, there are many ways to instruct the information to be instructed, such as, but not limited to, directly instructing the information to be instructed, such as the information to be instructed itself or its index. It can also indirectly instruct the information to be instructed by instructing other information, where there is a correlation between the other information and the information to be instructed. It can also instruct only a part of the information to be instructed, while the other parts are known or pre-agreed upon. For example, the instruction of specific information can be achieved by using a pre-agreed (e.g., protocol-defined) arrangement of various pieces of information, thereby reducing instruction overhead to some extent. Simultaneously, common parts of various pieces of information can be identified and uniformly indicated to reduce the instruction overhead caused by individually indicating the same information.

[0020] Furthermore, the specific indication method can also be any existing indication method, such as, but not limited to, the above-mentioned indication methods and their various combinations. Specific details of various indication methods can be found in existing technologies, and will not be elaborated upon here. As described above, for example, when multiple pieces of information of the same type need to be indicated, the indication methods for different pieces of information may differ. In specific implementation, the required indication method can be selected according to specific needs. This embodiment of the invention does not limit the selected indication method; therefore, the indication methods involved in this embodiment of the invention should be understood to cover various methods that enable the party to be indicated to obtain the information to be indicated.

[0021] It should be understood that the information to be indicated can be sent as a whole or divided into multiple sub-information messages sent separately, and the sending period and / or timing of these sub-information messages can be the same or different. The specific sending method is not limited in this embodiment of the invention. The sending period and / or timing of these sub-information messages can be predefined, for example, according to a protocol, or configured by the sending device by sending configuration information to the receiving device.

[0022] "Predefined" or "pre-configured" can be achieved by pre-saving corresponding codes, tables, or other means that can be used to indicate relevant information in the device. This embodiment of the invention does not limit the specific implementation method. "Saving" can refer to saving in one or more memories. These memories can be separate installations or integrated into the encoder, decoder, processor, or electronic device. Alternatively, some memories can be separately installed, while others are integrated into the decoder, processor, or electronic device. The type of memory can be any form of storage medium, and this embodiment of the invention does not limit this.

[0023] In the embodiments of this invention, the “protocol” may refer to a protocol family in the field of communication, a standard protocol with a similar protocol family frame structure, or a related protocol applied to a future physical-guided multimodal dynamic electron microscope video generation method and system. The embodiments of this invention do not specifically limit this.

[0024] In this embodiment of the invention, descriptions such as "when," "under the circumstances," "if," and "if" all refer to the device making corresponding processing under certain objective circumstances, and are not limited to a specific time. They do not require the device to make a judgment action during implementation, nor do they imply any other limitations.

[0025] In the description of the embodiments of the present invention, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. "And / or" in the embodiments of the present invention is merely a description of the relationship between the related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Furthermore, in the description of the embodiments of the present invention, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. Additionally, to facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or order of execution, and that "first," "second," etc., are not necessarily different. Furthermore, in the embodiments of this invention, words such as "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this invention should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0026] The network architecture and business scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0027] The above combination Figure 1 The method provided by the embodiments of the present invention is described in detail below. The following details a physically guided multimodal dynamic electron microscope video generation method provided by the embodiments of the present invention, the method comprising: acquiring in-situ dynamic low signal-to-noise ratio video data captured by a transmission electron microscope, and normalizing the video data to generate a normalized video sequence; Asymmetric spatiotemporal decomposition is performed on the normalized video sequence to extract the spatial noise field features of each frame and the temporal noise field features between frames, thereby obtaining asymmetric spatiotemporal noise parameters. Based on the currently generated local atomic configuration, a dynamic physical constraint gating factor is generated through a physical field calculation function that calculates the inter-atomic potential energy gradient, order entropy, and atomic jump frequency in real time. The activation intensity of the feature map channels in the deconvolution layer of the generator network is adjusted by the dynamic physical constraint gating factor, and random Gaussian noise is gradually restored to an electron microscope video frame sequence that conforms to the asymmetric spatiotemporal noise parameters and the dynamic physical constraint gating factor through a reverse asymmetric diffusion process, thereby generating a physically guided dynamic electron microscope synthetic video.

[0028] Optionally, the step of performing asymmetric spatiotemporal decomposition on the normalized video sequence includes: For each frame of the normalized video sequence, an anisotropic diffusion kernel with a diffusion rate slower along the atomic arrangement direction than in the vertical direction is used to generate a spatially anisotropic noise distribution. Based on the atomic displacement between adjacent frames of the normalized video sequence, the denoising step size is adaptively adjusted to generate a time-adaptive noise sequence. The asymmetric spatiotemporal noise parameters are generated by fusing the spatial anisotropic noise distribution with the time-adaptive noise timing sequence. The system adaptively adjusts the denoising step size based on the magnitude of the displacement vector field. This adjustment process follows the mapping relationship expressed by the following formula:

[0029] Where, Δt adaptive Δt represents the final time-adaptive denoising step size. base β is the system's preset base time step, which is a scalar constant. β is a learnable sensitivity coefficient used to adjust the degree of influence of displacement on the step size. V is the local average displacement magnitude extracted from the displacement vector field.

[0030] Optionally, the step of generating the dynamic physical constraint gating factor includes: The interatomic potential gradient of the local atomic configuration is calculated based on the empirical potential function, the order entropy is calculated based on the degree of lattice regularity, and the atomic hopping frequency is calculated based on the statistical characteristics of atomic hopping. The interatomic potential energy gradient, the order entropy, and the atomic hopping frequency are normalized to generate normalized physical indices. The normalized physical indices are weighted and summed to obtain the dynamic physical constraint gating factor. The system calculates the final dynamic physics constraint gating factor according to the following weighted summation formula: ; In the formula, G is the output gating factor. , , t In order, the normalized potential gradient, normalized order entropy, and normalized atomic jump frequency are represented by w. p w ss and w are the weighting coefficients for the three physical quantities, respectively.

[0031] Optionally, the step of adjusting the activation intensity of the feature map channels of the deconvolution layer in the generator network using the dynamic physical constraint gating factor includes: The dynamic physical constraint gating factor is broadcast to the same number of channels as the feature map output from the previous layer to generate a gating tensor. The feature map is multiplied element-wise with the gate tensor to suppress feature channels corresponding to high potential energy gradient, low order entropy or high atomic jump frequency, and to enhance feature channels corresponding to low potential energy gradient, high order entropy or low atomic jump frequency, thereby generating a physically adjusted feature map. The system performs the core modulation operation, which involves element-wise multiplication of the feature map and the broadcast-gated tensor, also known as the Hadamard product. This operation follows the formula: ; In this formula, F out It is the final generated physical-adjusted feature map, F in It is the feature map of the original output of the deconvolution layer, G broadcast It is the gating tensor after broadcasting. The symbol ⊙ represents the element-wise multiplication operation. Since the numerical design of the dynamic physical constraint gating factor is to reflect physical rationality, the region with a value close to 1 corresponds to a physically reasonable atomic configuration, while the region with a value close to 0 corresponds to a physically unreasonable configuration with excessive energy, disordered structure, or excessively fast atomic jumps.

[0032] Optionally, the steps of the reverse asymmetric diffusion process include: Initialize random Gaussian noise as the initial hidden variable; In each back-diffusion time step, anisotropic denoising is applied in the spatial domain and adaptive denoising is applied in the time domain, based on the asymmetric spatiotemporal noise parameters. The dynamic physical constraint gating factor is applied to the feature map correction of the time step and the initial latent variable is updated until the final physical-guided dynamic electron microscope synthesis video is generated. The final hidden variable update step, i.e., from state X t To X t-1 The transformation is achieved through a modified denoising formula that incorporates all the above constraints: ; In this formula, ΔX t γ represents the total update calculated at time step t. t N is a preset denoising rate coefficient that varies with time step t. pred，t M is the prediction noise output from the generator network, which has been modulated internally by the physical field adapter. ns，t It is a modulation matrix whose content originates from the asymmetric spatiotemporal noise parameters, used to achieve spatial anisotropy and time-adaptive denoising effects, by using the current latent variable X. t Subtract this update amount ΔX, which combines physical and structural guidance. t The system then obtains the hidden variable X for the next time step. t-1 .

[0033] Optionally, after generating the physically guided dynamic electron microscope composite video, the process further includes: Using the atomic displacement field calculated from the dynamic physical constraint gating factor and the order entropy, the atomic center coordinates and grain boundary positions of each frame are automatically extracted; Markov chain spatiotemporal consistency correction is performed on the atomic center coordinates of adjacent frames to remove isolated noise points; Connect the corrected atom center coordinates to generate a self-supervised atom trajectory annotation sequence; The core of this process is to construct and minimize a total loss function controlled by a dynamic weight adjuster. This total loss function is composed of three weighted components at each training iteration: ; In this formula, L total This is the total loss for the current batch, L rec The noise reconstruction loss is typically calculated using mean squared error to determine the pixel-level difference between the noise predicted by the generator network and the actual applied noise. It is primarily responsible for ensuring the visual fidelity of the generated video. phy The physical field consistency loss is used to quantify the difference between the physical field distribution of the generated video and the statistical distribution of real materials, ensuring physical plausibility. asy The asymmetric noise residual loss ensures that the noise model learned by the network conforms to the preset spatial anisotropy and time-adaptive characteristics.

[0034] Optionally, the method also includes the step of training the generative network: Construct a total loss function consisting of noise reconstruction loss, physical field consistency loss, and asymmetric noise residual loss; The weights of the noise reconstruction loss, the physical field consistency loss, and the asymmetric noise residual loss are automatically balanced by a dynamic weight adjuster that adjusts the weights based on the current generated quality and physical conformity. The generator network is trained end-to-end according to the total loss function until the total loss function converges. The physical constraint loss function is: ; in These are the weighting coefficients; The physical field consistency loss is a measure of the difference between these two distributions, and is typically calculated using the Kullback-Leibler divergence. ; In this formula, L phy This is the physical field consistency loss, P gen P represents the distribution of physical quantities generated by the generative model. real D represents a pre-defined physical statistical distribution extracted from real data. KL This represents the Kullback-Leibler divergence used to calculate the difference between two distributions. The smaller this loss value, the closer the generated video is to the real material in terms of macroscopic statistical properties.

[0035] Optionally, the step of training the generative network further includes: The physical field consistency loss is obtained by comparing the difference between the dynamic physical constraint gating factor and the physical statistical distribution extracted from real material data. The asymmetric noise residual loss is obtained by comparing the difference between the asymmetric spatiotemporal noise parameters and the preset anisotropy and adaptive diffusion characteristics. Predictive state transitions can be represented by the following formula: ; In this formula, F t+1 It is the predicted atomic configuration for the next frame, D θ It is a trained generative network containing the physical field adapter, F t It is the atomic configuration of the current frame, A t and B t It is a scalar coefficient related to the time step, controlling the ratio of information retention to noise introduction from the previous frame. E is a coefficient related to F. t A standard Gaussian noise tensor of the same dimension is used to simulate random thermal perturbations in a physical system, G. t This is the dynamic physical constraint gating factor generated by the physical field adapter in this prediction step. The system will generate F t+1 As input F for the next iteration tThis process is repeated until a future sequence of a preset length, such as 10 to 50 frames, is generated. These continuously generated prediction frames are combined to obtain the predictive dynamic electron microscope video. Finally, the system extracts the complete atomic motion trajectory from the video sequence.

[0036] Optionally, it also includes a step of predicting temporal evolution based on the physically guided dynamic electron microscopy synthesized video: Input the initial noisy frame into the trained generative network; The initial noise frame is directly extrapolated using the dynamic physical constraint gating factor. Output the atomic motion trajectories for multiple consecutive frames in the future to generate predictive dynamic electron microscopy videos.

[0037] Optionally, it includes: a data acquisition module, used to acquire in-situ dynamic low signal-to-noise ratio video data captured by a transmission electron microscope, and to normalize the video data to generate a normalized video sequence; The asymmetric spatiotemporal decomposition module is used to perform asymmetric spatiotemporal decomposition on the normalized video sequence, extracting the spatial noise field features of each frame and the temporal noise field features between frames to obtain asymmetric spatiotemporal noise parameters. The physics adapter is used to generate dynamic physical constraint gating factors based on the currently generated local atomic configuration by calculating the inter-atomic potential energy gradient, order entropy, and atomic hopping frequency in real time. The generation network module contains multiple deconvolutional layers, which are used to adjust the activation intensity of the feature map channels of the deconvolutional layers using the dynamic physical constraint gating factor, and through a reverse asymmetric diffusion process, gradually restore random Gaussian noise to an electron microscope video frame sequence that conforms to the asymmetric spatiotemporal noise parameters and the dynamic physical constraint gating factor, thereby generating a physically guided dynamic electron microscope synthetic video.

[0038] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0039] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0040] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0041] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0042] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0043] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0044] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0045] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0046] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0047] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0048] Engineering parameters and testing procedures The parameters of the raw TEM video data collected were: resolution 256x256 pixels, total 100 frames, frame rate 10 frames / second. Due to the limitations of the acquisition conditions, the initial peak signal-to-noise ratio (PSNR) of the video was only 15.2dB, the atomic structure was blurred and severely interfered with by random noise.

[0049] 1. Asymmetric spacetime decomposition (corresponding to claim 2) First, the input 100-frame normalized video sequence is subjected to asymmetric spatiotemporal decomposition to generate the asymmetric spatiotemporal noise parameters required for subsequent processes. Spatial anisotropic noise distribution generation: The system identifies the carbon-carbon bond direction of the graphene six-membered ring as the local principal axis through structural tensor analysis. Subsequently, anisotropic diffusion nuclei are applied, and the diffusion coefficient along this principal axis is set to D. para The diffusion coefficient perpendicular to the principal axis is D. perp Based on the rigidity of graphene materials, the diffusion coefficient ratio is set to D. perp / D para =3.5, this setting effectively smooths the noise in the direction perpendicular to the chemical bond, while preserving the atomic position information along the chemical bond direction to the greatest extent. The resulting spatial anisotropic noise distribution consists of 100 256x256 feature maps, whose numerical distribution accurately reflects the structural a priori of the atomic lattice. Time-adaptive noise timing generation: The Farneback dense optical flow algorithm is used to calculate the atomic displacement vector field between adjacent frames. Under electron beam irradiation, the average atomic displacement modulus V in the stable lattice region is obtained. stable Approximately 0.05 Å, while at the instant a carbon atom is ejected to form a vacancy defect, the displacement modulus V of its neighboring atoms... jump The maximum range is 0.7 Å, and the system has a preset base time step Δt. base=0.1, sensitivity coefficient β=0.8, according to the formula Δt adaptive =Δt base +β*V, the denoising step size in the stable region is calculated as Δt. stable =0.1 + 0.8 * 0.05 = 0.14, while in the atomic skip region, the denoising step size is adaptively adjusted to Δt. jump =0.1+0.8*0.7=0.66, this step size difference of more than 4.7 times ensures that no motion blur is generated when capturing the violent movement of atoms, and finally generates a one-dimensional time-adaptive noise temporal vector containing 99 elements representing the dynamic characteristics between frames; 2. Generation of dynamic physical constraint gating factors (corresponding to claim 3) In the subsequent back-diffusion generation process, the physics adapter calculates the dynamic physical constraint gating factor (G) in real time to guide the generation direction.

[0050] Physical index calculation: The system uses the Tersoff empirical potential function model to calculate the interactions between carbon atoms. For a given moment, it generates the local atomic configuration of the network output and calculates three physical indices in parallel: 1. Interatomic potential energy gradient: In a stable six-membered ring region, the calculated potential energy gradient is approximately 0.5 eV / Å; while in a high-energy unstable configuration formed due to the close proximity of atoms, this gradient increases to 5.0 eV / Å. 2. Order entropy: Calculated using the Steinhardt bond order parameter q6. For a perfect graphene lattice, the q6 value is approximately 0.665. Around vacancy defects, due to bond angle distortion, the q6 value drops to around 0.4. The order entropy is defined as the deviation from the ideal value of 0.665. 3. Atomic jump frequency: Within a 5-frame time window, the number of atoms whose displacement exceeds half the carbon-carbon bond length (approximately 0.71 Å) is counted. This frequency is 0 Hz in the stable region, but can reach 0.2 Hz during defect migration. Weighted summation: After normalizing the above three indicators using the minimum-maximum method, based on the prior knowledge that energy dominates, structure is secondary, and dynamics is secondary in the evolution of graphene, the weighting coefficient is set to w. p =0.5, w s =0.3, w f =0.2. A 256x256 gating factor (G) scalar field is generated through weighted summation. Its numerical distribution is shown in Table 2; the G value is close to 1.0 in physically reasonable regions and close to 0 in unreasonable regions.

[0051] 3. Physically guided reverse asymmetric diffusion generation (corresponding to claims 1, 4, and 5) The process starts with a 256x256x100 random Gaussian noise tensor, and iterates through T=500 discrete time steps to denoise, eventually generating a video. Feature map modulation: At each time step t, in the U-Net structure of the generator network, the feature map output by the deconvolution layer of each upsampling module (e.g., with 128 channels) is multiplied element-wise with the gating factor G broadcast to 128 channels. This operation suppresses the activation intensity of the feature map by more than 70% in regions with high potential gradients and low order (G value < 0.3), thereby preventing the formation of physically impossible atomic configurations. Asymmetric denoising: The noise N predicted by the network pred，t Before updating the latent variables, they are modulated by asymmetric spatiotemporal noise parameters from step 1. Spatially, an anisotropic kernel is used for smoothing, and temporally, the update amplitude is adjusted according to an adaptive step size. This ensures that the denoising process itself is also structure- and dynamically aware.

[0052] 4. Training and Validation (corresponding to claims 7 and 8) The training was performed using a dataset containing 50 sets of real TEM videos and corresponding high-precision molecular dynamics simulation videos (as baseline truth). The total loss function was controlled by a dynamic weight adjuster. In the initial training phase (the first 20 epochs), the weights were set to α. t =0.8,β t =0.1,γ t =0.1, prioritize learning visual reconstruction, when the noise reconstruction loss L rec After a 70% decrease, the weight is adjusted to α. t =0.2,β t =0.5,γ t =0.3, shifting the focus of learning to the physical field consistency L phy and asymmetric noise residual L asy After optimization, the model converged after training for 100 epochs; 5. Self-supervised annotation and evolutionary prediction (corresponding to claims 6 and 9) After generating the video, the system uses the order entropy map generated during the generation process to locate the atom center through local minima search, with a root mean square error (RMSE) as low as 0.12 pixels. Using Markov chain spatiotemporal consistency correction, it successfully tracked 98.5% of the atom trajectories. In addition, using the 50th frame of the input video as the initial condition, the system successfully predicted the atomic evolution of the next 20 frames (2 seconds) and accurately predicted the migration path of a vacancy defect.

[0053] Comparative Analysis of Experimental Results To quantify the technical effects of the present invention, the generated results are compared with the original video, a traditional deep learning-based denoising method (DAE), and molecular dynamics simulation data as the benchmark truth.

[0054] Table 1 Comparison of the effects of physical-guided generation and traditional methods:

[0055] Table 2 Numerical analysis of dynamic physical constraint gating factor (G):

[0056] Table 3. Verification of self-supervised atomic trajectory annotation and prediction accuracy:

[0057] As shown in Table 1, the method of this invention far surpasses traditional denoising methods in image quality metrics such as PSNR and SSIM, and the generated video has extremely high visual fidelity. More importantly, in terms of physical accuracy metrics such as atomic positioning accuracy and lattice constant deviation, the results of the method of this invention are very close to the baseline truth, proving the effectiveness of the physical guidance mechanism. Table 2 clearly shows how the gating factor G is quantified and assigned according to the physical rationality of the local atomic configuration, providing a precise "physical ruler" for the generation process. The data in Table 3 verifies the practical value of this invention in downstream scientific analysis tasks. Its high-precision self-supervised annotation and reliable temporal evolution prediction capabilities provide a powerful analytical tool for materials science research. This embodiment fully demonstrates that this invention successfully embeds physical laws forward into the generation model, realizing the transformation from noisy data to physically accurate dynamic electron microscopy videos. The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A physically guided method and system for generating multimodal dynamic electron microscope videos, characterized in that, Includes the following steps: In-situ dynamic low signal-to-noise ratio video data captured by transmission electron microscopy is acquired, and the video data is normalized to generate a normalized video sequence. Asymmetric spatiotemporal decomposition is performed on the normalized video sequence to extract the spatial noise field features of each frame and the temporal noise field features between frames, thereby obtaining asymmetric spatiotemporal noise parameters. Based on the currently generated local atomic configuration, a dynamic physical constraint gating factor is generated through a physical field calculation function that calculates the inter-atomic potential energy gradient, order entropy, and atomic jump frequency in real time. The activation intensity of the feature map channels in the deconvolution layer of the generator network is adjusted by the dynamic physical constraint gating factor, and random Gaussian noise is gradually restored to an electron microscope video frame sequence that conforms to the asymmetric spatiotemporal noise parameters and the dynamic physical constraint gating factor through a reverse asymmetric diffusion process, thereby generating a physically guided dynamic electron microscope synthetic video.

2. The method for generating physically guided multimodal dynamic electron microscope videos according to claim 1, characterized in that, The step of performing asymmetric spatiotemporal decomposition on the normalized video sequence includes: For each frame of the normalized video sequence, an anisotropic diffusion kernel with a diffusion rate slower along the atomic arrangement direction than in the vertical direction is used to generate a spatially anisotropic noise distribution. Based on the atomic displacement between adjacent frames of the normalized video sequence, the denoising step size is adaptively adjusted to generate a time-adaptive noise sequence. The asymmetric spatiotemporal noise parameters are generated by fusing the spatial anisotropic noise distribution with the time-adaptive noise timing.

3. The method for generating physically guided multimodal dynamic electron microscope videos according to claim 1, characterized in that, The step of generating the dynamic physical constraint gating factor includes: The interatomic potential gradient of the local atomic configuration is calculated based on the empirical potential function, the order entropy is calculated based on the degree of lattice regularity, and the atomic hopping frequency is calculated based on the statistical characteristics of atomic hopping. The interatomic potential energy gradient, the order entropy, and the atomic hopping frequency are normalized to generate normalized physical indices. The normalized physical indices are weighted and summed to obtain the dynamic physical constraint gating factor.

4. The method for generating physically guided multimodal dynamic electron microscope videos according to claim 1, characterized in that, The step of adjusting the activation intensity of feature map channels in the deconvolutional layer of the generator network using the dynamic physical constraint gating factor includes: The dynamic physical constraint gating factor is broadcast to the same number of channels as the feature map output from the previous layer to generate a gating tensor. The feature map is multiplied element-wise with the gate tensor to suppress feature channels corresponding to high potential energy gradients, low order entropy, or high atomic jump frequencies, and to enhance feature channels corresponding to low potential energy gradients, high order entropy, or low atomic jump frequencies, thereby generating a physically adjusted feature map.

5. The method for generating physically guided multimodal dynamic electron microscope videos according to claim 1, characterized in that, The steps of the reverse asymmetric diffusion process include: Initialize random Gaussian noise as the initial hidden variable; In each back-diffusion time step, anisotropic denoising is applied in the spatial domain and adaptive denoising is applied in the time domain, based on the asymmetric spatiotemporal noise parameters. The dynamic physical constraint gating factor is applied to the feature map correction of the time step and the initial latent variable is updated until the final physical-guided dynamic electron microscope synthesis video is generated.

6. The method for generating physically guided multimodal dynamic electron microscope videos according to claim 1, characterized in that, The process of generating physically guided dynamic electron microscope composite videos also includes: Using the atomic displacement field calculated from the dynamic physical constraint gating factor and the order entropy, the atomic center coordinates and grain boundary positions of each frame are automatically extracted; Markov chain spatiotemporal consistency correction is performed on the atomic center coordinates of adjacent frames to remove isolated noise points; By connecting the corrected coordinates of the atom centers, a self-supervised atomic trajectory annotation sequence is generated.

7. The method for generating physically guided multimodal dynamic electron microscope videos according to claim 1, characterized in that, It also includes the step of training the generative network: Construct a total loss function consisting of noise reconstruction loss, physical field consistency loss, and asymmetric noise residual loss; The weights of the noise reconstruction loss, the physical field consistency loss, and the asymmetric noise residual loss are automatically balanced by a dynamic weight adjuster that adjusts the weights based on the current generated quality and physical conformity. The generator network is trained end-to-end according to the total loss function until the total loss function converges.

8. The method for generating physically guided multimodal dynamic electron microscope videos according to claim 7, characterized in that, The step of training the generative network further includes: The physical field consistency loss is obtained by comparing the difference between the dynamic physical constraint gating factor and the physical statistical distribution extracted from real material data. The asymmetric noise residual loss is obtained by comparing the difference between the asymmetric spatiotemporal noise parameters and the preset anisotropy and adaptive diffusion characteristics.

9. The method for generating physically guided multimodal dynamic electron microscope videos according to claim 1, characterized in that, It also includes a step of predicting the temporal evolution of the dynamically synthesized electron microscopy video based on the physical guidance mentioned above: Input the initial noisy frame into the trained generative network; The initial noise frame is directly extrapolated using the dynamic physical constraint gating factor. Output the atomic motion trajectories for multiple consecutive frames in the future to generate predictive dynamic electron microscopy videos.

10. A physically guided multimodal dynamic electron microscope video generation system, applied to the physically guided multimodal dynamic electron microscope video generation method according to any one of claims 1-9, comprising: The data acquisition module is used to acquire in-situ dynamic low signal-to-noise ratio video data captured by a transmission electron microscope, and to normalize the video data to generate a normalized video sequence. The asymmetric spatiotemporal decomposition module is used to perform asymmetric spatiotemporal decomposition on the normalized video sequence, extracting the spatial noise field features of each frame and the temporal noise field features between frames to obtain asymmetric spatiotemporal noise parameters. The physics adapter is used to generate dynamic physical constraint gating factors based on the currently generated local atomic configuration by calculating the inter-atomic potential energy gradient, order entropy, and atomic hopping frequency in real time. The generation network module contains multiple deconvolutional layers, which are used to adjust the activation intensity of the feature map channels of the deconvolutional layers using the dynamic physical constraint gating factor, and through a reverse asymmetric diffusion process, gradually restore random Gaussian noise to an electron microscope video frame sequence that conforms to the asymmetric spatiotemporal noise parameters and the dynamic physical constraint gating factor, thereby generating a physically guided dynamic electron microscope synthetic video.

Images