Image synthesis method, related device, and computer program product

Abstract

The application discloses an image synthesis method, related equipment and a computer program product, and relates to the technical field of computer vision. The application obtains physical phenotype features of a plurality of target material particles extracted from real microscope images; the physical phenotype features of the target material particles are used to map the target material particles to a virtual three-dimensional space, and the spatial arrangement of the target material particles in the virtual three-dimensional space is constrained according to interaction rules between the target material particles, so that a simulation particle system is obtained; and the imaging effect of the simulation particle system under a microscope is simulated to generate a microscope simulation image. According to the application, a microscope simulation image with complete real physical meaning can be generated, and the realism and physical fidelity of the synthesized image are significantly improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to an image synthesis method, related equipment, and computer program product. Background Technology

[0002] In the interdisciplinary field of nanomaterials science, materials science, biomedicine, and microelectronics, researchers typically train various image processing models using large-scale microscope image datasets to improve the efficiency of material characterization, identification, classification, and analysis of microscope images. These trained models replace manual processing, significantly improving efficiency while reducing human error. However, the field currently faces the core challenge of scarce microscope image samples and high annotation costs, which severely restricts the training effectiveness and performance improvement of image processing models.

[0003] To address the core pain point of scarce microscope image samples, existing technologies have proposed several solutions, which can be broadly categorized into two types. One type employs traditional data augmentation techniques, while the other utilizes generative adversarial networks to generate simulated microscopic images.

[0004] However, the aforementioned existing technical solutions all have obvious limitations: they only process and optimize at the pixel level, making it difficult to generate new features of target materials (such as nanomaterials) with real physical meaning. This deficiency also directly leads to their inability to meet the core requirements of model training for sample diversity and realism, thus making it difficult to adapt to the application requirements of actual target material characterization scenarios. Summary of the Invention

[0005] In view of the above problems, this application is made to provide an image synthesis method, related equipment, and computer program product to improve the realism of synthesized new images. The specific solution is as follows:

[0006] In a first aspect, this application provides an image synthesis method, comprising:

[0007] To obtain the physical phenotypic characteristics of several target material particles extracted from real microscope images;

[0008] Based on the physical phenotypic characteristics of the target material particles, each target material particle is mapped to a virtual three-dimensional space, and the spatial arrangement of each target material particle in the virtual three-dimensional space is constrained according to the interaction rules between the target material particles to obtain a simulated particle system.

[0009] The imaging effect of the simulated particle system under a microscope is simulated to generate a simulated microscope image.

[0010] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of constraining the spatial arrangement of each target material particle in the virtual three-dimensional space according to the interaction rules between target material particles to obtain a simulated particle system includes:

[0011] In the virtual three-dimensional space, the positions of each target material particle are adjusted according to the preset inter-particle interaction rules to obtain a simulated particle system. The distance between any two target material particles in the simulated particle system satisfies the inter-particle interaction rules.

[0012] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of simulating the imaging effect of the simulated particle system under a microscope and generating a simulated microscope image includes:

[0013] The interaction between the target material particles and the virtual electron beam in the simulated particle system is simulated to generate a microscope simulation image of the simulated particle system.

[0014] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of simulating the interaction between the target material particles and the virtual electron beam in the simulated particle system to generate a microscope simulation image corresponding to the simulated particle system includes:

[0015] A virtual electron beam with a preset initial phase is injected into the simulated particle system along the imaging optical axis of the electron microscope corresponding to the simulated particle system.

[0016] Based on the phase delay generated when the virtual electron beam passes through the target material particles, the phase distribution of the electron wave function after the virtual electron beam penetrates the simulated particle system is calculated.

[0017] Based on the phase distribution of the electron wave function, a corresponding simulated microscope image is generated.

[0018] In one possible design, in another implementation of the first aspect of the embodiments of this application, the method further includes:

[0019] Based on the position and physical phenotypic characteristics of each target material particle in the simulated particle system in the microscope simulation image, a mask mark corresponding to each pixel of the microscope simulation image is generated. The mask mark is used to characterize the spatial structure information between the target material particles.

[0020] In one possible design, in another implementation of the first aspect of this application, the process of generating a mask marker corresponding one-to-one with the pixels of the microscope simulation image based on the three-dimensional spatial coordinates of each target material particle in the simulated particle system and the physical phenotypic features in the microscope simulation image includes:

[0021] For each target material particle, based on the three-dimensional spatial coordinates of the target material particle in the simulated particle system and the physical phenotypic features, the equivalent physical thickness of the target material particle at each occupied pixel is determined;

[0022] Based on the equivalent physical thickness of all the target material particles at each occupied pixel, the annotation value of each pixel in each annotation layer is calculated, wherein the target material particles correspond one-to-one with the annotation layers, and the annotation value is the proportion of the thickness contribution of the corresponding target material particles to the pixel.

[0023] The label values ​​of each pixel in each labeling layer are combined to generate a mask mark for the microscope image.

[0024] In one possible design, in another implementation of the first aspect of the embodiments of this application, the method further includes:

[0025] Determine the degree of overlap between the signal distribution of the simulated microscope image and the real microscope image in each frequency range;

[0026] When the overlap meets the preset fine-tuning conditions, the attribute parameters of the virtual electron beam are adjusted, and the adjusted virtual electron beam is used to return to the process of calculating the phase distribution of the electron wave function after the virtual electron beam penetrates the simulated particle system based on the phase delay generated when the virtual electron beam passes through the target material particles, until the overlap meets the preset fine-tuning conditions.

[0027] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of generating a corresponding microscope simulation image based on the phase distribution of the electron wavefunction includes:

[0028] A randomly generated defocus amount is used to modulate the phase distribution of the electronic wavefunction based on the defocus amount;

[0029] Microscopic simulation images are generated based on the phase distribution of the modulated electronic wavefunction.

[0030] In one possible design, in another implementation of the first aspect of the embodiments of this application, the method further includes:

[0031] Extract real particle texture features from real microscope images and simulated particle texture features from the simulated microscope images;

[0032] By comparing the real particle texture features with the simulated particle texture features, the uniformity of the simulated particle texture features is determined.

[0033] If the uniformity exceeds a preset threshold, noise is superimposed on the microscope simulation image so that the uniformity of the microscope simulation image after noise superposition is lower than the preset threshold.

[0034] In one possible design, in another implementation of the first aspect of the embodiments of this application, the method further includes:

[0035] Obtain energy spectrum data of several substrate materials used to support the actual prepared samples;

[0036] The image background of the microscope simulation image is randomly generated based on the energy spectrum data;

[0037] The image background is superimposed on the microscope simulation image to obtain the superimposed microscope simulation image.

[0038] Secondly, this application provides an electronic device, including: a memory and a processor;

[0039] The memory is used to store programs;

[0040] The processor is configured to execute the program to implement the image synthesis method described in any of the first aspects of this application.

[0041] Thirdly, this application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image synthesis method described in any of the first aspects of this application.

[0042] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the image synthesis method described in any of the first aspects of this application.

[0043] By means of the above technical solution, the image synthesis method proposed in this application firstly obtains the physical phenotypic features of the target material particles extracted from real microscope images, and uses the target material particles with real physical features as the synthesis object, no longer limited to the features of the target material particles at the pixel level, so that the subsequently generated image can correspond to the physical properties of the actual material.

[0044] Building upon this foundation, the physical phenotypic characteristics of the target material particles are mapped onto a virtual three-dimensional space, accurately reproducing their three-dimensional physical morphology. This allows the synthesized object to possess three-dimensional spatial properties consistent with real materials, thus aligning with actual imaging scenarios. Simultaneously, the spatial arrangement of the target material particles in the virtual three-dimensional space is constrained to conform to the interaction rules of real material particles, simulating the natural arrangement process of particles in real materials. This enables the virtual sample to reproduce the true physical state of the raw materials, ensuring that the generated sample not only possesses physical phenotypic characteristics but also reflects the rules of physical change in materials, thus possessing complete and realistic physical meaning.

[0045] Finally, by simulating the physical process of microscope imaging of target material particles in the particle system, the final generated microscope simulation image closely matches the real imaging scene in terms of imaging rules and pixel distribution logic, significantly improving the realism and physical fidelity of the synthesized image. It can be used as a high-quality sample image for training image processing models, effectively improving the model's task performance. Attached Figure Description

[0046] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0047] Figure 1 A schematic diagram of an implementation system architecture for the image synthesis method provided in this application embodiment;

[0048] Figure 2 This is a schematic flowchart of an image synthesis method provided in an embodiment of this application;

[0049] Figure 3 A schematic diagram of a simulated particle system provided in an embodiment of this application;

[0050] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] This application provides an image synthesis method, and the image synthesis method of the embodiment of this application will be described in detail below with reference to the accompanying drawings.

[0053] See Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200, wherein the server 200 may include one or more servers (…). Figure 1 (This example uses a server as an illustration).

[0054] Either terminal 100 or server 200 can be used independently to execute the image compositing method provided in the embodiments of this application. Alternatively, terminal 100 and server 200 can also be used collaboratively to execute the image compositing method provided in the embodiments of this application.

[0055] In this application embodiment, the terminal 100 can be a mobile phone, tablet computer, teaching screen, laptop computer, ultra-mobile personal computer (UMPC), personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0056] This application provides an image compositing method, illustrated by applying the method to a computer device. Specifically, the computer device may be... Figure 1 The system consists of terminal 100 or a combination of terminal 100 and server 200. (Refer to...) Figure 2 The present application provides a schematic flowchart of an image synthesis method, which may include steps S110 to S140, and these steps are described in detail below.

[0057] Step S110: Obtain the physical phenotypic characteristics of several target material particles extracted from real microscope images.

[0058] This step can be initiated by using microscopic imaging equipment such as scanning transmission electron microscopes and optical microscopes to obtain a series of real microscopic images from multiple samples containing target material particles. Then, algorithms such as deep learning, image segmentation, and feature extraction are used to extract the physical phenotypic features of each target material particle from the real microscopic images.

[0059] Furthermore, to avoid redundant data collection and identification, a particle database can be pre-constructed, storing the physical phenotypic characteristics of target material particles derived from a large number of real microscopic images. During actual modeling, the corresponding target material particle gene codes can be selected from this database based on the target material type, particle size range, or specific application scenario. This allows for the rapid acquisition of complete physical phenotypic characteristic information of several target material particles, providing reliable support for subsequent steps and enabling the final generated microscope simulation images to more closely resemble the imaging effects of real nanoparticles.

[0060] In this context, "target material particles" refers to specific material particles of interest in the current scenario, such as nanomaterial particles. Physical phenotypic features are a set of parameters that quantitatively describe the physical properties of these target material particles, including their morphology, size, and surface state. These parameters not only include features such as aspect ratio, tortuosity, and surface defect density, but also supplementary features such as interplanar spacing, edge curvature, and ligand-related scattering parameters. By acquiring these multi-dimensional feature information, a comprehensive and accurate description of the target material particle properties can be achieved, thus providing sufficient data support for subsequent modeling work.

[0061] Step S120: Based on the physical phenotypic characteristics of the target material particles, map each target material particle to a virtual three-dimensional space.

[0062] First, a solid planar model is set up in the virtual three-dimensional space to receive and attach particles, such as... Figure 3 The base model is then constructed. Based on the physical phenotypic characteristics of the target material particles obtained in step S110, a geometric model of the virtual three-dimensional particles is built. Subsequently, algorithms such as coordinate mapping and feature-driven mapping are used to place each constructed virtual three-dimensional particle model into its corresponding position in the virtual three-dimensional space.

[0063] It should be noted that, Figure 3 This diagram is provided solely to aid the reader's understanding of the modeling process. In practice, the virtual modeling process described in steps S120 to S140 can be implemented entirely through background data processing, without necessarily requiring the construction of a visual model.

[0064] Step S130: Based on the interaction rules between target material particles, constrain the spatial arrangement of each target material particle in the virtual three-dimensional space to obtain the simulated particle system.

[0065] In the process of placing each completed virtual 3D particle model into the constructed virtual 3D space, it is necessary to determine the interaction rules between the target material particles. These rules can be formulated based on the physical properties of the target material and the interaction laws between real particles, or they can be customized by researchers according to their research needs. Specifically, interaction rules may include, for example: repulsion rules between particles (such as particles not being able to penetrate each other, and minimum distance constraints between particles), attraction rules (such as distance constraints corresponding to van der Waals forces and electrostatic forces between particles), and contact rules (such as angular constraints and pressure constraints when particles are in contact). In addition, the influence coefficients of environmental factors (such as temperature and humidity) on the interaction rules can also be introduced.

[0066] Then, based on the aforementioned interaction rules, simulation algorithms such as the discrete element method and molecular dynamics are used to constrain the spatial arrangement of the target material particles mapped to the virtual three-dimensional space in step S120. This simulates the interaction process between particles in a real scene, recreating the dynamic process during sample preparation where target material particles settle in the liquid and adhere to the substrate, and where particles naturally arrange themselves or aggregate due to interactions. Finally, the stabilized three-dimensional virtual particle arrangement system is defined as the simulated particle system, and parameters such as the spatial coordinates, morphology, and relative positions of each particle in this system are recorded for subsequent imaging simulations.

[0067] This application does not employ simple arrangement methods such as random or uniform distribution. Instead, it uses simulations based on the interactions between real particles, ensuring that the resulting simulated particle system accurately reflects the three-dimensional distribution of the material particles. Compared to simple arrangement methods like random or uniform distribution, this arrangement method offers significant technical advantages, as detailed below:

[0068] Understandably, simply placing virtual 3D particles according to probability or fixed intervals without considering the interaction forces (such as repulsion and attraction) between target material particles in a real scene can easily lead to virtual samples exhibiting issues such as particle surface penetration, unreasonable spacing, and abnormal contact angles, which do not conform to real physical laws, resulting in an arrangement lacking real physical meaning. Step S130, based on the physical characteristics of the target material, formulates rules that conform to the real particle interaction laws. Through iterative adjustments using simulation algorithms, the spatial arrangement and mutual positional relationships of the virtual 3D particles are consistent with the real sample, avoiding arrangement defects that do not conform to physical logic and making the simulated particle system closer to the three-dimensional distribution of real material particles. Even if the final distribution is uniform after the above-mentioned inter-particle interaction constraints, this distribution is still a simulation result obtained under the premise of satisfying real physical interaction mechanisms such as particle repulsion, attraction, and boundary contact constraints, possessing complete and rigorous physical constraint logic.

[0069] Secondly, the "random or uniform distribution" method is too simplistic and cannot distinguish the physical properties of different target material particles (such as the magnetic field effect of magnetic particles and the interaction between porous particles and pores), resulting in poor adaptability. Step S130, however, can flexibly adjust the specific interaction rules (such as the magnetic field constraint of magnetic particles and the pore effect constraint of porous particles) according to the specific type of target material, making the arrangement constraints more closely match the physical properties of the specific material, improving the accuracy of the simulated particle system, and adapting to the simulation needs of various types of target material particles.

[0070] Furthermore, simulated particle systems generated by "random or uniform distribution" lack realistic physical constraints in their arrangement, failing to conform to the actual particle stacking and interaction patterns. This leads to significant deviations between real microscope images and key features such as particle occlusion, scattering, and imaging contrast in the subsequently generated simulated microscope images, making them unsuitable as effective substitutes for real images and thus unable to address the problem of scarce microscope image samples. More seriously, using synthetic images lacking realistic physical meaning to train the model may actually degrade its performance. In contrast, the simulated particle system generated in step S130 has particle arrangement and interaction states consistent with the real scene, allowing the imaging simulation in step S140 to more accurately reproduce the imaging effects under a real microscope (such as ligand effects and imaging characteristics after particle contact), ensuring that the information content and visual features of the simulated image are highly matched with the real image, truly realizing the technical value of "simulated images replacing real images."

[0071] Step S140: Simulate the imaging effect of the particle system under a microscope to generate a simulated microscope image.

[0072] This step simulates the imaging effect of a particle system under a microscope, transforming the simulated particle system into a visualized microscopic image. This virtual image, which has similar visual characteristics and information content to a real microscopic image, is used to replace the real microscopic image for relevant analysis and research.

[0073] In practice, this step can directly estimate the signal intensity based on the geometric relationship between the particle surface normal and the detector to simulate stereo imaging contrast at low cost; or it can simulate the microscope imaging effect by convolving the particle distribution map in the simulated particle system with the point spread function of the microscope imaging system to generate a microscope simulation image; or it can use a preset material texture and the mapping to the camera plane to replace physical scattering calculation to quickly render the particle appearance; or it can bypass physical simulation by learning the mapping relationship from the virtual particle image to the real microscope image based on the neural network.

[0074] In summary, the image synthesis method proposed in this application first obtains the physical phenotypic features of the target material particles extracted from real microscope images, and uses the target material particles with real physical features as the synthesis object, no longer limited to the pixel-level features of the target material particles, so that the subsequently generated image can correspond to the physical properties of the actual material.

[0075] Building upon this foundation, the physical phenotypic characteristics of the target material particles are mapped onto a virtual three-dimensional space, accurately reproducing their three-dimensional physical morphology. This allows the synthesized object to possess three-dimensional spatial properties consistent with real materials, thus aligning with actual imaging scenarios. Simultaneously, the spatial arrangement of the target material particles in the virtual three-dimensional space is constrained to conform to the interaction rules of real material particles, simulating the natural arrangement process of particles in real materials. This enables the virtual sample to reproduce the true physical state of the raw materials, ensuring that the generated sample not only possesses physical phenotypic characteristics but also reflects the rules of physical change in materials, thus possessing complete and realistic physical meaning.

[0076] Finally, by simulating the physical process of microscopic imaging of target material particles in a particle system, the final generated simulated microscopic images closely resemble real imaging scenarios in terms of imaging patterns and pixel distribution logic. This significantly improves the realism and physical fidelity of the synthesized images, making them suitable as replacements for real microscopic images in subsequent material analysis, particle detection, and other scenarios. This effectively reduces the cost of real experiments and avoids sample loss. Furthermore, imaging parameters can be flexibly adjusted to simulate imaging effects under different experimental conditions, providing support for experimental design and parameter optimization.

[0077] Next, through the following embodiments, other possible implementations of the image synthesis method proposed in this application will be described in detail.

[0078] To improve the operability of this application, the following section takes widely used nanomaterials as the specific research object and clearly explains a possible specific extraction process of physical phenotypic characteristics in step S110 above.

[0079] First, the three-dimensional structure of the particles is obtained through multi-scale analysis. Multi-scale decomposition processing of real microscope images accurately separates nanoparticles from the background while also enabling in-depth analysis of the microstructure within the nanoparticles (such as lattice structure). For example, by analyzing the shape proportions and symmetry of the two-dimensional projection and combining this with the statistical laws of material growth, its equivalent aspect ratio in three-dimensional space can be calculated. Alternatively, after the two-dimensional features of the particles are extracted, the optimal set of three-dimensional parameters that can produce that specific projection (two-dimensional feature) is searched from a pre-maintained library of three-dimensional parametric models, thereby determining the three-dimensional structural features of the nanoparticles.

[0080] Furthermore, the surface morphology of each separated nanoparticle is precisely quantified. Considering that the edges of nanoparticles are not perfect geometric lines, this embodiment can use a curvature feature extraction algorithm to capture the atomic step distribution at the corners of the nanoparticles and automatically record the radius data of each corner, thereby reflecting the degree of distortion at various points on the edges of the nanoparticles.

[0081] Furthermore, considering the influence of ligands on the nanoparticle surface on imaging contrast, the differences in electron beam scattering at various locations in the ligand layer are transformed into a continuous probability distribution function. This function can accurately describe the blurring degree of the nanoparticle edges. Based on this probability distribution function, the contribution of nanoparticle surface defects to contrast can be deduced, thereby quantifying the nanoparticles.

[0082] The extracted physical phenotypic parameters, such as the three-dimensional structural features, tortuosity, and surface defect density of the nanoparticles, are used to generate multidimensional feature vectors, which serve as the genetic code for the nanoparticles, thus creating a particle database. It should be noted that the embodiments of this application can automatically collect personalized parameters based on material type and particle shape. For example, for silver nanowires, the focus is on aspect ratio and degree of curvature; for quantum dots, the focus is on the thickness ratio of the core to the shell, to flexibly adapt to different application scenarios.

[0083] Furthermore, target material particles are randomly sampled from the particle database or selected according to experimental requirements, and a virtual substrate is constructed in a virtual three-dimensional space. The physical phenotypic characteristics of the obtained target material particles are then mapped onto this virtual substrate. Simultaneously, in the virtual three-dimensional space, the positions of each target material particle are adjusted according to preset inter-particle interaction rules to obtain a simulated particle system, ensuring that the distance between any two target material particles in the simulated particle system satisfies the inter-particle interaction rules.

[0084] It is understandable that complex interaction forces will occur between particles during sample preparation. In this embodiment, equation (1), namely the Leonard-Jones potential energy model, can be used to calculate the interparticle distance constraints corresponding to the interparticle interaction rules.

[0085] (1)

[0086] Where ε represents the maximum strength of the attraction between the two target material particles, σ represents the finite distance when the interaction potential energy between the two target material particles is exactly zero, and r represents the distance between the two target material particles. Describe the repulsive force generated when two target material particles are close together. Describe the attractive force generated between two target material particles at a distance.

[0087] Based on the distance constraints between particles, the spatial position of the target material particles is adjusted in a virtual three-dimensional space to simulate the balance of attractive and repulsive forces between particles, so as to ensure that there is no non-physical overlap between particles and restore the spatial arrangement of the real particle system.

[0088] It is understandable that the interactions between particles exhibit a segmented control characteristic depending on the spacing. For example, when there is an attractive force between particles, they will spontaneously approach each other; when the particle spacing is less than the preset minimum safety constraint distance, strong repulsive force becomes dominant, effectively preventing non-physical overlap of particles; when the particle spacing is within a moderate range, weak attractive force will drive particles to spontaneously cluster, replicating the particle clustering behavior in the real system. The spatial arrangement of particles generated in this way possesses both natural random distribution characteristics and statistical regularity consistent with material properties, accurately reproducing the particle distribution characteristics experimentally observed during sample preparation.

[0089] Based on the aforementioned inter-particle interaction rules, the spatial positions of the particles are continuously iteratively optimized until the spatial arrangement of all particles in the system satisfies the preset interaction constraints, and the spatial configuration of the entire particle system reaches convergence and stability. Finally, the converged and stable three-dimensional virtual particle arrangement system is defined as the simulated particle system. Key characteristic parameters such as the spatial coordinates, morphological parameters, and relative positional relationships of all particles within the system are recorded simultaneously, providing basic input data for subsequent imaging simulation.

[0090] Furthermore, using the simulated particle system obtained above, the imaging effect of the simulated particle system under a microscope is simulated, and a simulated microscope image is generated. During the imaging simulation process, the real physical interactions that occur when rays emitted from a virtual light source (such as a photon source or electron source) pass through the simulated particle system—including refraction, absorption, scattering, diffraction, and the generation of transmitted electron signals—are reproduced. Finally, based on the physical quantities such as the intensity, phase, angle, or energy distribution of the rays after penetrating the simulated particle system, they are converted into corresponding pixel grayscale or signal intensity, thereby generating a simulated microscope image, successfully transforming the simulated particle system into a visualized simulated microscope image.

[0091] This application uses the generation of simulated images under an electron microscope as an example to illustrate the simulation process of the above-mentioned imaging effect. In this example, the imaging principle of a real electron microscope can be simulated by simulating the interaction between the target material particles and the virtual electron beam in the simulated particle system, and finally the simulated particle system is transformed into a visualized simulated image under an electron microscope.

[0092] Specifically, the first step is to obtain the microscope imaging parameters corresponding to real electron microscope images, such as magnification, accelerating voltage, and imaging mode, to improve the consistency between simulated and real imaging effects. Then, based on the set imaging parameters, simulation algorithms that match the physical principles of imaging are used to simulate the imaging effect of real electron microscopes, such as electron beam propagation simulation and Monte Carlo electron scattering simulation algorithms.

[0093] Optionally, simulating the imaging process of a real electron microscope may include: injecting a virtual electron beam with a preset initial phase into the simulated particle system along the imaging optical axis of the electron microscope corresponding to the simulated particle system; calculating the phase distribution of the electron wave function after the virtual electron beam penetrates the simulated particle system based on the phase delay generated when the virtual electron beam passes through the target material particles; and generating a corresponding simulated microscope image based on the phase distribution of the electron wave function.

[0094] As is understandable, the imaging principle of a transmission electron microscope is to use a high-energy electron beam to penetrate the sample, generate contrast through the interaction between electrons and the sample, and then magnify and image the image through multiple electromagnetic lenses. This embodiment follows this imaging principle, injecting a virtual electron beam with a preset initial phase into the simulated particle system to simulate the emission and propagation characteristics of an electron beam in a real electron microscope.

[0095] In addition to setting the initial phase of the incident electron beam based on the calibration values ​​of the electron microscope, appropriate noise components can be introduced to characterize the non-deterministic factors present in the actual imaging process. For example, a randomly generated defocus amount can be used as a perturbation parameter to modulate the phase distribution of the electron wave function accordingly. The simulated image generated based on the modulated phase distribution will exhibit alternating bright and dark stripes in its edge regions due to the defocus effect, thus more closely resembling the physical process of real microscopic imaging in terms of image detail and effectively improving the realism of the simulation results.

[0096] When a virtual electron beam propagates within a simulated particle system, the electron matter wave experiences a differential phase modulation effect as its propagation occurs due to the difference in electron scattering characteristics between the virtual particle material and the background virtual medium. This is combined with... Figure 3As shown in the propagation path, the incident electron beam a propagates entirely within the background virtual medium without being obstructed by any particles. Using the equivalent refractive index of the background virtual medium as the initial phase calibration reference, electron beam a experiences no additional potential field or path change during propagation, thus maintaining its initial incident phase after penetrating the entire simulated particle system. In contrast, electron beam b passes through particles in its propagation path. The local thickness and energy distribution of the particles scatter the incident virtual electron beam b, altering its propagation path, modulating the electron wavefunction, and creating a cumulative phase delay. Ultimately, after penetrating the simulated system, electron beam b exhibits a phase difference compared to its initial incident phase.

[0097] During the simulation, the phase distribution of the electron wavefunction after the virtual electron beam penetrates the simulated particle system is calculated. This phase distribution should be consistent with the phase superposition pattern of electron waves in different regions after the electron beam penetrates the sample in a real scene. Based on this, the phase distribution of the electron wavefunction of the emitted electron beam is mapped to pixel-level grayscale differences to generate a simulated electron microscope image.

[0098] The phase distribution of the electron wave function of the emitted electron beam can also be calculated using methods such as the weak phase object approximation method, the global projection method, the angular spectrum propagation method, and the process function approximation.

[0099] In addition, in an alternative implementation, the phase distribution of the electron wave function of the emitted electron beam can be calculated using a multi-slice method. Specifically, the simulated particle system is divided into a series of thin slices along the incident direction of the electron beam. The electron wave function interacts with the projection potential of each thin slice during propagation. Referring to the following equation (2), the modulation effect of each thin slice on the electron wave function is calculated. After multiple propagations and transmissions, the final wave function is formed in the emission layer. .

[0100] (2)

[0101] Where V(x,y,z) represents the electrostatic potential energy distribution inside the particle in layer z. σ represents the phase distribution of the electron beam in the emission layer of the simulated particle system, and σ represents the interaction constant between the electrons and the target material particles.

[0102] Based on the above implementation, the phase distribution of the electron wavefunction of the emitted electron beam, after being transmitted by the subsequent imaging system, can generate a simulated microscope image with true contrast. This image exhibits physical contrast related to atomic number and mass thickness within the particle, while clearly displaying Fresnel fringe characteristics at the boundary between the particle and the background virtual medium, effectively improving the realism and accuracy of the simulated image in representing microscopic morphological details.

[0103] To further enhance the realism of the simulated microscope images, this application also proposes other possible methods for optimizing images.

[0104] In one possible implementation, the above image synthesis method may further include: acquiring energy spectrum data of several substrate materials used to support the actual prepared sample; randomly generating an image background for a microscope simulation image based on the energy spectrum data; and superimposing the image background with the microscope simulation image to obtain the superimposed microscope simulation image.

[0105] Specifically, energy spectrum data of substrate materials (such as carbon films) of different thicknesses and amorphous structures used to support actual sample preparation are collected, and a random background with fractal characteristics is constructed based on this energy spectrum data. This background differs from the traditional uniform grayscale background; it contains grayscale fluctuations and statistical signal fluctuations, which can simulate the quantum statistical effect of the detector during electronic signal acquisition. Moreover, the background noise follows the Poisson distribution law shown in formula (3). The constructed background noise is superimposed on the microscope simulation image to reproduce the inherent particle noise characteristics of the image under low-dose imaging conditions, thereby broadening the imaging scenarios that the microscope simulation image can be adapted to.

[0106] (3)

[0107] Where N represents the expected number of electrons per unit pixel, and n is the actual number of electrons captured.

[0108] In another possible implementation, the above image synthesis method may further include: determining the overlap of signal distribution between the simulated microscope image and the real microscope image in each frequency range; adjusting the attribute parameters of the virtual electron beam when the overlap meets the preset fine-tuning conditions, and using the adjusted virtual electron beam to return to the execution of the process of calculating the phase distribution of the electron wave function after the virtual electron beam penetrates the simulated particle system based on the phase delay generated when the virtual electron beam passes through the target material particles, until the overlap meets the preset fine-tuning conditions.

[0109] It is important to clarify that the imaging system of a real microscope possesses a specific modulation transfer function, which causes signal attenuation in the image within a certain frequency range. This embodiment focuses on optimizing the simulated microscope image from the frequency domain.

[0110] Specifically, algorithms such as Fourier transform are first used to analyze the energy distribution characteristics of simulated microscope images and real microscope images in the frequency domain, and then the overlap of signal distribution between the two in each frequency range is quantitatively calculated.

[0111] A pre-set overlap threshold is used as a criterion for parameter fine-tuning. This threshold can be flexibly set and adjusted based on factors such as actual imaging requirements, target material particle characteristics, and the imaging accuracy of the real microscope. If the overlap between the simulated microscope image and the real microscope image in the high-frequency region exceeds this threshold, it indicates that the energy in the high-frequency region of the simulated image is too strong, and the realism of the noise in the image is insufficient. In this case, the imaging effect simulation process can be re-executed by fine-tuning the attribute parameters of the virtual electron beam (such as pump parameters), introducing shot noise that is more in line with the actual hardware characteristics into the process, thereby improving the realism of the final generated simulated microscope image.

[0112] In another possible implementation, the above image synthesis method may further include: extracting real particle texture features from a real microscope image and simulated particle texture features from a microscope simulation image; comparing real texture features with simulated texture features to determine the uniformity of simulated texture features; and if the uniformity exceeds a preset threshold, superimposing noise onto the microscope simulation image so that the uniformity of the microscope simulation image after superimposing noise is lower than the preset threshold.

[0113] This process optimizes simulated microscope images from a spatial domain perspective. First, texture features are extracted from both the real microscope image and the simulated microscope image. These features include, but are not limited to, light and shadow, line direction, and surface roughness. Specifically, this involves extracting the local gradient histogram of the real microscope image of the nanomaterial to obtain the grayscale evolution process at the edges of the nanoparticles, and extracting the subtle shadows caused by electron scattering in the real particle image. Similarly, shadow features are extracted from the simulated microscope image.

[0114] Furthermore, the texture features of the real microscope image are compared with the texture features of the microscope simulation image. If the regularity of the microscope simulation image is higher than the preset threshold, it indicates that the microscope simulation image is too clean or the edges are too neat. Then the correction program will be activated, sending an adjustment signal to the physics engine to increase the surface interference factor in the simulation environment, so that the generated microscope simulation image is closer to the result of laboratory shooting in terms of visual details.

[0115] It should be noted that during the image optimization process in the frequency domain and spatial domain, the real microscope image can be any real microscope image corresponding to any target material particle in the simulated particle system, or it can be a real microscope image of other microscope types, both of which can meet the requirements for comparative analysis.

[0116] The aforementioned image optimization process uses real microscope images as a reference to quantitatively determine the deviation between the simulated image and the real image. It then adjusts the simulation environment accordingly, effectively correcting the deviation between the simulated and real images. In the frequency and spatial domains, this ensures that the synthesized simulated microscope image matches the real microscope imaging results, improving the fit between the simulated and real microscope images. Furthermore, this embodiment employs a closed-loop adjustment design of "quantitative judgment - parameter adjustment - iterative optimization." Through iterative optimization, it can automatically find the optimal simulation environment, ensuring the stability and reliability of the synthesized image, further enhancing the practicality and operability of the entire image synthesis method.

[0117] In summary, the image synthesis method described above can stably generate high-quality microscopic simulation images. To fully utilize these simulation images, standardize image information annotation, and construct a standardized sample dataset, thereby providing reliable data support for subsequent visual analysis and model training, it is necessary to standardize the annotation of each microscopic simulation image.

[0118] Traditional annotation methods typically use binary masks, that is, using 0s and 1s to distinguish between the background and material particles in an image. At the nanoscale, this simple annotation method loses a lot of crucial information. To address this, embodiments of this application design a mask annotation method with depth-sensing capabilities. Specifically, based on the position and physical phenotypic characteristics of each target material particle in the simulated particle system (3D) in the microscope simulation image, mask markers are generated that correspond one-to-one with the pixels of the microscope simulation image. These mask markers are used to characterize the spatial structural information between the target material particles.

[0119] The simulated particle system from the aforementioned image synthesis process is retrieved, and the physical phenotypic features of each target material particle in the system, as well as the specific spatial position parameters of each target material particle in the virtual three-dimensional space, such as three-dimensional spatial coordinates, particle centroid coordinates, particle angular orientation, and relative positional relationships between particles, are extracted. Based on the above information, a mask marker corresponding to each pixel in each microscope simulation image is generated, and this mask marker is used to characterize the spatial structural information between target material particles in the image.

[0120] The form of masking is not unique; one option is to use a multi-channel mask. For example, a binary tensor of dimension (n, w, y) can be constructed, where N is the total number of particles. The i-th channel is 1 only within the projection region of the i-th particle, and 0 in the remaining regions. If a pixel is 1 in channels n, w, and y simultaneously, it indicates that three particles numbered n, w, and y are superimposed on that pixel. This multi-channel representation can unambiguously preserve the overlap order and spatial superposition structure between particles in the image.

[0121] However, considering that in the physical process of simulated microscope imaging, the pixel intensity of the image decreases exponentially with the cumulative mass thickness along the electron beam penetration path, the multi-channel mask mentioned above only contributes 1 to the pixel as long as the particles exist (full occlusion). The superposition of multiple particles is merely a logical AND operation of 1s and 1s, which will lose the gradient effect at the particle edges and the internal density differences.

[0122] Continuing this line of thought, this application proposes a mask generation mechanism based on the contribution of equivalent physical thickness in another possible implementation. The specific process may include: for each target material particle, determining the equivalent physical thickness of the target material particle at each occupied pixel based on its three-dimensional spatial coordinates and physical phenotypic characteristics in the simulated particle system; calculating the annotation value of each pixel in each annotation layer based on the equivalent physical thickness of all target material particles at each occupied pixel, where each target material particle corresponds one-to-one with an annotation layer, and the annotation value represents the proportion of the target material particle's thickness contribution to the pixel; and combining the annotation values ​​of each pixel in each annotation layer to generate a mask marker for the microscope image.

[0123] Specifically, for each target material particle in the simulated particle system, based on its spatial coordinates and physical phenotypic characteristics within the simulated particle system, the equivalent physical thickness of the particle on each pixel within the coverage area along the projection path is calculated. This equivalent physical thickness is directly related to the material density of the particle in the optical axis direction and the degree of particle packing on the optical path corresponding to that pixel.

[0124] Furthermore, a labeling layer system corresponding one-to-one with the target particle material is constructed. Consistent with the multi-channel masking logic described above, assuming the simulated particle system contains N target material particles, an independent labeling layer is set for each target material particle. In this case, the mask marker corresponding to each pixel can be represented as (h1, h2, ..., h...). n ), where h i This represents the annotation value of the annotation layer corresponding to the i-th target material particle at this pixel.

[0125] For any pixel in a simulated microscope image, the thickness contribution ratio of each annotation layer at that pixel can be calculated by aggregating the equivalent physical thickness of all particles covering that pixel. Here, the annotation values ​​are no longer limited to binary logic (either 0 or 1), but rather are continuous weighted values ​​within the range [0,1]. Their magnitude represents the relative physical proportion of the corresponding target material particle in the pixel's projection signal. For example, if a pixel is simultaneously covered by three target material particles with equivalent physical thicknesses of 0.2, 0.3, and 0.5 respectively, then the thickness contribution ratios of the three annotation layers are 0.2, 0.3, and 0.5, respectively. The mask label for this pixel would then be (0.2, 0.3, 0.5), clearly reflecting the difference in contribution of each particle to the pixel's projection signal.

[0126] Finally, the thickness contribution ratio of each pixel across all labeled layers is integrated to generate the final mask marker for the microscopic simulation image. In addition, other marker features, such as diffraction feature annotations, can be introduced into the mask. It is understandable that Fresnel diffraction fringes often appear in nanoscale imaging, and these fringes are often mistaken for the true edges of particles. By creating an interference feature region in the mask to mark the phase fluctuation range of the diffraction fringes, probability values ​​are used to identify which areas are real material and which areas are visual illusions produced by wave interference. Thus, this mask can not only characterize the spatial structural relationships between particles but also the changes in particle boundaries under different defocus conditions.

[0127] In summary, this annotation method not only fully records the spatial superposition order and topological relationship between particles, but also retains the contrast information of simulated microscopic images caused by differences in particle morphology and density. This provides higher-precision prior knowledge about spatial depth for subsequent tasks such as quantitative image analysis and model training, helping to accurately understand the spatial relationship between occluders and occluded objects in microscope images.

[0128] This application also provides an electronic device in its embodiments. (See reference...) Figure 4 The diagram illustrates a structure suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, tablet computers, and large-screen teaching displays. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0129] like Figure 4As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 2 or a program loaded from a storage device 8 into a random access memory (RAM) 3, to implement the image compositing method of the foregoing embodiments of this application. When the electronic device is powered on, the RAM 3 also stores various programs and data required for the operation of the electronic device. The processing unit 1, ROM 2, and RAM 3 are interconnected via a bus 4. An input / output (I / O) interface 5 is also connected to the bus 4.

[0130] Typically, the following devices can be connected to I / O interface 5: input devices 6 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 7 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 8 including, for example, memory cards, hard drives, etc.; and communication devices 9. Communication device 9 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0131] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the image synthesis methods provided in this application.

[0132] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the image synthesis methods provided in this application.

[0133] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0135] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0136] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0137] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

Claims

1. An image synthesis method, characterized in that, include: To obtain the physical phenotypic characteristics of several target material particles extracted from real microscope images; Based on the physical phenotypic characteristics of the target material particles, each target material particle is mapped to a virtual three-dimensional space, and the spatial arrangement of each target material particle in the virtual three-dimensional space is constrained according to the interaction rules between the target material particles to obtain a simulated particle system. The imaging effect of the simulated particle system under a microscope is simulated to generate a simulated microscope image.

2. The image synthesis method according to claim 1, characterized in that, The process of constraining the spatial arrangement of the target material particles in the virtual three-dimensional space based on the interaction rules between the target material particles to obtain the simulated particle system includes: In the virtual three-dimensional space, the positions of each target material particle are adjusted according to the preset inter-particle interaction rules to obtain a simulated particle system. The distance between any two target material particles in the simulated particle system satisfies the inter-particle interaction rules.

3. The image synthesis method according to claim 1, characterized in that, The process of simulating the imaging effect of the simulated particle system under a microscope and generating a simulated microscope image includes: The interaction between the target material particles and the virtual electron beam in the simulated particle system is simulated to generate a microscope simulation image of the simulated particle system.

4. The image synthesis method according to claim 3, characterized in that, The process of simulating the interaction between the target material particles and the virtual electron beam in the simulated particle system, and generating a corresponding microscopic simulation image of the simulated particle system, includes: A virtual electron beam with a preset initial phase is injected into the simulated particle system along the imaging optical axis of the electron microscope corresponding to the simulated particle system. Based on the phase delay generated when the virtual electron beam passes through the target material particles, the phase distribution of the electron wave function after the virtual electron beam penetrates the simulated particle system is calculated. Based on the phase distribution of the electron wave function, a corresponding simulated microscope image is generated.

5. The image synthesis method according to any one of claims 1-4, characterized in that, The method further includes: Based on the position and physical phenotypic characteristics of each target material particle in the simulated particle system in the microscope simulation image, a mask mark corresponding to each pixel of the microscope simulation image is generated. The mask mark is used to characterize the spatial structure information between the target material particles.

6. The image synthesis method according to claim 5, characterized in that, Based on the three-dimensional spatial coordinates of each target material particle in the simulated particle system and the physical phenotypic features in the simulated microscope image, the process of generating a mask marker corresponding one-to-one with the pixels of the simulated microscope image includes: For each target material particle, based on the three-dimensional spatial coordinates of the target material particle in the simulated particle system and the physical phenotypic features, the equivalent physical thickness of the target material particle at each occupied pixel is determined; Based on the equivalent physical thickness of all the target material particles at each occupied pixel, the annotation value of each pixel in each annotation layer is calculated, wherein the target material particles correspond one-to-one with the annotation layers, and the annotation value is the proportion of the thickness contribution of the corresponding target material particles to the pixel. The label values ​​of each pixel in each labeling layer are combined to generate a mask mark for the microscope image.

7. The image synthesis method according to claim 4, characterized in that, The method further includes: Determine the degree of overlap between the signal distribution of the simulated microscope image and the real microscope image in each frequency range; When the overlap meets the preset fine-tuning conditions, the attribute parameters of the virtual electron beam are adjusted, and the adjusted virtual electron beam is used to return to the process of calculating the phase distribution of the electron wave function after the virtual electron beam penetrates the simulated particle system based on the phase delay generated when the virtual electron beam passes through the target material particles, until the overlap meets the preset fine-tuning conditions.

8. The image synthesis method according to claim 4, characterized in that, The process of generating a corresponding microscopic simulation image based on the phase distribution of the electron wavefunction includes: A randomly generated defocus amount is used to modulate the phase distribution of the electronic wavefunction based on the defocus amount; Microscopic simulation images are generated based on the phase distribution of the modulated electronic wavefunction.

9. The image synthesis method according to any one of claims 1-4, characterized in that, The method further includes: Extract real particle texture features from real microscope images and simulated particle texture features from the simulated microscope images; By comparing the real particle texture features with the simulated particle texture features, the uniformity of the simulated particle texture features is determined. If the uniformity exceeds a preset threshold, noise is superimposed on the microscope simulation image so that the uniformity of the microscope simulation image after noise superposition is lower than the preset threshold.

10. The image synthesis method according to any one of claims 1-4, characterized in that, The method further includes: Obtain energy spectrum data of several substrate materials used to support the actual prepared samples; The image background of the microscope simulation image is randomly generated based on the energy spectrum data; The image background is superimposed on the microscope simulation image to obtain the superimposed microscope simulation image.

11. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the image synthesis method as described in any one of claims 1 to 10.

12. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the image synthesis method as described in any one of claims 1 to 10.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the various steps of the image synthesis method as described in any one of claims 1 to 10.

Images