Computer system based on electron microscope image simulation calculation model

Abstract

The present disclosure relates to the field of computer systems based on specific computational models, and more particularly to the field of computer systems based on electron microscope image simulation computational models. The present disclosure describes a computer system based on an electron microscope image simulation computational model, the computer system comprising: an electron microscope image simulation computational model; a simulation display unit; and a simulation control unit, wherein the computer system provides the same operation mode and output display as a prototype electron microscope, wherein the prototype electron microscope comprises a prototype control unit and a prototype display unit, the number and types of operable control devices in the simulation control unit and the prototype control unit are the same, and operating the operable control devices in the simulation control unit causes the same changes in the image on the simulation display unit as operating the operable control devices in the prototype control unit causes the same changes in the image on the prototype display unit.

Description

Technical Field

[0001] The present disclosure relates to the field of computer systems based on specific computational models, and more particularly to the field of computer systems based on electron microscope image simulation computational models. This computer system, together with the simulation control unit (i.e., operating device) contained therein, may also be referred to as or used as an electron microscope simulation training device. This device simulates the full operation of a prototype scientific instrument and provides the same operational results and graphical feedback as the prototype instrument, while costing only approximately one-tenth of the prototype instrument. Furthermore, it is robust, safe, and reliable, unlike the delicate and fragile prototype instrument. Through the computer system, it provides a reliable tool for training expensive, advanced, and sophisticated instruments. Background Art

[0002] Electron microscopy, as a cutting-edge tool for microscopic observation and analysis, is an indispensable method in scientific research and theoretical teaching. Through electron microscopy, researchers can explore the microstructure of materials and understand their properties at an unprecedented level of detail. Electron microscopy instruction is not limited to imparting theoretical knowledge but also emphasizes the cultivation of practical skills.

[0003] Electron microscope equipment is expensive, and practical training can easily damage the equipment and take up already tight scientific research work time, resulting in high training costs. Therefore, it is necessary to conduct training on the operation of the electron microscope before use. Chinese invention patent CN104537903B discloses an interactive transmission electron microscope virtual simulation teaching system, in which "the user is prompted with text and voice to follow the operating steps and precautions. The user performs operations on the input module according to the prompt information, and the video output module outputs the corresponding operation results of the transmission electron microscope three-dimensional model. When the user makes an error in the operation, the video output module and the audio output module respectively give an error alarm prompt." Yan Zhen, Yang Jun, Peng Rui, Li Chenghua, Yi Peishan, Lin Honghui, Zhang Dawei, and Liu Huanhuan in "Construction and Application of Virtual Simulation Experiments of Electron Microscopes [J]; Chinese Journal of Cell Biology; Issue 10, 2022" and Wang Kunlun in "Construction and Application of Virtual Simulation Experiment Teaching System of Scanning Electron Microscopes [J]; Science and Technology Style; Issue 30, 2023" also disclosed similar virtual simulation experiment systems, in which "the instrument operation process needs to be strictly followed according to the steps." The aforementioned virtual simulation experiment systems primarily rely on the training methods of linear virtual simulation teaching systems. These linear virtual simulation teaching systems typically include a user interface, instrument operation guides, a three-dimensional model of the electron microscope, and a database of fixed operating steps. The system prompts the user with operating steps and precautions through text and voice. The user performs pre-set operations on the user interface according to the prompts. When the user makes an error, the system issues an error alarm. After the user completes all pre-set steps, the system provides an operational evaluation. In this type of linear virtual simulation training, users cannot master all the key operational points of the electron microscope operation process, nor can they master the operational experience of quickly handling various problems that arise during the operation process. This type of virtual simulation training, divorced from actual operation, cannot meet users' needs for electron microscope operation training. In addition, these virtual simulation experiments can only perform simple step-by-step process operations using a mouse and keyboard. They completely lack the complex free control and adjustment functions in the electron microscope image generation and adjustment process, and cannot achieve true operational training. To achieve the same multi-degree-of-freedom operations as those on a real electron microscope's operating panel, it's clearly unrealistic to store and retrieve all the changes in electron microscope images corresponding to different operations in a database, as the amount of data required would increase exponentially. Therefore, it's necessary to develop a specific computational model that, starting from a limited number of electron microscope images, can calculate the image effects that a real electron microscope would produce under other circumstances, and then couple them with the corresponding operation and control elements.This particular computational model needs to be able to simulate the entire electron microscope operation and provide the same operational results and graphical feedback as an electron microscope. The technical challenge lies in implementing algorithms that simulate the complex operating principles of the prototype electron microscope system. These include: Electron microscopes precisely control electron beam focusing, scanning, and signal processing to adjust image brightness, contrast, magnification, and focus to ensure high-quality imaging. The electron beam is accelerated and focused, then scanned across the sample surface. The generated secondary electrons and other signals are amplified and modulated to adjust the picture tube brightness, achieving brightness adjustment. Contrast is influenced by multiple factors, including scanning speed, signal quality, sample characteristics, and post-processing software. Magnification is controlled by adjusting the electron beam focusing, scanning range, and electron optical system parameters. A corrective magnetic field is introduced to eliminate astigmatism, and precise focus is achieved through coarse and fine focusing, followed by destigmatization, ultimately producing clear, high-resolution images. Because electron microscopes have a vast range of magnification, their images can reveal images of the atomic-scale microscopic world, requiring an incalculable amount of data and computation. Therefore, it is necessary to simplify the algorithm model as much as possible while maintaining acceptable computing power, thereby developing a computer system based on an electron microscope image simulation model. This simplified algorithm model is combined with a physical controller that fully corresponds to a real electron microscope to simulate the electron microscope photography operation and image display process. This solution is the computer system based on the electron microscope image simulation model provided by the present disclosure. Summary of the Invention

[0004] To address these issues, the present invention provides a computer system based on an electron microscope image simulation model. This system, comprised of both computational model software and specialized control hardware, simulates the actual electron microscope operation process. Users can operate this computer system using the same techniques as a real electron microscope, simplifying the algorithm for simulating the electron microscope image acquisition process. This reduces the cost of electron microscope training and improves its effectiveness.

[0005] The present disclosure provides a computer system based on an electron microscope image simulation computing model, the computer system comprising:

[0006] Computational models for electron microscope image simulation;

[0007] an analog display portion; and

[0008] Analog control unit,

[0009] The computer system provides the same operation mode and output display as the prototype electron microscope, wherein the prototype electron microscope includes a prototype control unit and a prototype display unit, the simulation control unit and the number and types of operable control devices in the prototype control unit are the same, and operating the operable control devices in the simulation control unit causes the changes in the image in the simulation display unit to be the same as the changes in the image in the prototype display unit caused by operating the operable control devices in the prototype control unit.

[0010] The computer system disclosed in the present invention also includes standard hardware and software configurations included in traditional computer systems, such as keyboards, mice, hard disks, transmission lines, etc. Since these parts are conventional technical means in this field, they will not be repeated in this disclosure. In the present disclosure, the parts of the computer system based on the electron microscope image simulation calculation model disclosed in the present invention that are different from traditional computer systems are mainly described, including the electron microscope image simulation calculation model, the simulation control unit corresponding to the actual electron microscope, and the simulation display unit. In addition to including conventional display device hardware, the simulation display unit also includes the process of processing, rendering and performing other conventional processing on the results calculated by the electron microscope image simulation calculation model, wherein the conventional display device hardware includes displays, especially liquid crystal displays, VR helmets, XR glasses, MR glasses, large screens, ring screens, three-dimensional stereo displays, naked-eye 3D display devices, panoramic CAVE systems, desktop XR or naked-eye 3D display stations and other display output devices.

[0011] In one embodiment, the operable control device in the simulation control unit includes a switching button, which can switch between high-magnification mode and low-magnification mode. When the mode is switched by the switching button, the electron microscope image simulation calculation model generates an image through a multi-level mapping texture algorithm, thereby achieving the same change in the image in the simulation display unit as that in the prototype display unit.

[0012] In one embodiment, the operable control device in the simulation control section includes a magnification knob. When the magnification knob is rotated, the electron microscope image simulation calculation model calculates the distance between the simulated camera and the simulated sample and generates an image through a multi-level mapping texture algorithm, thereby achieving the same scaling change of the image in the simulation display section as that in the prototype display section.

[0013] In one embodiment, the operable control device in the simulation control section includes an X-direction position knob and a Y-direction position knob. When the X-direction position knob and / or the Y-direction position knob are rotated, the electron microscope image simulation calculation model simulates and calculates the coordinate position of the electron beam in the three-dimensional scene through a coordinate conversion formula, thereby achieving the same movement change of the image in the simulation display section and the image in the prototype display section.

[0014] In one embodiment, the operable control device in the simulation control unit includes a trackball and a trackball lock. When the trackball is rotated, the electron microscope image simulation calculation model simulates and calculates the coordinate position of the sample in the three-dimensional scene through a coordinate conversion formula, thereby achieving the same movement change of the image in the simulation display unit and the image in the prototype display unit; the trackball lock can lock or unlock the trackball so that rotating the trackball has no or produces no operational effect.

[0015] In one embodiment, the operable control device in the simulation control section includes a brightness knob. When the brightness knob is rotated, the electron microscope image simulation calculation model simulates and calculates the signal intensity of the electron beam incident on the sample surface through a light and dark contrast algorithm, thereby achieving the same light and dark changes in the image in the simulation display section as in the image in the prototype display section.

[0016] In one embodiment, the operable control device in the simulation control section includes a contrast knob. When the contrast knob is rotated, the electron microscope image simulation calculation model simulates and calculates the scanning speed of the electron beam, the resolution and signal-to-noise ratio of the electron beam signal through a light-dark contrast algorithm, thereby achieving the same contrast change between the image in the simulation display section and the image in the prototype display section.

[0017] In one embodiment, the operable control device in the simulation control section also includes a brightness and contrast automatic adjustment button. When the brightness and contrast are automatically adjusted by the brightness and contrast automatic adjustment button, the electron microscope image simulation calculation model simulates and calculates the signal intensity of the electron beam emitted to the sample surface through a light and dark contrast algorithm, and simulates and calculates the scanning speed of the electron beam, the resolution and signal-to-noise ratio of the electron beam signal through a light and dark contrast algorithm, thereby achieving the same brightness and contrast changes of the image in the simulation display section and the image in the prototype display section.

[0018] In one embodiment, the operable control device in the analog control unit includes an astigmatism and correction mode selection button, an astigmatism mode indicator light, and a correction mode indicator light. When the astigmatism mode is selected through the astigmatism and correction mode selection button, the astigmatism mode indicator light lights up and the astigmatism mode is entered. When the correction mode is selected through the astigmatism and correction mode selection button, the correction mode indicator light lights up and the correction mode is entered.

[0019] In one embodiment, the operable control device in the simulation control section includes an X-direction astigmatism and correction knob and a Y-direction astigmatism and correction knob. When the X-direction astigmatism and correction knob and / or the Y-direction astigmatism and correction knob are rotated, the electron microscope image simulation calculation model simulates and calculates the electron beam focusing ability through an astigmatism and focusing algorithm and a noise generation algorithm, thereby achieving the same astigmatism and correction changes in the image in the simulation display section and the image in the prototype display section in the astigmatism mode and the correction mode, respectively.

[0020] In one embodiment, the operable control device in the simulation control unit includes a coarse focusing knob and a fine focusing knob. When the coarse focusing knob and / or the fine focusing knob are rotated, the electron microscope image simulation calculation model simulates and calculates the ability of the electromagnetic lens to focus the electron beam through an astigmatism and focusing algorithm and a noise generation algorithm, thereby achieving the same focusing change of the image in the simulation display unit as that in the prototype display unit.

[0021] In one embodiment, the operable control device in the simulation control unit includes a first-gear scanning speed button, a second-gear scanning speed button, a third-gear scanning speed button, a fourth-gear scanning speed button, an image saving button and a photo taking button. When the first-gear scanning speed button or the second-gear scanning speed button or the third-gear scanning speed button or the fourth-gear scanning speed button and the photo taking button and / or the image saving button are pressed, the electron microscope image simulation calculation model simulates and calculates the scanning speed of the electron beam and the image photo taking effect in the three-dimensional scene through a scanning photo taking algorithm, thereby realizing the scanning photo taking of the image in the simulation display unit that is the same as the image in the prototype display unit.

[0022] In one embodiment, the low magnification mode is 20 to 2000 times, and the high magnification mode is 500 to 1 million times.

[0023] In one embodiment, in the multi-level map texture algorithm,

[0024] ,

[0025] in, is the tangent space normal vector, N is the normal vector in model space, T is the tangent vector in model space, B is the bitangent vector in model space, B=T×N, T, B and N are perpendicular to each other, where:

[0026] ,

[0027] Among them, T x 、T y and T z are the x, y, and z components of the tangent vector T, respectively, and B x 、B y and B z are the x, y, and z components of the bitangent vector B, N x 、N y and N z are the x, y and z components of the normal vector N, respectively, where the model space is the coordinate system for defining and storing the geometric data of the sample model, and the bitangent vector is perpendicular to the tangent vector and the normal vector.

[0028] In one embodiment, in the multi-level map texture algorithm,

[0029] ,

[0030] in, I 镜面 is the color intensity of the specular highlight component, I s is the specular intensity of the light source, K s is the mirror reflection coefficient of the object surface, H is the half-angle vector, which is used to approximate the specular highlight effect, and N is the normal vector of the object surface. dot To find the dot product, pow For the power function, max is a function that finds the maximum value of the elements in the following brackets.

[0031] In one embodiment, the calculation formula of the half-angle vector H is:

[0032] ,

[0033] Among them, L is the incident direction vector of the light, V is the observer direction vector, Normalization (L+V) is the normalized function of (L+V).

[0034] In one embodiment, the noise generation algorithm disturbs the source image through an irregular noise image, and uses the average value of the pixel values ​​at the same position in the two images as the pixel information of the corresponding position in the generated image.

[0035] In one embodiment, in the light-dark contrast algorithm, the main texture color is sampled, the brightness is adjusted, the brightness value is calculated, the grayscale is mixed with the original color to adjust the saturation, the grayscale is mixed with the final color to adjust the contrast, and the final color and the original transparency are returned.

[0036] In one embodiment, in the astigmatism and focus algorithm, the position colors of four different UV coordinates are sampled from the main texture, different weights are applied to these colors for blending, and the final blended color is returned as the pixel color output.

[0037] In one embodiment, in the pan-zoom algorithm,

[0038] ,

[0039] Where x and y are the coordinates before translation, x' and y' are the coordinates after translation, and Δx and Δy are the translation distances in the x and y directions.

[0040] In one embodiment, .

[0041] In one embodiment, in the scanning and photographing algorithm, image data is selectively processed according to the scanning mode, a temporary texture buffer is created, the threshold is adjusted, material parameters are set: diffuse reflection, reflection, specular glossiness, reflection glossiness, bump mapping, subdivision, transparency, refraction, and the contents of the source buffer are rendered to the target buffer, and finally the buffer reference is updated.

[0042] The present disclosure provides a computer system based on an electron microscope image simulation computing model. The specific technical solutions are as follows:

[0043] The overall system includes a simulation calculation model based on electron microscope images and a control system;

[0044] The electron microscope image simulation calculation model includes a display unit and an algorithm library;

[0045] The algorithm library includes a multi-level mapping texture algorithm, a noise generation algorithm, a light and dark contrast algorithm, an astigmatism and focusing algorithm, a translation and magnification algorithm, and a scanning and photographing algorithm;

[0046] The control unit system includes a communication unit and a control unit;

[0047] The control unit includes a trackball control unit, a knob control unit, a switch control unit and an indicator light display;

[0048] The control unit exchanges data with the electron microscope image simulation calculation model through the communication unit and serial port message transmission and reception to achieve the setting of each algorithm parameter in the algorithm library and display the image changes in real time on the display unit;

[0049] The input signal of the button control part of the control part switches the high and low magnification modes. The input signal of the knob control part of the control part simulates and controls the parameters of the electron optical system, generates an image through a multi-level mapping texture algorithm, and calculates the distance between the simulated camera and the simulated sample. The calculation result is transmitted to the display part to realize the image zoom function;

[0050] The input signal of the knob control part of the control part simulates and controls the coordinate position of the electron beam in the three-dimensional scene, and the coordinate conversion formula is used to calculate and transmit the calculation result to the display part to realize the simulated electron beam movement function;

[0051] The input signal of the control part trackball control part simulates the coordinate position of the sample in the three-dimensional scene, and the coordinate conversion formula is used to calculate the coordinate position. The calculation result is transmitted to the display part to realize the simulation sample image movement function;

[0052] The input signal of the knob control part of the control part simulates and controls the signal intensity of the electron beam incident on the sample surface, and is calculated by the light-dark contrast algorithm, and the calculation result is transmitted to the display part to realize the simulated brightness adjustment function;

[0053] The input signal of the knob control part of the control part simulates and controls the scanning speed of the electron beam, the resolution and signal-to-noise ratio of the electron beam signal, and is calculated through the light-dark contrast algorithm. The calculation result is transmitted to the display part to realize the simulated contrast adjustment function;

[0054] The input signal of the knob control part of the control part simulates and controls the focusing ability of the electron beam, and is calculated through the astigmatism and focusing algorithm and the noise generation algorithm, and the calculation result is transmitted to the display part to realize the simulated astigmatism adjustment function;

[0055] The input signal of the control knob control part simulates the ability of the electromagnetic lens to focus the electron beam, and is calculated through the astigmatism and focusing algorithm and the noise generation algorithm, and the calculation result is transmitted to the display part to realize the simulated focus adjustment function;

[0056] According to the input signal of the button control part of the control part, the scanning speed of the electron beam in the three-dimensional scene is simulated and controlled, and the calculation is performed through the scanning and photographing algorithm, and the calculation result is transmitted to the display part to realize the simulated scanning and photographing function;

[0057] In the multi-level mapping texture algorithm,

[0058] ,

[0059] Among them, N t is the tangent space normal vector, N is the normal vector in model space, T is the tangent vector in model space, B is the bitangent vector in model space, B=T×N, T, B and N are perpendicular to each other, where:

[0060] ,

[0061] Among them, T x 、T y and T z are the x, y, and z components of the tangent vector T, B x 、B y and B z are the x, y, and z components of the bitangent vector B, N x 、N y and N z are the x, y, and z components of the normal vector N, respectively, where the model space is the coordinate system that defines and stores the geometric data of the sample model, and the bitangent vector is perpendicular to the tangent vector and the normal vector;

[0062] Among them, in the multi-level mapping texture algorithm,

[0063] ,

[0064] in, I 镜面 is the color intensity of the specular highlight component, I s is the specular intensity of the light source, K s is the mirror reflection coefficient of the object surface, H is the half-angle vector, which is used to approximate the specular highlight effect, and N is the normal vector of the object surface. dot To find the dot product, pow For the power function, max is the function of the maximum value of the elements in the following brackets;

[0065] The calculation formula of the half-angle vector H is:

[0066] ,

[0067] Among them, L is the incident direction vector of the light, V is the observer direction vector, Normalization (L+V) is the normalized function of (L+V);

[0068] The noise generation algorithm is implemented by disturbing the source image through an irregular noise image. The specific method is: taking the average of the pixel values ​​at the same position in the two images as the pixel information at the corresponding position in the generated image.

[0069] The implementation process of the light and dark contrast algorithm is: sampling the main texture color, adjusting the brightness, calculating the brightness value, mixing grayscale with the original color to adjust the saturation, mixing grayscale with the final color to adjust the contrast, and returning the final color and the original transparency.

[0070] The implementation process of the astigmatism and focusing algorithm in the algorithm library is: sampling the position colors of four different UV coordinates from the main texture, applying different weights to these colors to mix them, and returning the final mixed color as the pixel color output.

[0071] The formula of the translation and amplification algorithm is:

[0072] Translation formula:

[0073] ,

[0074] Where x, y, x', and y' are the coordinates before and after translation respectively.

[0075] Magnification formula:

[0076] ,

[0077] The implementation process of the scanning and photographing algorithm is as follows: selectively processing image data according to the scanning mode, creating a temporary texture buffer, adjusting the threshold, setting material parameters: diffuse reflection, reflection, specular glossiness, reflection glossiness, bump mapping, subdivision, transparency, refraction, rendering the contents of the source buffer to the target buffer, and finally updating the buffer reference.

[0078] The control unit exchanges data with the electron microscope image simulation calculation model through the communication unit and serial port message transmission and reception.

[0079] Compared to practical training for electron microscope equipment and linear virtual simulation training, this computer system based on the electron microscope image simulation model offers a simpler simulation algorithm. Utilizing the display unit's scene rendering capabilities, it achieves a fully simulated display of electron microscope sample images, resulting in simulation results that are closer to physical simulations. The simulation data can support real-world physical simulations, is repeatable, and is relatively low-cost. Key algorithms include a multi-level texture mapping algorithm, a noise generation algorithm, a light-dark contrast algorithm, an astigmatism and focusing algorithm, a translational magnification algorithm, and a scanning and imaging algorithm. Furthermore, by establishing a communication module between the electron microscope image simulation model and the control system, and utilizing UART-USB conversion technology for data transmission and a custom-defined optimized data format, this system ensures efficient and real-time information exchange between the control system and the electron microscope image simulation model. This allows for a fully simulated electron microscope operation process in both software and hardware, reducing the cost of practical electron microscope training and improving its effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The accompanying drawings are only examples of an embodiment and cannot be regarded as all necessary conditions of the present invention. In the accompanying drawings:

[0081] Figure 1is a comparison diagram of control devices operable by the simulated control unit and the prototype control unit of the present disclosure;

[0082] Figure 2 It is a schematic diagram of the overall system architecture of the present disclosure;

[0083] Figure 3 This is the imaging simulation effect diagram of the multi-level mapping texture algorithm disclosed in the present invention;

[0084] Figure 4 This is a simulation effect diagram of the noise generation algorithm disclosed in the present invention;

[0085] Figure 5 This is a simulation effect diagram of the brightness and darkness contrast algorithm imaging disclosed in the present invention;

[0086] Figure 6 This is a simulation effect diagram of the astigmatism and focusing algorithm disclosed in the present invention;

[0087] Figure 7 This is a simulation effect diagram of the translation and magnification algorithm of the present invention.

[0088] In the figure: 001 is the trackball lock; 002 is the trackball; 003 is the X-direction position knob; 004 is the Y-direction position knob; 005 is the high and low magnification mode switching button; 006 is the magnification switching knob; 007 is the first-speed scan speed button; 008 is the second-speed scan speed button; 009 is the third-speed scan speed button; 010 is the fourth-speed scan speed button; 011 is the image save button; 012 is the photo button; 013 is the astigmatism mode indicator; 014 is the correction mode indicator; 015 is the astigmatism and correction mode selection button; 016 is the X-direction astigmatism and correction knob; 017 is the Y-direction astigmatism and correction knob; 018 is the brightness and contrast automatic adjustment button; 019 is the brightness knob; 020 is the contrast knob; 021 is the coarse focus knob; 022 is the fine focus knob. DETAILED DESCRIPTION

[0089] The following description clearly and completely describes the technical solutions in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts are within the scope of protection of the present disclosure.

[0090] In the description of the embodiments of the present disclosure, it should be noted that the indicated orientation or position relationship is based on the orientation or position relationship shown in the accompanying drawings, or is the orientation or position relationship in which the disclosed product is usually placed when in use, or is the orientation or position relationship commonly understood by those skilled in the art, or is the orientation or position relationship commonly placed when the disclosed product is in use. This is only for the convenience of describing the present disclosure and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present disclosure.

[0091] In the description of the embodiments of the present disclosure, it should be noted that, unless otherwise expressly specified or limited, the terms "disposed" and "connected" should be understood in a broad sense. For example, they may refer to a fixed connection, a detachable connection, or an integral connection, and may be a direct connection or an indirect connection via an intermediate medium. Those skilled in the art will understand the specific meanings of the above terms in this disclosure based on specific circumstances. Example

[0092] [Example 1]

[0093] Embodiment 1 of the present disclosure discloses a computer system based on an electron microscope image simulation calculation model. The electron microscope image simulation calculation model is implemented using the Unity engine, and the simulation control unit is implemented using 3D printing technology and an STM32 microcontroller chip. The simulation control unit communicates with the PC-side Unity engine via USB3.0 and is compatible with a Type-C data cable, and links the Unity engine to display image changes on a computer monitor.

[0094] like Figure 1 As shown, the number and types of operable control components in the simulation control unit and the prototype control unit are the same, including: 001, trackball lock; 002, trackball; 003, X-direction position knob; 004, Y-direction position knob; 005, high and low magnification mode switching button; 006, magnification switching knob; 007, first-gear scanning speed button; 008, second-gear scanning speed button; 009, third-gear scanning speed button; 010, fourth-gear scanning speed button; 011, image save button; 012, photo button; 013, astigmatism mode indicator; 014, correction mode indicator; 015, astigmatism and correction mode selection button; 016, X-direction astigmatism and correction knob; 017, Y-direction astigmatism and correction knob; 018, brightness and contrast automatic adjustment button; 019, brightness knob; 020, contrast knob; 021, coarse focus knob; 022, fine focus knob.

[0095] like Figure 2As shown, in the computer system based on the electron microscope image simulation calculation model, the electron microscope image simulation calculation model includes the following algorithms: multi-level mapping texture algorithm, noise generation algorithm, light and dark contrast algorithm, astigmatism and focusing algorithm and translation magnification algorithm, scanning and photography algorithm.

[0096] The simulation control unit exchanges data with the Unity engine through the controller, communication middleware, and serial port message transmission and reception, and realizes the parameter setting of the post-rendering processing of the visualization window and the indicator signal sent by the display electron microscope image simulation calculation model through the serial port message, so as to realize that the changes of the image in the simulation display unit caused by operating the operable control device in the simulation control unit are the same as the changes of the image in the prototype display unit caused by operating the operable control device in the prototype control unit. The specific process is as follows: the simulation control unit sends the device model information in a loop until it receives the Unity engine connection confirmation message; the serial port manager is turned on, and polls the serial port with a separate thread and searches for the simulation control unit with a matching model; after the model is matched, the serial port manager starts the waiting thread for receiving messages and suspends the serial port polling thread; the serial port manager sends a connection confirmation message to the matched simulation control unit at regular intervals to maintain the connection; after receiving the connection confirmation message, the simulation control unit reports the status of its own controllers; after receiving the status information of the simulation control unit, the serial port manager sets the event processing flag; after the main loop of the electron microscope image simulation calculation model processes the event, it sends a message with a separate thread to update the status of the indicator light on the simulation control unit. In this process:

[0097] The mathematical formula for calculating the tangent space normal vector in the multi-level mapping texture algorithm is: In the multi-level mapping texture algorithm,

[0098] ,

[0099] Among them, N t is the tangent space normal vector, N is the normal vector in model space, T is the tangent vector in model space, B is the bitangent vector in model space, B=T×N, T, B and N are perpendicular to each other, where:

[0100] ,

[0101] Among them, T x 、T y and T z are the x, y, and z components of the tangent vector T, B x 、B y and B z are the x, y, and z components of the bitangent vector B, N x 、N y and N zare the x, y, and z components of the normal vector N, respectively, where the model space is the coordinate system that defines and stores the geometric data of the sample model, and the bitangent vector is perpendicular to the tangent vector and the normal vector;

[0102] Among them, in the multi-level mapping texture algorithm,

[0103] ,

[0104] in, I 镜面 is the color intensity of the specular highlight component, I s is the specular intensity of the light source, K s is the mirror reflection coefficient of the object surface, H is the half-angle vector, which is used to approximate the specular highlight effect, and N is the normal vector of the object surface. dot To find the dot product, pow For the power function, max is the function of the maximum value of the elements in the following brackets;

[0105] The calculation formula of the half-angle vector H is:

[0106] ,

[0107] Among them, L is the incident direction vector of the light, V is the observer direction vector, Normalization (L+V) is the normalized function of (L+V);

[0108] Figure 3 This is an imaging simulation effect obtained through a multi-level mapping texture algorithm.

[0109] The noise generation algorithm is specifically implemented by using a noise texture and calculating the mean of each point in the image with this noise texture to achieve the noise perturbation effect of the image. In the mean calculation, the larger the noise texture weight is, the greater the noise perturbation of the image is. This is used to simulate the imaging quality of electron microscopes at different scanning speeds. In this embodiment, the brightness of a single pixel on the target image is calculated using the following formula:

[0110] Target image brightness = weight factor × noise image brightness + (1-weight factor) × source image brightness

[0111] The weight factor represents the weight of gray, and its value is not greater than 1. The larger the gray weight, the grayer the image.

[0112] In this embodiment, images are stored in the computer as a brightness matrix and accessed via UV coordinates during post-processing. For example, if the center of a 1080×720 image is white—that is, the brightness of the pixel at column 540 and row 360 is 100%—then the grayscale of the color searched by UV coordinates (0.5, 0.5) is 1. During image post-processing, the brightness of pixels at different UV coordinates is reassigned. The noise map is constantly changing, so the UV coordinates also change at any moment. This embodiment simulates this change using the following formula:

[0113] The u coordinate after the change = the u coordinate before the change + a random number between 0 and 1,

[0114] The v coordinate after the change = the v coordinate before the change + the remainder of time divided by 200 × the remainder of time divided by 200,

[0115] In the uv coordinate system, the horizontal direction is called u and the vertical direction is called v.

[0116] Figure 4 This is a simulation of the imaging effect after the noise generation algorithm.

[0117] In this embodiment, the specific implementation process of the brightness / darkness contrast algorithm is as follows: the brightness of each pixel in the source image is averaged with the pure gray value to adjust the contrast. The more the image leans toward gray, the lower the contrast. The contrast is controlled by adjusting the weight of gray in the weighted average value. The specific implementation formula is as follows:

[0118] Target image brightness = (1-weight factor) × source image brightness + weight factor × 0.5

[0119] The weight factor represents the weight of gray, and its value is not greater than 1. The larger the gray weight, the grayer the image.

[0120] Figure 5 This is a simulation effect of imaging through a light-dark contrast algorithm.

[0121] In this embodiment, astigmatism and focusing are calculated using the following formula:

[0122] ,

[0123] Among them, η 中心 represents the center weight of the point to be blurred in the source image, η 左 ,η 右 ,η 上、 η 下Represents the weights of the four points on the left, right, top and bottom. These weights are multiplied by the brightness of the center point and its nearby points and the sum is the average brightness. Among them, C(u-Δ,v) represents the brightness of the point slightly to the left of the center point, and Δ represents the distance to the left. For the case of defocus, the weights and the distances of the sampling points are symmetrically distributed. When astigmatism occurs, asymmetric weights or unequal sampling distances are used for simulation. When unequal sampling distances are used for simulation, the sum of the distances is a fixed value, and its direction angle is θ As the focus knob rotates, Δx=r* is recalculated. θ and Δy=r* θ For the case of misalignment, fix the above (r, θ ) θ , and let r change periodically with time, that is, .

[0124] Figure 6 This is the imaging simulation effect after astigmatism and focusing algorithm.

[0125] In this embodiment, the specific formula for implementing the translation and magnification algorithm is as follows:

[0126] Translation formula:

[0127] ,

[0128] Where x, y, x', and y' are the coordinates before and after translation respectively.

[0129] Magnification formula:

[0130] ,

[0131] The specific implementation process of the scanning and photographing algorithm is as follows: selectively process the image data according to the scanning mode, create a temporary texture buffer, adjust the threshold, set the material parameters: diffuse reflection, reflection, specular glossiness, reflection glossiness, bump mapping, subdivision, transparency, refraction, and render the contents of the source buffer to the target buffer, and finally update the buffer reference.

[0132] Figure 7 This is the imaging simulation effect of the translation and magnification algorithm.

[0133] When the control unit does not receive a connection confirmation message from the electron microscope image simulation calculation model for a period of time, it stops sending its own status information and restarts to send model information in a loop to wait for the electron microscope image simulation calculation model to be reopened;

[0134] When the electron microscope image-based simulation calculation model captures a "disconnection" related exception when attempting to receive a serial port message, a disconnection event flag is set, and serial port polling is restarted to wait for the control unit to reconnect.

[0135] The present disclosure is not limited to the aforementioned specific embodiments, but extends to any new features or any new combination disclosed in this specification, as well as any new method or process steps or any new combination disclosed.

Claims

1. A computer system based on an electron microscope image simulation computing model, the computer system comprising: Computational models for electron microscope image simulation; an analog display portion; and An analog control unit, which is a physical control device that is the same as or similar to the controller of the prototype electron microscope and includes operable control devices. The controller of the prototype electron microscope is referred to as a prototype control unit, and the display device of the prototype electron microscope is referred to as a prototype display unit; The number and types of control devices operable in the simulation control unit and the prototype control unit are the same, wherein the computer system provides the same operation and output display as the prototype electron microscope, operating the operable control device in the simulation control unit so that the image on the simulation display unit changes in the same way as the image on the prototype display unit changes when the operable control device in the prototype control unit is operated, the image being generated by a multi-level mapping texture algorithm; In the multi-level mapping texture algorithm, ， in, N t is the tangent space normal vector, N is the normal vector in model space, T is the tangent vector in model space, B is the bitangent vector in model space, B=T×N, T, B and N are perpendicular to each other, where: ， Among them, T x 、T y and T z are the x, y, and z components of the tangent vector T, respectively, and B x 、B y and B z are the x, y, and z components of the bitangent vector B, N x 、N y and N z are the x, y, and z components of the normal vector N, respectively, where the model space is the coordinate system that defines and stores the geometric data of the sample model, and the bitangent vector is perpendicular to the tangent vector and the normal vector; Among them, in the multi-level mapping texture algorithm, ， in, I 镜面 is the color intensity of the specular highlight component, I s is the specular intensity of the light source, K s is the mirror reflection coefficient of the object surface, H is the half-angle vector, which is used to approximate the specular highlight effect, and N is the normal vector of the object surface. dot To find the dot product, pow For the power function, max is the function of the maximum value of the elements in the following brackets; The calculation formula of the half-angle vector H is: ， Among them, L is the incident direction vector of the light, V is the observer direction vector, Normalization (L+V) is the normalized function of (L+V).

2. The computer system based on the electron microscope image simulation computing model as described in claim 1, wherein the operable control device in the simulation control unit includes a switching button, and the switching button can switch between high-magnification mode and low-magnification mode. When the mode is switched by the switching button, the electron microscope image simulation computing model generates an image through a multi-level mapping texture algorithm, thereby achieving the same change in the image in the simulation display unit as that in the prototype display unit.

3. The computer system based on the electron microscope image simulation computing model as described in claim 1, wherein the operable control device in the simulation control unit includes a magnification knob. When the magnification knob is rotated, the electron microscope image simulation computing model calculates the distance between the simulated camera and the simulated sample and generates an image through a multi-level mapping texture algorithm, thereby achieving the same scaling change of the image in the simulation display unit and the image in the prototype display unit.

4. The computer system based on the electron microscope image simulation calculation model of claim 1, wherein the operable control device in the simulation control unit includes an X-direction position knob and a Y-direction position knob, and when the X-direction position knob and / or the Y-direction position knob are rotated, the electron microscope image simulation calculation model simulates and calculates the coordinate position of the electron beam in the three-dimensional scene through a coordinate conversion formula, thereby achieving the same movement change of the image in the simulation display unit and the image in the prototype display unit.

5. The computer system based on the electron microscope image simulation calculation model of claim 1, wherein the operable control device in the simulation control unit includes a trackball and a trackball lock; when the trackball is rotated, the electron microscope image simulation calculation model simulates and calculates the coordinate position of the sample in the three-dimensional scene through a coordinate conversion formula, thereby achieving the same movement change of the image in the simulation display unit and the image in the prototype display unit; the trackball lock can lock or unlock the trackball so that rotating the trackball has no or a no operation effect.

6. The computer system based on the electron microscope image simulation calculation model as described in claim 1, wherein the operable control device in the simulation control unit includes a brightness knob. When the brightness knob is rotated, the electron microscope image simulation calculation model simulates and calculates the signal intensity of the electron beam incident on the sample surface through a light and dark contrast algorithm, thereby achieving the same light and dark changes in the image in the simulation display unit and the image in the prototype display unit.

7. The computer system based on the electron microscope image simulation calculation model as described in claim 1, wherein the operable control device in the simulation control unit includes a contrast knob. When the contrast knob is rotated, the electron microscope image simulation calculation model simulates and calculates the scanning speed of the electron beam, the resolution and signal-to-noise ratio of the electron beam signal through a light-dark contrast algorithm, thereby achieving the same contrast change between the image in the simulation display unit and the image in the prototype display unit.

8. The computer system based on the electron microscope image simulation calculation model as described in claim 1, wherein the operable control device in the simulation control unit also includes a brightness and contrast automatic adjustment button. When the brightness and contrast are automatically adjusted by the brightness and contrast automatic adjustment button, the electron microscope image simulation calculation model simulates and calculates the signal intensity of the electron beam emitted to the sample surface through a light and dark contrast algorithm, and simulates and calculates the scanning speed of the electron beam, the resolution and signal-to-noise ratio of the electron beam signal through a light and dark contrast algorithm, thereby achieving the same brightness and contrast changes of the image in the simulation display unit and the image in the prototype display unit.

9. The computer system based on the electron microscope image simulation calculation model of claim 1, wherein the operable control components in the simulation control unit include an astigmatism and correction mode selection button, an astigmatism mode indicator light, and a correction mode indicator light; when the astigmatism mode is selected by the astigmatism and correction mode selection button, the astigmatism mode indicator light turns on, and the system enters the astigmatism mode; when the correction mode is selected by the astigmatism and correction mode selection button, the correction mode indicator light turns on, and the system enters the correction mode.

10. The computer system based on the electron microscope image simulation calculation model of claim 9, wherein the operable control device in the simulation control unit includes an X-direction astigmatism and correction knob and a Y-direction astigmatism and correction knob. When the X-direction astigmatism and correction knob and / or the Y-direction astigmatism and correction knob are rotated, the electron microscope image simulation calculation model simulates and calculates the electron beam focusing ability using an astigmatism and focusing algorithm and a noise generation algorithm, thereby achieving the same astigmatism and correction changes in the image on the simulation display unit and the image on the prototype display unit in the astigmatism mode and the correction mode, respectively.

11. The computer system based on the electron microscope image simulation calculation model of claim 1, wherein the operable control device in the simulation control unit includes a coarse focus knob and a fine focus knob. When the coarse focus knob and / or the fine focus knob are rotated, the electron microscope image simulation calculation model simulates and calculates the ability of the electromagnetic lens to focus the electron beam through an astigmatism and focusing algorithm and a noise generation algorithm, thereby achieving the same focus change of the image in the simulation display unit and the image in the prototype display unit.

12. The computer system based on the electron microscope image simulation calculation model as described in claim 1, wherein the operable control device in the simulation control unit includes a first-gear scanning speed button, a second-gear scanning speed button, a third-gear scanning speed button, a fourth-gear scanning speed button, an image save button and a photo button. When the first-gear scanning speed button, the second-gear scanning speed button, the third-gear scanning speed button or the fourth-gear scanning speed button and the photo button and / or the image save button are pressed, the electron microscope image simulation calculation model simulates and calculates the scanning speed of the electron beam and the image photo saving effect in the three-dimensional scene through a scanning and photo algorithm, thereby realizing scanning and photo taking of the image in the simulation display unit that is the same as the image in the prototype display unit.

13. The computer system based on the electron microscope image simulation calculation model according to claim 2, wherein: The low-magnification mode is 20 to 2,000 times, and the high-magnification mode is 500 to 1,000,000 times.

14. The computer system based on the electron microscope image simulation calculation model according to claim 10 or 11, in, The noise generation algorithm disturbs the source image through an irregular noise image, and uses the average value of the pixel values ​​at the same position in the two images as the pixel information of the corresponding position in the generated image.

15. The computer system based on the electron microscope image simulation calculation model according to claim 6, 7 or 8, in, In the light and dark contrast algorithm, the main texture color is sampled, the brightness is adjusted, the brightness value is calculated, the grayscale is mixed with the original color to adjust the saturation, the grayscale is mixed with the final color to adjust the contrast, and the final color and the original transparency are returned.

16. The computer system based on the electron microscope image simulation calculation model according to claim 10 or 11, in, In the astigmatism and focusing algorithm, the position colors of four different UV coordinates are sampled from the main texture, different weights are applied to these colors for mixing, and the final mixed color is returned as the pixel color output.

17. The computer system based on the electron microscope image simulation calculation model according to claim 4 or 5, in, The coordinate transformation formula is a translation and magnification algorithm. In the translation and magnification algorithm, ， Where x and y are the coordinates before translation, x' and y' are the coordinates after translation, and Δx and Δy are the translation distances in the x and y directions; in, .

18. The computer system based on the electron microscope image simulation calculation model according to claim 12, in, In the scanning and photographing algorithm, the image data is selectively processed according to the scanning mode, a temporary texture buffer is created, the threshold is adjusted, the material parameters are set: diffuse reflection, reflection, specular glossiness, reflection glossiness, bump mapping, subdivision, transparency, refraction, and the contents of the source buffer are rendered to the target buffer, and finally the buffer reference is updated.

Images