Fault simulation-based electron microscope virtual training method, equipment and medium

Abstract

The invention relates to the technical field of computers, in particular to an electron microscope virtual training method and device based on fault simulation and a medium, and the method comprises the steps: determining a constructed virtual electron microscope model; generating and issuing a corresponding electron microscope operation task; generating a simulation fault for the electron microscope operation task according to a pre-constructed electron microscope fault knowledge graph, and adjusting the virtual electron microscope model according to the simulation fault; collecting real-time component operation of the user in the process of executing the electron microscope operation task in the virtual environment; and generating a real-time observation image corresponding to the observation target object based on the real-time component operation. Therefore, the user can carry out diagnosis and repair exercises of various faults in a virtual environment without risks, dependence and loss of training on physical equipment are reduced, and safe and low-cost high-fidelity fault simulation training is realized.

Description

Technical Field

[0001] This invention relates to a virtual training method, equipment, and medium for electron microscopy based on fault simulation, and belongs to the field of computer technology. Background Technology

[0002] Electron microscopes (EMs) are high-precision instruments that play an irreplaceable role in many fields such as materials science, biomedicine, and semiconductor testing. However, EMS is complex to operate, expensive, and prone to malfunctions in practical use, placing high demands on the professional skills and troubleshooting abilities of operators.

[0003] Traditional electron microscope operation training relies on physical equipment and instructor guidance, resulting in high costs for training on physical equipment and the risk of equipment damage due to misoperation. Furthermore, simulating faults on actual equipment is difficult, making it hard to artificially create diverse and controllable fault scenarios, leaving trainees lacking practical experience in handling abnormal situations. Moreover, the training content and effectiveness are limited by the instructor's individual experience, making it difficult to systematically and standardizedly cover common fault types.

[0004] Although some electron microscope operation simulation software exists, most of them can only perform normal procedural operation simulations. The fault modes are mostly pre-set fixed scripts, lacking dynamism and intelligence. They are difficult to flexibly generate faults that are close to reality according to training tasks, and they are also difficult to conduct in-depth analysis and real-time guidance of the user's operation process. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by providing a virtual training method, equipment, and medium for electron microscopy based on fault simulation.

[0006] The technical solution of this invention to solve the above-mentioned technical problems is as follows: a virtual training method for electron microscopy based on fault simulation, comprising:

[0007] In a virtual environment, a virtual electron microscope model that has been constructed is determined; the virtual electron microscope model includes multiple model components;

[0008] Generate and publish corresponding electron microscope operation tasks for users currently undergoing virtual electron microscope training;

[0009] Based on a pre-built electron microscope fault knowledge graph, simulated faults for the electron microscope operation task are generated through a large language model, and the virtual electron microscope model is adjusted according to the simulated faults.

[0010] Real-time component operations are collected during the process of the user performing the electron microscope operation task in the virtual environment;

[0011] Based on the operation of the real-time component, a real-time observation image corresponding to the observed target object is generated.

[0012] The beneficial effects of this invention are:

[0013] By constructing a virtual electron microscope model and adjusting component parameters in a digital twin data model to introduce simulated faults, users can practice diagnosing and repairing various faults in a virtual environment without risk. This reduces the dependence on and wear and tear of physical equipment for training, and achieves safe, low-cost, high-fidelity fault simulation training.

[0014] Based on a pre-built electron microscope fault knowledge graph, and leveraging the analysis and generation capabilities of a large language model, it is possible to dynamically generate realistic simulated faults for different electron microscope operation tasks. This breaks through the limitations of fixed and limited fault scenarios in traditional simulation software, enhances the diversity and relevance of training, and provides dynamic and intelligent fault generation capabilities.

[0015] By collecting users' real-time component operations and generating corresponding real-time observation images, the system provides users with intuitive feedback on their operation results, simulates the operation experience of a real electron microscope, and lays a solid foundation for further evaluation of user operations and provision of real-time guidance, thus achieving effective interaction and feedback in the training process.

[0016] Based on the above technical solution, the present invention can be further improved as follows:

[0017] Furthermore, the construction process of the electron microscopy fault knowledge graph includes:

[0018] Acquire knowledge related to electron microscope malfunctions;

[0019] Construct a meta-model corresponding to the electron microscope fault knowledge graph, and define the types and attributes of nodes and edges in the meta-model;

[0020] Natural language processing is performed on the knowledge content to obtain text content, and the corresponding entities and relationships are extracted from the text content.

[0021] Based on the entities and relationships, the nodes and edges in the meta-model are filled to generate an electron microscopy fault knowledge graph.

[0022] The beneficial effects of adopting the above-mentioned further solutions are:

[0023] By constructing a knowledge graph of electron microscope (EM) faults, relevant knowledge can be systematically and structurally integrated. This allows for the generation of more accurate simulated faults that closely match real-world conditions, based on the rich information within the graph. Simultaneously, the knowledge graph provides a clear and accurate knowledge foundation for large language models, enabling them to better analyze and process EM operation tasks. This results in more targeted and diverse simulated faults, improving the realism and effectiveness of fault simulations and further enhancing the quality and impact of virtual training. Furthermore, the knowledge graph facilitates the management and updating of EM fault knowledge, allowing for the timely integration of new fault information and handling methods into the training system, ensuring the timeliness and practicality of the training content.

[0024] Furthermore, the virtual electron microscope model is adjusted based on the simulated fault, specifically including:

[0025] Generate a corresponding digital twin data model for the virtual electron microscope model;

[0026] The conventional parameter ranges of the component parameters of each model component in the virtual electron microscope model are defined using the digital twin data model.

[0027] Based on the simulated fault, the parameter values ​​of the component are adjusted so that they do not conform to the normal parameter range.

[0028] Furthermore, after adjusting the virtual electron microscope model based on the simulated fault, the method further includes:

[0029] Based on the adjustment of the component parameters, the corresponding reverse adjustment is taken as the corresponding repair action;

[0030] Based on the preset priority of each repair action, the corresponding repair action sequence is obtained;

[0031] Based on the real-time component operation, the user's real-time completion level of the electron microscope operation task is determined by the preset standard component operation sequence corresponding to the electron microscope operation task and the repair action sequence corresponding to the simulated fault.

[0032] Based on the changing trend of the real-time completion rate, the user is given corresponding prompts.

[0033] Furthermore, based on the real-time component operations, the user's real-time completion level of the electron microscope operation task is determined through a preset standard component operation sequence corresponding to the electron microscope operation task and a repair action sequence corresponding to the simulated fault. Specifically, this includes:

[0034] Determine the standard component operation sequence corresponding to the preset electron microscope operation task, and the first actions included in the repair action sequence corresponding to the simulated fault;

[0035] Based on the operation of each real-time component, determine whether there is a corresponding second action in the first action;

[0036] If it exists, the real-time adjustment value of the running parameters corresponding to the real-time component operation is compared with the standard adjustment value of the running parameters corresponding to the second action to obtain the sub-completion degree corresponding to the real-time component operation.

[0037] Based on the sub-completion degree corresponding to each real-time component operation and their respective weights, the real-time completion degree of the user's electron microscope operation task is obtained by weighting.

[0038] The beneficial effects of adopting the above-mentioned further solutions are:

[0039] By accurately calculating the real-time completion rate of electron microscope operations, users can receive quantitative operational assessments. This helps users clearly understand their performance during the operation, identifying which steps were performed well and which still have room for improvement. For trainers, this data allows for targeted guidance and feedback, improving training efficiency and effectiveness. Furthermore, prompts based on trends in real-time completion rates can promptly correct user errors, further enhancing training quality and the user's learning experience. Moreover, this quantitative assessment and real-time prompting mechanism makes the training process more scientific and standardized, contributing to the development of electron microscope operators with higher professional competence and troubleshooting abilities.

[0040] Furthermore, generating a real-time observation image corresponding to the observed target object based on the operation of the real-time component specifically includes:

[0041] Generate the corresponding original observation image based on the observed target object;

[0042] For the model components contained in the virtual electron microscope model, according to their corresponding component parameters and operating parameters, the corresponding rendering method is selected to render the original observation image to obtain the real-time observation image;

[0043] Based on the real-time component operation, the rendering level of the corresponding rendering method is adjusted according to the adjustment of the component parameters and / or running parameters of the model component, so as to regenerate the real-time observation image.

[0044] Furthermore, based on the changing trend of the real-time completion rate, corresponding prompts are given to the user, specifically including:

[0045] If the real-time completion rate does not reach the preset growth rate after a preset time period, the trend of the real-time completion rate is determined.

[0046] If the trend of change is an oscillating trend or an upward trend, then determine the highest sub-completion level corresponding to the current real-time component operation;

[0047] If the highest sub-completion level is higher than a preset threshold, then a prompt is generated for the user based on the real-time adjustment value corresponding to the highest sub-completion level, and prompts are also generated for the user based on the operation of several other real-time components with the lowest sub-completion levels.

[0048] If the highest completion level is lower than a preset threshold, a prompt will be generated for the user based on the current real-time component operation.

[0049] If the trend is downward, then a prompt is generated for the user based on the current real-time component operation.

[0050] The beneficial effects of adopting the above-mentioned further solutions are:

[0051] By providing prompts to users based on the real-time progress trend, potential problems during operation can be identified promptly. When the real-time progress does not increase as expected, corresponding prompts are given based on different trends and sub-completion levels, helping to guide users to optimize their operations. For situations with fluctuating or upward trends and high highest sub-completion levels, prompts that refer to both the highest completion level and low completion levels allow users to fully understand their operational status and further improve their skills. Conversely, for situations with low highest sub-completion levels or a downward trend, prompts directly address the current operation, helping users quickly correct errors and prevent further deterioration. This personalized and targeted prompting mechanism greatly enhances the learning effectiveness of users in virtual training, making training more efficient and targeted, enabling users to master electron microscope operation skills and troubleshooting capabilities more quickly, and adapt to various complex situations in real-world work.

[0052] Furthermore, the method also includes:

[0053] If, based on the user's historical records, it is determined that the user's real-time completion rate in multiple electron microscope operation tasks meets the preset requirements and that the user has not performed any preset prohibited component operations, then, based on the Internet of Things, the user is associated with an actual electron microscope device.

[0054] The actual electron microscope equipment is operated based on the real-time component operation.

[0055] The beneficial effects of adopting the above-mentioned further solutions are:

[0056] By associating qualified users with actual electron microscope equipment, users can smoothly transition to operating the actual equipment after thorough virtual training, closely integrating virtual training with practical application. Troubleshooting and operational experience accumulated in the virtual environment can be directly applied to the actual equipment, reducing the probability of errors in real-world operation and improving the safety and efficiency of using the equipment. Simultaneously, this provides users with a more realistic operating experience, helping to further enhance their professional skills and practical abilities, ensuring that training results are effectively transformed into productivity in actual work.

[0057] On the other hand, this application also proposes a virtual training device for electron microscopy based on fault simulation, comprising:

[0058] At least one processor; and,

[0059] A memory communicatively connected to the at least one processor; wherein,

[0060] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the fault simulation-based electron microscopy virtual training method as described in any of the above examples.

[0061] On the other hand, this application also proposes a non-volatile computer storage medium storing computer-executable instructions configured to implement the fault simulation-based electron microscope virtual training method as described in any of the above examples. Attached Figure Description

[0062] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0063] Figure 1 This is a flowchart illustrating the electron microscope virtual training method based on fault simulation in the embodiments of this application;

[0064] Figure 2 This is a schematic diagram of the system framework for the application of the electron microscope virtual training method based on fault simulation in one scenario of this application embodiment;

[0065] Figure 3 This is a partial structural diagram of an electron microscope fault knowledge graph under one scenario in an embodiment of this application;

[0066] Figure 4 This is a schematic diagram of an electron microscope virtual training device based on fault simulation in an embodiment of this application. Detailed Implementation

[0067] The specific embodiments of the present invention will be described in detail below. The present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used is for describing particular embodiments only and is not intended to limit the invention.

[0069] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0070] like Figure 1 As shown, embodiments of this application provide a virtual training method for electron microscopy based on fault simulation, including:

[0071] S101: In a virtual environment, determine the constructed virtual electron microscope model; the virtual electron microscope model includes multiple model components.

[0072] Virtual environments can include computer software environments, virtual reality (VR) environments, etc., and can be built using smart devices such as computers, smartphones, and VR devices.

[0073] The virtual electron microscope model comprises multiple model components. Each component can be generated by 3D scanning of a physical electron microscope or by researchers through 3D design and fabrication. These model components are then assembled according to the actual structure to create the virtual electron microscope model in a virtual environment.

[0074] To ensure realism and accuracy when constructing virtual electron microscope models, high-precision 3D scanning technology can be used on each model component of the physical electron microscope equipment to obtain precise shape and size data. For components that require frequent operation and adjustment in actual use, researchers can perform more detailed 3D design and fabrication to ensure that the operation feel and feedback in the virtual environment are consistent with the real equipment.

[0075] S102: Generate and publish corresponding electron microscope operation tasks for users currently performing virtual electron microscope training.

[0076] like Figure 2 As shown, the system framework includes a client (which can be mounted on smartphones, computers, VR glasses, etc.) and a cloud. Users log in to the virtual environment in the client using an account and password. The client verifies the user's identity by interacting with the cloud. Once the identity verification is successful, the user logs into the virtual environment.

[0077] The system has a pre-set electron microscope operation task library, which can select and publish corresponding electron microscope operation tasks to users based on user confirmation, user's historical task records, user's random selection, etc.

[0078] Electron microscope operation tasks can be designed to take into account users' different skill levels and training objectives. For novice users, tasks may focus on familiarizing themselves with and mastering basic operations, such as powering on and off the equipment and setting basic parameters. For more experienced users, tasks will focus more on improving fault diagnosis and handling capabilities, and may include some complex simulated faults, requiring users to locate and repair the faults within a specified time.

[0079] During task assignment, the system can detail the task's objectives and requirements, and based on user selections, explain the corresponding operational steps. It can also provide relevant prompts and guidance based on user needs. The difficulty of the task can be dynamically adjusted according to the user's learning progress and performance, ensuring that users are trained in a challenging yet achievable task environment.

[0080] Furthermore, to enhance the fun and interactivity of the training, the system may incorporate gamification elements. For example, a task leaderboard could be set up, allowing users to see their ranking among all participants; or users who excel at completing tasks could be rewarded with virtual badges or points, which could be redeemed for more learning resources or to unlock more advanced training tasks. These methods can stimulate users' learning enthusiasm and competitive spirit, thereby better achieving the training objectives.

[0081] S103: Based on the pre-constructed electron microscope fault knowledge graph, generate simulated faults for the electron microscope operation task through a large language model, and adjust the virtual electron microscope model according to the simulated faults.

[0082] The electron microscope malfunction knowledge graph describes the malfunctions that electron microscopes may encounter during actual operation using structured language.

[0083] Specifically, the process of constructing an electron microscopy fault knowledge graph may include:

[0084] Acquire knowledge related to electron microscope malfunctions. This can be done through expert experience (including senior electron microscope engineers, maintenance experts, etc.), equipment documentation, equipment maintenance records, academic literature, and forums. Based on this acquired knowledge, organize the malfunctions, symptoms, and solutions that may be encountered in actual work, and include these as relevant knowledge content.

[0085] A meta-model corresponding to the electron microscopy fault knowledge graph is constructed, and the types and attributes of nodes and edges in the meta-model are defined. The type of a node can include fault phenomena, equipment components, fault manifestations, solutions, etc., and its attribute is the corresponding value. The type of an edge can vary depending on the nodes it connects to.

[0086] Natural language processing is performed on the knowledge content to obtain text content, and corresponding entities and relations are extracted from the text content. For example, from the knowledge content "Image jitter is usually caused by an unstable sample stage," we can extract the entities "image jitter" and "sample stage," and the relations "image jitter" [caused by...] - "unstable sample stage." At this point, the node for the equipment component is the sample stage, the node for the fault phenomenon is "image jitter," and the nodes corresponding to the fault manifestation and solution are not yet obtained from this knowledge content and need to be obtained in conjunction with other knowledge content.

[0087] Based on entities and relationships, nodes and edges in the meta-model are populated to generate an electron microscopy fault knowledge graph. For example... Figure 3 As shown, a partial structure of the electron microscope fault knowledge graph is provided as an example. The node corresponding to the equipment component has the attribute of objective aperture. The possible fault phenomena it can cause include aperture contamination, aperture misalignment, and physical damage to the aperture. For the aperture contamination node, the fault manifestations are blurred image, fixed and irregular shadows or spots in the image, and severe astigmatism. The corresponding solutions are to attempt cleaning, aperture alignment, and replacement of the aperture.

[0088] Meanwhile, the Large Language Model (LLM) can be deployed locally, either by selecting an existing open-source LLM or by training it using its own collected samples. It can generate simulated faults in a fixed format for the current electron microscopy operation task (e.g., acquiring clear images of the observed object, adjusting a model component, etc.).

[0089] Specifically, a corresponding digital twin data model is generated for each virtual electron microscope model. Within this digital twin data model, corresponding component parameters are set for each model component in the virtual electron microscope model. These component parameters describe the properties of the model component itself; for example, the objective aperture's component parameters may include aperture size, centering position, and degree of contamination.

[0090] Using a digital twin data model, the standard parameter ranges for each component in the virtual electron microscope model are defined. Within these standard parameter ranges, the model component can operate normally. Taking the objective aperture as an example, the standard parameter range corresponding to the aperture size can include {20μm, 30μm, 50μm, 100μm}, the standard parameter range corresponding to the centering position can be (0±x, 0±x), where x is the allowable deviation value, typically not exceeding 5μm, and the standard parameter range corresponding to the degree of contamination can be 0%~y%, where y% is the allowable contamination value, typically not exceeding 10%.

[0091] Based on the simulated faults, the parameter values ​​of the components are adjusted to deviate from the normal parameter range. For example, if the simulated fault is objective aperture contamination, the parameter value corresponding to the degree of contamination needs to be adjusted by more than 10%; if the simulated fault is that the aperture size selection does not match the current requirements (for example, the currently selected aperture size is too low, resulting in a dark final observation image that does not meet the requirements), the default aperture size needs to be selected to the corresponding aperture size that does not meet the requirements.

[0092] S104: Collect real-time component operations during the process of the user performing the electron microscope operation task in the virtual environment.

[0093] The system is pre-configured with corresponding user-executable component operations. These include adjusting, replacing, moving, and cleaning model components within the virtual electron microscope model. Different operations produce different effects on the final observed image of the target object, such as making the image clearer, more blurred, or moving the target object's position within the lens. For instance, real-time component operations can include the corresponding operation object (usually a knob within the device component) and its corresponding parameter changes (including adjustment range and direction). It can also collect other data such as time consumption as part of the real-time component operation.

[0094] In addition, users may perform other operations not configured by the system during system use, which can also be used as real-time component operations for data acquisition. However, these operations are not included in the completion calculation or the generation of subsequent real-time observation images. For example, these operations could include changing the observation angle or viewing exploded views of the structure.

[0095] S105: Generate a real-time observation image corresponding to the observed target object based on the operation of the real-time component.

[0096] The target object to be observed is the object that needs to be observed in this electron microscopy operation. For example, it can be a corresponding cellular structure, inorganic structure, etc. If the electron microscopy operation does not include a target object to be observed, such as if the operation is only used for a simple task like starting up a device component, then this step is not required.

[0097] An observation image refers to an image of a target object observed under an electron microscope. The specific process of generating a real-time observation image may include the following:

[0098] The original observation image is generated based on the observed target object. This original observation image is a high-definition image of the original observed target object itself, which can clearly reflect the shape, size, state, etc. of the observed target object.

[0099] For each model component within the virtual electron microscope model, a corresponding rendering method is selected based on its component parameters and runtime parameters to render the original observation image, resulting in a real-time observation image. Different component parameters and runtime parameters will lead to different rendering effects. When the user does not perform any component operations, the component parameters and runtime parameters are set to the default initial values ​​for each model component.

[0100] Generally, each model component corresponds to at least one rendering process. For example, parameters such as the aperture size and contamination level of the objective aperture affect light transmission and image quality, thus producing different filter effects during rendering. When the aperture size is inappropriate, the image may become blurry or have uneven brightness; in this case, the rendering method can be a blurring effect such as Gaussian blur. When the contamination level is high, the image may have blemishes or shadows; in this case, the rendering effect can be adding blemishes or shadows. In this case, the objective aperture corresponds to two rendering processes.

[0101] The rendering process is used to give the original observation image visual effects such as blurring and shadows, thereby simulating the appearance of the observation image when the virtual electron microscope model is not adjusted to the appropriate values.

[0102] In real-time component operations, adjustments to the component parameters and / or runtime parameters of model components affect the rendering level of the corresponding rendering method to regenerate the real-time observation image. When a user adjusts component parameters and / or runtime parameters in real-time component operations, the corresponding rendering method does not need to be adjusted, but the rendering level during the rendering process needs to be adjusted. For example, if a user adjusts the aperture size of the objective lens, and this adjustment is a positive adjustment that meets the task requirements, the Gaussian blur level can be reduced.

[0103] By constructing a virtual electron microscope model and adjusting component parameters in a digital twin data model to introduce simulated faults, users can practice diagnosing and repairing various faults in a virtual environment without risk. This reduces the dependence on and wear and tear of physical equipment for training, and achieves safe, low-cost, high-fidelity fault simulation training.

[0104] Based on a pre-built electron microscope fault knowledge graph, and leveraging the analysis and generation capabilities of a large language model, it is possible to dynamically generate realistic simulated faults for different electron microscope operation tasks. This breaks through the limitations of fixed and limited fault scenarios in traditional simulation software, enhances the diversity and relevance of training, and provides dynamic and intelligent fault generation capabilities.

[0105] By collecting users' real-time component operations and generating corresponding real-time observation images, the system provides users with intuitive feedback on their operation results, simulates the operation experience of a real electron microscope, and lays a solid foundation for further evaluation of user operations and provision of real-time guidance, thus achieving effective interaction and feedback in the training process.

[0106] In one embodiment, during the user's actual operation, corresponding auxiliary reminders can be given based on the assessment of the user's actual skill level.

[0107] At this point, based on the adjustment of component parameters, the corresponding reverse adjustment is used as the corresponding repair action. Taking the objective aperture as an example again, if the simulated fault is objective aperture contamination, the corresponding component parameter adjustment is to adjust the parameter value of the contamination level by more than 10%. The reverse adjustment would then be to adjust the parameter value of the contamination level to less than 10%. When the user wants to perform this reverse adjustment, the repair actions that can be performed include replacing the objective aperture and cleaning the objective aperture. From the virtual environment, the user can select the model component corresponding to the objective aperture. In the pop-up dialog box, the user can choose to observe the contamination level to obtain the parameter value of the component corresponding to the current contamination level, or, based on their own experience, judge that the contamination level is high based on the performance of the real-time observed image. Simultaneously, in the adjacent dialog box, the user can select buttons such as "Replace Objective Aperture" and "Clean Objective Aperture" (selectable for objective apertures with different contamination levels) to implement the repair action.

[0108] Based on the preset priority of each repair action, a corresponding repair action sequence is obtained. Generally, if only one simulated fault is set, the repair action sequence contains only one repair action. If multiple simulated faults are set, the corresponding repair actions can be combined to generate a repair action sequence according to the priority of each simulated fault. The higher the priority, the earlier the repair action is ordered, indicating that it should be repaired more early to prevent affecting the repair of other simulated faults later.

[0109] Based on real-time component operation, the system determines the user's real-time completion rate of electron microscope operation tasks by using preset standard component operation sequences corresponding to the electron microscope operation tasks and repair action sequences corresponding to simulated faults. For each electron microscope operation task, a corresponding standard component operation sequence is set, which includes the optimal component operations and their corresponding order for that electron microscope operation task.

[0110] When determining the degree of completion, the standard component operation sequence corresponding to the preset electron microscope operation task is determined, as well as the first actions contained in the repair action sequence corresponding to the simulated fault. The first action refers to the component operation contained in the standard component operation sequence and the repair action sequence, and there are usually multiple first actions.

[0111] Based on each real-time component operation, it is determined whether a corresponding second action exists within the first action. The second action is a single action within the first action; that is, it is determined based on the action type of the real-time component operation (e.g., rotating a knob, adjusting the position of an object), without considering specific parameter values, to determine if the real-time component operation has a similar action within the first action. If so, the real-time adjustment value of the operating parameters corresponding to the real-time component operation is compared with the standard adjustment value of the operating parameters corresponding to the second action to obtain the sub-completion level of the real-time component operation. If multiple second actions of the same action type exist within the first action (e.g., all rotating a specific knob), the corresponding second action can be determined based on the action sequence of the real-time component operations and adjacent actions.

[0112] In the repair action sequence, taking the objective aperture as an example, when the simulated fault is objective aperture contamination, if the real-time component operation is to replace the objective aperture, so that the real-time adjustment value of its corresponding operating parameter is adjusted from more than 10% to 0%, which is consistent with the standard adjustment value of the operating parameter corresponding to the second action of 0%, then the sub-completion degree corresponding to the real-time component operation is considered to be 100%.

[0113] In the standard component operation sequence, assuming the real-time component operation is to move the carrier containing the observed target object a distance a, and the standard adjustment value of the corresponding second action's operation parameter is to move the carrier a distance b so that the observed target object can reach the center position of the lens, then the sub-completion degree corresponding to this real-time component operation is min{a / b,b / a}.

[0114] Based on the sub-completion levels corresponding to each real-time component operation and their respective weights, a weighted average is calculated to obtain the user's completion level for the electron microscope operation task. Each first action is pre-assigned a weight, and the sum of the weights of all first actions is 1. Here, only real-time component operations with corresponding second actions are selected, and their corresponding sub-completion levels and weights are weighted and summed to obtain the current real-time completion level. Real-time component operations that are not considered first actions are considered invalid operations and are not included in the completion level calculation.

[0115] Based on the real-time progress trend, users are given corresponding prompts. This real-time progress can be shown or hidden from the user based on their needs. If the progress remains unchanged or even decreases after a certain period, it is assumed that the user is encountering difficulties, and a more detailed prompt can be given. The prompt can also be upgraded accordingly. The initial prompt is a simple prompt, such as "Try checking the aperture," and if the user still does not improve the progress after a certain period, a more complex prompt can be given, such as showing the user the correct operation process.

[0116] Specifically, if the real-time completion rate does not reach the preset growth rate after a preset time period, the trend of the real-time completion rate is determined. The preset time period can be set based on requirements, for example, 30-60 seconds. The real-time completion rate is the completion rate at the moment the preset time period is reached, or at the moment the user clicks confirm.

[0117] The trend can be determined using relevant indicators. For example, the real-time completion rate can be collected over a recent period, tracking changes in the real-time completion rate after each user performs a real-time component operation, resulting in a time-series real-time completion rate sequence. A trend line is then obtained by linearly fitting this sequence. If the slope of the trend line is positive and higher than a preset slope value, it indicates an upward trend; if the slope is negative and its absolute value is higher than a preset slope value, it indicates a downward trend; if the absolute value of the trend line's slope is lower than the preset slope value and the slope is near 0, it may indicate an oscillating or stable trend. Further, the standard deviation of the real-time completion rate in the sequence can be determined. A large standard deviation indicates an oscillating trend, while a small standard deviation indicates a stable trend.

[0118] If the trend is either oscillating or upward, it indicates that the user's current action is effective, but the effect is slow, causing the upward trend to fall short of the preset growth level, or the action is too drastic, resulting in an oscillating trend. In this case, determine the highest sub-completion level corresponding to the current real-time component action; that is, determine the highest sub-completion level the user can achieve while continuously executing this real-time component action.

[0119] If the highest sub-completion level is higher than a preset threshold, it indicates that the user has actually achieved a relatively optimal adjustment effect in the process. Based on the real-time adjustment value corresponding to the highest sub-completion level, a prompt is generated for the user, such as informing the user that in the current real-time component operation, the corresponding real-time adjustment value has already achieved a relatively optimal adjustment effect, and no further adjustment is needed. For the other real-time component operations with the lowest sub-completion levels, prompts are also generated for the user, such as reminding the user which real-time component operations still have low sub-completion levels and require adjustment.

[0120] If the completion level of the highest sub-component is lower than the preset threshold, it means that the current real-time component operation still needs to be adjusted. For the current real-time component operation, a prompt is generated for the user, such as prompting the user to adjust the direction or the adjustment range.

[0121] If the trend is downward or stable, the adjustment performed by the user is considered to be a reverse adjustment or an invalid adjustment. For the current real-time component operation, a prompt is generated for the user, such as prompting the user to adjust the direction or magnitude of the adjustment.

[0122] Furthermore, if, based on the user's historical records, it is determined that the user's real-time completion rate in multiple electron microscope operation tasks all meet preset requirements (for example, it can be set to an increasing trend of speed higher than the preset speed), and the user has not performed any preset prohibited component operations, then the user is considered to have a high level of knowledge.

[0123] At this point, based on the Internet of Things (IoT), the system connects users to actual electron microscope equipment; and based on real-time component operations, it allows users to operate the actual electron microscope equipment. For example, by pre-connecting the actual electron microscope equipment to the system using IoT technology, real-time component operations performed by users on model components in the system can be directly fed back to the actual electron microscope equipment, automatically executing the corresponding operations, thereby achieving virtual-real linkage and providing users with a more realistic training scenario.

[0124] like Figure 4 As shown in the illustration, this application also provides a virtual training device for electron microscopy based on fault simulation, comprising:

[0125] At least one processor; and,

[0126] A memory communicatively connected to the at least one processor; wherein,

[0127] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the fault simulation-based electron microscopy virtual training method as described in any of the above embodiments.

[0128] This application also provides a non-volatile computer storage medium storing computer-executable instructions configured to implement the fault simulation-based electron microscope virtual training method as described in any of the above embodiments.

[0129] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0130] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0131] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are exhaustively listed. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] For those skilled in the art, various modifications and improvements can be made without departing from the concept of the present invention, and these modifications and improvements are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the appended claims.

Claims

1. A virtual training method for electron microscopy based on fault simulation, characterized in that, include: In a virtual environment, the already constructed virtual electron microscope model is identified; The virtual electron microscope model includes multiple model components; Generate and publish corresponding electron microscope operation tasks for users currently undergoing virtual electron microscope training; Based on a pre-built electron microscope fault knowledge graph, simulated faults for the electron microscope operation task are generated through a large language model, and the virtual electron microscope model is adjusted according to the simulated faults. Real-time component operations are collected during the process of the user performing the electron microscope operation task in the virtual environment; Based on the operation of the real-time component, a real-time observation image corresponding to the observed target object is generated.

2. The method according to claim 1, characterized in that, The process of constructing the electron microscope fault knowledge graph includes: Acquire knowledge related to electron microscope malfunctions; Construct a meta-model corresponding to the electron microscope fault knowledge graph, and define the types and attributes of nodes and edges in the meta-model; Natural language processing is performed on the knowledge content to obtain text content, and the corresponding entities and relationships are extracted from the text content. Based on the entities and relationships, the nodes and edges in the meta-model are filled to generate an electron microscopy fault knowledge graph.

3. The method according to claim 1, characterized in that, Adjusting the virtual electron microscope model based on the simulated fault specifically includes: Generate a corresponding digital twin data model for the virtual electron microscope model; The conventional parameter ranges of the component parameters of each model component in the virtual electron microscope model are defined using the digital twin data model. Based on the simulated fault, the parameter values ​​of the component are adjusted so that they do not conform to the normal parameter range.

4. The method according to claim 3, characterized in that, After adjusting the virtual electron microscope model based on the simulated fault, the method further includes: Based on the adjustment of the component parameters, the corresponding reverse adjustment is taken as the corresponding repair action; Based on the preset priority of each repair action, the corresponding repair action sequence is obtained; Based on the real-time component operation, the user's real-time completion level of the electron microscope operation task is determined by the preset standard component operation sequence corresponding to the electron microscope operation task and the repair action sequence corresponding to the simulated fault. Based on the changing trend of the real-time completion rate, the user is given corresponding prompts.

5. The method according to claim 4, characterized in that, Based on the real-time component operations, the user's real-time completion rate of the electron microscope operation task is determined through a preset standard component operation sequence corresponding to the electron microscope operation task and a repair action sequence corresponding to the simulated fault. Specifically, this includes: Determine the standard component operation sequence corresponding to the preset electron microscope operation task, and the first actions included in the repair action sequence corresponding to the simulated fault; Based on the operation of each real-time component, determine whether there is a corresponding second action in the first action; If it exists, the real-time adjustment value of the running parameters corresponding to the real-time component operation is compared with the standard adjustment value of the running parameters corresponding to the second action to obtain the sub-completion degree corresponding to the real-time component operation. Based on the sub-completion degree corresponding to each real-time component operation and their respective weights, the real-time completion degree of the user's electron microscope operation task is obtained by weighting.

6. The method according to claim 3, characterized in that, Based on the operation of the real-time component, a real-time observation image corresponding to the observed target object is generated, specifically including: Generate the corresponding original observation image based on the observed target object; For the model components contained in the virtual electron microscope model, according to their corresponding component parameters and operating parameters, the corresponding rendering method is selected to render the original observation image to obtain the real-time observation image; Based on the real-time component operation, the rendering level of the corresponding rendering method is adjusted according to the adjustment of the component parameters and / or running parameters of the model component, so as to regenerate the real-time observation image.

7. The method according to claim 5, characterized in that, Based on the changing trend of the real-time completion rate, the user will be given corresponding prompts, specifically including: If the real-time completion rate does not reach the preset growth rate after a preset time period, the trend of the real-time completion rate is determined. If the trend of change is an oscillating trend or an upward trend, then determine the highest sub-completion level corresponding to the current real-time component operation; If the highest sub-completion level is higher than a preset threshold, then a prompt is generated for the user based on the real-time adjustment value corresponding to the highest sub-completion level, and prompts are also generated for the user based on the operation of several other real-time components with the lowest sub-completion levels. If the highest completion level is lower than a preset threshold, a prompt will be generated for the user based on the current real-time component operation. If the trend is downward, then a prompt is generated for the user based on the current real-time component operation.

8. The method according to claim 4, characterized in that, The method further includes: If, based on the user's historical records, it is determined that the user's real-time completion rate in multiple electron microscope operation tasks meets the preset requirements and that the user has not performed any preset prohibited component operations, then, based on the Internet of Things, the user is associated with an actual electron microscope device. The actual electron microscope equipment is operated based on the real-time component operation.

9. A virtual training device for electron microscopy based on fault simulation, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the fault simulation-based electron microscopy virtual training method as described in any one of claims 1 to 8.

10. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to implement the electron microscope virtual training method based on fault simulation as described in any one of claims 1 to 8.

Images