Large language model for providing user assistance to a user of a microscope

Abstract

The present disclosure relates to large language models for providing user assistance to users of microscopes. The present disclosure relates generally to techniques for providing user assistance when studying samples using a microscope. According to the disclosed techniques, a trained large language model is used to generate user assistance information. The disclosed techniques enable user assistance information to be provided that is tailored to a graphical user interface used by a user to interact with a microscope.

Description

Technical Field

[0001] Various examples of this disclosure relate to providing user assistance to users of a microscope when acquiring images of samples using a microscope. Background Technology

[0002] Microscopes are used in a variety of applications, such as imaging semiconductor samples, imaging biological samples such as cells or tissues, inspecting composite materials, online testing on production lines, end-of-line testing on production lines, and in academia and manufacturing.

[0003] There are also a wide variety of microscope types, including but not limited to optical microscopes, particle microscopes, and atomic force microscopes. Even within a single type of microscope, there are multiple subtypes. For example, there are many different types of optical microscopes, using different imaging modalities, different illumination configurations, different filters in the detection path, etc. Phase or amplitude imaging is possible. Fluorescence imaging or light sheet imaging are other possible approaches. Sometimes a single microscope can be controlled to provide many different such imaging modalities, for example, by activating or deactivating the use of certain optical filters in the imaging path, by using a certain illumination configuration, and / or by using some post-processing of the image.

[0004] This diversity among use cases, coupled with the complexity of the hardware / software, makes it difficult to provide customized user support to microscope users. For example, microscope manufacturers may face the task of providing extremely comprehensive user manuals that cover all different types of use cases and microscope hardware / software configurations. On the other hand, such complex manuals make it difficult for users to find the specific information they are interested in. Summary of the Invention

[0005] Therefore, advanced user-assistance technologies are needed when studying samples using a microscope. Specifically, these technologies need to be tailored to allow users to retrieve specific information useful for the current task in a short time.

[0006] This requirement is met through the features of the independent claims. The features of the dependent claims define the embodiments.

[0007] Techniques are disclosed for generating contextual data for the current settings of a microscope, either statically or dynamically, and organizing it into a standardized, machine-understandable format. A large language model (LLM) (e.g., a chat assistant) can then leverage the contextual data to provide accurate user guidance to the microscope user, such as instructing which graphical user interface (GUI) elements should be used or configured in a certain way to achieve the desired results.

[0008] A method for providing user assistance using a computing device is disclosed. The user assistance is provided when studying samples using a microscope. The method includes providing a GUI to the user. The GUI is adapted to depict a microscopic image of the sample acquired using the microscope. The GUI is alternatively or additionally adapted to configure the imaging process associated with the microscopic image. The method also includes generating contextual data for the microscope based on its current settings. The method further includes triggering inference from a trained LLM based on the contextual data to generate user assistance information. The method also includes providing the user assistance information to the user.

[0009] A GUI can refer to a visual interface that allows users to interact with a microscope through graphical elements such as icons, menus, and windows. A GUI can display the microscopic images acquired by the microscope and provide interface elements as controls for adjusting imaging parameters or configurations. For example, a GUI may include tools for setting exposure times, selecting filters, or activating specific imaging modalities. Alternatively, or outside of such hardware setups, interface elements may enable the setting of one or more software post-processing steps applied to the acquired images.

[0010] The microscope can be an optical microscope or a particle microscope. It can also be a scanning electron microscope. Transmission microscopy or reflection microscopy can be used. The techniques disclosed herein can be applied to various types and kinds of equipment that provide microscopic images.

[0011] Microscopic images are visual representations of samples captured using a microscope. Such images can be generated using various imaging techniques, such as bright-field microscopy, fluorescence microscopy, phase-contrast microscopy, or other modalities, depending on the type and configuration of the microscope. Microscopic images can be displayed in real-time during acquisition or stored for subsequent analysis.

[0012] Contextual data refers to information describing the current state of the microscope and the imaging environment. This data can include settings such as magnification level, illumination intensity, filter configuration, imaging mode selection, and other parameters related to microscope operation. During imaging, contextual data is used to infer user needs or provide customized assistance.

[0013] LLMs can include sequence-to-sequence deep neural networks employing a transformer architecture. The transformer architecture can include one or more multi-head self-attention layers. These layers allow joint attention to information from different representational subspaces at different locations, thereby capturing complex contextual relationships in the contextual data. LLMs may have been trained using unsupervised training methods such as masked language modeling or next-sentence prediction. This allows LLMs to learn general language patterns and relationships that are not specific to a particular domain or task. Therefore, LLMs can be considered domain-agnostic, meaning they are not limited to specific applications or use cases. In some examples, LLMs may have been trained using large-scale textual data corpora (e.g., books, articles, and websites). This training data may not be limited to information related to microscopy or microscopy but can be general language patterns and relationships applicable to a wide range of tasks. LLMs may have a large number of parameters, such as tens of millions or even hundreds of millions. These parameters are learned during training and allow LLMs to capture complex patterns and relationships in the contextual data.

[0014] By triggering LLM inference to generate user assistance information that takes into account contextual data, user assistance information can be provided in a state-aware manner tailored to the specific situations the user faces when interacting with the user interface. Therefore, more customized user assistance information can be provided.

[0015] As an example, contextual data can indicate the current hardware settings of the microscope. The current hardware settings of the microscope may include information about, for example, the illumination used, the activated or deactivated filters, the illumination wavelength, the objective lens, the magnification, etc.

[0016] Microscope hardware settings refer to the configuration or adjustments made to the physical components of the microscope, which affect the imaging process and the resulting microscopic images. Such settings can include, but are not limited to, adjustments to objectives, illumination sources, filters, polarizers, and other optical elements. Microscope hardware settings can affect various aspects of the imaging process, such as resolution, contrast, brightness, and color accuracy. For example, adjusting the focal length of an objective or changing its magnification can be considered a change in hardware settings. Similarly, switching between different illumination sources (e.g., bright field, dark field, or fluorescence) can also be considered a modification of hardware settings. The specific settings used for imaging can depend on various factors, including the type of sample being observed, the required resolution level, and the intended application.

[0017] Contextual data can indicate the microscope's current image acquisition settings. Image acquisition settings can include parameters such as camera sensor parameters, including exposure time, frame rate, and sensitivity.

[0018] As understood from the above, the current hardware settings and / or current image acquisition settings relate to the current state of the microscope and describe properties that affect the appearance of the acquired microscopic images. For example, depending on the hardware settings and / or image acquisition settings, different structures may appear in the resulting microscopic images with different contrasts. For example, the current hardware settings and / or current image acquisition settings may affect the current imaging mode, such as bright-field imaging, dark-field imaging, phase-contrast imaging, fluorescence imaging, and so on, to name a few.

[0019] Alternatively, in addition to this information that directly affects the appearance of the micrograph, which may be included in the context data, the context data may also include other or additional information that does not directly affect the appearance of the micrograph.

[0020] For example, contextual data may include the microscope's log file. Such a log file may include status monitoring information detected by one or more control units of the microscope and / or one or more sensors of the microscope.

[0021] A microscope log file can refer to a data record or archive that documents various events, activities, or changes related to the operation and performance of the microscope over time. Such a log can include information about instrument settings, user interactions, image acquisition parameters, system errors or warnings, maintenance records, and other relevant details. In the context of providing user assistance, log files serve as a resource for understanding historical patterns and trends in microscope usage and performance. By analyzing data from log files, an LLM can gain insights into how the microscope has been used in the past, including commonly used settings, common errors or problems, and other relevant information that can inform its user assistance recommendations. Log files can be stored locally on the microscope itself, or they can be transferred to a remote server for centralized storage and analysis. In some cases, multiple microscopes can contribute data to a single shared log file, enabling broader insights into patterns and trends across different instruments and users. Log files may also include information about software updates, firmware revisions, and other changes to the microscope's operating system or applications over time. This information helps the LLM understand how changes to microscope configuration can affect its performance and behavior, leading to more accurate and evidence-based user assistance recommendations.

[0022] Contextual data may include error messages output via the microscope's system software. These error messages may relate to hardware errors, such as mechanical or electrical errors in one or more components of the microscope. Error messages may also relate to software errors, such as missing access permissions or control task failures.

[0023] Contextual data may include sensor readings from the microscope's sensors. For example, the microscope's temperature or humidity sensors may provide repetitive sensor readings at a certain sampling rate, and the log file may include this information.

[0024] Providing information relevant to the overall operation of the microscope, but not strictly tied to the imaging process, allows for the revelation of underlying issues or problems manifested in the appearance of the microscopic images. For example, degraded image quality might be the root cause of some hardware or software defect in the microscope. To perform such root cause analysis, it is helpful to obtain additional contextual data characterizing the overall state of the microscope's operation, beyond the optical path and image processing. For instance, if a user is experiencing difficulties with image quality, an LLM can analyze various information, including sensor readings, error messages, and log file data, to determine if the root cause is a malfunction in the temperature control system. This understanding can then inform the LLM's recommendations for troubleshooting and resolving the problem. Therefore, in some cases, identifying the root cause of a problem may involve analyzing data from multiple sources and using techniques such as correlation analysis or causal inference.

[0025] Contextual data can indicate the current software configuration of one or more image processing software modules of the microscope.

[0026] For example, a microscope may include one or more cameras that require one or more raw images. These raw images can then be post-processed using one or more image processing software modules of the microscope. For example, multiple raw images can be acquired using different illumination settings (e.g., different illumination directions). The raw images can then be combined using digital post-processing in the image processing software modules to obtain a microscopic image. The resulting microscopic image may have digital phase difference. Image post-processing techniques are conceivable, such as denoising, anti-aliasing, etc., to name just a few. Certain digital filters can be applied.

[0027] By providing the LLM with this information about the image post-processing currently being used, the LLM can also provide customized user assistance for one or more settings of such image processing software modules.

[0028] Contextual data can indicate the current settings of the GUI. The GUI can be user-adjustable. For example, certain interface elements can be rearranged, or fully activated or deactivated. For example, the toolbar of an interface element can be hidden or expanded. Contextual data can indicate which interface elements are visible, which are hidden, or where a visible interface element is placed / positioned. Contextual data can indicate the arrangement of different interface elements on the screen. This allows LLM to provide simplified user guidance, for example, indicating to the user the location of a specific interface element on the screen that the user should click or modify. Especially for scenarios where the GUI includes a large number of interface elements, and these elements are configurable in terms of their appearance and / or visibility, such technology can greatly improve the usability of user assistance provided to the user.

[0029] Contextual data can indicate user interactions with the GUI. For example, contextual data can include a list of interface elements the user has recently interacted with. Sequences of interface elements can also be indicated. An "action history" listing certain actions taken by the user can be included in the contextual data. This allows the LLM to understand which actions the user previously attempted while controlling the microscope. For example, this can enable the LLM to incorporate further user assistance information and instructions to these previously attempted actions, thereby providing user-specific assistance tailored to the user's prior activity.

[0030] More generally, contextual data can provide information associated with a specific user who triggered the request for user assistance. This information may not be related to, for example, a specific query entered by the user that identifies the user assistance requested; rather, the user-related information may be related to general attributes of the user beyond a specific query. Contextual data can be user-specific. Contextual data can indicate information associated with the user of the GUI. Such techniques enable the provision of user-specific user assistance, for example, user-specific assistance tailored to the user's specific skill level and / or user access permissions.

[0031] Contextual data can indicate user-related information, including a unique user identifier. This unique user identifier can be associated with a user repository of users authorized to use the GUI. Therefore, a user identifier can be a local user identifier associated with a specific microscope or a set of microscopes in the local environment. Alternatively or additionally, a global user identifier can also be used, which is associated, for example, with a global user repository of users registered with the manufacturer of the GUI and / or the microscope. Typically, this global user repository can be stored on an online server, allowing it to be accessed by multiple different instances of the GUI deployed on different sites.

[0032] Contextual data can indicate a user's skill level. For example, a user's skill level can indicate the length of time they have interacted with the GUI. It can also be a categorization metric that groups user experience into a predefined set of skill level categories. Furthermore, it can indicate whether a user has previously performed a task via the GUI or not; this can be tracked through the GUI's background monitoring processes. This information about the user's skill level can be linked to the unique user identifier discussed above. By providing the user's skill level, the complexity of user assistance can be customized accordingly. For instance, certain relatively complex user guidance information may not be provided to relatively inexperienced users. Similarly, certain interaction patterns or instructions may not be provided to certain users. Alternatively, in addition to customizing the complexity of the user assistance itself, it is conceivable to customize the presentation of user assistance based on the user's skill level. For example, while a relatively detailed and advanced set of instructions for using the GUI and / or manipulating the microscope may be sufficient for a highly experienced user to execute these instructions, a more detailed, step-by-step set of instructions may be needed for a less experienced user to follow them. In this way, the amount of information or the granularity of user assistance can be adjusted according to the user's skill level.

[0033] Contextual data can indicate anomalous user interaction patterns with the GUI. An anomalous user interaction pattern can refer to a pattern that behaves as an outlier compared to a reference set of user interaction patterns. It can also refer to a pattern that has not been previously seen in the reference set. This reference set can be user-specific for a particular user; in other words, the anomalous user interaction pattern can be anomalous within that particular user's scope. Alternatively, this reference set can be globally valid for a group of users, making the anomalous user interaction pattern anomalous not only for a specific user but also for multiple other users. Various techniques exist for detecting such anomalous user interaction patterns. For example, the actual user interaction pattern can be compared to a reference catalog of user interaction patterns. A heuristic distance metric can be used. If the actual user interaction pattern is not found in the reference catalog, it can be flagged as anomalous. Machine learning anomaly detectors, such as autoencoders, can be used. Here, a sequence of encoded user interactions is fed into the encoder of the autoencoder. The encoder then determines the latent representation at the bottleneck layer and subsequently attempts to reconstruct the feature vector using a decoder. The autoencoder has been trained using a training dataset that includes a reference set of user interaction patterns. For anomalous user interaction patterns, the reconstructed feature vectors will show significant deviations from the original feature vectors, allowing the associated user interaction patterns to be detected as anomalous. This technique is based on the finding that anomalous user interaction patterns can be associated with certain problematic or flawed system settings in the microscope. Users may have activated or deactivated settings that are normally deactivated or activated. Users may have performed unexpected activities that are outside the scope of typical user manuals.

[0034] Contextual data can indicate user permission levels. User permission levels adjust which microscope and / or post-processing settings a user can change, or more generally, which GUI settings can be altered. This is particularly important for relatively complex microscopes that are at risk of damage, where user permission levels can be restricted for inexperienced users. Certain settings may be disallowed for inexperienced users. By providing this information to the LLM (Limited Management System), the LLM can ensure that user assistance is not provided that would require the user to perform actions outside the permitted set of actions, thus avoiding user confusion.

[0035] The contextual data has been disclosed above in relation to scenarios such as microscopes, GUIs, software executed based on image data acquired from microscopes, and / or users. Alternatively or additionally, contextual data can be related to the sample being studied. For example, contextual data can indicate the sample type, sample processing history, sample size, and / or structural dimensions of the sample's structures. A textual description of the microscopic images depicting the sample can be provided. Different types of samples often require different imaging strategies. For example, some biological structures are barely visible when using amplitude imaging. For such structures (e.g., cells), phase-contrast imaging may be beneficial. Furthermore, some structures may not be depicted, if appropriate, when selecting the field of view and / or magnification, because they are below resolution or because they are too large to be perceived in a limited field of view. All such information can be considered by the LLM and, if information about the sample is provided, can assist the user.

[0036] User assistance information (MAI) generally refers to any information provided to microscope users to assist them in operating the microscope and / or interpreting images acquired using the microscope. This information may include, but is not limited to, guidance on how to adjust microscope settings, how to select imaging modalities, how to use post-processing software, or how to interpret results obtained from image analysis. In this context, MAI can be provided in various forms, such as text-based instructions, visual aids like charts or images, or even interactive tools guiding users through specific procedures. The type and format of MAI can depend on the user's specific needs, the complexity of the microscope, and the level of expertise required to operate it. For example, in some cases, MAI may be provided as a series of step-by-step instructions guiding the user through a specific imaging protocol. In other cases, MAI may be more general, providing an overview of the different imaging techniques or modalities available on the microscope and allowing the user to select the most appropriate option for their specific application. Furthermore, MAI may include troubleshooting guidance, which provides instructions on how to resolve common problems or errors that may occur during microscope operation. This guidance can be particularly useful for users new to microscopy or with limited experience with a particular type of microscope.

[0037] There are various options available for implementing user assistance information. For example, tutorial-guided workflows can be provided for user interaction. Tutorial / guided workflows may involve user interaction with a GUI. For instance, users may be provided with a step-by-step set of instructions on which interface elements to activate or manipulate and / or what settings are needed to achieve a certain goal. This goal may be queried by the user or typically defined by logic that determines the need for user assistance. As mentioned above, user assistance information (and even more generally, the way user assistance information is provided) can depend on the user's skill level.

[0038] There are various options available for how to provide user assistance information to users. For example, user assistance information can be provided via a GUI. For instance, user assistance information may include graphical highlighting of at least one interface element of the GUI. For example, graphical highlighting can be provided for those interface elements that will be clicked, activated, or otherwise manipulated by the user. Alternatively, a tooltip pop-up associated with a particular interface element can be provided. Such a tooltip pop-up may include a text description. The corresponding text can be generated by an LLM. The text description may include, for example, the functionality of the interface element.

[0039] Such a technique can be achieved by providing certain text through an LLM (Linux Virtual Machine), which can be parsed and post-processed by a program associated with the GUI. The format of this text can be specified in the prompt. For example, an LLM can provide a table indicating a specific sequence of interface elements that need to be highlighted. This information can then be input into the program associated with the GUI to provide that highlighting or other emphasis on certain interface elements. By providing information directly within the GUI, particularly intuitive and easy-to-follow user assistance can be provided.

[0040] User assistance information can also be provided outside the GUI. For example, it can be provided on a separate computing device (e.g., a separate handheld user device, such as a smartphone). This enables backward compatibility with different types of GUIs. No specific software interface is required for interfacing between the LLM output and the GUI. This allows for the decoupling of the computing device controlling the microscope and executing the GUI from external data sources.

[0041] As a general rule, the provision of user assistance can be queried by the user. In other words, the user can provide a corresponding manual query for user assistance. Then, based on the corresponding user-triggered request, LLM inference is triggered. Alternatively, or in addition to the manual and explicit triggers for generating user assistance, semi-automatic or fully automatic triggering can be envisioned. Implicit triggering events can be detected. LLM inference can more generally be triggered by at least one triggering event. For example, user interaction with the GUI can be monitored to determine whether at least one triggering event is met. For example, a background process can monitor user interaction with one or more interface elements of the GUI. The background process can then automatically determine whether the user needs user assistance; if so, LLM inference can be triggered. To give a concrete example, abnormal user interaction patterns can be detected, and at least one triggering event can include abnormal user interaction patterns. Techniques for detecting abnormal user interaction patterns have been explained above; although these techniques have been discussed above in conjunction with contextual data, they can also be readily applied to determine whether triggers for LLM inference are met. This technique is based on the understanding that certain abnormal user situations may easily lead to user errors, and therefore, users may particularly need user assistance when acting outside of standard patterns. At the same time, by identifying abnormal user situations, we can avoid having to provide users with repetitive user assistance too frequently, which would otherwise risk causing user resentment.

[0042] Monitoring user interaction may include detecting changes in the settings of GUI elements. For example, when a user changes an interface element, user assistance tailored for that specific change in setting can be queried at the LLM. User interaction may include changes in the visibility of GUI elements. For example, a user may activate the visibility of a given interface element; in this case, user assistance for that specific interface element can be queried at the LLM.

[0043] Monitoring user interactions may include monitoring changes to the GUI software version. For example, upon detecting a change in the GUI software version, user assistance may be queried. For instance, user assistance tailored to the differences between the updated GUI software version and the previously installed GUI software version may be queried.

[0044] This method can also include determining prompts for the LLM based on user queries. For example, a user may input a query using, for example, text input or voice-to-text input. The prompt can encapsulate the user query, for example, along with contextual data. The user query can also be preprocessed, and then the preprocessed user query can be included in the prompt. The prompt typically includes such a user query along with contextual data. It is not necessary to construct prompts based on user queries in all cases. For example, techniques for detecting trigger criteria for obtaining user assistance information based on monitoring user interactions with the GUI have been disclosed above. In this scenario, providing the LLM with a descriptor indicating the user interaction along with contextual data in the prompt may be sufficient to provide user assistance information. In this and other scenarios, explicit formulation of the user query is not required.

[0045] According to the example, the GUI is executed within a packaged operating system. For example, a computing device including a processor and memory storing program code for executing the GUI can be located on a local area network disconnected from the internet. In this scenario, the LLM can be executed locally on that computing device. In another scenario, the LLM can be deployed in the cloud, accessible via the internet. The triggering of inference for the LLM can then be initiated by one or more processes executing on a computing device other than the one providing the GUI. This other computing device, which then triggers the inference of the cloud-deployed LLM, can be connected to the internet, allowing the corresponding commands for the LLM inference to be transmitted via the internet. By encapsulating the programs used for microscope control and the GUI with the programs required to trigger the LLM inference, the security of the encapsulated operating system for controlling the microscope and providing the GUI is improved. For example, data leakage can be effectively prevented.

[0046] The method may also include: performing data transfer of at least a portion of context data between an operating system (i.e., the operating system providing the GUI) and another operating system (i.e., the operating system that triggers the LLM inference). This data transfer can be secure. Secure data transfer means that by imposing appropriate restrictions and constraints on data transfer, it can be ensured that only the intended information, i.e., at least a portion of the context data, is transmitted; while other information (particularly microscopic images including information about the sample, etc.) cannot be transmitted using data transfer. For example, data transfer can be via one or more optical codes, such as QR codes. Data transfer can be via near-field wireless transmission and / or self-organizing wireless transmission. This type of data transfer may have inherently limited data transfer rates, making it impossible to carry payload data (particularly microscopic images) within a limited duration. Data transfer may require manual user interaction, for example, to approve certain information to be shared. Thus, the user retains control over information shared outside the encapsulated operating system.

[0047] The method may also include: detecting that the microscope is not connected to the internet, and then, in response to detecting that the microscope is not connected to the internet, locally storing context data on the microscope. The method may also include: transmitting the stored context data to an assistive device, for example, using optical code scanning technology, Universal Serial Bus (USB) connection, Near Field Communication (NFC) transmission, Bluetooth® transmission, or WLAN connection. The assistive device then triggers LLM inference. The transmission of the stored context data can be performed using dynamic QR codes or video QR codes.

[0048] Inference that triggers LLM may include providing appropriate prompts to the computing device performing the LLM. These prompts may include at least a portion of the context data, for example, in a compressed format. For instance, a latent representation of the context data may be determined based on a pre-trained encoder. This latent representation may reside in a latent space of machine learning that can be interpreted by LLM.

[0049] Alternatively, contextual data can request the LLM to retrieve information from one or more encoded and structured knowledge elements. Such encoded and structured knowledge elements may include, for example, microscope manuals, GUI manuals, performance specification databases of one or more optical components of the microscope, microscope log files, technical white papers, and / or microscope error databases. Retrieval Augmented Generation (RAG) can be employed. RAG refers to a method by which the LLM uses external information or data and its own training to generate responses. In the context of this disclosure, RAG may involve retrieving relevant information from a document database based on contextual data. This method improves the accuracy and relevance of the user assistance provided because it leverages both the general knowledge of the LLM and specific contextual information.

[0050] A performance specification database for one or more optical elements can be a collection of data describing the characteristics and capabilities of the optical elements used in a microscope. Such databases are typically compiled by the manufacturers of the optical elements and provide detailed information on various parameters, such as numerical aperture, magnification, resolution, and spectral transmission range. This information helps in understanding how different optical elements contribute to the overall imaging performance of the microscope.

[0051] A technical white paper can refer to a scientific or technical document that provides an in-depth analysis or explanation of a specific topic (such as a new technology, method, or process) related to microscopy. These documents are typically written by experts in the field and provide detailed information on the theoretical aspects, experimental results, and practical applications of various microscopic imaging modalities. In the context of providing user assistance, technical white papers can serve as a valuable resource for understanding complex concepts and providing accurate guidance to users.

[0052] Microscope or GUI manuals can be documents that provide comprehensive instructions and explanations on how to use the microscope or interact with the GUI. Such manuals typically cover topics such as setup and calibration procedures, operating modes, troubleshooting guides, and maintenance recommendations. These documents can provide valuable context for understanding user behavior and intent when interacting with the microscope.

[0053] According to the example, the method may further include providing a user feedback mechanism to add user input to a user feedback database. User input may relate to the usefulness and / or relevance of the user assistance information. This allows users to evaluate or measure the quality of the user assistance information. Further contextual data can then be generated based on this user feedback database. For example, certain user assistance information previously flagged as particularly useful by a user can be reused in another instance by another user who needs the user assistance information. On the other hand, negative samples where user assistance information was not found to be particularly helpful to the user can be used as negative examples, allowing the LLM to provide alternative solutions.

[0054] It should be understood that the features mentioned above and described below can be used not only in the indicated combinations, but also in other combinations or individually, without departing from the scope of the invention. Attached Figure Description

[0055] Figure 1 The system, including microscopes and computing devices, is illustrated schematically according to various examples.

[0056] Figure 2 The illustrations schematically depict systems including microscopes, computing devices running packaged operating systems, and other computing devices connected to the Internet, based on various examples.

[0057] Figure 3 It is a flowchart based on various examples of methods.

[0058] Figure 4 The GUI is illustrated schematically based on various examples. Detailed Implementation

[0059] Some examples in this disclosure generally provide multiple circuits or other electrical devices. All references to circuits and other electrical devices and the functions provided by each are not intended to be limited to what is illustrated and described herein. While specific labels may be assigned to the various circuits or other electrical devices disclosed, such labels are not intended to limit the scope of operation of the circuits and other electrical devices. Such circuits and other electrical devices may be combined and / or separated from each other in any way, depending on the desired specific type of electrical implementation. It will be understood that any circuit or other electrical device disclosed herein may include any number of microcontrollers, graphics processing units (GPUs), tensor processing units (TPUs), integrated circuits (e.g., application-specific integrated circuits or field-programmable gate arrays (FPGAs)), memory devices (e.g., FLASH, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or other suitable variations thereof), and software that cooperates with each other to perform one or more of the operations disclosed herein. Furthermore, any one or more of the electrical devices may be configured to execute program code implemented in a non-transitory computer-readable medium and programmed to perform any number of the disclosed functions.

[0060] In the following description, embodiments of the invention will be illustrated in detail with reference to the accompanying drawings. It will be understood that the following description of the embodiments should not be construed as limiting. The scope of the invention is not intended to be limited by the embodiments or drawings described herein, which are for illustrative purposes only.

[0061] The accompanying drawings are intended to be schematic representations, and the elements shown in the drawings are not necessarily shown to scale. Rather, the various elements are shown so that their function and general purpose will become apparent to those skilled in the art. Any connection or coupling between functional blocks, devices, components, or other physical or functional units shown in the drawings or described herein may also be achieved through indirect connections or couplings. Coupling between components may also be established via wireless connections. Functional blocks may be implemented in hardware, firmware, software, or a combination thereof.

[0062] Based on various examples, LLM is used to provide user assistance when studying samples using a microscope. For this purpose, LLM is prompted with contextual data generated based on the current microscope settings.

[0063] The techniques disclosed herein enable the realization of user assistance by sensing specific states of the microscope and / or by allowing the user to control the microscope via a GUI. Furthermore, the techniques disclosed herein enable user assistance in scenarios where the microscope is in a protected environment that is enclosed and not connected to the internet.

[0064] The disclosed technology can combine configuration data, the microscope's current status, and / or data from the microscope control software as additional context for LLM. This comprehensive contextual information can be provided along with any user query to improve searching within the document database. By potentially narrowing down relevant documents and improving the quality of responses, this method ensures that user assistance is accurate and effective. Furthermore, the disclosed technology enables offline-to-online data transfer: in scenarios where the microscope lacks internet connectivity but requires a cloud-based solution, this disclosure proposes local storage of configuration data, the microscope's current status information, and control software details. This stored data can then be transferred to an internet-connected assistive device via various methods. For example, QR code transmission can be used, where the data can be encrypted into a QR code. The user scans the QR code using a smartphone, accesses the cloud solution, uploads data, and interacts with a chatbot or similar AI assistant. For larger datasets, dynamic QR codes (e.g., video or color QR codes) can be used to carry greater data volumes. Alternative transmission methods include USB connectivity, NFC, Bluetooth, or Wi-Fi, providing flexible options based on user preferences and available hardware. By integrating these features, this disclosure ensures that users receive timely and relevant assistance tailored to their specific microscopy tasks, regardless of the system's connectivity status.

[0065] Various techniques are based on the finding that LLM-based co-pilot user assistance can provide step-by-step instructions, including descriptions of the GUI. To provide such instructions, the LLM providing these instructions is prompted in the context of understanding the GUI. In the reference implementation, this is provided by a user manual. Such a manual may be incomplete and inaccurate. Therefore, according to the disclosed techniques, context data may include a machine-readable description of the user interface. This enables providing the LLM with accurate information regarding GUIs that display differences across multiple software versions, and further enables the LLM to dynamically adjust user assistance information based on user settings. According to examples of this disclosure, the generation of context data can occur locally at the computing device executing the GUI for controlling the microscope, ensuring that the context data remains accurate and up-to-date.

[0066] Figure 1A system 105 according to various examples is schematically illustrated. System 105 includes a microscope 130. Microscope 130 may be an optical microscope, for example, configurable for bright-field imaging, dark-field imaging, phase-contrast imaging, fluorescence imaging, etc. Microscope 130 includes control circuitry 131 for controlling imaging hardware 132. The control circuitry may execute system software for the microscope. This system software may generate log files for microscope 130. Microscope 130 may include sensors for status monitoring (e.g., health monitoring). Imaging hardware 132 may include, for example, an illumination module, a sample holding module, objectives, a camera, one or more filters, etc. Some of these components may be electrically powered, such as the sample holding module, for moving and positioning the sample holder and sample attachment data relative to the optical path defined by the objectives. The illumination module may be controlled to activate different illumination modes. The camera may be controlled to achieve different image acquisition settings. These are merely some examples of possible settings for the components of microscope 130 that can be adjusted by control circuitry 131. For this purpose, control circuitry 131 is configured to receive appropriate control data from control interface 125 of computing device 120. The computing device 120 includes a processor 121 and a memory 122. The computing device 120 also includes a human-machine interface (HMI) 123 and a communication interface 124. The processor 121 can load and execute program code stored in the memory 122. While loading and executing the program code, the processor 121 can perform techniques disclosed herein, such as providing a GUI to a user via the HMI 123, controlling the appearance of the GUI, providing control data to the control circuitry 131 of the microscope 130 based on one or more settings of one or more interface elements of the GUI, post-processing raw image data acquired from the microscope 130 (e.g., by filtering, combining, denoising, regularizing, and other operations), and communicating with a database 150 and / or a server 151 (e.g., to trigger LLM inference). The processor 121 can communicate with the Internet 159 via the communication interface 124. For example, certain knowledge elements can be retrieved from the database 150. These knowledge elements can be encapsulated in a hint to the LLM, which is inferred at the server 151. RAG can be used.

[0067] Although Figure 1 The computing device 120 of the system 105 shown is directly connected to the Internet 159, but in other scenarios, the processor 121 of the computing device 120 can execute a packaged operating system. Figure 2 This scenario is illustrated in the image.

[0068] Figure 2 The total corresponds to Figure 1 More specifically, Figure 2 The system 106 shown typically corresponds to Figure 1 The system 105 shown. However, in Figure 2 In this scenario, computing device 120 is not directly connected to the Internet 159. Instead, secure data transfer 148 can be performed between computing device 120 and another computing device 140. This secure data transfer can be achieved, for example, via optical code, NFC, WLAN, or Bluetooth. For this purpose, computing device 140 includes a corresponding communication interface 149, which can communicate with the communication interface 124 of computing device 120 using the secure data transfer 148. Computing device 140 also includes a processor 141, memory 142, HMI 143, and communication interface 144. Communication interface 144 is connected to the Internet 159 and can retrieve knowledge elements from knowledge database 150 and / or trigger LLM inference at server 151.

[0069] Figure 3 This is a flowchart illustrating various methods based on different examples. Optional boxes are shown with dashed lines. Figure 3 The method can be executed by a computing device. More specifically, Figure 3 The method can be executed by the processor when the program code is loaded from memory and executed. For example, Figure 3 The method can be executed on multiple distributed computing devices. For example, Figure 3 At least a portion of the method can be executed by the processor 121 of the computing device 120 (see...) Figure 1 and Figure 2 ); alternative or additional land, Figure 3 At least a portion of the method may be made by the processor 141 of the computing device 140 (see...) Figure 2 )implement. Figure 3 The methods generally involve using a microscope (see...) Figure 1 , Figure 2 When using microscope 130, user assistance is provided to the user. This user assistance is achieved through user assistance information generated at least in part based on the output of the LLM. The LLM can be cloud-deployed, for example, by a device connected to the execution system via the Internet 159. Figure 3 The method is executed on one or more computing devices or one or more servers (e.g., server 151). In some examples, the LLM may also be deployed locally, for example, executing... Figure 3 One or more computing devices can also perform LLM inference in other steps.

[0070] At box 3005, a GUI is provided. The GUI can be accessed via an HMI (e.g., previously integrated). Figure 1 and Figure 2The HMI (Hybrid Management Interface) of the computing device 120 discussed herein is provided. The GUI includes multiple interface elements. Manipulation of these interface elements enables the setting of properties of the image acquisition process at the microscope 130, for example, by controlling one or more hardware components of the imaging hardware 132. Manipulation of these interface elements may alternatively or additionally enable the setting of properties of digital image post-processing performed by the control circuitry 1311 of the microscope 130 and / or the processor 121 of the computing device 120. Figure 4 An example implementation of GUI 400 is shown. GUI 400 includes a toolbar 405 containing multiple interface elements through which image acquisition and / or image post-processing properties can be adjusted. Furthermore, GUI 400 includes an area 410 for displaying microscopic images. Figure 4 In this scenario, the microscopic image is overlaid with certain positioning information, referring here to the cell segmentation mask obtained through post-processing. The GUI 400 also includes a multi-line text form 415, through which users can input user-aided queries. Figure 4 The GUI 400 shown is just one of many possible examples, and various modifications are conceivable. First, the GUI 400 may not include the multi-line text form 415; in some examples, user assistance may be provided not based on explicit user queries, but on monitoring user interactions with the GUI. Even if the text query is obtained from the user, it can be obtained through a separate GUI (e.g., a GUI provided by computing device 140 via HMI 143). Furthermore, the type of microscopic image depicted, as well as the type and number of interface elements, can vary depending on the implementation. In some cases, the GUI 400 itself may be reconfigurable, allowing the number and functionality of interface elements visible in the GUI 400 to be reconfigured by the user.

[0071] Refer again Figure 3 In box 3010, user interactions with the GUI can be monitored to determine whether a triggered event has been met. For example, abnormal user interactions can be detected in box 3010. Changes in the settings of GUI interface elements can be detected. Changes in the visibility of GUI interface elements can be detected. Changes in the GUI software version can be detected.

[0072] Then, at box 3015, it is determined whether a user-assisted trigger exists. For example, determining whether a trigger event exists can be based on monitoring user interaction at box 3010. Alternatively or additionally, it can be determined whether the user has provided a user query to the system for user assistance. For example, it can be obtained as previously mentioned... Figure 4The explicit text user query discussed in the multi-line text form 415. Here, user-assisted triggers obtained from the user can be obtained not only from the specific operating system and computing device of the GUI at execution box 3005; but also from the assistive device (see [link to relevant documentation]). Figure 2 (Computing device 140) User queries serve as user assistance.

[0073] If a triggering event is detected at box 3015, the method continues at box 3020.

[0074] At box 3020, context data is generated. Context data is typically associated with the microscope. Context data can indicate the microscope's current hardware settings. Context data can indicate the microscope's current image acquisition settings. Context data can indicate the microscope's log files. Context data can indicate error messages output by the microscope's system software. Context data can indicate sensor readings from the microscope's sensors.

[0075] Contextual data can indicate the current software configuration of the microscope's image processing module. This image processing module can be provided by the microscope's system software and / or by a computing device attached to the microscope (e.g., previously integrated). Figure 1 and combination Figure 2 The computing device 120 under discussion performs the operation. Context data can indicate the current settings of the GUI provided at box 3005. For example, this could involve an indication of which interface elements are currently visible or hidden in the GUI. Alternatively or additionally, context data can indicate the location information of the interface elements, such as their arrangement in the GUI. All such information can help provide customized user assistance adapted to the specific settings of the microscope and its software, as well as the specific settings of the GUI.

[0076] Contextual data can indicate user interactions with a GUI. For example, it can indicate which specific interface elements a user has activated. The history of user interactions for a particular user with the GUI can also be indicated by contextual data.

[0077] Contextual data can indicate user-related information, such as user identifier, user skill level, unusual user interaction patterns of users interacting with the GUI, and / or user permission level.

[0078] Contextual data can indicate the sample type, sample processing history, sample size, and / or structural dimensions of the sample currently being studied under a microscope.

[0079] As understood from the above, context data can be closely related to the microscope's current state, the GUI, the sample being studied, and / or the microscope's user. Therefore, in some scenarios, context data can be generated at the computing device attached to the microscope. Sometimes, this computing device may be running an operating system in a protected environment. Then, in such scenarios, at box 3025, the context data can be transferred to another computing device.

[0080] At box 3025, the data transfer of at least a portion of the context data can be performed between the operating system executing the GUI and another operating system. This data transfer can be protected in the following sense: it can be restricted to this portion of the context data. For example, it can be encrypted. For example, the data rate throughput can be limited. For example, it may be subject to one or more information constraints that limit the type of information that can be transmitted in the data transfer. For example, it can be encrypted. Manual user interaction may be required, for example, manual user approval. For example, it can be limited in its scope, for example, short-range communication can be employed. For example, reference... Figure 2 Data transmission 148 between computing device 120 and computing device 140 can be performed at box 3025. One or more optical codes can be used to perform the data transmission. Alternatives include near-field wireless transmission and / or self-organizing wireless transmission.

[0081] At box 3030, RAG can be executed. For example, box 3030 may include triggering the retrieval of information from one or more encoded and structured knowledge elements. This may include accessing the appropriate knowledge repository or triggering an LLM to retrieve the information. Example knowledge elements include a microscope manual, a GUI manual, a performance specification database of one or more optical components of the microscope, a technical white paper, a microscope log file, or a microscope error database. If this information is retrieved directly, it can be incorporated into the prompt subsequently generated at box 3035.

[0082] Optionally, a prompt can be generated for the LLM at box 3035. For example, the prompt can be determined based on a user query and / or on alternative information describing the environment in which user assistance is required (e.g., obtained from monitoring at box 3010).

[0083] Box 3035 is optional because in some scenarios, if the user query can be obtained in text form, it can be directly provided to the LLM as a prompt. By specifically generating the prompt, certain knowledge about the operation of the LLM can be considered. For example, the prompt can be generated based on a prompt database that stores certain acceptable prompts. Then, these predefined questions can be modified to some extent based on the current situation.

[0084] At box 3040, inference for the LLM is triggered. The LLM may be cloud-deployed, i.e., executed on a server. In this case, triggering inference for the LLM may include: providing a hint to the server hosting the LLM via the Internet and querying the server to infer the LLM based on that hint. In other examples, the LLM may be locally deployed. In this scenario, box 3040 may include the actual inference for the LLM at the local computing device hardware.

[0085] LLM inference enables the acquisition of user assistance information. Then, at box 3045, user assistance information is provided to the user.

[0086] As a general rule, user assistance information may include text information generated by an LLM. At least a portion of this text information can be converted into audio output using a text-to-speech model. Playback of the corresponding audio waveform can then be triggered.

[0087] In other scenarios, user assistance information may be provided alternatively or additionally via a GUI provided at box 3005, for example, as a graphical highlight of an interface element of the GUI or as a tooltip pop-up associated with the interface element. Such a tooltip pop-up may include a textual description of the function of the interface element. However, it is not required to provide user assistance information via a GUI. For example, user assistance information may be provided via a separate HMI, such as another GUI provided on another device, so that interaction with the encapsulated operating system is not required at box 3045. For example, user assistance information may be displayed to the user in a GUI running on the user's smartphone.

[0088] User assistance information may include, for example, tutorials or guided workflows for user interaction with a GUI. The provision of user assistance information may be dependent on the user's skill level.

[0089] Optionally, at box 3050, user feedback can be obtained. User feedback can indicate the usefulness and / or relevance of the user assistance information previously provided at box 3045. This user feedback can be collected through a user feedback mechanism. User feedback can be explicit, for example, explicitly requesting feedback from the user regarding the usefulness and / or relevance of the user assistance information. Alternatively or additionally, user feedback can be implicit, for example, based on the user's usage patterns at the time the feedback is obtained. For example, if the user follows the instructions provided by the user assistance information, it can be determined that the user assistance information is helpful; conversely, if the user decides to deviate from the user instructions provided via the user assistance information, it can be determined that the user assistance information is not helpful. The user feedback mechanism can add this user feedback to a user feedback database, allowing subsequent contextual data to be generated based on the user feedback database.

[0090] like Figure 3 As indicated by the feedback arrows, after providing user assistance information at box 3045 and optionally obtaining user feedback at box 3050, further iterations of box 3010 can be executed. This allows for continuous machine-user interaction. The GUI can continuously guide the user through multiple control steps and image acquisition of the microscope.

[0091] In summary, techniques generally involving providing user assistance when acquiring images of samples using a microscope have been disclosed. These techniques include generating contextual data for the microscope's current settings and organizing it into a standardized, machine-understandable format. The LLM then uses this contextual data to provide accurate user guidance, such as how to use or set which interface elements to achieve the desired results. The contextual data may include information about the microscope's current hardware settings, image acquisition settings, software configuration, and user interactions with the GUI. This data may also include log files, error messages, sensor readings, and other information relevant to the overall operation of the microscope. The LLM uses this contextual data to provide customized user assistance that takes into account the specific situations the user faces when interacting with the GUI. User assistance can be provided in various forms, such as text-based instructions, visual aids, or interactive tools that guide the user through specific processes.

[0092] In some examples, users can manually query user assistance information, while in others, it can be automatically triggered based on monitoring user interactions with the GUI. User assistance information can also be provided outside the GUI (e.g., on a separate handheld device).

[0093] These techniques may include: performing data transfer of at least a portion of context data between an operating system providing a GUI and another operating system that triggers LLM inference. Such data transfer may be secure and limited to intended information, thereby preventing the transmission of other information.

[0094] Although the invention has been shown and described with reference to certain preferred embodiments, equivalents and modifications will arise for those skilled in the art upon reading and understanding this specification. The invention encompasses all such equivalents and modifications and is defined only by the scope of the appended claims.

[0095] To illustrate, while the above scenario using a single LLM has been disclosed, in some scenarios, the task of providing user assistance information can be decomposed into multiple sub-tests and sub-tasks that can be assigned to different LLMs. Furthermore, a multi-agent system can be envisioned, where agents defined by multiple LLMs collaborate to solve a task for providing user assistance information. A multi-agent system is an architectural framework in which multiple agents interact and collaborate to achieve a shared goal. Such a system is particularly useful when the current task can be decomposed into sub-tasks that can be assigned to different agents (each with different capabilities or expertise). For example, in the context of user assistance, one agent might specialize in natural language processing, while another agent might focus on retrieving technical documents or operating system logs. Each agent in a multi-agent system can operate with varying degrees of autonomy, ranging from tightly coupled collaboration to loosely coordinated interaction.

[0096] To further illustrate, although various examples have been disclosed in conjunction with optical microscopy, similar techniques can be applied to, for example, scanning particle microscopy, such as scanning electron microscopy.

[0097] To further illustrate this, various examples have been disclosed in the context of GUIs used to control microscopes. However, similar techniques can also be applied to other mechanical devices, such as volumetric imaging tools (e.g., computed tomography scanners, magnetic resonance imaging scanners), lithography tools (e.g., exposure equipment), and micromanipulators, to name just a few.

Claims

1. A computer-implemented method for providing user assistance when studying samples using a microscope, the method comprising: A graphical user interface is provided to the user, which depicts microscopic images of the sample acquired using the microscope and configures the imaging process associated with the microscopic images. Based on the microscope's current settings, generate contextual data for the microscope. Based on the context data, inference from a trained large language model is triggered to generate user assistance information, and Provide the user assistance information to the user.

2. The computer-implemented method according to claim 1, in, The context data indicates at least one of the microscope's current hardware settings or the microscope's current image acquisition settings.

3. The computer-implemented method according to claim 1 or 2, in, The context data indicates one or more of the following: The microscope's log file; Error messages output by the system software of the microscope; The sensor readings of the microscope's sensor.

4. The computer-implemented method according to any one of the preceding claims, in, The context data indicates the current software configuration of one or more image processing software modules of the microscope.

5. The computer-implemented method according to any one of the preceding claims, in, The context data indicates the current settings of the graphical user interface.

6. The computer-implemented method according to claim 5, wherein, The current settings of the graphical user interface are defined by one or more of the following: Visual interface elements; Hide interface elements; The position of UI elements.

7. The computer-implemented method according to any one of the preceding claims, in, The context data indicates user interaction with the graphical user interface.

8. The computer-implemented method according to any one of the preceding claims, in, The context data indicates user-related information, which includes one or more of the following: Unique user identifier; User skill level; Abnormal user interaction patterns with the graphical user interface; User permission levels.

9. The computer-implemented method according to any one of the preceding claims, in, The context data indicates sample-related information, which includes one or more of the following: Sample type; Sample processing history; Sample size; The structural dimensions of the sample.

10. The computer-implemented method according to any one of the preceding claims, in, The user assistance information is provided to the user via the graphical user interface.

11. The computer-implemented method according to claim 10, in, The user assistance information includes at least one of the following: The graphical user interface elements are highlighted. Tooltips associated with interface elements include textual descriptions of the functionality of those interface elements.

12. The computer-implemented method according to any one of the preceding claims further includes: in, The inference of the trained large language model is triggered by at least one triggering event. The method further includes: Monitor user interactions with the graphical user interface to determine whether the at least one triggering event is satisfied.

13. The computer-implemented method according to claim 12, The monitoring of the user interaction includes: Detect abnormal user interaction patterns. The at least one triggering event includes the abnormal user interaction mode.

14. The computer-implemented method according to claim 12 or 13, in, Monitoring the user interaction includes detecting at least one of the following: Changes to the settings of the interface elements of the graphical user interface; The change in the visibility of the interface elements of the graphical user interface; and The change in the software version of the graphical user interface.

15. The computer-implemented method according to any one of the preceding claims further includes: Based on the user query, a prompt is determined for the trained large language model.

16. The computer-implemented method according to any one of the preceding claims, in, The provision of the graphical user interface is executed within the packaged operating system. The trained large language model is deployed in the cloud. The reasoning triggered by the trained large language model is executed in another operating system connected to the Internet.

17. The computer-implemented method according to claim 16, further comprising: At least a portion of the context data is transferred between the operating system and the other operating system via one or more optical codes.

18. The computer-implemented method according to claim 16 or 17, further comprising: At least a portion of the context data is transmitted between the operating system and the other operating system via near-field wireless transmission and / or self-organizing wireless transmission.

19. The computer-implemented method according to claim 17 or 18, in, The data transmission requires manual user interaction.

20. The computer-implemented method according to any one of the preceding claims, in, The contextual data is at least partially included in the prompts given to the trained large language model.

21. The computer-implemented method according to any one of the preceding claims further includes: Triggers the retrieval of information from one or more encoded and structured knowledge elements.

22. The computer-implemented method according to claim 21, in, The one or more encoded and structured knowledge elements include at least one of the following: The manual for the microscope; The manual for the graphical user interface; The performance specification database of one or more optical elements of the microscope; Technical white paper; The microscope's log file; or The microscope's error database.

23. The computer-implemented method according to any one of the preceding claims further includes: Provide a user feedback mechanism to add user feedback to the user feedback database, providing input from the user regarding the usefulness and / or relevance of the user assistance information. The context data is generated based on the user feedback database.

24. The computer-implemented method according to any one of the preceding claims, wherein, The user assistance information includes tutorials or guided workflows for user interaction with the graphical user interface.

25. The computer-implemented method according to any one of the preceding claims, in, The user assistance information and / or the provision of the user assistance information depends on the user's skill level.

26. One or more computing devices, the one or more computing devices comprising at least one processor and at least one memory, the at least one processor being configured to load program code from the at least one memory and execute the program code, the execution of the program code causing the at least one processor to perform the method according to any one of the preceding claims.

Images