Non-contact microscope control method and system based on brain-computer interface

Abstract

The invention provides a non-contact microscope control method and system based on a brain-computer interface, and relates to the technical field of microscope control, and the method comprises the steps: collecting a brain wave signal generated when a user executes a target psychological task through a brain wave signal collection device, recognizing the brain wave signal by using a pre-trained target intention recognition model to determine a control instruction, and controlling the microscope body to execute a corresponding operation function or controlling the microscope sample processor to execute a sample processing function based on the instruction; the training set of the target intention recognition model is a data set containing brain wave signal samples and corresponding idea instruction samples, and the brain wave signal samples are signals generated based on multiple brain-computer interface normal forms. Various brain-computer interface paradigms are fused, the requirement of complex operation of the microscope for instruction diversity is met, different brain wave signals are mapped to various operation functions of the microscope respectively, and full-function control without touching the microscope body by hands can be achieved through the brain wave signals.

Description

Technical Field

[0001] This invention relates to the field of microscope control technology, and more particularly to a non-contact microscope manipulation method and system based on a brain-computer interface. Background Technology

[0002] Microscopes, as key equipment for basic scientific research and clinical testing, have played an important role in many fields such as biology, medicine, and materials science.

[0003] Currently, microscope operation mainly relies on manual adjustment of the stage, objective lens, or external buttons, which presents the following problems: First, in sterile, highly clean, or hazardous sample (such as virus, radioactive material) environments, manual operation is prone to introducing contamination or risks; Second, users with hand disabilities or motor dysfunction cannot use traditional microscopes; Third, existing eye-tracking, gesture, or voice control solutions still suffer from low recognition accuracy, significant environmental interference, and unnatural user experience.

[0004] Although brain-computer interface technology has been explored in the control of mobile platforms such as wheelchairs and robotic arms, and has provided a cutting-edge solution for non-contact control of microscopes, there are still significant gaps in its application to intelligent control of microscopes. Furthermore, a single BCI (Brain-Computer Interface) paradigm has drawbacks such as a limited instruction set, inability to achieve full-function control, and conflicts between interactive logic and observation tasks. For example, the P300 paradigm and the SSVEP paradigm can disrupt the immersion and continuity of observation, while the motor imagery paradigm, although not dependent on vision, has a limited instruction set that makes it difficult to support full-function control such as panoramic observation, fine focusing, image capture, and sample processing.

[0005] Therefore, in the specific scenario of microscope manipulation, no single existing brain-computer interface paradigm can provide sufficiently rich and precise control commands while ensuring the continuity of observation. There is an urgent need for a technological solution for intelligent microscope manipulation that can integrate the advantages of multiple paradigms and systematically solve the aforementioned problems. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a non-contact microscope control method and system based on brain-computer interface, which can integrate multiple BCI paradigms and realize integrated non-contact intelligent control covering the entire microscope operation process, in order to address the above-mentioned deficiencies of the prior art.

[0007] The technical solution adopted by this invention to solve the technical problem is as follows: In a first aspect, the present invention discloses a non-contact microscope manipulation method based on a brain-computer interface, wherein the method includes: The brainwave signals generated when a user performs a target psychological task are collected using an EEG signal acquisition device. The brainwave signals are identified using a pre-trained target intent recognition model to determine the control commands corresponding to the brainwave signals; Based on the control commands, the microscope body is manipulated to perform corresponding operation functions or the microscope sample processor is manipulated to perform sample processing functions. The training set of the target intent recognition model is a dataset containing brainwave signal samples and their corresponding thought command samples. The brainwave signal samples are signals generated based on various brain-computer interface paradigms, and the brainwave signal samples correspond one-to-one with the thought command samples.

[0008] Optionally, the step of acquiring brainwave signals generated when a user performs a target psychological task via an EEG signal acquisition device includes: The brainwave signals generated by the user when performing a target mental task under the guidance of motor imagination or visual stimulation provided by the display screen are collected by the brainwave signal acquisition device.

[0009] Optionally, when a nine-grid virtual control interface is displayed on a screen that provides visual stimulation to the user, and the EEG signal is a first P300 event-related potential generated when the user focuses on the region of interest corresponding to the target movement mode on the nine-grid virtual control interface, the control command is a movement control command that moves in the target movement mode on the XY plane. The central area and the eight surrounding areas of the nine-grid virtual control interface correspond to different movement modes, and the central area and the eight surrounding areas flash in turn according to a pseudo-random sequence. The XY plane is a plane parallel to the electric stage on the microscope body. The step of controlling the microscope body to perform corresponding operation functions or controlling the microscope sample processor to perform sample processing functions based on the control commands includes: Based on the aforementioned movement control commands, the electric stage on the microscope body is moved in the XY plane in the manner described in the target movement.

[0010] Optionally, when the triggering frequency of the first P300 event-related potential is a single trigger or multiple triggers, the step of manipulating the motorized stage on the microscope body to move in the XY plane in the target movement manner based on the movement control command includes: Based on the movement control command, the electric stage on the microscope body is manipulated to move in the XY plane in the target movement mode at the target movement speed corresponding to the single trigger or the multiple triggers.

[0011] Optionally, when a nine-grid virtual control interface and a mode switching button interface are displayed on a screen that provides visual stimulation to the user, and the EEG signal is a second P300 event-related potential generated when the user focuses on the region of interest corresponding to the mode switching button interface, the control command is a mode switching command. The mode switching button interface is highlighted and flashed in a pseudo-random sequence in the corresponding area of ​​the display screen. The non-contact microscope manipulation method based on brain-computer interface further includes: Control the display screen to switch visual feedback modes based on mode switching commands; The visual feedback modes include a coarse adjustment mode and a fine adjustment mode. In the coarse adjustment mode, the main area of ​​the display screen shows the nine-grid virtual control interface, the middle area of ​​the nine-grid virtual control interface shows its corresponding movement mode and real-time microscopic image stream, and the other areas outside the main area of ​​the display screen show the mode switching button interface. In the fine adjustment mode, the main area of ​​the display screen shows the real-time microscopic image stream, and the other areas outside the main area of ​​the display screen show the nine-grid virtual control interface and the mode switching button interface.

[0012] Optionally, when the display screen is equipped with an SSVEP virtual control panel to be activated, and the EEG signal is an EEG signal generated by the user through first-type motor imagery, and the control command is a wake-up or hide control command for the function panel, the non-contact microscope manipulation method based on the brain-computer interface further includes: The SSVEP virtual control panel is activated or hidden on the display screen based on the wake-up or hide control commands of the function panel. The SSVEP virtual control panel includes virtual buttons corresponding to the operation functions of the microscope body and virtual buttons corresponding to the sample processing functions of the microscope sample processor, and each of the virtual buttons flashes at a different fixed frequency. Furthermore, the SSVEP virtual control panel is activated on the display screen, and the EEG signal is a first steady-state visual evoked potential generated when the user focuses on the target virtual button on the display screen, which has the same frequency as the flashing frequency of the target virtual button. The control command is a function switching control command. Based on the function switching control command, the current function is switched to the operation function or sample processing function corresponding to the target virtual button.

[0013] Optionally, when the brainwave signal is a brainwave signal generated by the user through a second type of motor imagery, the control command is a platform lifting control command, a photo capture control command, or a video recording control command; the platform lifting control command, the photo capture control command, or the video recording control command each correspond to different types of second-type motor imagery. The step of controlling the microscope body to perform corresponding operation functions or controlling the microscope sample processor to perform sample processing functions based on the control commands includes: The electric stage on the microscope body is moved in the z-axis direction based on the stage lifting control command; the z-axis direction is the direction perpendicular to the electric stage. Alternatively, the digital imaging system on the microscope body may be triggered to capture the current field of view based on the photographing control command; Alternatively, the recording control command can trigger the digital imaging system on the microscope body to start or stop recording the current field of view.

[0014] Optionally, when the display screen providing visual stimulation to the user shows a sample loading button interface and a sample unloading button interface, and the EEG signal is a second steady-state visual evoked potential generated by the user focusing on the region of interest corresponding to the sample loading button interface or the sample unloading button interface, and the flashing frequency of the region of interest is the same, the control command is a sample loading control command or a sample unloading control command; the corresponding areas of the sample loading button interface and the sample unloading button interface on the display screen flash brightly at different fixed frequencies. Wherein, when the microscope sample processor is a multi-degree-of-freedom robotic arm or an automated sample loader, the step of controlling the microscope body to perform corresponding operational functions or controlling the microscope sample processor to perform sample processing functions based on the control commands includes: Based on the loading sample control command or the unloading sample control command, the multi-degree-of-freedom robotic arm or the automatic sample loader is manipulated to perform the sample loading function or the sample unloading function.

[0015] Optionally, when a virtual control panel for objective lens selection is displayed on a screen that provides visual stimulation to the user, and the EEG signal is a third steady-state visual evoked potential generated when the user focuses on the region of interest corresponding to the target objective lens button on the virtual control panel, and the flashing frequency of the region of interest is the same, the control command is an objective lens switching control command; the virtual objective lens buttons on the virtual control panel are highlighted and flashed at different fixed frequencies in their corresponding areas on the screen. The step of controlling the microscope body to perform corresponding operation functions or controlling the microscope sample processor to perform sample processing functions based on the control commands includes: The microscope body is operated by an electric objective lens switcher based on objective lens switching control commands to switch the corresponding target objective lens.

[0016] Secondly, the present invention also discloses a non-contact microscope manipulation system based on a brain-computer interface, wherein the system comprises: EEG signal acquisition equipment is used to collect brainwave signals generated when a user performs a target mental task; An embedded system device for recognizing the brainwave signals using a pre-trained target intent recognition model to determine control commands corresponding to the brainwave signals; A control device is used to manipulate the microscope body to perform corresponding operation functions or to manipulate the microscope sample processor to perform sample processing functions based on the control commands. The training set of the target intent recognition model is a dataset containing brainwave signal samples and their corresponding thought command samples. The brainwave signal samples are signals generated based on various brain-computer interface paradigms, and the brainwave signal samples correspond one-to-one with the thought command samples.

[0017] This invention provides a non-contact microscope manipulation method and system based on a brain-computer interface. The non-contact microscope manipulation method includes: acquiring brainwave signals generated when a user performs a target mental task using an electroencephalogram (EEG) signal acquisition device; identifying the EEG signals using a pre-trained target intent recognition model to determine control commands corresponding to the EEG signals; and manipulating the microscope body to perform corresponding operation functions or manipulating the microscope sample processor to perform sample processing functions based on the control commands. The training set of the target intent recognition model is a dataset containing EEG signal samples and their corresponding thought command samples, and the EEG signal samples are signals generated based on various brain-computer interface paradigms, with each EEG signal sample corresponding to a thought command sample. Therefore, this invention utilizes a dataset containing brainwave signal samples and corresponding thought command samples generated based on multiple brain-computer interface paradigms in training the intention recognition model. It integrates multiple brain-computer interface paradigms, meeting the requirements of diverse and reliable commands for complex microscope operations. This overcomes the limitations of a single brain-computer interface paradigm's limited command set. Furthermore, different brainwave signals are mapped to various microscope operation functions or sample processing functions controlling the microscope's sample processor. This enables full-function control of the microscope without manual contact, achieving integrated non-contact intelligent control covering the entire microscope operation process. Users can trigger control commands by performing mental tasks, reducing cumbersome physical operations and realizing a non-contact intelligent human-computer interaction paradigm. This allows for precise thought control of microscope functions such as photography, video recording, stage movement, and sample loading / unloading, fundamentally avoiding contamination and risks to sterile and high-risk samples during operation, and providing a feasible control solution for people with hand disabilities. Attached Figure Description

[0018] Figure 1 This is a flowchart of a preferred embodiment of the non-contact microscope manipulation method based on a brain-computer interface in this invention; Figure 2 This is a schematic diagram of the XY plane movement control logic of a specific electric platform disclosed in this invention; Figure 3 This is a schematic diagram of a specific function switching control logic disclosed in this invention; Figure 4 This is a schematic diagram of the Z-axis movement control logic of a specific electric platform disclosed in this invention; Figure 5 This is a specific image operation control diagram disclosed in this invention; Figure 6 This is a schematic diagram of a specific sample loading / unloading control logic disclosed in this invention; Figure 7 This is a schematic diagram of a specific objective lens selection control logic disclosed in this invention; Figure 8 This is a functional principle block diagram of a preferred embodiment of the non-contact microscope control system based on a brain-computer interface in this invention; Figure 9 This is a schematic diagram of a specific brain-controlled microscope system based on a display screen disclosed in this invention; Figure 10 This is a schematic diagram of another specific brain-controlled microscope system based on a display screen disclosed in this invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0020] Currently, microscope operation mainly relies on manual adjustment of the stage, objective lens, or external buttons, which presents the following problems: First, in sterile, highly clean, or hazardous sample (such as virus, radioactive material) environments, manual operation is prone to introducing contamination or risks; Second, users with hand disabilities or motor dysfunction cannot use traditional microscopes; Third, existing eye-tracking, gesture, or voice control solutions still suffer from low recognition accuracy, significant environmental interference, and unnatural user experience.

[0021] Although brain-computer interface technology has been explored in the control of mobile platforms such as wheelchairs and robotic arms, and has provided a cutting-edge solution for non-contact control of microscopes, there are still significant gaps in its application to intelligent control of microscopes. Furthermore, a single BCI (Brain-Computer Interface) paradigm has drawbacks such as a limited instruction set, inability to achieve full-function control, and conflicts between interactive logic and observation tasks. For example, the P300 paradigm and the SSVEP paradigm can disrupt the immersion and continuity of observation, while the motor imagery paradigm, although not dependent on vision, has a limited instruction set that makes it difficult to support full-function control such as panoramic observation, fine focusing, image capture, and sample processing.

[0022] For example, the mainstream paradigms of brain-computer interface technology mainly include: the P300-based paradigm, the motor imagery-based paradigm, and the steady-state visual evoked potential (SSVEP)-based paradigm. The P300-based paradigm indicates that when a user notices a rare, meaningful stimulus (such as a randomly flashing icon), a positive EEG fluctuation, i.e., the P300 event-related potential, occurs approximately 300 milliseconds after the stimulus appears. This paradigm is suitable for making discrete choices from multiple options, but its interaction efficiency depends on the visual stimulus interface, which can compete for the user's visual attention and interfere with continuous observation of microscopic images. The motor imagery-based paradigm indicates that when a user imagines the movement of a body part (such as a hand or foot) in their mind, the energy of specific rhythms (such as μ and β rhythms) in the sensorimotor cortex of the brain will show significant changes. This paradigm requires no external visual stimulation and is suitable as a "mental shortcut" to trigger commands. However, it has a limited range of commands that it can independently and accurately distinguish. The Steady-State Visual Evoked Potential (SSVEP) paradigm indicates that when a user gazes at a visual stimulus that flashes at a specific frequency, their EEG signal will generate a response synchronized with the flashing frequency and its harmonic frequencies—that is, a steady-state visual evoked potential. This paradigm has high recognition accuracy and fast response, making it very suitable for quickly selecting from multiple fixed options. However, similar to the P300-based paradigm, it requires the user to gaze at a specific flashing target, thus interrupting the main observation task.

[0023] To this end, this application provides a non-contact microscope control scheme based on brain-computer interface, which can integrate multiple BCI paradigms to achieve integrated non-contact intelligent control covering the entire microscope operation process.

[0024] Please see Figure 1 , Figure 1 This is a flowchart of the non-contact microscope manipulation method based on a brain-computer interface in this invention. Figure 1 As shown, the non-contact microscope manipulation method based on brain-computer interface described in this embodiment of the invention includes: Step S11: Collect the brainwave signals generated when the user performs the target psychological task using an EEG signal acquisition device.

[0025] In this embodiment, the user wears an EEG signal acquisition device, such as a dry electrode EEG cap, and then generates EEG signals by performing a specific mental task. These EEG signals are then acquired by the EEG signal acquisition device. Specifically, the EEG signal acquisition device acquires the EEG signals generated by the user when performing a target mental task under the influence of motor imagery or visual stimuli provided by a display screen. That is, the user generates corresponding EEG signals through motor imagery, attention regulation, and response to visual stimuli, and these signals are acquired by the EEG signal acquisition device.

[0026] Step S12: Use a pre-trained target intent recognition model to identify the brainwave signals to determine the control commands corresponding to the brainwave signals; wherein, the training set of the target intent recognition model is a dataset containing brainwave signal samples and their corresponding thought command samples, and the brainwave signal samples are signals generated based on multiple brain-computer interface paradigms, and the brainwave signal samples correspond one-to-one with the thought command samples.

[0027] In this embodiment, after acquiring the brainwave signals generated by the user when performing the target psychological task using an EEG signal acquisition device, a pre-trained target intent recognition model can be used to directly identify the brainwave signals and determine the control commands corresponding to them. It is understood that the identified brainwave signals are converted into control commands.

[0028] Step S13: Based on the control command, manipulate the microscope body to perform the corresponding operation function or manipulate the microscope sample processor to perform the sample processing function.

[0029] In this embodiment, after identifying the corresponding control command by recognizing the EEG signal, the microscope body is manipulated to perform the corresponding operation function or the microscope sample processor is manipulated to perform the sample processing function, such as controlling the movement of the microscope's motorized stage, performing automatic focusing, switching objectives, performing photography and video recording, and sample loading / unloading operations, thereby realizing intelligent observation of the sample being tested.

[0030] As can be seen, in this embodiment of the invention, the training intention recognition model utilizes a dataset containing brainwave signal samples and corresponding thought command samples generated based on multiple brain-computer interface paradigms. This integrates multiple brain-computer interface paradigms, meeting the requirements of diverse and reliable instructions for complex microscope operations. It overcomes the limitations of a single brain-computer interface paradigm's instruction set. Furthermore, different brainwave signals are mapped to various microscope operation functions or sample processing functions controlling the microscope's sample processor. This enables full-function control of the microscope without manual contact through brainwave signals, achieving integrated non-contact intelligent control covering the entire microscope operation process. Users can trigger control commands by performing mental tasks, reducing cumbersome physical operations. This achieves a non-contact intelligent human-computer interaction paradigm, enabling precise thought control of microscope functions such as photography, video recording, stage movement, and sample loading / unloading. This fundamentally avoids contamination and risks to sterile and high-risk sample operation environments and provides a feasible control solution for people with hand disabilities.

[0031] In the first specific implementation, when a nine-grid virtual control interface is displayed on a screen providing visual stimulation to the user, and the EEG signal is the first P300 event-related potential generated when the user focuses on the region of interest corresponding to the target movement mode on the nine-grid virtual control interface, the target intent recognition model identifies the control command determined by the first P300 event-related potential as a movement control command to move in the target movement mode on the XY plane. The central area of ​​the nine-grid virtual control interface and its surrounding eight areas correspond to different movement modes, and the central area and its surrounding eight areas flash brightly in turn according to a pseudo-random sequence. The XY plane is a plane parallel to the motorized stage on the microscope body. Based on the movement control command, the motorized stage on the microscope body is manipulated to move in the target movement mode on the XY plane. It should be noted that at any given time, only one area flashes brightly, and the area that flashes brightly each time is unpredictable. For example, the central area of ​​the nine-grid virtual control interface and its surrounding eight areas flash brightly (i.e., turn on and off) in sequence, and the duration of each flash is the same (e.g., 100 milliseconds), and the target area flashing is random. Users can induce a P300 potential by focusing on a target area and recognizing that area when it flashes. The target intent recognition model then analyzes the P300 potential to determine the flashing target area. (For example, if a P300 potential occurs at a certain moment, and at the same moment the area where the marker on the nine-grid virtual control interface moves to the right also flashes, it can be determined which area the user is observing that caused the P300 potential.) This leads to the generation of corresponding control commands.

[0032] Furthermore, when the triggering frequency of the first P300 event-related potential is a single trigger or multiple triggers, the electric stage on the microscope body is controlled by the movement control command to move in the XY plane in a target movement manner according to the target movement speed corresponding to the single trigger or multiple triggers.

[0033] Furthermore, when the screen providing visual stimulation to the user displays a nine-grid virtual control interface and a mode switching button interface, and the EEG signal is the second P300 event-related potential generated when the user focuses on the region of interest corresponding to the mode switching button interface, the target intent recognition model identifies the control command determined by the second P300 event-related potential as a mode switching command; the corresponding area of ​​the mode switching button interface on the screen flashes in a pseudo-random sequence; the screen switches visual feedback modes based on the mode switching command. As with the pseudo-random sequence flashing of the central area and the eight surrounding areas of the nine-grid virtual control interface, the corresponding area of ​​the mode switching button interface on the screen also flashes in a pseudo-random sequence. That is, the central area and the eight surrounding areas of the nine-grid virtual control interface, as well as the corresponding area of ​​the mode switching button interface on the screen, flash sequentially (i.e., turn on and off), and only one area flashes at any given time. The duration of each flash is the same (e.g., 100 milliseconds), and the target area of ​​the flashing is random.

[0034] The visual feedback modes include coarse adjustment mode and fine adjustment mode. In coarse adjustment mode, the main area of ​​the display screen shows a nine-grid virtual control interface, the middle area of ​​the nine-grid virtual control interface shows the corresponding movement mode and real-time microscopic image stream, and the other areas outside the main area of ​​the display screen show the mode switching button interface. In fine adjustment mode, the main area of ​​the display screen shows the real-time microscopic image stream, and the other areas outside the main area of ​​the display screen show the nine-grid virtual control interface and the mode switching button interface.

[0035] For example, see Figure 2 As shown, to achieve contactless movement control of the microscope's motorized stage in the XY plane, a nine-grid virtual control interface is displayed on the screen. The central area of ​​this interface is marked "Stationary," and the eight surrounding areas correspond to eight movement directions (up, down, left, right, and four diagonal directions), i.e., movement modes. The left and right directions represent the X-axis, and the up and down directions represent the Y-axis. These areas flash alternately in a pseudo-random sequence. Two parallel processes are running: First, the stimulus presentation process: control the flashing of the nine-square grid and accurately record the flashing time of each area each time.

[0036] Second, the signal detection process: real-time acquisition and analysis of EEG signals to detect P300 potential characteristics and the precise time of their occurrence.

[0037] Subsequently, time-locked analysis is performed, comparing the time point at which the P300 potential is detected with the flashing time records of all regions. Only when the flashing time of a specific region is highly correlated with the time of the P300 potential occurrence is the user's selection intent determined to be that region, and a corresponding directional movement control command is immediately generated and sent to the motorized stage. That is, when the user needs to move their field of vision, they focus their attention on the region of interest corresponding to the target direction, actively counting or identifying it, thereby inducing the first P300 event-related potential in the brain. This P300 potential is detected by the EEG signal acquisition device, and its corresponding target region is locked. Then, the target intent recognition model identifies the corresponding movement control command determined by the first P300 event-related potential and sends it to the motorized stage's actuator, so that the actuator drives the motorized stage to move in the target direction in the XY plane, and the movement speed can be controlled by the first P300 event. The triggering frequency of the relevant potentials is controlled in stages. For example, this function is achieved by analyzing the time interval between consecutive valid first P300 event relevant potentials. When an isolated, valid first P300 event relevant potential (i.e., a single trigger) is detected, the stage is controlled to move continuously at a low speed. When multiple consecutive valid first P300 event relevant potentials for the same direction of movement are detected within a preset short time window (e.g., 1 second) (rapid continuous trigger), the stage is controlled to switch to a high-speed movement mode, allowing the user to intuitively adjust the movement speed of the stage by controlling the rhythm of their own attention.

[0038] Furthermore, see the above. Figure 2 As shown: To optimize user experience and resolve line-of-sight conflicts during control, the display screen can present two switchable visual feedback modes. A specific, functionally fixed area is designated as a "mode switching button." When the user needs to switch visual feedback modes, they simply need to focus their attention on this "mode switching button" and recognize it as it flashes. After detecting the P300 potential (the second P300 event-related potential) for this button, the mode switching command is executed, and the corresponding control mode update is displayed on the interface. First, coarse adjustment mode: In this mode, the main area of ​​the display screen is used to display the P300 nine-grid control interface (nine-grid virtual control interface). The central area of ​​the P300 nine-grid control interface not only displays "static" but also overlays a real-time microscopic image stream, allowing the user to still roughly understand the current field of view during positioning. The user issues a movement command by looking at any area in the surrounding area of ​​the nine-grid virtual control interface, and the motorized stage moves accordingly, suitable for quickly searching and locating areas of interest over a large area.

[0039] Second, Fine-tuning Mode: In this mode, the main area of ​​the display screen continuously shows a full-screen real-time microscopic image, while a small nine-grid virtual control interface is displayed in one corner of the image. While observing the sample in the main field of view, the user can use peripheral vision or briefly glance at the nine-grid virtual control interface to issue movement commands. In this mode, the stage is controlled to make small, step-by-step movements, allowing the user to make fine adjustments to the field of view without taking their eyes off the sample details.

[0040] It should be noted that the flashing of each cell in the nine-grid virtual control interface is random, but the system knows which cell is flashing at any given moment. The system compares the time of the P300 peak in the EEG signal with the flashing times of each cell in the recording. If, and only if, the flashing time of a specific cell (such as the cell on the right) highly coincides with the time of the P300 potential, it is determined that the user was surprised or attracted during that flash, and therefore the user intends to select the direction corresponding to that cell.

[0041] In the second specific implementation, when the display screen is set with an SSVEP virtual control panel to be activated, and the EEG signal is the EEG signal generated by the user through the first type of motor imagery, and the control command is the activation or deactivation control command of the function panel, the SSVEP virtual control panel on the display screen is activated or deactivated based on the activation or deactivation control command of the function panel. Furthermore, when the SSVEP virtual control panel is activated on the display screen, and the EEG signal is the first steady-state visual evoked potential generated when the user focuses on the target virtual button on the display screen, with the same frequency as the flashing frequency of the target virtual button, the control command is a function switching control command; based on the function switching control command, the current function is switched to the operation function or sample processing function corresponding to the target virtual button. It is understood that, see [link to relevant documentation]. Figure 3 As shown, users can bring up or hide the function panel by executing a specific, unintentionally triggered mode-switching command (i.e., a function-switching control command), such as visualizing the movement of a specific body part (e.g., the tongue) (e.g., imagining the tip of the tongue moving towards the palate), thus effectively avoiding confusion with routine operations and unconscious physiological activities. The panel is brought up or hidden by recognizing this unique EEG pattern, and corresponding operations are triggered by gazing at the function buttons flashing at different frequencies on the panel.

[0042] See above. Figure 3 As shown, the SSVEP virtual control panel includes virtual buttons corresponding to the various operation functions of the microscope body and virtual buttons corresponding to the sample processing functions of the microscope sample processor, and each virtual button flashes at a different fixed frequency. In other words, functions such as objective lens switching, sample loading / unloading (robotic arm control), and stage movement mode switching are managed through a single SSVEP virtual control panel (dedicated function).

[0043] In the third specific implementation, when the EEG signal is generated by the user through the second type of motor imagery, the control command is a stage lifting control command, a photographing control command, or a video recording control command. Each of these commands corresponds to a different type of second-type motor imagery. Based on the stage lifting control command, the electric stage on the microscope body is moved in the z-axis direction; the z-axis direction is perpendicular to the electric stage. Alternatively, the photographing control command triggers the digital imaging system on the microscope body to capture the current field of view; or, the video recording control command triggers the digital imaging system on the microscope body to start or stop recording the current field of view.

[0044] For example, see Figure 4 As shown, Z-axis movement is based on the motor imagery paradigm, where the user controls the Z-axis (focusing) by performing different, pre-set mental tasks. For example, "Type I motor imagery" (such as imagining the left foot moving) can be mapped to "stage lowering / objective moving away from the sample," and "Type II motor imagery" (such as imagining the right foot moving) can be mapped to "stage rising / objective moving closer to the sample." By analyzing the energy changes of the sensorimotor rhythms (such as μ rhythms and β rhythms) corresponding to the motor imagery in real time through a target intent recognition model, the user's intent can be identified and Z-axis movement can be controlled.

[0045] For example, see Figure 5 As shown, the image-based operation control is based on the motor imagery paradigm. When a user performs a mental task involving a "Type I instruction" (e.g., imagining left hand movement), and the specific EEG pattern is recognized, a "take a picture" command is immediately sent to the microscope's digital imaging system to capture the current visual field. When the user performs a mental task involving a "Type II instruction" (e.g., imagining right hand movement), a "start recording" command is triggered upon recognition. When the user performs the same "Type II instruction" task again, a "stop recording" command is triggered. The images captured by the microscope are transmitted to a display screen via a CCD module for real-time viewing, and the current operation status (e.g., "photo taken," "recording") is displayed on the screen in real time, enhancing the user experience.

[0046] In the fourth specific implementation, when the display screen providing visual stimulation to the user shows a sample loading button interface and a sample unloading button interface, and the EEG signal is a second steady-state visual evoked potential generated by the user focusing on the region of interest corresponding to the sample loading button interface or the sample unloading button interface, which has the same flashing frequency as the region of interest, the control command is a sample loading control command or a sample unloading control command; the corresponding areas of the sample loading button interface and the sample unloading button interface on the display screen are brightly flashed at different fixed frequencies; wherein, when the microscope sample processor is a multi-degree-of-freedom robotic arm or an automatic sample loader, the multi-degree-of-freedom robotic arm or the automatic sample loader is controlled to perform the sample loading function or the sample unloading function based on the sample loading control command or the sample unloading control command.

[0047] See example diagram. Figure 6 As shown, the sample loading / unloading control based on the steady-state visual evoked potential (SSVEP) paradigm involves two virtual buttons on the display screen that flash at different frequencies (e.g., 12Hz and 15Hz), labeled "Load Sample" and "Unload Sample" respectively. When a user looks at one of the buttons, their visual cortex generates a steady-state visual evoked potential at the same frequency as the flashing. The target awareness recognition model can identify the frequency the user is looking at through frequency analysis (e.g., Fast Fourier Transform), thereby triggering the corresponding sample processing control command and controlling the automated sample loading / unloading device to complete the operation.

[0048] In the fifth specific implementation, when the screen providing visual stimulation to the user displays a virtual control panel for objective lens selection, and the EEG signal is a third steady-state visual evoked potential generated when the user focuses on the region of interest corresponding to the target objective lens button on the virtual control panel, and the flashing frequency of the region of interest is the same, the control command is an objective lens switching control command; each virtual objective lens button on the virtual control panel flashes brightly at a different fixed frequency in the corresponding area on the screen; and the electro-optical objective lens switcher on the microscope body is operated based on the objective lens switching control command to switch the corresponding target objective lens.

[0049] See example diagram. Figure 7As shown, the objective lens selection control based on the Steady-State Visual Evoked Potential (SSVEP) paradigm provides a virtual control panel for objective lens selection on the user feedback interface to achieve switching between different magnifications. This panel contains multiple (e.g., three) virtual buttons that flash at different frequencies. Each button is clearly labeled with its corresponding objective lens magnification (e.g., "10X", "40X", "100X"), and these buttons correspond one-to-one with the objective lens positions on the microscope objective lens turret. When the user needs to switch objectives, their gaze is fixed on the target button representing the target magnification (e.g., on the "40X" button flashing at 15Hz). The user's visual cortex generates a steady-state visual evoked potential (SVP) with the same flashing frequency as the target button. The target intent recognition model identifies this characteristic frequency through real-time EEG signal frequency analysis (e.g., Fast Fourier Transform), maps the identified characteristic frequency to a preset objective lens switching control command, and then sends a control command to the electro-optical objective lens switcher, driving the objective lens turret to rotate and precisely switch the target objective lens into the optical path. After the objective lens is switched, visual feedback is provided on the user interface (such as highlighting the currently selected objective lens button) and a soft beep is emitted to inform the user that the operation is complete.

[0050] In one embodiment, such as Figure 8 As shown, based on the above-described non-contact microscope manipulation method based on a brain-computer interface, the present invention also provides a corresponding non-contact microscope manipulation system based on a brain-computer interface, comprising: The EEG signal acquisition device 11 is used to acquire the brainwave signals generated when the user performs a target psychological task.

[0051] An embedded system device 12 is used to identify the brainwave signals using a pre-trained target intention recognition model to determine control commands corresponding to the brainwave signals; wherein the training set of the target intention recognition model is a dataset containing brainwave signal samples and their corresponding intention command samples, and the brainwave signal samples are signals generated based on multiple brain-computer interface paradigms, and the brainwave signal samples correspond one-to-one with the intention command samples.

[0052] The control device 13 is used to control the microscope body to perform corresponding operation functions or to control the microscope sample processor to perform sample processing functions based on the control commands.

[0053] Furthermore, it is worth noting that the working process of the non-contact microscope control system based on brain-computer interface provided in this embodiment is the same as the working process of the non-contact microscope control method based on brain-computer interface described above, and will not be repeated here. For details, please refer to the working process of the non-contact microscope control method based on brain-computer interface described above.

[0054] For example, see Figure 9As shown, a brain-controlled microscope system based on a display screen may specifically include a microscope stand, an illumination system, an electro-optical microscope switcher, a CCD / CMOS image detector, a three-dimensional motorized stage and its driver, a display screen, an electroencephalogram (EEG) signal acquisition device, an embedded master controller (i.e., an embedded system device), an automatic sample loader, and a power supply. The EEG signal acquisition device is connected to the embedded master controller, and the embedded master controller is connected to the display screen, the driver, the automatic sample loader, the CCD / CMOS image detector, and the electro-optical microscope switcher, respectively. Users generate brainwave signals by performing specific mental tasks under the guidance of motor imagery or visual stimuli provided by the display screen. These signals are acquired by an EEG acquisition device and transmitted to an embedded central controller. The embedded central controller uses a built-in target intent recognition model to identify the signals and convert them into control commands. Based on these control commands, the microscope's motorized stage is moved, autofocus is performed, objectives are switched, and operations such as taking photos / recording videos or loading / unloading samples are performed, thus enabling intelligent observation of the sample. In addition to providing visual stimulation, the display screen can also receive images or videos of the current field of view transmitted from the digital imaging system via the CCD module and display the current image or video and the current operating status, such as "Photo taken" or "Recording." Alternatively, see [link to other documentation]. Figure 10 As shown, a brain-controlled microscope system based on a display screen can specifically include a microscope stand, an illumination system, an electro-optical microscope switcher, a CCD / CMOS image detector, a three-dimensional motorized stage and its driver, a display screen, an electroencephalogram (EEG) signal acquisition device, an embedded central controller, a robotic arm system, and a power supply. The EEG signal acquisition device is connected to the embedded central controller, which in turn is connected to the display screen, the driver, the motion controller in the robotic arm system, the CCD / CMOS image detector, and the electro-optical microscope switcher. The robotic arm system is integrated next to the microscope and consists of the robotic arm body, a gripper, a motion controller, and a power module. The gripper can be specially designed according to the sample carrier (such as a slide or culture dish). The motion controller receives the EEG signals identified by the embedded central controller, determines the control commands, and calculates the commands into the motion trajectories of each joint. The power module supplies power to the entire robotic arm system. Users generate brainwave signals by performing specific mental tasks. These signals are then acquired by an EEG acquisition device and transmitted to an embedded central controller. The embedded central controller uses a built-in target intent recognition model to identify the signals and convert them into control commands. In addition to controlling the microscope body (such as moving the motorized stage, autofocusing, switching objectives, and taking photos and videos) based on these commands, it can also precisely control a multi-degree-of-freedom robotic arm to perform complex sample processing tasks, including but not limited to picking up a specific slide from the sample box, placing it in a designated position on the stage, and moving the slide back after observation.

[0055] It should be noted that contactless microscope control, either on-site or remotely, can be achieved using a browser / server (B / S) architecture. This means users access an application running on a remote server via a browser, without needing to install dedicated client software locally. The display-based brain-controlled microscope system supports B / S architecture deployment, using the EEG signal processing unit and microscope control unit as the server. Clients can access the remote interface via a browser, while the server runs the EEG signal decoding algorithm and microscope control program. After authentication (entering account and password), users can access the control interface on-site or remotely via a browser to view microscope images or control the microscope in real time. The client needs to be equipped with an EEG acquisition device to collect the user's brainwave signals. The browser transmits the collected user data (EEG) to the EEG recognition module deployed on the server side via the network for EEG command recognition, thereby enabling contactless control of the microscope on-site or remotely, such as moving the stage, switching objectives, taking pictures and recording videos, loading / unloading samples, etc. The HTTP protocol is also used to ensure low-latency communication between the user's browser and the microscope server, enabling data communication and synchronization, and real-time synchronization of images and operating status. Even if the user and the microscope system are not in the same space, efficient and stable remote control and monitoring operations can be achieved.

[0056] It should also be noted that after the system starts, it first guides the user to record a period of resting state to obtain baseline EEG data. Subsequently, the system enters a personalized calibration mode, guiding the user to perform a series of standard mental tasks (such as imagining left hand movement, imagining right hand movement, and focusing on flashing stimuli at specific locations on a screen), while simultaneously recording their EEG signals. This data is used to train a personalized intention recognition model for the user, completing the system initialization.

[0057] Therefore, the above-mentioned technical solution of this application can produce the following technical effects, namely: First, true non-contact and high safety: Users do not need to touch the microscope body or any physical controller with their bare hands. They can achieve full-function control through brainwave signals, which fundamentally avoids the contamination and risks to sterile samples and high-risk samples during operation. It is especially suitable for high-level biosafety laboratories, clean rooms and scenarios where radioactive or toxic samples are handled.

[0058] Second, high-precision and low-latency control: Modern signal processing and machine learning algorithms (such as EEGNet and Transformer) are used to decode EEG features in real time, which enables rapid recognition of commands and high accuracy, ensuring the real-time performance and precision of microscope manipulation.

[0059] Third, multi-functional integration and paradigm fusion: By integrating multiple brain-computer interface paradigms such as P300, motor imagination and steady-state visual evoked potentials, a rich "mind command set" is constructed, enabling a single system to support the entire process of operation, including stage movement, multi-scale imaging (objective lens switching), image capture (photography / video recording) and sample processing.

[0060] Fourth, immersive experience and high-efficiency operation: During the critical precise positioning and observation stage, the user's eyes do not need to leave the microscopic field of view, and the hands are also freed up to simultaneously perform recording or other operations, which greatly reduces the cognitive load and time cost of switching between different tasks, and realizes an immersive workflow that integrates "observation-control".

[0061] Fifth, strong system adaptability and scalability: The system provides a user calibration mode, which can adapt to the physiological differences of different users and improve the universality of control. At the same time, the system architecture is open and can be easily integrated with other interaction modalities such as eye tracking and gesture recognition. It also supports remote control based on B / S architecture, providing a solid foundation for future functional expansion and cross-platform applications.

[0062] Sixth, providing barrier-free solutions for people with disabilities: providing feasibility for users with hand or arm motor dysfunction to operate high-precision scientific instruments.

[0063] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein.

[0064] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0065] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can read and execute instructions from and from an instruction execution system, apparatus or device).

[0066] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0067] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A non-contact microscope manipulation method based on a brain-computer interface, characterized in that, The method includes: The brainwave signals generated when a user performs a target psychological task are collected using an EEG signal acquisition device. The brainwave signals are identified using a pre-trained target intent recognition model to determine the control commands corresponding to the brainwave signals; Based on the control commands, the microscope body is manipulated to perform corresponding operation functions or the microscope sample processor is manipulated to perform sample processing functions. The training set of the target intent recognition model is a dataset containing brainwave signal samples and their corresponding thought command samples. The brainwave signal samples are signals generated based on various brain-computer interface paradigms, and the brainwave signal samples correspond one-to-one with the thought command samples.

2. The non-contact microscope manipulation method based on brain-computer interface according to claim 1, characterized in that, The acquisition of brainwave signals generated when a user performs a target psychological task via an EEG signal acquisition device includes: The brainwave signals generated by the user when performing a target mental task under the guidance of motor imagination or visual stimulation provided by the display screen are collected by the brainwave signal acquisition device.

3. The non-contact microscope manipulation method based on brain-computer interface according to claim 2, characterized in that, When a nine-square grid virtual control interface is displayed on a screen that provides visual stimulation to the user, and the EEG signal is the first P300 event-related potential generated when the user focuses on the region of interest corresponding to the target movement mode on the nine-square grid virtual control interface, the control command is a movement control command that moves in the target movement mode on the XY plane. The central area and the eight surrounding areas of the nine-grid virtual control interface correspond to different movement modes, and the central area and the eight surrounding areas flash in turn according to a pseudo-random sequence. The XY plane is a plane parallel to the electric stage on the microscope body. The step of controlling the microscope body to perform corresponding operation functions or controlling the microscope sample processor to perform sample processing functions based on the control commands includes: Based on the aforementioned movement control commands, the electric stage on the microscope body is moved in the XY plane in the manner described in the target movement.

4. The non-contact microscope manipulation method based on brain-computer interface according to claim 3, characterized in that, When the triggering frequency of the first P300 event-related potential is a single trigger or multiple triggers, the step of manipulating the motorized stage on the microscope body to move in the XY plane in the target movement manner based on the movement control command includes: Based on the movement control command, the electric stage on the microscope body is manipulated to move in the XY plane in the target movement mode at the target movement speed corresponding to the single trigger or the multiple triggers.

5. The non-contact microscope manipulation method based on a brain-computer interface according to claim 3, characterized in that, When a nine-grid virtual control interface and a mode switching button interface are displayed on a screen that provides visual stimulation to the user, and the EEG signal is the second P300 event-related potential generated when the user focuses on the region of interest corresponding to the mode switching button interface, the control command is a mode switching command. The mode switching button interface is highlighted and flashed in a pseudo-random sequence in the corresponding area of ​​the display screen. The non-contact microscope manipulation method based on brain-computer interface further includes: Control the display screen to switch visual feedback modes based on mode switching commands; The visual feedback modes include a coarse adjustment mode and a fine adjustment mode. In the coarse adjustment mode, the main area of ​​the display screen shows the nine-grid virtual control interface, the middle area of ​​the nine-grid virtual control interface shows its corresponding movement mode and real-time microscopic image stream, and the other areas outside the main area of ​​the display screen show the mode switching button interface. In the fine adjustment mode, the main area of ​​the display screen shows the real-time microscopic image stream, and the other areas outside the main area of ​​the display screen show the nine-grid virtual control interface and the mode switching button interface.

6. The non-contact microscope manipulation method based on brain-computer interface according to claim 2, characterized in that, When the display screen is set with an SSVEP virtual control panel to be activated, and the EEG signal is the EEG signal generated by the user through first-type motor imagery, and the control command is the activation or deactivation control command of the function panel, the non-contact microscope manipulation method based on brain-computer interface further includes: The SSVEP virtual control panel is activated or hidden on the display screen based on the wake-up or hide control commands of the function panel. The SSVEP virtual control panel includes virtual buttons corresponding to the operation functions of the microscope body and virtual buttons corresponding to the sample processing functions of the microscope sample processor, and each of the virtual buttons flashes at a different fixed frequency. Furthermore, the SSVEP virtual control panel is activated on the display screen, and the EEG signal is a first steady-state visual evoked potential generated when the user focuses on the target virtual button on the display screen, which has the same frequency as the flashing frequency of the target virtual button. The control command is a function switching control command. Based on the function switching control command, the current function is switched to the operation function or sample processing function corresponding to the target virtual button.

7. The non-contact microscope manipulation method based on brain-computer interface according to claim 2, characterized in that, When the brainwave signal is a brainwave signal generated by the user through the second type of motor imagery, the control command is a platform lifting control command, a photo control command, or a video recording control command; the platform lifting control command, the photo control command, or the video recording control command each correspond to different types of the second type of motor imagery; The step of controlling the microscope body to perform corresponding operation functions or controlling the microscope sample processor to perform sample processing functions based on the control commands includes: The electric stage on the microscope body is moved in the z-axis direction based on the stage lifting control command; the z-axis direction is the direction perpendicular to the electric stage. Alternatively, the digital imaging system on the microscope body may be triggered to capture the current field of view based on the photographing control command; Alternatively, the recording control command can trigger the digital imaging system on the microscope body to start or stop recording the current field of view.

8. The non-contact microscope manipulation method based on brain-computer interface according to claim 2, characterized in that, When the display screen providing visual stimulation to the user shows the loading and unloading sample button interfaces, and the EEG signal is a second steady-state visual evoked potential generated by the user focusing on the region of interest corresponding to the loading or unloading sample button interface, with the flashing frequency of the region of interest being the same, the control command is a loading sample control command or an unloading sample control command; the corresponding areas of the loading and unloading sample button interfaces on the display screen flash brightly at different fixed frequencies. Wherein, when the microscope sample processor is a multi-degree-of-freedom robotic arm or an automated sample loader, the step of controlling the microscope body to perform corresponding operational functions or controlling the microscope sample processor to perform sample processing functions based on the control commands includes: Based on the loading sample control command or the unloading sample control command, the multi-degree-of-freedom robotic arm or the automatic sample loader is manipulated to perform the sample loading function or the sample unloading function.

9. The non-contact microscope manipulation method based on brain-computer interface according to claim 2, characterized in that, When the screen providing visual stimulation to the user displays a virtual control panel for objective lens selection, and the EEG signal is a third steady-state visual evoked potential generated when the user focuses on the region of interest corresponding to the target objective lens button on the virtual control panel and the flashing frequency of the region of interest is the same, the control command is an objective lens switching control command. The virtual buttons for each objective lens on the virtual control panel are highlighted and flashed at different fixed frequencies in the corresponding areas of the display screen. The step of controlling the microscope body to perform corresponding operation functions or controlling the microscope sample processor to perform sample processing functions based on the control commands includes: The microscope body is operated by an electric objective lens switcher based on objective lens switching control commands to switch the corresponding target objective lens.

10. A non-contact microscope manipulation system based on a brain-computer interface, characterized in that, The system includes: EEG signal acquisition equipment is used to collect brainwave signals generated when a user performs a target mental task; An embedded system device for recognizing the brainwave signals using a pre-trained target intent recognition model to determine control commands corresponding to the brainwave signals; A control device is used to manipulate the microscope body to perform corresponding operation functions or to manipulate the microscope sample processor to perform sample processing functions based on the control commands. The training set of the target intent recognition model is a dataset containing brainwave signal samples and their corresponding thought command samples. The brainwave signal samples are signals generated based on various brain-computer interface paradigms, and the brainwave signal samples correspond one-to-one with the thought command samples.

Images