Teleoperation task-oriented multi-mode asynchronous brain-computer interaction method, system and terminal

Abstract

The invention belongs to the technical field of brain-computer interfaces and equipment teleoperation, and relates to a teleoperation task-oriented multi-mode asynchronous brain-computer interaction method, which is characterized in that a control instruction-return picture double-interface surrounding layout is used, an operator can select an instruction stimulation block according to a return picture, and the stimulation block presents visual stimulation to induce a specific electroencephalogram signal; establishing a line-of-sight error model capable of correcting fixed errors and estimating random error distribution, and calculating effective line-of-sight coordinates; using an implicit asynchronous switch to switch a system observation / control mode according to the effective sight line coordinates of the observation state; and calculating an alternative instruction classification coefficient based on the electroencephalogram signal, determining an alternative instruction range and a weight according to the task state effective line-of-sight coordinate, performing fusion decision, and outputting an instruction to a controlled object. The invention further provides a teleoperation task-oriented multi-mode asynchronous brain-computer interaction system and a teleoperation task-oriented multi-mode asynchronous brain-computer interaction terminal. The method reduces the learning cost and cognitive load of an operator, and has the advantages of flexible control, real-time response, accurate recognition and the like.

Description

Technical Field

[0001] This invention relates to the fields of brain-computer interfaces and device teleoperation technology, and in particular to a multimodal asynchronous brain-computer interaction method, system and terminal for teleoperation tasks. Background Technology

[0002] Teleoperation refers to the technology of human operators controlling remote robots, mechanical systems, or electronic devices in real-time or near real-time via remote control devices. During teleoperation, the operator and the controlled object are physically separated; the operator is in a safe area, while the controlled object is in a dangerous, inaccessible, or unsafe environment. Because teleoperation combines human decision-making capabilities with machine execution precision, it is suitable for tasks requiring high flexibility and environmental adaptability, especially in high-risk environments (such as bomb disposal and rescue) and special space environments (such as space and the deep sea). In short, the core of teleoperation is to remotely "extend" human intelligence to machine systems to overcome spatial limitations and improve operational safety. In mission scenarios requiring teleoperation, operators typically need to simultaneously handle multiple pieces of information and operate complex control systems. Traditional control and communication methods often rely on continuous control and communication via hand gestures (such as joysticks and buttons) or voice channels, which may not be able to handle certain unexpected events or the operational needs of parallel tasks during operation.

[0003] Brain-computer interface (BCI) is a novel control and communication technology that establishes a connection between the brain and a computer or other electronic device, independent of conventional brain information output pathways (peripheral nerves and muscle tissue). BCI allows operators to interact directly with the system through thought, reducing reliance on traditional user interfaces (such as manual control and voice commands), thereby improving efficiency or enabling parallel task execution under conditions where traditional channels are occupied. Among these, visual BCI, due to its diverse encoding methods, can support larger instruction sets and offers advantages such as faster recognition speed and higher accuracy, thus demonstrating greater development potential.

[0004] However, most existing visual BCI human-computer interaction interfaces use a matrix layout, arranging instructions according to rows and columns on a single panel. This is incompatible with the characteristics of teleoperation tasks, which require constant observation of environmental information and the user's own state. Placing the feedback screen and instruction panel on the same interface, whether arranged vertically or horizontally, leads to wasted space due to screen size limitations, thus compressing the instruction set and failing to leverage the advantage of visual BCI in building larger instruction sets through stimulus encoding. Conversely, placing the feedback screen and instruction panel on two separate interfaces requires frequent switching between them, reducing ease of use and flexibility. Furthermore, using a matrix-style instruction layout in teleoperation tasks is ergonomically unsound, adding extra learning costs for the operator. Moreover, due to the trade-off between the number of instructions and recognition accuracy—systems with more instructions often have lower recognition accuracy—current solutions to this problem include extending the visual stimulus time. However, this increases operational latency, further increasing the difficulty and complexity of teleoperation tasks. Therefore, the accuracy of multi-command BCI in recognizing commands still needs further improvement to meet the demands for operational precision while maintaining real-time performance in teleoperation scenarios. Furthermore, existing BCIs are often synchronous, requiring operators to follow system prompts. However, in actual teleoperation scenarios, operators often need to make real-time judgments based on environmental and status information and respond to unexpected events. Therefore, the human-computer interaction strategy of synchronous BCIs is also insufficient to meet the rapid response requirements of teleoperation scenarios.

[0005] Therefore, there is an urgent need to propose a high-precision brain-computer interface solution that can adapt to and meet the rapid response requirements in teleoperation task scenarios. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing matrix-based visual BCI interfaces, which are unable to adapt to the interactive requirements of teleoperation tasks. Current multi-command BCI systems suffer from limitations in response speed and recognition accuracy, making it difficult to simultaneously meet the real-time and precision requirements of teleoperation tasks. Furthermore, existing synchronous BCI human-computer interaction strategies rely entirely on the system's time settings, neglecting the suddenness of events and the real-time nature of operations, thus failing to meet the rapid response demands of teleoperation scenarios. This invention proposes a multimodal asynchronous brain-computer interface method, system, and terminal for teleoperation tasks. The invention designs a dual-interface surround presentation mode for control commands and feedback images, establishing a mapping relationship between command functions and spatial positions, facilitating operator observation and operation. It also proposes a decision-making mode that fuses gaze estimation and EEG signal recognition, establishing a gaze error correction model that can partially correct random errors. This model reflects the deviation of gaze detection coordinates from the true gaze point coordinates under the influence of system errors, and dynamically and adaptively delineates the range of alternative commands based on the effective gaze estimation coordinates during the command decoding stage, thereby improving recognition accuracy without increasing visual stimulation time. In addition, this invention proposes an asynchronous control method for decision-making based on the fusion of eye movement and electroencephalogram (EEG) signals, which can enable the brain-computer interface system to respond quickly to sudden events and improve the robustness of the system, avoiding false triggering caused by untimely eye movement or inability to change instructions midway when using a synchronous system.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a multimodal asynchronous brain-computer interface method for teleoperation tasks. Before performing the interaction method, the operator wears an EEG acquisition device to collect calibration data and construct the operator's EEG signal classification and decoding model and gaze error correction model. Perform the multimodal asynchronous brain-computer interface method as follows: S10. The control commands and the feedback screen of the controlled object are presented to the operator at the same time. The feedback screen is located in the center of the human-computer interaction interface and is updated in real time. The control commands are presented as stimulation blocks around the feedback screen. S20. Switch to observation mode. At this time, the stimulus block does not flash. The operator looks at the feedback screen or stimulus block according to the control requirements. The coordinates of the operator's gaze point are detected in real time, and the coordinates of the gaze point are corrected to obtain the effective gaze estimation coordinates in the observation state. S30. Determine the estimated coordinates of the effective gaze in the observation state. If it is located in the return screen area, return to step S20. If it is located in the control command area, switch to control mode. The operator looks at the stimulus block to be selected. The flashing of the stimulus block induces specific EEG signals in the operator. Detect and record the coordinates of the operator's gaze point during the flashing of the stimulus block. After the stimulus block stops flashing, execute steps S40 to S60. S40. Correct the coordinates of the operator's gaze point during the flashing of the stimulus block to obtain the estimated coordinates of the effective gaze in the task state. Determine the alternative control commands based on the estimated coordinates of the effective gaze in the task state and set a weight for each alternative control command. Extract the operator's EEG signal features and perform pattern recognition on the EEG signal features based on the EEG signal classification and decoding model to obtain the classification coefficients. S50. Calculate the fusion decision coefficient of each candidate control command by combining the classification coefficient and the weight of the candidate control command, and output the control command with the highest fusion decision coefficient to the controlled object; S60. The controlled object responds to the control command and returns to the execution step S20.

[0008] As one possible implementation, the line-of-sight error correction model includes a fixed offset and a random error distribution for line-of-sight estimation. The operator's line-of-sight error correction model is constructed using the following method: S01. Subtract the actual coordinates of the target point from the coordinates of each line of sight landing point to obtain the error data of all line of sight landing point coordinates relative to the actual coordinates of the target point; S02. Calculate the mean and standard deviation of the error data; S03. Calculate the calibration score of the error data based on the average value and standard deviation of the error data, configure the preset error value, remove the error data with a calibration score greater than the preset error value, and obtain a new dataset; S04. Repeat steps S02 to S03 for the new dataset until no more data with calibration scores greater than the preset error value are found. The dataset obtained at this point is the valid error dataset. S05. Construct a two-dimensional Gaussian probability density function based on the mean and standard deviation of the effective error data set. Its expected value is the fixed offset of the line-of-sight estimation. Subtract the fixed offset of the line-of-sight estimation from the two-dimensional Gaussian probability density function to obtain the random error distribution of the line-of-sight estimation.

[0009] As one possible implementation, correcting the coordinates of the line of sight to obtain the effective line-of-sight estimation coordinates in the observation state includes the following sub-steps: S200. Real-time detection of the operator's line of sight coordinates to obtain a set of line of sight coordinates per unit time. S201. Based on the random error distribution of line-of-sight estimation, outliers in the set of line-of-sight coordinates are removed to obtain the effective set of line-of-sight coordinates; S202. Calculate the average coordinates of the set of effective line-of-sight coordinates, and subtract the fixed offset of the line-of-sight estimation from the average coordinates to obtain the estimated coordinates of the effective line-of-sight in the observation state.

[0010] As one possible implementation, S201 includes the following sub-steps: S2010. Calculate the average coordinates of the set of coordinates of the points where the line of sight falls per unit time; S2011. Calculate the two-dimensional Gaussian probability density function. Shaft standard deviation and Shaft standard deviation; S2012. Based on coordinate average value, Shaft standard deviation and The standard deviation of the axis is used to calculate the coordinates of each line of sight point. Shaft calibration fraction and Shaft calibration score, obtain Axis calibration fraction set and Axis calibration fraction set; S2013. Elimination based on the three-standard-deviation principle Axis calibration fraction set and Outliers in the axis calibration fraction set are used to obtain a new set of line-of-sight coordinates. S2014. Repeat steps S2010 to S2013 for the new set of line-of-sight coordinates until no more outliers need to be removed. The set of line-of-sight coordinates obtained at this time is the valid set of line-of-sight coordinates.

[0011] As one possible implementation, the following method is used to determine alternative control commands based on the effective line-of-sight estimation coordinates in the task state: A selection window is drawn with the effective line-of-sight estimation coordinates in the task state as the center. All control commands corresponding to stimulus blocks whose center points are located within the selection window are determined as alternative control commands.

[0012] As one possible implementation, the selection window is drawn centered on the effective line-of-sight estimation coordinates in the task state, including: Using the effective line-of-sight estimation coordinates of the task state as the center of the ellipse, with times Shaft standard deviation and times The standard deviation of the axes is two semi-axes; draw an ellipse selection window; or, with the effective line-of-sight estimated coordinates of the task state as the center, ... times Shaft standard deviation or times Draw a circular selection window with the axis standard deviation as the radius; or, center the selection window on the estimated coordinates of the effective line of sight in the task state. times Shaft standard deviation or times Using the standard deviation of the axis as the side length, draw a square selection window; or, using the effective line-of-sight estimation coordinates of the task state as the center, with... times Shaft standard deviation and times The standard deviation of the axes is defined by two sides; draw a rectangular selection window. , It is the set of real numbers.

[0013] As one possible implementation, the fusion decision coefficient for each candidate control command is calculated by combining the classification coefficient and the weights of the candidate control commands: S500. Calculate the probability of the stimulus block coverage area corresponding to each candidate control instruction based on the two-dimensional Gaussian probability density function, and set a weight for each candidate control instruction based on the probability distribution of the candidate control instructions; or, calculate the distance of each candidate control instruction from the center of the selection window, and set a weight for each candidate control instruction based on the distance between each candidate control instruction and the center of the selection window. S501. Multiply the classification coefficients by the weights of each candidate control command to obtain the fusion decision coefficients for each candidate control command.

[0014] Secondly, the present invention provides a multimodal asynchronous brain-computer interface system for teleoperation tasks, comprising: The calibration module is used to build an EEG signal classification and decoding model and an operator's gaze error correction model. The human-computer interaction module presents the control commands and the feedback screen of the controlled object to the operator at the same time. The feedback screen is located in the center of the human-computer interaction interface and is updated in real time. The control commands are presented as stimulation blocks around the feedback screen. The gaze detection module, in observation mode, is used to detect the coordinates of the operator's gaze point in real time, correct the gaze point coordinates, and obtain the effective gaze estimation coordinates in observation mode; in control mode, it is used to detect and record the coordinates of the operator's gaze point during the flashing of the stimulus block, correct the coordinates of the operator's gaze point during the flashing of the stimulus block, and obtain the effective gaze estimation coordinates in task mode. The EEG data acquisition module is used to collect the operator's EEG signals during calibration and operation. The instruction recognition and decision-making module determines candidate control instructions based on the effective line-of-sight estimation coordinates in the task state and sets a weight for each candidate control instruction; it extracts the operator's EEG signal features, performs pattern recognition on the EEG signal features to obtain classification coefficients; and it calculates the fusion decision coefficient for each candidate control instruction by combining the classification coefficients and the weights of the candidate control instructions. The remote operation control module is used to output control commands obtained from the fusion decision to the controlled object and to acquire the images returned by the controlled object. It also includes an asynchronous control module that switches between control and observation modes based on the estimated coordinate position of the effective line of sight in the observation state and the status flags sent by the control commands.

[0015] Thirdly, the present invention provides a terminal including a processor and a communication interface coupled to the processor, the processor being used to run computer programs or instructions to implement the multimodal asynchronous brain-computer interaction method for teleoperation tasks provided in the first aspect.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention proposes a wraparound dual-interface layout for control commands and feedback images, which can solve the incompatibility problem between traditional matrix-style visual BCI interfaces and teleoperation tasks. Placing the feedback image in the center of the screen ensures that the operator can focus on observing the task scene, while reducing the distance of eye movement when switching between "observation / control" states. The wraparound control command area can maximize the use of screen space while establishing a strong mapping relationship between "command function and spatial position" to improve interactive intuitiveness. This eliminates the need for the operator to memorize the position of commands, reducing the operator's learning cost and significantly reducing the operator's cognitive load.

[0017] 2. Based on pre-collected calibration data, this invention constructs a two-dimensional Gaussian distribution model of the line-of-sight error. According to the coordinates of the operator's line-of-sight, invalid data such as head movements and blinking can be adaptively eliminated, correcting the fixed error in line-of-sight detection, and estimating the error range of the corrected coordinates, thereby improving the accuracy of line-of-sight detection.

[0018] 3. This invention integrates a gaze estimation correction method based on a two-dimensional Gaussian distribution with EEG recognition technology. It uses a multimodal fusion decision-making command recognition method of "eye movement pre-screening - EEG fine decoding" to identify target commands. First, a dynamic candidate command selection window is constructed based on the effective gaze estimation coordinates to perform preliminary screening of control commands and assign weights to the candidate commands. Then, the weights are fused with the EEG pattern recognition classification coefficients to make a decision, thereby improving the decoding accuracy of commands without increasing the stimulation duration and breaking through the limitations of response speed and recognition accuracy of multi-command BCI systems.

[0019] 4. This invention employs an implicit asynchronous state switching method, which does not require the introduction of an additional gaze switch or the specification of a specific gaze area and can respond at any time. This solves the problem that synchronous BCI is not flexible enough and has difficulty responding to sudden operations. Operators can naturally and autonomously switch the system's control / observation mode according to the current control needs without performing additional operations. This can effectively avoid false triggering caused by factors such as the decision time exceeding the fixed time interval between two rounds of operations in synchronous systems, thereby improving the system's robustness. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of a multimodal asynchronous brain-computer interface method for teleoperation tasks proposed in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the process of constructing an operator's line-of-sight error correction model in an embodiment of the present invention. Figure 3 This is a schematic diagram of the control commands and feedback images of the unmanned vehicle in an embodiment of the present invention. Figure 4 This is a schematic diagram of the control commands and the surround display interface of the UAV in an embodiment of the present invention; Figure 5 This is a flowchart of the process for removing outliers from the set of line-of-sight coordinates based on the random error distribution of line-of-sight estimation in this embodiment of the invention to obtain a valid set of line-of-sight coordinates. Figure 6 This is a schematic diagram of the structure of a multimodal asynchronous brain-computer interface system for teleoperation tasks proposed in an embodiment of the present invention; Figure 7 This is a schematic diagram illustrating the process of an operator controlling a drone to perform flight control tasks in an embodiment of the present invention. Detailed Implementation

[0021] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first threshold and the second threshold are merely used to distinguish different thresholds and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0022] It should be noted that in this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0023] In this invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one" or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, "at least one of a, b, or c" can represent: a, b, c, a combination of a and b, a combination of a and c, a combination of b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0024] This invention aims to provide a multimodal asynchronous brain-computer interface (BCI) method, system, and terminal for teleoperation tasks. It designs a dual-plane, surround-style presentation of control commands and transmitted visuals, establishing a mapping relationship between command function and spatial location for ease of observation and operation. This invention proposes a decision-making mode that fuses gaze estimation and EEG signal recognition, establishing a gaze error correction model that can partially correct random errors. This model reflects the deviation of gaze detection coordinates from the true gaze point coordinates under the influence of systematic errors. Based on the effective gaze estimation coordinates during the command decoding stage, it dynamically and adaptively defines the range of alternative commands, improving recognition accuracy without increasing visual stimulation time. Furthermore, this invention proposes an asynchronous control method that fuses eye movement and EEG signals for decision-making. This enables the BCI system to respond quickly to sudden events and improves system robustness, avoiding false triggering caused by untimely gaze shifts or the inability to change commands mid-process, as seen in synchronous systems.

[0025] In a first aspect, embodiments of the present invention provide a multimodal asynchronous brain-computer interface method for teleoperation tasks, see [link to previous document]. Figure 1 Before performing the interaction method, the operator wears an EEG acquisition device to collect calibration data and construct the operator's EEG signal classification and decoding model and gaze error correction model. Current gaze detection technologies are limited by recognition algorithms or hardware performance, often resulting in errors influenced by random errors, head movements, and blinking. Current hybrid brain-computer interfaces combining eye movements and EEG often only use eye movement data for coarse region selection, rather than using it to improve overall performance. Regarding gaze detection errors, head movements and blinking generally cause significant and irregular shifts in recognition results, and such results should be considered invalid. Random errors, on the other hand, tend to exhibit a normal distribution, symmetrically distributed around a center with higher frequency closer to the center; these results should be considered valid but with inherent errors. This invention proposes a method for correcting gaze detection data based on probability estimation. This method improves gaze recognition performance by eliminating invalid data, estimating and correcting errors in valid data, and fusing eye movement (gaze detection) data with EEG signals at a decision level.

[0026] As one possible implementation, the line-of-sight error correction model includes a fixed offset and a random error distribution for line-of-sight estimation, see [link to relevant documentation]. Figure 2 The operator's line-of-sight error correction model is constructed using the following method: S00. Configuration Each target point is presented to the operator one by one, and the coordinates of the operator's gaze at each target point are collected to obtain... Coordinates of the line of sight; S01. Subtract the actual coordinates of the target point from the coordinates of each line of sight landing point to obtain the error data of all line of sight landing point coordinates relative to the actual coordinates of the target point; S02. Calculate the mean and standard deviation of the error data; S03. Calculate the calibration score of the error data based on the average value and standard deviation of the error data, configure the preset error value, remove the error data with a calibration score greater than the preset error value, and obtain a new dataset; S04. Repeat steps S02 to S03 for the new dataset until no more data with calibration scores greater than the preset error value are found. The dataset obtained at this point is the valid error dataset. S05. Construct a two-dimensional Gaussian probability density function based on the mean and standard deviation of the effective error data set. Its expected value is the fixed offset of the line-of-sight estimation. Subtract the fixed offset of the line-of-sight estimation from the two-dimensional Gaussian probability density function to obtain the random error distribution of the line-of-sight estimation.

[0027] As an example of constructing an operator's line-of-sight error correction model, step 1: configuration Each target point is presented to the operator one by one. There are 1 target points, and the presentation time for each target point is 1. Each second, the operator needs to focus on the currently displayed target point. During this time, with The sampling rate continuously collects the coordinates of the operator's gaze point at each target point, obtaining... The coordinates of the line of sight landing point are represented as follows: ,in, ; ; ; , The coordinates of the point where the line of sight falls. Axial components, The coordinates of the point where the line of sight falls. Axial components. Since current visual brain-computer interface systems also require the acquisition of EEG data of the operator's gaze at the target stimulus for modeling before use, in practical applications, this step can be performed simultaneously with the step of acquiring offline EEG modeling (calibration) data.

[0028] Step 2: Subtract the corresponding actual target point coordinates from the coordinates of each line-of-sight point to obtain the error data of all line-of-sight point coordinates relative to the actual target point coordinates. : in, This represents the actual coordinates of the target point.

[0029] Step 3: Transfer the error data remove , Expanding the dimensions outside the component and reducing them to one dimension, we obtain... , , ; Step 4: Calculate the current error data average and standard deviation , , This represents the number of iterations, initially set to 0. The set of natural numbers; Step 5: Calculate the calibration score of the error data based on the Z-score method. The preset error value is 3, meaning that the components on any coordinate axis... Points with a value greater than 3 are considered invalid coordinate points. After removing these invalid coordinate points, a new dataset is constructed from the remaining data. The amount of data in this dataset is , This indicates the number of invalid coordinate points removed in this round.

[0030] Repeat steps 4 and 5 until no more points are removed. The resulting dataset is the valid error dataset. Let's assume this valid error dataset is... This can be considered as the distribution of gaze detection results unaffected by head movements and blinking. Since the random error has a two-dimensional Gaussian component, its average value... and Construct a two-dimensional Gaussian probability density function: , It is the set of real numbers; The above two-dimensional Gaussian probability density function is denoted as ,in, , , , , .

[0031] Mathematical expectation of a two-dimensional Gaussian probability density function The fixed offset for line-of-sight estimation is given by the two-dimensional Gaussian probability density function. Subtracting the fixed offset from the line-of-sight estimation gives the random error distribution of the line-of-sight estimation.

[0032] See Figure 1 Perform the multimodal asynchronous brain-computer interface method as follows: S10. The control commands and the feedback screen of the controlled object are presented to the operator at the same time. The feedback screen is located in the center of the human-computer interaction interface and is updated in real time. The control commands are presented as stimulation blocks around the feedback screen. This invention proposes a human-computer interaction interface with a surround layout of control commands and the feedback screen of the controlled object. The feedback screen is located in the center of the interface, and the control commands are distributed around the feedback screen in the form of stimulus blocks according to different functions. Considering the flexibility and precision of brain-controlled telemanipulation tasks, this invention designs an operator-friendly multi-instruction set brain-computer interface paradigm, which can be used to meet the needs of manipulating external controlled objects with different movement commands in different directions (degrees of freedom) and at different paces. The small area that presents each command content and its corresponding visual stimulus to the operator is called a stimulus block, and each stimulus block encodes a different visual stimulus.

[0033] As an example, see Figure 3 and Figure 4 The paper demonstrates the interface layout of the human-computer interaction interface with a wraparound layout proposed in this invention when operating two remotely controlled objects with different degrees of freedom: unmanned vehicles and drones. Figure 3The interface shown displays the camera feed from the autonomous vehicle in the center, surrounded by a control command area containing 32 stimulus blocks. The control commands use a similar layout logic to operating a real car. The left side displays braking commands of varying lengths, corresponding to different deceleration effects when the brake pedal is pressed with varying force. The right side displays accelerator commands of varying lengths, corresponding to different acceleration effects when the accelerator pedal is pressed with varying force. The upper left and right sides display directional commands, corresponding to small turns of the steering wheel to the left and right for different durations, with the steering wheel automatically returning to center. The lower right side displays gear commands simulating an automatic transmission car, including Park (P), Drive (D), Reverse (R), and Neutral (N). The lower left side displays auxiliary function commands, in this example, for autonomous driving, emergency braking, turning on headlights, and honking the horn.

[0034] Figure 4 The interface shown displays the drone's camera feed in the center, surrounded by a control command area containing 48 stimulus blocks. The control commands use a similar layout logic to those used for operating the drone. The second column on the left contains ascent / descent commands of varying lengths, corresponding to ascent and descent speeds. The second column on the right contains forward / backward commands of varying lengths, corresponding to horizontal flight at different speeds. The left and right sides of the upper area contain turning commands, corresponding to turning the drone to the left and right by different amounts. The left and right sides of the lower area contain side-flying commands, corresponding to horizontal flight at different speeds to the left and right. The first column on the left and right sides contains diagonal horizontal flight commands, corresponding to horizontal flight at different speeds to the left front / back and right front / back, respectively.

[0035] It should be noted that using single-modal visual stimulus encoding (such as steady-state visual evoked potential, SSVEP) or mixed-modal visual encoding (such as SSVEP-P300 mixed visual stimulus encoding, etc.) to visually encode functional instructions does not affect the implementation of this method, and the number and size of stimulus blocks do not affect the implementation of this invention.

[0036] By designing a visual brain-computer interface with a wraparound layout, a dual-interface wraparound layout panel of "image feedback - control command area" is adopted. The image feedback is placed in the center of the interface, and the visual stimulation commands are arranged around the image feedback in a wraparound manner according to their functions. While maximizing the use of screen space, a mapping relationship between command functions and spatial positions is established, which is convenient for the operator to observe and operate.

[0037] S20. Switch to observation mode. At this time, the stimulus block does not flash. The operator looks at the feedback screen or stimulus block according to the control requirements. The coordinates of the operator's gaze point are detected in real time, and the coordinates of the gaze point are corrected to obtain the effective gaze estimation coordinates in the observation state. As one possible implementation, correcting the coordinates of the line of sight to obtain the effective line-of-sight estimation coordinates in the observation state includes the following sub-steps: S200. Real-time detection of the operator's line of sight coordinates to obtain a set of line of sight coordinates per unit time. S201. Based on the random error distribution of line-of-sight estimation, outliers in the set of line-of-sight coordinates are removed to obtain the effective set of line-of-sight coordinates; As one possible implementation, see Figure 5 S201 includes the following sub-steps: S2010. Calculate the average coordinates of the set of coordinates of the points where the line of sight falls per unit time; S2011. Calculate the two-dimensional Gaussian probability density function. Shaft standard deviation and Shaft standard deviation; S2012. Based on coordinate average value, Shaft standard deviation and The standard deviation of the axis is used to calculate the coordinates of each line of sight. Shaft calibration fraction and Shaft calibration score, obtain Axis calibration fraction set and Axis calibration fraction set; S2013. Elimination based on the three-standard-deviation principle Axis calibration fraction set and Outliers in the axis calibration fraction set are used to obtain a new set of line-of-sight coordinates. S2014. Repeat steps S2010 to S2013 for the new set of line-of-sight coordinates until no more outliers need to be removed. The set of line-of-sight coordinates obtained at this time is the valid set of line-of-sight coordinates.

[0038] As an example of obtaining a set of effective line-of-sight coordinates, the current stimulus time period is first obtained. The set of coordinates of the point where the inner line of sight falls , , , ; Step 1: Calculate the average coordinates of the set: Step 2: Based on the average value , Shaft standard deviation and Shaft standard deviation Calculate the coordinates of each line of sight point. Shaft calibration fraction and Shaft calibration score, obtain Axis calibration fraction set and Axis calibration fraction set : Step 3: Eliminate candidates based on the three-standard-deviation rule. Axis calibration fraction set and Outliers in the axis calibration fraction set are used to obtain a new set of line-of-sight coordinates. The number of data in this set is ,in This indicates the number of invalid points removed in this round.

[0039] Repeat steps one through three until no more points are removed.

[0040] S202. Calculate the average coordinates of the set of effective line-of-sight coordinates, and subtract the fixed offset of the line-of-sight estimation from the average coordinates to obtain the estimated coordinates of the effective line-of-sight in the observation state.

[0041] It should be noted that using other methods to remove outliers, including Z-score methods and IQR (interquartile range) methods with different thresholds, will not affect the implementation of this method.

[0042] S30. Determine the estimated coordinates of the effective gaze in the observation state. If it is located in the return screen area, return to step S20. If it is located in the control command area, switch to control mode. The operator looks at the stimulus block to be selected. The flashing of the stimulus block induces specific EEG signals in the operator. Detect and record the coordinates of the operator's gaze point during the flashing of the stimulus block. After the stimulus block stops flashing, execute steps S40 to S60. S40. Correct the coordinates of the operator's gaze point during the flashing of the stimulus block to obtain the estimated coordinates of the effective gaze in the task state. Determine the alternative control commands based on the estimated coordinates of the effective gaze in the task state and set a weight for each alternative control command. Extract the operator's EEG signal features and perform pattern recognition on the EEG signal features based on the EEG signal classification and decoding model to obtain the classification coefficients. In step S40, the coordinates of the operator's gaze point during the flashing of the stimulus block are corrected to obtain the estimated coordinates of the effective gaze in the task state. The correction method used is the same as in step S20. For the specific correction process, please refer to the sub-steps of step S20, which will not be repeated here.

[0043] As one possible implementation, the following method is used to determine the candidate control commands based on the effective line-of-sight estimation coordinates in the task state: a selection window is drawn with the effective line-of-sight estimation coordinates in the task state as the center, and the control commands corresponding to all stimulus blocks whose center points are located within the selection window are determined as candidate control commands.

[0044] As one possible implementation, the selection window is drawn centered on the effective line-of-sight estimation coordinates in the task state, including: Using the effective line-of-sight estimation coordinates of the task state as the center of the ellipse, with times Shaft standard deviation and times The standard deviation of the axes is two semi-axes; draw an ellipse selection window; or, with the effective line-of-sight estimated coordinates of the task state as the center, ... times Shaft standard deviation or times Draw a circular selection window with the axis standard deviation as the radius; or, center the selection window on the estimated coordinates of the effective line of sight in the task state. times Shaft standard deviation or times Using the standard deviation of the axis as the side length, draw a square selection window; or, using the effective line-of-sight estimation coordinates of the task state as the center, with... times Shaft standard deviation and times The standard deviation of the axes is defined by two sides; draw a rectangular selection window. , It is the set of real numbers.

[0045] As an example, the effective line-of-sight estimation coordinates in the task state are: ,by Centered on the ellipse, with and An elliptical selection window is created for the two semi-axes. The standard equation of this selection window is: Then, all instruction blocks whose center point is inside the ellipse are selected as candidate instructions. The specific implementation method is as follows: Iterate through all control commands, and for any stimulus block corresponding to a control command, calculate its center coordinates one by one according to the expression in the selection window. Whether it is within the selection window; if the center point coordinates of the instruction are within the elliptical selection window, i.e. If so, then include it in the alternative instruction set C.

[0046] It should be noted that using circles, rectangles, or squares instead of ellipses to draw the selection window, or replacing the method of filtering candidate instructions based on the selection window, such as requiring the selection window to completely cover the entire instruction block to be selected, or allowing the instruction block to be selected as long as part of it is within the selection window, does not affect the implementation of this invention.

[0047] S50. Calculate the fusion decision coefficient of each candidate control command by combining the classification coefficient and the weight of the candidate control command, and output the control command with the highest fusion decision coefficient to the controlled object; As one possible implementation, the fusion decision coefficient for each candidate control command is calculated by combining the classification coefficient and the weights of the candidate control commands: S500. Calculate the probability of the stimulus block coverage area corresponding to each candidate control instruction based on the two-dimensional Gaussian probability density function, and set a weight for each candidate control instruction based on the probability distribution of the candidate control instructions; or, calculate the distance of each candidate control instruction from the center of the selection window, and set a weight for each candidate control instruction based on the distance between each candidate control instruction and the center of the selection window. As an example, when assigning weights to candidate control commands based on probability distribution, the control command with a higher probability has a greater weight; when assigning weights to candidate control commands based on distance, the candidate control command with a smaller distance from the center of the selection window has a greater weight. If the number of commands in the candidate command set is 0, then no weight needs to be assigned.

[0048] S501. Multiply the classification coefficients by the weights of each candidate control command to obtain the fusion decision coefficients for each candidate control command.

[0049] S60. The controlled object responds to the control command and returns to the execution step S20.

[0050] This invention employs an implicit mode switching method, which does not require the introduction of an additional gaze switch or the specification of a specific gaze area and can respond at any time. It solves the problem that synchronous BCI is not flexible enough and has difficulty responding to sudden operations. Operators can naturally and autonomously switch the system's control / observation mode according to the current control needs without performing additional operations. This can effectively avoid false triggering caused by factors such as the decision time exceeding the fixed time interval between two rounds of operation in synchronous systems, thereby improving the system's robustness.

[0051] Secondly, embodiments of the present invention provide a multimodal asynchronous brain-computer interface system for teleoperation tasks, comprising: The calibration module is used to build a classification and decoding model of the operator's EEG signals and a model for correcting gaze errors. The human-computer interaction module presents the control commands and the feedback screen of the controlled object to the operator at the same time. The feedback screen is located in the center of the human-computer interaction interface and is updated in real time. The control commands are presented as stimulation blocks around the feedback screen. The gaze detection module, in observation mode, is used to detect the coordinates of the operator's gaze point in real time, correct the gaze point coordinates, and obtain the effective gaze estimation coordinates in observation mode; in control mode, it is used to detect and record the coordinates of the operator's gaze point during the flashing of the stimulus block, correct the coordinates of the operator's gaze point during the flashing of the stimulus block, and obtain the effective gaze estimation coordinates in task mode. The EEG data acquisition module is used to collect the operator's EEG signals during calibration and operation. The instruction recognition and decision-making module determines candidate control instructions based on the effective line-of-sight estimation coordinates in the task state and sets a weight for each candidate control instruction; it extracts the operator's EEG signal features, performs pattern recognition on the EEG signal features to obtain classification coefficients; and it calculates the fusion decision coefficient for each candidate control instruction by combining the classification coefficients and the weights of the candidate control instructions. The remote operation control module is used to output control commands obtained from the fusion decision to the controlled object and to acquire the images returned by the controlled object. It also includes an asynchronous control module that switches between control and observation modes based on the estimated coordinate position of the effective line of sight in the observation state and the status flags sent by the control commands.

[0052] In practical implementation, the human-computer interaction module, gaze detection module, command recognition and decision-making module, remote operation control module, and asynchronous control module are all deployed in a host computer. There can be one or multiple host computers. When using multiple host computers, they can be deployed functionally on different devices within the same network, or some processing modules can be deployed in the cloud and communicate with other modules via a wireless network. The controlled object can be any type of unmanned equipment suitable for remote operation scenarios, such as drones, unmanned vehicles, robots, or robotic arms.

[0053] Thirdly, the present invention provides a terminal including a processor and a communication interface coupled to the processor, the processor being used to run computer programs or instructions to implement the multimodal asynchronous brain-computer interaction method for teleoperation tasks provided in the first aspect.

[0054] The technical solution of the present invention will be further illustrated below with reference to specific embodiments.

[0055] This embodiment uses a drone and its flight control system, scalp electrodes, an EEG amplifier, and a host computer with a camera to form a remote operating system. The host computer presents the operator with a human-computer interaction interface containing control commands that include drone-transmitted images and visual stimulus codes. In this embodiment, the control commands are a 48-command system using a hybrid encoding of P300 and SSVEP. The layout of the human-computer interaction interface is as follows: Figure 4As shown. The operator can control a drone to fly in 4 degrees of freedom using this system. During operation, the operator can keep the drone hovering by looking at the transmitted screen, or control the drone's movement by looking at different control command blocks. If an operational error occurs during the viewing period, the operator can shift their gaze back to the transmitted screen to cancel the current output. After the visual stimulus ends, the system will not output any commands.

[0056] Before use, the operator wears scalp electrodes and is in a bright environment so that the camera can detect the operator's eye position. See also Figure 7 After the device is worn, the operator needs to perform calibration, which is done in a multi-threaded, parallel manner: EEG Acquisition Sub-thread: Responsible for continuously acquiring the operator's EEG signals. It synchronously acquires EEG signals, including those during the operator's gaze commands, via scalp electrodes. The acquired EEG signals undergo analog signal processing such as noise reduction and amplification, as well as A / D conversion, within the EEG amplifier. Finally, the acquired EEG signals are saved to the host computer. The gaze detection sub-thread is responsible for continuously detecting the coordinates of the operator's gaze point. It converts the operator's facial information synchronously collected by the host computer's camera into gaze detection coordinates and saves them in the host computer. Main thread: Responsible for handling human-computer interaction, including the following steps during calibration: 1) Presenting visual stimuli: The system presents a human-computer interaction interface, including a schematic diagram of the drone's transmitted images and control commands, to the operator through the screen of the host computer. The system first prompts the operator to focus on the stimulus block through prompts. All stimulus blocks flash according to the coded visual stimuli, inducing the corresponding EEG signals in the operator. When the stimulation reaches the predetermined duration, the visual stimulation ends. The operator needs to focus on the corresponding stimulus block according to the system prompts. During the focusing process, the operator should remain focused and try to avoid head movement or blinking. After a 1-second rest, the human-computer interaction module prompts the operator for the next stimulus block.

[0057] Repeat the above process until all calibration data for all stimulus blocks have been collected.

[0058] 2) Establish a classification and decoding model for task-state EEG signals: Establish classification and decoding models for SSVEP and P300 based on the calibrated EEG data stored in the host computer, and save them to the host computer.

[0059] 3) Establish a line-of-sight estimation model based on a two-dimensional Gaussian distribution: Establish a line-of-sight estimation model based on the line-of-sight detection calibration data stored locally, and save it to the host computer.

[0060] Steps 2) and 3) in the calibration process can be run in parallel.

[0061] After calibration, the operator can use the system to control the drone. A single flight control task comprises multiple individual control subtasks. The system operates in a multi-threaded, parallel manner. EEG Acquisition Sub-thread: Responsible for continuously acquiring the operator's EEG signals. It synchronously acquires EEG signals, including those during the operator's gaze commands, via scalp electrodes. The acquired EEG signals undergo analog signal processing such as noise reduction and amplification, as well as A / D conversion, within the EEG amplifier. Finally, the acquired EEG signals are saved to the host computer. The gaze detection sub-thread is responsible for continuously detecting the coordinates of the operator's gaze point. It converts the operator's facial information synchronously collected by the host computer's camera into gaze detection coordinates and saves them in the host computer. The feedback video synchronization sub-thread is responsible for acquiring the drone's feedback video in real time. It obtains the drone's feedback video from the drone's flight control system in real time and displays it in the feedback video area at the center of the human-machine interface. Main thread: Responsible for handling core functions such as human-computer interaction, system status control, command recognition, and flight control. A single control subtask includes the following processes: 1) Determine the current drone flight status and select commands: The host computer screen presents the operator with a human-machine interface containing drone-transmitted images and control commands. The control commands do not flash. Based on the calibrated gaze estimation model, the gaze detection coordinates for the most recent second are corrected. The system state is adjusted in real time according to the corrected gaze estimation coordinates. If the coordinates are in the transmitted image area, the system remains in observation mode; if they are in the function command area, the system switches to control mode. The operator continuously watches the transmitted image and determines the operation to be performed. After determining the operation, the operator watches the corresponding command stimulus block.

[0062] 2) Presenting visual stimuli: The operator continuously gazes at the stimulus block to be selected. At this time, the control instruction flashes according to the coded visual stimulus, inducing the corresponding EEG signal of the operator. When the stimulus reaches the predetermined duration, the visual stimulus ends.

[0063] 3) Constructing a command selection window: The gaze detection coordinates collected during the visual stimulus process are corrected to obtain effective gaze estimation coordinates. If the coordinates are in the feedback screen area, a cancellation command is sent to the host computer, and the host computer switches the system to observation mode. If the coordinates are in the control command area, a command selection window is constructed based on the effective gaze estimation coordinates, and a set of alternative commands is constructed based on the center coordinates of the command block. Weights are set based on the distance from the center of the selection window. At this time, the operator looks at the drone's feedback screen and waits for the command execution status.

[0064] 4) Identify EEG signals: Identify the EEG signals collected during the visual stimulation process and save the decision coefficients of all instructions.

[0065] 5) Fusion decision: Select the candidate instruction with the highest weight from the candidate instruction set as the output instruction based on the saved decision coefficients.

[0066] 6) Responding to control commands and providing feedback: The host computer briefly displays a prompt below the corresponding stimulus block based on the received output command to provide feedback to the operator. The output command controls the drone to make corresponding movements through the drone flight control system. The operator observes the prompt appearing on the output command stimulus block and the change in the transmitted image, and determines the subsequent operation to be performed.

[0067] This concludes a single control subtask process. The operator and system need to repeat the above steps until the flight control task is completed. Once the flight control task is completed, all threads terminate.

[0068] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the description of the drawings, in carrying out the claimed invention. In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several of the functions listed in the specification. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0069] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the invention and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications fall within the scope of the invention and its equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A multimodal asynchronous brain-computer interface method for teleoperation tasks, characterized in that, Before performing the interactive method, the operator wears an EEG acquisition device to collect calibration data and construct the operator's EEG signal classification and decoding model and gaze error correction model; Perform the multimodal asynchronous brain-computer interface method as follows: S10. The control commands and the feedback screen of the controlled object are presented to the operator at the same time. The feedback screen is located in the center of the human-computer interaction interface and is updated in real time. The control commands are presented as stimulation blocks around the feedback screen. S20. Switch to observation mode. At this time, the stimulus block does not flash. The operator looks at the feedback screen or stimulus block according to the control requirements. The coordinates of the operator's gaze point are detected in real time, and the coordinates of the gaze point are corrected to obtain the effective gaze estimation coordinates in the observation state. S30. Determine the estimated coordinates of the effective gaze in the observation state. If it is located in the return screen area, return to step S20. If it is located in the control command area, switch to control mode. The operator looks at the stimulus block to be selected. The flashing of the stimulus block induces specific EEG signals in the operator. Detect and record the coordinates of the operator's gaze point during the flashing of the stimulus block. After the stimulus block stops flashing, execute steps S40 to S60. S40. Correct the coordinates of the operator's line of sight during the flashing of the stimulus block to obtain the estimated coordinates of the effective line of sight in the task state. Determine the range of alternative control commands based on the estimated coordinates of the effective line of sight in the task state and set a weight for each alternative control command. Extract the operator's EEG signal features, perform pattern recognition on the EEG signal features based on the EEG signal classification and decoding model, and obtain classification coefficients; S50. Calculate the fusion decision coefficient of each candidate control command by combining the classification coefficient and the weight of the candidate control command, and output the control command with the highest fusion decision coefficient to the controlled object; S60. The controlled object responds to the control command and returns to the execution step S20.

2. The multimodal asynchronous brain-computer interface method for teleoperation tasks according to claim 1, characterized in that, The line-of-sight error correction model includes a fixed offset and a random error distribution for line-of-sight estimation. The operator's line-of-sight error correction model is constructed using the following method: S00. Configuration Each target point is presented to the operator one by one, and the coordinates of the operator's gaze at each target point are collected to obtain... Coordinates of the line of sight; S01. Subtract the actual coordinates of the target point from the coordinates of each line of sight landing point to obtain the error data of all line of sight landing point coordinates relative to the actual coordinates of the target point; S02. Calculate the mean and standard deviation of the error data; S03. Calculate the calibration score of the error data based on the average value and standard deviation of the error data, configure the preset error value, remove the error data with a calibration score greater than the preset error value, and obtain a new dataset; S04. Repeat steps S02 to S03 for the new dataset until no more data with calibration scores greater than the preset error value are found. The dataset obtained at this point is the valid error dataset. S05. Construct a two-dimensional Gaussian probability density function based on the mean and standard deviation of the effective error data set. Its expected value is the fixed offset of the line-of-sight estimation. Subtract the fixed offset of the line-of-sight estimation from the two-dimensional Gaussian probability density function to obtain the random error distribution of the line-of-sight estimation.

3. The multimodal asynchronous brain-computer interface method for teleoperation tasks according to claim 2, characterized in that, Correcting the coordinates of the line of sight point to obtain the effective line of sight estimation coordinates in the observation state includes the following sub-steps: S200. Real-time detection of the operator's line of sight coordinates to obtain a set of line of sight coordinates per unit time. S201. Based on the random error distribution of line-of-sight estimation, outliers in the set of line-of-sight coordinates are removed to obtain the effective set of line-of-sight coordinates; S202. Calculate the average coordinates of the set of effective line-of-sight coordinates, and subtract the fixed offset of the line-of-sight estimation from the average coordinates to obtain the estimated coordinates of the effective line-of-sight in the observation state.

4. The multimodal asynchronous brain-computer interface method for teleoperation tasks according to claim 3, characterized in that, S201 includes the following sub-steps: S2010. Calculate the average coordinates of the set of coordinates of the points where the line of sight falls per unit time; S2011. Calculate the two-dimensional Gaussian probability density function. Shaft standard deviation and Shaft standard deviation; S2012. Based on coordinate average value, Shaft standard deviation and The standard deviation of the axis is used to calculate the coordinates of each line of sight. Shaft calibration fraction and Shaft calibration score, obtain Axis calibration fraction set and Axis calibration fraction set; S2013. Elimination based on the three-standard-deviation principle Axis calibration fraction set and Outliers in the axis calibration fraction set are used to obtain a new set of line-of-sight coordinates. S2014. Repeat steps S2010 to S2013 for the new set of line-of-sight coordinates until no more outliers need to be removed. The set of line-of-sight coordinates obtained at this time is the valid set of line-of-sight coordinates.

5. The multimodal asynchronous brain-computer interface method for teleoperation tasks according to claim 2, characterized in that, The following method is used to determine alternative control commands based on the effective line-of-sight estimation coordinates in the task state: A selection window is drawn with the effective line-of-sight estimation coordinates in the task state as the center. All control commands corresponding to stimulus blocks whose center points are located within the selection window are determined as alternative control commands.

6. The multimodal asynchronous brain-computer interface method for teleoperation tasks according to claim 5, characterized in that, A selection window is drawn centered on the effective line-of-sight estimation coordinates in the task state, including: Using the effective line-of-sight estimation coordinates of the task state as the center of the ellipse, with times Shaft standard deviation and times The standard deviation of the axes is two semi-axes; draw an ellipse selection window; or, with the effective line-of-sight estimated coordinates of the task state as the center, ... times Shaft standard deviation or times Draw a circular selection window with the axis standard deviation as the radius; or, center the selection window on the estimated coordinates of the effective line of sight in the task state. times Shaft standard deviation or times Using the standard deviation of the axis as the side length, draw a square selection window; or, using the effective line-of-sight estimation coordinates of the task state as the center, with... times Shaft standard deviation and times The standard deviation of the axes is defined by two sides; draw a rectangular selection window. , It is the set of real numbers.

7. The multimodal asynchronous brain-computer interface method for teleoperation tasks according to claim 5, characterized in that, The fusion decision coefficient for each candidate control command is calculated using the following method, combining the classification coefficient and the weights of the candidate control commands: S500. Calculate the probability of the stimulus block coverage area corresponding to each candidate control instruction based on the two-dimensional Gaussian probability density function, and set a weight for each candidate control instruction based on the probability distribution of the candidate control instructions; or, calculate the distance of each candidate control instruction from the center of the selection window, and set a weight for each candidate control instruction based on the distance between each candidate control instruction and the center of the selection window. S501. Multiply the classification coefficients by the weights of each candidate control command to obtain the fusion decision coefficients for each candidate control command.

8. A multimodal asynchronous brain-computer interface system for teleoperation tasks, characterized in that, include: The calibration module is used to build an EEG signal classification and decoding model and an operator's gaze error correction model. The human-computer interaction module presents the control commands and the feedback screen of the controlled object to the operator at the same time. The feedback screen is located in the center of the human-computer interaction interface and is updated in real time. The control commands are presented as stimulation blocks around the feedback screen. The gaze detection module, in observation mode, is used to detect the coordinates of the operator's gaze point in real time, correct the gaze point coordinates, and obtain the effective gaze estimation coordinates in observation mode; in control mode, it is used to detect and record the coordinates of the operator's gaze point during the flashing of the stimulus block, correct the coordinates of the operator's gaze point during the flashing of the stimulus block, and obtain the effective gaze estimation coordinates in task mode. The EEG data acquisition module is used to collect the operator's EEG signals during calibration and operation. The instruction recognition and decision-making module determines candidate control instructions based on the effective line-of-sight estimation coordinates in the task state and sets a weight for each candidate control instruction; Extract the operator's EEG signal features, perform pattern recognition on the EEG signal features, and obtain classification coefficients; The fusion decision coefficient for each candidate control instruction is calculated by combining the classification coefficient and the weight of the candidate control instruction. The remote operation control module is used to output control commands obtained from the fusion decision to the controlled object and to acquire the images returned by the controlled object. It also includes an asynchronous control module that switches between control and observation modes based on the estimated coordinate position of the effective line of sight in the observation state and the status flags sent by the control commands.

9. A terminal comprising a processor and a communication interface coupled to the processor, the processor being configured to run computer programs or instructions to implement the multimodal asynchronous brain-computer interface method for teleoperation tasks as described in any one of claims 1 to 7.

Images