A hybrid brain-computer interaction system and method for air-ground collaborative robot control

Abstract

The application discloses a kind of hybrid brain-computer interaction systems and methods for air-ground collaborative robot control, belong to brain-computer fusion, human-computer interaction science field, system includes: electroencephalogram acquisition and analysis module, eye tracking module, fine control subsystem, video point navigation subsystem.The application can realize the hands-free operation behavior of operator to ground robot by the human-computer interaction system based on hybrid brain-computer interface.

Description

Technical Field

[0001] This invention belongs to the field of brain-computer interface and human-computer interaction science and technology, and in particular relates to a hybrid brain-computer interaction system and method for controlling air-ground collaborative robots. Background Technology

[0002] In recent years, air-ground collaborative systems have played a vital role in transportation, search and rescue missions due to their high flexibility and speed. Compared with traditional single-robot systems, air-ground collaborative systems integrate the complementary advantages of aerial and ground robots, significantly improving the overall performance of the system.

[0003] However, current research on automatic control in air-to-ground collaborative systems reveals that machine intelligence is insufficient for emergency control in abnormal situations and independent decision-making under complex tasks. Therefore, human intervention in control is indispensable. In existing robotic systems, operator control is geared towards a single object; the operator can only control a single robot to achieve a goal. If a single operator could control multiple robots collaboratively, it would not only improve operator efficiency but also prevent errors caused by incomplete communication between operators. Given the complex control requirements of air-to-ground collaborative systems, a single operator cannot achieve complete control through manual means. Therefore, it is necessary to explore other efficient control methods and design reasonable interaction mechanisms to further improve the control accuracy of air-to-ground collaborative systems. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a hybrid brain-computer interface system and method for controlling air-ground collaborative robots, thereby resolving the issues present in the prior art.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a hybrid brain-computer interface system for controlling air-ground collaborative robots, comprising:

[0006] The EEG acquisition and analysis module is used to present visual stimuli to the operator, acquire the operator's EEG signals, analyze the EEG signals, and obtain EEG analysis commands.

[0007] An eye-tracking module is used to collect the operator's eye movement signals and obtain the eye movement coordinates corresponding to the eye movement signals;

[0008] A fine control subsystem is used to perform decision-level fusion of the EEG analysis commands and the eye movement coordinates to generate control commands for controlling the ground robot.

[0009] The video point selection and navigation subsystem is used to receive the returned video from the ground robot and the aerial robot, construct a navigation interaction based on the returned video according to the control command, generate navigation target points to be sent to the ground robot, and realize global navigation of the ground robot.

[0010] Preferably, the EEG acquisition and analysis module includes:

[0011] The visual stimulus unit is used to present several visual stimuli of different frequencies, and each visual stimulus corresponds to a command for fine control of the ground robot.

[0012] A signal acquisition unit is used to acquire the electroencephalogram (EEG) signals of the operator.

[0013] The signal parsing unit is used to decode the EEG signal using the filter bank canonical correlation analysis (FBCCA) method to obtain the EEG parsing command.

[0014] Preferably, the fine control subsystem includes:

[0015] A statistical analysis unit is used to perform statistical analysis on the eye movement coordinates using a sliding window with the same step size as the EEG signal decoding;

[0016] The control command output module is used to analyze the EEG signal decoding results and the eye movement coordinates, perform decision-level fusion, and output fine control commands.

[0017] Preferably, the fine control commands include: start / stop, accelerate, decelerate, turn left, and turn right.

[0018] Preferably, the decision-level fusion includes:

[0019] When the previous control command was non-control, it is determined that the current state is non-control. At this time, only the eye movement control area discrimination and eye movement tracking 0.25s window length command output are performed.

[0020] When the last control command was one of the five control commands, the current state is determined to be under control. At this time, the eye movement control area, the eye movement 0.25s window length, the eye movement 0.5s window length, and the SSVEP correlation coefficient will be comprehensively considered.

[0021] Preferably, the video point selection navigation subsystem includes:

[0022] The coordinate recording unit is used to decode the EEG's intention to confirm the selection of points and record the eye movement coordinates at that moment;

[0023] The coordinate calculation unit is used to collect the camera intrinsic parameters of the ground robot and the aerial robot, and calculate the ground coordinates pointed to by the video coordinate points.

[0024] The coordinate mapping unit is used to calculate the relative coordinate relationship between the aerial robot and the ground robot, and to map the ground coordinates to the target point coordinates in the ground robot coordinate system;

[0025] The navigation unit is used for global navigation using the ground robot's navigation system.

[0026] Preferably, the human-computer interaction system includes:

[0027] A fine-grained control interface is used to display the returned video, the visual stimuli, and command feedback indicators;

[0028] The video point selection navigation interface is used to display the returned video, navigation point visualization indicators, and navigation status indicators;

[0029] The function switching interface is used to provide functions for switching control modes, video sources, and interface layouts.

[0030] Status indicator component, used to display system connection status and task status.

[0031] In a second aspect, the present invention provides a hybrid brain-computer interface method for controlling air-ground collaborative robots, used to implement the system described in the first aspect, comprising the following steps:

[0032] Visual stimuli are presented to the operator through a display device, and the operator's brainwave signals are collected through an EEG acquisition device. The brainwave signals are then analyzed to obtain brainwave analysis commands.

[0033] The operator's eye movement signals are collected using an eye tracker, and the corresponding eye movement coordinates are obtained.

[0034] A decision-level fusion approach is adopted to fuse the EEG analysis commands and the eye movement coordinates to generate control commands for controlling the ground robot;

[0035] The system receives video feeds from ground and aerial robots, constructs navigation interactions based on the video feeds according to control commands, generates navigation target points to be sent to the ground robots, and achieves global navigation for the ground robots.

[0036] Compared with the prior art, the present invention has the following advantages and technical effects:

[0037] This invention provides a hybrid brain-computer interface system for controlling air-ground collaborative robots, comprising: an EEG acquisition and analysis module for presenting visual stimuli to the operator, acquiring the operator's EEG signals, analyzing the EEG signals, and obtaining EEG analysis commands; an eye-tracking module for acquiring the operator's eye-tracking signals and obtaining the eye-tracking coordinates corresponding to the eye-tracking signals; a fine control subsystem for performing decision-level fusion of the EEG analysis commands and the eye-tracking coordinates to generate control commands for controlling the ground robot; and a video point selection and navigation subsystem for receiving video feedback from the ground robot and the air robot, constructing navigation interaction based on the video feedback according to the control commands, generating navigation target points to be sent to the ground robot, and realizing global navigation of the ground robot.

[0038] Through the technical solution of this invention, an operator, based on a hybrid brain-computer interface-assisted air-to-ground collaborative human-machine interaction system, can achieve hands-free operation of a ground robot. A global navigation mode provides the operator with higher-level control capabilities, reducing control load; pose control allows the operator to adjust the end effector pose of the ground robot, achieving precise control. Combined with hands-free remote control of an aerial robot, complete control of the air-to-ground collaborative unmanned system is achieved. The air-to-ground collaborative unmanned system with hybrid brain-computer interface-assisted control described in this invention can be used in scenarios such as transportation, search and rescue missions, improving operational efficiency and reducing labor costs. It is also applicable to other scenarios requiring hands-free control. Attached Figure Description

[0039] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0040] Figure 1 This is a system schematic diagram according to an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of decision-level fusion according to an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the human-computer interaction interface according to an embodiment of the present invention. Detailed Implementation

[0043] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0044] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0045] Example 1

[0046] like Figure 1 As shown, this embodiment provides a hybrid brain-computer interface system for controlling air-ground collaborative robots, which enables efficient single-person control of air-ground collaborative unmanned systems. The operator can achieve global navigation and pose control of the ground robot without occupying both hands. Combined with hand-controlled drones, it meets the control requirements of air-ground collaborative unmanned systems while reducing the operator's workload.

[0047] The system specifically includes:

[0048] The EEG acquisition and analysis module is used to present visual stimuli to the operator, acquire the operator's EEG signals, analyze the EEG signals, and obtain EEG analysis commands.

[0049] Specifically, the EEG acquisition and analysis module collects and decodes the operator's EEG signals. The display device presents five visual stimuli, with corresponding commands used for precise control of the ground robot.

[0050] In this embodiment, the EEG acquisition system uses an EEG cap to acquire EEG signals from eight channels: PO5, PO3, PO2, PO4, PO6, O1, O2, and O2, at a frequency of 1000 Hz. Using the Steady-State Visual Evoked Potential (SSVEP) paradigm, visual flashing stimuli for five commands are provided through augmented reality glasses at frequencies of 9 Hz, 10 Hz, 11 Hz, 12 Hz, and 13 Hz, achieving the output of five-category commands. Signal analysis employs Filter Bank Canonical Correlation Analysis (FBCCA) for classification, using a 2-second window and a 0.25-second step size for command analysis. The weighted average of the correlation coefficients for each command across all filter bands is output as the command's correlation coefficient. Simultaneously, the EEG signals are also analyzed to interpret peak trigger intentions, enabling confirmation of navigation target points.

[0051] An eye-tracking module is used to collect the operator's eye movement signals and obtain the eye movement coordinates corresponding to the eye movement signals;

[0052] Specifically, the eye-tracking module obtains eye-tracking data through an eye tracker on the augmented reality eye, capturing the operator's gaze direction at a frequency of 60Hz. For the same five command outputs as SSVEP, the eye-tracking data provides a classification of five commands and non-control outputs. If the coordinates match the area of ​​the flashing stimulus, the command is output; otherwise, a non-control output is given. Statistical analysis of the eye-tracking data is performed using window lengths of 0.25s and 0.5s. For navigation target point selection and gaze buttons on the human-computer interaction interface, the raw eye-tracking data is smoothed using a moving average method before being input into the video point selection navigation subsystem and the human-computer interaction subsystem for corresponding eye-tracking function processing.

[0053] A fine control subsystem is used to perform decision-level fusion of the EEG analysis commands and the eye movement coordinates to generate control commands for controlling the ground robot.

[0054] Specifically, the precision control subsystem integrates EEG and eye-tracking signals to output five categories of commands, including non-control commands, to achieve pose control of the ground robot. The five categories of commands include: acceleration, deceleration, left turn, right turn, and emergency stop. The non-control commands maintain the current linear velocity while setting the steering angular velocity to zero.

[0055] In this embodiment, the signals received by the fine control subsystem include: the correlation coefficient matrix from the EEG acquisition and analysis module, the number of command hits and current coordinates for 0.25s and 0.5s window lengths from the eye-tracking module. Considering the characteristics of the air-to-ground collaborative scenario, the fine control subsystem uses a hybrid brain-computer interface to asynchronously fuse EEG and eye-tracking modalities at the decision-making level. The data used in this fusion scheme includes the command with the highest correlation coefficient obtained from the SSVEP brain-computer interface, the coordinates of the highest hit rate obtained from the 0.25s and 0.5s window lengths of eye-tracking, and the current coordinates.

[0056] The steps to obtain commands include: performing EEG decoding using filter bank canonical correlation analysis, statistically analyzing eye movement coordinates using a sliding window with the same step size as the EEG decoding, and outputting fine-grained control commands based on the relevant data obtained from the EEG and eye movement processing methods using a decision-level fusion approach.

[0057] like Figure 2As shown, in the decision-level fusion method, the state of historical commands needs to be determined first. When the last control command was non-control, the current state is determined to be non-control, and only eye-tracking control area determination and eye-tracking 0.25s window length command output are performed. If the eye-tracking coordinates are outside the eye-tracking control area, non-control output is directly performed; if they are within the control area, the eye-tracking 0.25s window length command output is used as the command output to maintain or change the command. This scheme can mitigate the impact of eye-tracking data jitter on the continuity of the non-control state and utilize the rapid response characteristics of eye-tracking to enable users of the air-ground cooperative system to quickly switch from observation to control. When the last control command was one of the five control commands, the current state is determined to be control, and the eye-tracking control area, eye-tracking 0.25s window length, eye-tracking 0.5s window length, and SSVEP correlation coefficient are comprehensively considered. If the eye-tracking coordinates are outside the eye-tracking control area, non-control output is directly performed; if they are within the control area, the eye-tracking 0.25s window length data is considered. If the original command is retained, the result is accepted. If the command is changed or the output is uncontrolled, further verification is performed using the SSVEP result. If the SSVEP command output changes and is consistent with the output of the 0.25s eye-tracking window, or if the 0.25s eye-tracking window output is uncontrolled, the new output is accepted. If the SSVEP command retains the original command, it indicates that there may be eye-tracking data jitter, and the output of the 0.5s eye-tracking window will be used as the final fusion decision output.

[0058] This embodiment ensures the temporal stability of control command output when controlling the air-to-ground system, and switches between command and non-control modes with a relatively fast response speed. The commands output by the decision-level fusion model are the output of the fine-grained control subsystem, corresponding to five categories of control commands, thus achieving pose control of the ground robot.

[0059] The video point selection and navigation subsystem is used to receive the returned video from the ground robot and the aerial robot, construct a navigation interaction based on the returned video according to the control command, generate navigation target points to be sent to the ground robot, and realize global navigation of the ground robot.

[0060] Specifically, achieving global navigation for ground robots includes:

[0061] The intention to confirm the selection point is determined by EEG decoding, and the eye movement coordinates at this moment are recorded. The ground coordinates pointed to by the video coordinate points are calculated by the camera intrinsic parameters of the ground robot and the aerial robot. The ground coordinates are mapped to the target point coordinates in the ground robot coordinate system by the relative coordinate relationship between the aerial robot and the ground robot. Global navigation is then performed using the ground robot's navigation system.

[0062] In this embodiment, the video point selection subsystem selects points in the transmitted video using smoothed eye-tracking coordinates and confirms navigation behavior at these points using peak EEG signal triggering. This triggers a command to be sent to the air-ground collaborative unmanned system, transmitting the navigation intent. The signals received by the video point selection subsystem include: peak EEG signal triggering from the EEG acquisition and analysis module and smoothed eye-tracking coordinates from the eye-tracking module. When the operator gazes at the transmitted video area and triggers the navigation intent via EEG, the eye-tracking coordinates at that moment are recorded. These coordinates must correspond to the ground area in the video; otherwise, the input is rejected. After verification, the eye-tracking coordinates are converted to relative coordinates in a pixel coordinate system with the top-left corner as the origin. By transforming the coordinates of the transmitted video, the target point is converted into a navigation target point in the ground robot coordinate system.

[0063] Both the ground and aerial robots transmit 16:9 aspect ratio videos, but their coordinate transformations differ. For both types of transmitted videos, the cameras need to be calibrated beforehand to obtain the distortion-free intrinsic parameter matrix. and and transform relative coordinates pixel coordinates :

[0064] ;

[0065] in, The intrinsic parameter matrix of the camera after distortion removal. The pixel focal length in the x-direction. The pixel focal length in the y-direction. Let x be the x-coordinate of the camera's optical center in the pixel coordinate system. Let y be the y-coordinate of the camera's optical center in the pixel coordinate system. Let x be the pixel coordinate. The relative coordinate x was entered earlier. The number of pixels in the horizontal direction of the image. Let y be the pixel coordinate. The relative coordinate y was entered earlier. The number of pixels in the vertical direction of the image. It is a pixel coordinate vector.

[0066] Convert the pixel coordinates of both to the ray direction in the camera-normalized coordinates:

[0067] ;

[0068] in, This is the ray direction vector in the camera's normalized coordinate system.

[0069] For ground robots, their cameras are fixed on a platform at a height of [missing information]. In front of the robot center At this location, facing forward of the ground robot, the pitch angle is... The current position of the ground robot in the world coordinate system is... yaw angle is The ray direction in the camera coordinate system is transformed into the intersection point with the ground in the world coordinate system using the following formula, which is the navigation target point. :

[0070] ;

[0071] ;

[0072] ;

[0073] ;

[0074] ;

[0075] ;

[0076] in, Let be the rotation matrix from the camera coordinate system to the world coordinate system. Let be the rotation matrix from the ground robot's coordinate system to the world coordinate system. Let be the rotation matrix from the camera coordinate system to the ground robot coordinate system. Let be the rotation matrix of the yaw angle ψ of the robot rotating about the z-axis of the world coordinate system. Let be the rotation matrix of the camera around the y-axis, representing the pitch angle ϕ. The fixed-mount rotation matrix between the camera coordinate system and the robot coordinate system when the camera is at zero pitch angle. Let be the position vector of the camera coordinate system origin in the world coordinate system. Let be the position vector of the camera in the robot's coordinate system. Let be the position vector of the robot's coordinate system origin in the world coordinate system. Let be the forward horizontal distance of the camera relative to the robot's center in the robot's coordinate system. Height above the ground Let be the direction vector of the ray in the world coordinate system. Let be the position vector of the camera's optical center in the world coordinate system. The scaling factor for the intersection of the ray and the ground plane. Let be the coordinate vector of the intersection point of the ray and the ground plane.

[0077] For aerial robots, the camera is also fixed on the platform, in front of the robot's center. Below At this location, facing forward of the ground robot, the pitch angle is... The aerial robot's current position in the world coordinate system is... The direction vector is At this point, regarding the formula above... and By replacing it with the following formula, the current navigation target point can be obtained. :

[0078] ;

[0079] ;

[0080] in, Let be the rotation matrix of the aerial robot's roll angle α around the x-axis. Let be the rotation matrix of the aerial robot's pitch angle β around the y-axis. Let γ be the rotation matrix for the yaw angle γ of the aerial robot rotating around the z-axis.

[0081] In the method described in this embodiment, the navigation target point in the world coordinate system corresponding to the returned video can be calculated using the above formula, or it can be directly obtained through ROS's TF transformation tree. The latter requires establishing a coordinate transformation relationship between the aerial robot and the ground robot to achieve a complete coordinate transformation. Through video point selection and coordinate transformation, the operator's global navigation intent can be transmitted to the ground robot, realizing upper-level control functions.

[0082] In this embodiment, the aerial robot is controlled by the operator's body movements.

[0083] This embodiment integrates the visual stimuli, the generation of control commands, and the construction of navigation interactions to form a human-computer interaction system.

[0084] The human-computer interaction system integrates the above functions through augmented reality glasses, forming a hands-free interactive platform that is convenient for the user. The human-computer interaction system includes:

[0085] The fine control interface includes a video feedback display, five visual stimuli designed for corresponding positions, and five command feedback indicators, which are used to realize the fine control functions of the ground robot.

[0086] The video point selection and navigation interface includes a return video display, navigation point visualization, invalid point selection markers, and navigation markers, which are used to realize the video point selection and navigation function of the ground robot.

[0087] The function switching interface includes control mode switching buttons, video switching buttons, and interface follow and fix buttons, which are used to switch the modes of various functions and components.

[0088] The status and prompt components include a device connection status indicator and a target confirmation button, which are used to indicate the connection status of the air-ground cooperative system and the identification status of the target to be searched.

[0089] like Figure 3 As shown, the gaze buttons on the right side of the human-computer interaction interface (HCI) contain three functions. The top gaze button toggles the current control mode, determining whether the current control method is global navigation or pose control; the middle gaze button toggles the current video feed, determining whether the video displayed on the white screen is from the ground robot or the aerial robot; and the bottom gaze button toggles the HCI's follow mode, determining whether the HCI is fixed in the real space or follows the operator's movements. These three gaze buttons use eye-tracking data and loading animations; a 1-second gaze on a button triggers a switch to change the current mode.

[0090] The main interactive window of the human-computer interface is located in the center. The white screen is the display area for the returned video. The five red squares are the visual flashing stimuli required for the SSVEP paradigm, flashing between red and black at a predetermined frequency. The r-value changes using a sine function to successfully trigger the EEG signals required for the SSVEP paradigm at a 60Hz display. The five flashing stimuli will appear when the current control mode is pose control, and will be hidden when switching to global navigation mode. In global navigation mode, during navigation interaction, the selected location will be displayed as a small dot within 1 second after selection, or the words "invalid selection" will be displayed to provide feedback on the selection. When the ground robot is navigating, a rotating polyhedron will appear in the upper left corner to indicate that navigation is currently in progress.

[0091] In this embodiment, the operator's EEG signals are decoded by the EEG acquisition and analysis module, and eye movement features are acquired by the eye tracking module. Both signals are input to the fine control subsystem and the video point selection and navigation subsystem. The fine control subsystem inputs fine control commands to the ground robot to achieve its pose control. The video point selection and navigation subsystem inputs navigation coordinate data to the aerial or ground robot, and through corresponding coordinate transformation relationships, issues global navigation control commands to the ground robot. All of the above components are implemented through the human-computer interaction interface provided by the virtual reality glasses, forming a complete human-computer interaction system for the ground robot.

[0092] Example 2

[0093] Based on the same inventive concept, this embodiment also provides a hybrid brain-computer interface method for controlling air-ground collaborative robots, used to implement the system described in Embodiment 1, including the following steps:

[0094] Visual stimuli are presented to the operator through a display device, and the operator's brainwave signals are collected through an EEG acquisition device. The brainwave signals are then analyzed to obtain brainwave analysis commands.

[0095] The operator's eye movement signals are collected using an eye tracker, and the corresponding eye movement coordinates are obtained.

[0096] A decision-level fusion approach is adopted to fuse the EEG analysis commands and the eye movement coordinates to generate control commands for controlling the ground robot;

[0097] The system receives video feeds from ground and aerial robots, constructs navigation interactions based on the video feeds according to control commands, generates navigation target points to be sent to the ground robots, and achieves global navigation for the ground robots.

[0098] The hybrid brain-computer interface method for controlling air-ground collaborative robots provided in this embodiment has all the advantages of the hybrid brain-computer interface system for controlling air-ground collaborative robots provided in Embodiment 1.

[0099] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A hybrid brain-computer interface system for controlling air-to-ground collaborative robots, characterized in that, include: The EEG acquisition and analysis module is used to present visual stimuli to the operator, acquire the operator's EEG signals, analyze the EEG signals, and obtain EEG analysis commands. The EEG acquisition and analysis module includes: The visual stimulus unit is used to present several visual stimuli of different frequencies, and each visual stimulus corresponds to a command for fine control of the ground robot. A signal acquisition unit is used to acquire the electroencephalogram (EEG) signals of the operator. The signal parsing unit is used to decode the EEG signal using the filter bank canonical correlation analysis (FBCCA) method to obtain the EEG parsing command. An eye-tracking module is used to collect the operator's eye movement signals and obtain the eye movement coordinates corresponding to the eye movement signals; A fine control subsystem is used to perform decision-level fusion of the EEG analysis commands and the eye movement coordinates to generate control commands for controlling the ground robot. The precision control subsystem includes: A statistical analysis unit is used to perform statistical analysis on the eye movement coordinates using a sliding window with the same step size as the EEG signal decoding; The control command output module is used to analyze the EEG signal decoding results and the eye movement coordinates, perform decision-level fusion, and output fine control commands; the fine control commands include: start / stop, accelerate, decelerate, turn left, and turn right; The decision-making level integration includes: When the previous control command was non-control, it is determined that the current state is non-control. At this time, only the eye movement control area discrimination and eye movement tracking 0.25s window length command output are performed. When the last control command was one of the five control commands, it is determined that the current state is under control. At this time, the eye movement control area, the eye movement 0.25s window length, the eye movement 0.5s window length, and the SSVEP correlation coefficient will be comprehensively considered. The video point selection and navigation subsystem is used to receive the returned video from the ground robot and the aerial robot, construct the navigation interaction based on the returned video according to the control command, generate the navigation target point to be sent to the ground robot, and realize the global navigation of the ground robot. The video point selection and navigation subsystem includes: The coordinate recording unit is used to decode the EEG's intention to confirm the selection of points and record the eye movement coordinates at that moment; The coordinate calculation unit is used to collect the camera intrinsic parameters of the ground robot and the aerial robot, and calculate the ground coordinates pointed to by the video coordinate points. The coordinate mapping unit is used to calculate the relative coordinate relationship between the aerial robot and the ground robot, and to map the ground coordinates to the target point coordinates in the ground robot coordinate system; The navigation unit is used for global navigation using the ground robot's navigation system.

2. The system according to claim 1, characterized in that, It also includes a human-computer interaction subsystem: A fine-grained control interface is used to display the returned video, the visual stimuli, and command feedback indicators; The video point selection navigation interface is used to display the returned video, navigation point visualization indicators, and navigation status indicators; The function switching interface is used to provide functions for switching control modes, video sources, and interface layouts. Status indicator component, used to display system connection status and task status.

3. A hybrid brain-computer interface method for controlling air-to-ground collaborative robots, characterized in that, For implementing the system according to any one of claims 1-2, the following steps are included: Visual stimuli are presented to the operator through a display device, and the operator's brainwave signals are collected through an EEG acquisition device. The brainwave signals are then analyzed to obtain brainwave analysis commands. The operator's eye movement signals are collected using an eye tracker, and the corresponding eye movement coordinates are obtained. A decision-level fusion approach is adopted to fuse the EEG analysis commands and the eye movement coordinates to generate control commands for controlling the ground robot; The system receives video feeds from ground and aerial robots, constructs navigation interactions based on the video feeds according to control commands, generates navigation target points to be sent to the ground robots, and achieves global navigation for the ground robots.

Images