Multi-robot connectivity control system and method based on multi-modal interaction interface

By using a multimodal interactive interface system that combines brain-computer interface, eye tracking, and hand controller, the problem of insufficient communication connectivity of multi-robot systems in dynamic environments is solved, enabling flexible formation control and autonomous collaboration, and enhancing the operator's interactive capabilities.

WO2024244569A1PCT designated stage expired Publication Date: 2024-12-05SOUTHEAST UNIV

Patent Information

Application Number
PCT/CN2024/079270
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-31
Filing Date
2024-02-29
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing multi-robot collaborative control systems cannot flexibly switch formation topologies in dynamic and complex environments, have insufficient communication connectivity, lack effective operator-multi-robot system interaction interfaces, and are difficult to achieve autonomous collaborative control.

Method used

By employing a multimodal interactive interface system that combines brain-computer interface, gaze tracking, and hand controller, and embedding connectivity-maintaining actions through perception constraints, expected distance constraints, and obstacle avoidance constraints, the autonomous and collaborative control of the robot team is achieved.

Benefits of technology

It enables flexible connection and maintenance of robot teams under constraints, enhances the operator's interactive capabilities, allows for switching between dispersed and close formations, and achieves autonomous collaborative control by combining operator intelligent decision-making with machine task execution.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024079270_05122024_PF_FP_ABST
    Figure CN2024079270_05122024_PF_FP_ABST
Patent Text Reader

Abstract

A multi-robot connectivity control system and method based on a multi-modal interaction interface. The system comprises: an interaction interface module, a communication module and a multi-robot cooperative control module. A method involved comprises: an operator remotely controlling a human-computer interaction module to implement an input of a multi-modal control instruction, transmitting the control instruction to a multi-robot cooperative control module by means of a communication module, and the multi-robot cooperative control module finally implementing cooperative control of a robot team by means of a connectivity preservation algorithm and a consistency algorithm.
Need to check novelty before this filing date? Find Prior Art

Description

Multi-robot connectivity control system and method based on multimodal interactive interface Technical Field

[0001] The present invention belongs to the field of human-machine fusion interaction and multi-robot team collaborative control, and specifically relates to a multi-robot connectivity control system and method based on a multimodal interaction interface. Background Art

[0002] Maintaining connectivity in multi-robot communication is one of the most fundamental and crucial issues in multi-robot control. It is a necessary condition for the convergence of most multi-robot collaborative control algorithms, such as consensus control, group convergence and aggregation, formation maintenance and transformation, and collaborative area exploration and coverage. In dynamic, complex, communication-restricted, and GPS-denied non-open environments (e.g., urban streets and underground castles), the main limitation of current multi-robot collaborative control lies in its reliance on the rigidity of the multi-robot formation. This inflexibility prevents the free switching of formation topologies in particularly chaotic and dynamic environments. Therefore, it is urgent to explore new multi-robot collaborative control methods that can achieve dynamic topological evolution of multi-robot movement while maintaining connectivity.

[0003] Fully autonomous control of multi-robot teams remains a distant prospect, requiring operator intervention / assistance. However, current research on operator-multi-robot interaction is insufficient, particularly lacking effective interfaces. Due to the high flexibility of multi-robot systems, current single-modal interaction methods often fail to provide multi-degree-of-freedom control. Therefore, research on multimodal interaction methods is urgently needed to achieve more flexible control of multi-robots.

[0004] Summary of the Invention

[0005] To solve the above problems, the present invention discloses a multi-robot connectivity control system and method based on a multimodal interactive interface, which embeds perception constraints, desired distance constraints and obstacle avoidance constraints into a unique connectivity maintenance action, allowing robot teams to perform connection maintenance in a flexible manner under certain constraints, and realize autonomous collaborative control of robot teams.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] A multi-robot connectivity control system based on a multimodal interactive interface, including an interactive interface module, a communication module, and a multi-robot collaborative control module;

[0008] The interactive interface module includes a brain-computer interface module, a sight tracking module, a fusion module, and a hand controller module; the brain-computer interface module includes a visual stimulation presentation module, an electroencephalogram (EEG) signal acquisition module, and an EEG signal processing module. The visual stimulation presentation module induces the operator's cerebral cortex to generate SSVEP signals by designing a stimulus source that flashes at a specific frequency on the interactive interface. The EEG signal acquisition module detects the SSVEP signals generated by the operator and transmits them to the EEG signal processing module. The EEG signal processing module recognizes the SSVEP signals, obtains the recognition results, and sends them to the fusion module. The sight tracking module includes an eye movement signal. The eye movement signal acquisition module and the eye movement signal processing module, the eye movement signal acquisition module detects the operator's gaze point position on the interactive interface through an eye tracker and transmits it to the eye movement signal processing module, the eye movement signal processing module identifies the gaze point position, obtains the identification result and sends it to the fusion module; the fusion module fuses the output results of the brain-computer interface module and the eye tracking module, obtains the target selection instruction according to the fusion result, maps it into a formation connectivity instruction and sends it to the robot collaborative control module; the hand controller module maps the hand controller end position into linear velocity and angular velocity control instructions and sends them to the robot collaborative control module to control the navigator's movement.

[0009] The communication module is used to realize data interaction between the interactive interface module and the robot collaborative control module;

[0010] The multi-robot collaborative control module includes a connectivity maintenance module and an adaptive control module; the connectivity maintenance module includes three modules: a sensing module, an obstacle avoidance module, and a formation distance module, wherein the sensing module represents a specific inter-robot sensing / communication model, the formation module realizes group requirements such as formation control of the desired distance between robots, the obstacle avoidance module realizes collision avoidance between robots, and the connectivity maintenance module enables the robot team to perform connection maintenance in a flexible manner under the above three constraints; the adaptive module realizes the consistency of speed between all followers and between followers and the leader.

[0011] The control method of the multi-robot connectivity control system based on the multimodal interactive interface comprises the following steps:

[0012] Step 1: The operator remotely starts the robot team driver through the human-machine interface, and the robot team enters the standby state until receiving the next motion control instruction;

[0013] Step 2: The operator controls the hand controller to send continuous linear velocity and angular velocity control instructions to the multi-robot collaborative control module to control the movement of the leader.

[0014] Step 3: The minimum threshold of the increase or decrease of the algebraic connectivity of the operator's gaze on the human-computer interaction interface The eye tracking module can track eye movements and gaze movements, and realize eye tracking control. The eye tracking module detects the operator's effective gaze events on the interactive interface and sends them to the fusion module.

[0015] Step 4: The operator wears an EEG cap, which collects the gaze-related potential signals generated by the cerebral cortex, detects the SSVEP signals generated by the operator and transmits them to the EEG signal processing module. The EEG signal processing module recognizes the SSVEP signals, obtains the recognition results and sends them to the fusion module.

[0016] Step 5: The fusion module can fuse the output results of the brain-computer interface module and the gaze tracking module, obtain the target selection instruction based on the fusion result, and map it to the minimum threshold of increasing or decreasing the algebraic connectivity The second type of control instructions are sent to the multi-robot collaborative control module.

[0017] Step 6: The minimum threshold for increasing or decreasing algebraic connectivity sent by the fusion module through TCP communication The discrete control instructions are sent to the multi-robot collaborative control module.

[0018] Step 7: After receiving the control command, the multi-robot collaborative control module uses the connectivity maintenance module to switch between the dispersed and close formations of the robot teams and maintain the connection between the robot teams. represents the minimum connectivity measure of the group, The lower it is, the more dispersed the group is, and vice versa. The higher the speed, the tighter the group. At the same time, the adaptive module ensures that all followers have the same speed as each other and between followers and the leader.

[0019] Furthermore, the following three constraints and requirements are embedded into a unique connectivity preservation action: (1) a specific inter-robot sensing / communication model; (2) group requirements such as formation / desired distance control; and (3) individual requirements such as obstacle avoidance control. This enables the robot team to maintain connectivity in a flexible manner under certain constraints, allowing interactive links to be established / deleted at any time while maintaining global connectivity.

[0020] Furthermore, the multi-robot collaborative control module includes a connectivity maintenance module and an adaptive module. The connectivity maintenance module adopts a connectivity maintenance algorithm based on graph theory to ensure the connection of the robot team, and the adaptive module adopts a consistency algorithm to achieve consistent speed between all followers and between followers and leaders. The two together realize autonomous collaborative control of the robot team.

[0021] Furthermore, while the robot team is autonomously controlling multiple robots through the connectivity maintenance module and the adaptive module, humans can intervene / assist the multi-robot team. Through the multimodal interaction interface, users can achieve (1) the control of the linear velocity and angular velocity of the leader by the hand controller, and (2) the fusion module sends the formation connectivity command to change the minimum threshold of algebraic connectivity online. Enable multi-robot teams to complete the switching between dispersed and close formations on their own. represents the minimum connectivity measure of the group, The lower it is, the more dispersed the group is, and vice versa. The higher the value, the tighter the group. This allows users to better control the behavior of the robot team, effectively combining the operator's intelligent decision-making ability with the machine's task execution capability.

[0022] The beneficial effects of the present invention are:

[0023] (1) Embedding perception constraints, desired distance constraints, and obstacle avoidance constraints into a unique connectivity maintenance action enables the robot team to perform connectivity maintenance in a flexible manner under certain constraints, thus achieving autonomous collaborative control of the robot team;

[0024] (2) The present invention adopts a command input method that integrates brain-computer interface and eye tracking, using EEG and eye tracking signals as a powerful supplement to traditional input modalities, providing operators with additional command input channels and enhancing the operator's interactive capabilities;

[0025] (3) While the robot team is autonomously and collaboratively controlled through the connectivity maintenance module and the adaptive module, humans can intervene / assist the robot team. Through the multimodal interaction interface, users can achieve:

[0026] 1) Linear and angular velocity control of the navigator;

[0027] 2) Online change of the minimum threshold of algebraic connectivity

[0028] It enables multi-robot teams to complete the formation dispersion and close switching independently, allowing users to better control the behavior of the robot team and effectively combining the operator's intelligent decision-making ability with the machine's task execution capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG1 is a system architecture diagram of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0031] The present invention designs a multi-robot connectivity control system based on a multimodal interactive interface, which includes an interactive interface module, a communication module and a multi-robot collaborative control module; the interactive interface module includes a brain-computer interface module, a line of sight tracking module, a fusion module, and a hand controller module; the multi-robot collaborative control module includes a connectivity maintenance module and an adaptive control module; the communication module is used to realize data interaction between the interactive interface module and the multi-robot collaborative control module.

[0032] The brain-computer interface module includes a visual stimulation presentation module, an EEG signal acquisition module and an EEG signal processing module. The visual stimulation presentation module induces the operator's cerebral cortex to generate SSVEP signals by designing a stimulus source that flashes at a specific frequency on the interactive interface. The EEG signal acquisition module detects the SSVEP signals generated by the operator and transmits them to the EEG signal processing module. The EEG signal processing module recognizes the SSVEP signals, obtains the recognition results and sends them to the fusion module.

[0033] The eye tracking module includes an eye movement signal acquisition module and an eye movement signal processing module. The eye movement signal acquisition module detects the operator's gaze point position on the interactive interface through an eye tracker and transmits it to the eye movement signal processing module. The eye movement signal processing module identifies the gaze point position, obtains the recognition result, and sends it to the fusion module.

[0034] The fusion module fuses the output results of the brain-computer interface module and the gaze tracking module, obtains the target selection instruction based on the fusion result, maps it into a formation connectivity instruction and sends it to the multi-robot collaborative control module; the connectivity maintenance module receives the formation connectivity instruction sent by the fusion module and switches the robot system between formation dispersion and tightness.

[0035] The connectivity maintenance module comprises three modules: a sensing module, an obstacle avoidance module, and a formation module. The sensing module represents a specific inter-robot sensing / communication model, the formation module implements group requirements such as formation control at a desired distance between robots, and the obstacle avoidance module implements collision avoidance between robots. The connectivity maintenance module enables a team of robots to maintain connectivity in a flexible manner within these three constraints.

[0036] The adaptive module achieves speed consistency among all followers and between followers and the navigator;

[0037] The hand controller module maps the hand controller end position into linear velocity and angular velocity control instructions and sends them to the robot collaborative control module to control the movement of the navigator.

[0038] The control method of the multi-robot connectivity control system based on the multimodal interactive interface comprises the following steps:

[0039] Step 1: The operator remotely starts the robot team driver through the human-machine interface, and the robot team enters the standby state until receiving the next motion control instruction;

[0040] Step 2: The operator controls the hand controller to send continuous linear velocity and angular velocity control instructions to the multi-robot collaborative control module to control the movement of the leader.

[0041] Step 3: The minimum threshold of the increase or decrease of the algebraic connectivity of the operator's gaze on the human-computer interaction interface The eye tracking module can track eye movements and gaze movements, and realize eye tracking control. The eye tracking module detects the operator's effective gaze events on the interactive interface and sends them to the fusion module.

[0042] Step 4: The operator wears an EEG cap, which collects the gaze-related potential signals generated by the cerebral cortex, detects the SSVEP signals generated by the operator and transmits them to the EEG signal processing module. The EEG signal processing module recognizes the SSVEP signals, obtains the recognition results and sends them to the fusion module.

[0043] Step 5: The fusion module can fuse the output results of the brain-computer interface module and the gaze tracking module, obtain the target selection instruction based on the fusion result, and map it to the minimum threshold of increasing or decreasing the algebraic connectivity The second type of control instructions are sent to the multi-robot collaborative control module.

[0044] Step 6: The minimum threshold for increasing or decreasing algebraic connectivity sent by the fusion module through TCP communication The discrete control instructions are sent to the multi-robot collaborative control module.

[0045] Step 7: After receiving the control command, the multi-robot collaborative control module uses the connectivity maintenance module to switch between the dispersed and close formations of the robot teams and maintain the connection between the robot teams. represents the minimum connectivity measure of the group, The lower it is, the more dispersed the group is, and vice versa. The higher the speed, the tighter the group. At the same time, the adaptive module ensures that all followers have the same speed as each other and between followers and the leader.

[0046] The working process and principle of the present invention are as follows:

[0047] 1. The operator issues control instructions through the multimodal interactive interface

[0048] The operator sends the minimum threshold of discrete formation algebraic connectivity through the brain-computer interface module and the gaze tracking module Control instructions, send continuous speed control instructions through the hand controller module.

[0049] First, the brain-computer interface module obtains the visual stimulus source by modulating the screen brightness. The stimulus source sequence calculation formula is as follows:

[0050] In the above formula, f represents the frequency of the stimulus source, i represents the frame coordinate of the stimulus source sequence, and Refresh_Rate represents the refresh frequency of the interactive interface.

[0051] The brain-computer interface module establishes a brain-computer interface recognition accuracy model. Canonical correlation analysis (CCA) is used to analyze and process EEG signals. For K stimulation sources with different frequencies, the probability of each stimulation source being selected is the correlation coefficient between its template signal and SSVEP signal. Let the i-th target T i The flashing frequency is f i , i=1,2,…,K, its template signal is as follows:

[0052] In the above formula, N h is the number of harmonics, t represents time;

[0053] The SSVEP signal X is compared with Perform linear combination, x=X T W X , W X and To combine X and Projected to a one-dimensional linear coefficient vector, the superscript T indicates transposition; through canonical correlation analysis, find the relationship between x and y i The correlation coefficient ρ i The largest W X and

[0054] Then the i-th target T i Probability of being selected:

[0055] In the above formula, is W found through canonical correlation analysis X and The corresponding x and y i The correlation coefficient of .

[0056] The gaze tracking module establishes a gaze tracking accuracy model and uses Gaussian distribution to model the accuracy of the gaze point:

[0057] In the above formula, σ is the detection error of the eye tracker in this dimension, μ is the coordinate of the target in this dimension, and x is the coordinate of the gaze point detected by the eye tracker in this dimension;

[0058] Set target T i The coordinates on the interactive interface are (x i ,y i ), then the probability of the target being selected is: P gaze (T i )=N(σ1,x g , x i )·N(σ2,y g ,y i )

[0059] In the above formula, σ1 is the detection error of the eye tracker in the vertical direction, σ2 is the detection error of the eye tracker in the horizontal direction, (x g ,y g ) is the operator’s gaze point position obtained from the eye tracker test.

[0060] The fusion module uses the naive Bayes method to fuse the recognition results of the brain-computer interface module and the gaze tracking module. i Probability of being selected: P(T i )=P BCI (T i )·P gaze (T i )

[0061] The final target selected is T:

[0062] The position information of the hand controller end is mapped to the speed control signal of the navigator robot, using the two degrees of freedom of the hand controller end, namely X and Z directions. The Z direction is used to control the linear speed of the navigator robot, and the X direction is used to control the angular speed. In order to ensure that the user can control multiple robots with a single degree of freedom, set the z direction. b 、x b It is a buffer zone. When the end position of the hand controller is within this zone, no control command is issued.

[0063] The specific control methods are as follows:

[0064] In the above formula, k v is the proportionality coefficient, K w is a constant, g1(z, z b ) and g2(x, x b )The functional relationship is as follows:

[0065] 2. Multi-robot collaborative control

[0066] The multi-robot cooperative control module includes a connectivity maintenance module and an adaptive module. The connectivity maintenance module adopts a connectivity maintenance algorithm based on graph theory.

[0067] The dynamic communication graph G(t) = {V, E(t)} is a time-varying undirected graph consisting of a vertex set V = {n1, n2, ..., n N} and edge set E(t)={(n i , n j )|||x ij (t)||<R}, the vertices represent the multiple robots in the group, and the edge set contains the unordered pairs of vertices with neighbor relationships. First, the dynamic communication of the multi-robot team is abstracted into an undirected graph and the adjacency matrix of the graph is designed.

[0068] When designing the adjacency matrix, we embed a number of constraints and requirements into the design of the adjacency matrix: (1) specific inter-robot sensing / communication models; (2) group requirements, such as formation control; and (3) individual requirements, such as obstacle avoidance. ij =α ij β ij γ ij

[0069] In the above formula, A ij is the adjacency matrix element, α ij To avoid obstacles, β ij is the formation control constraint, γ ij For perception constraints.

[0070] γ ij As the perception constraint, the robot's maximum perception range constraint is considered. The design is as follows:

[0071] In the above formula, d ij represents the distance between the i-th robot and the j-th robot, D represents the maximum sensing range of the robot, μ a is the proportional coefficient, v a To set the constant, the purpose is to make the function smooth.

[0072] β ij As the expected distance constraint, β ij (d ij ) is defined as d ij =d0 has a unique maximum value and when |d ij A function whose smoothness vanishes as -d0|→∞.

[0073] α ijTo avoid obstacles, consider the minimum safe distance between robots. The design is as follows:

[0074] In the above formula, d min Indicates the minimum safe distance between robots, d max Indicates the maximum safe distance between robots, μ a is the proportional coefficient, v a To set the constant, the purpose is to make the function smooth.

[0075] Then, calculate the corresponding Laplace matrix as follows:

[0076] In the above formula, L is the Laplace matrix, diag(δ i ) is the degree matrix, and A is the adjacency matrix.

[0077] Furthermore, establish V λ (λ Z ) is the second smallest eigenvalue λ of the graph Laplacian operator Z (also called algebraic connectivity) and obey the gradient of this potential to always maintain λ Z >0 to maintain connectivity.

[0078] Furthermore, when the multi-robot collaborative control module receives the minimum threshold for changing algebraic connectivity sent by the fusion module When given instructions, the robot team can control the tightness of the formation. A measure representing the minimum degree of connectivity of a group. The lower it is, the more dispersed the group is, and vice versa. The higher it is, the tighter the group.

[0079] The adaptive module uses a consistency algorithm to achieve speed consistency between all followers and between followers and the leader. The design is as follows: i (k+1)=V i (k)+a*u i (k)

[0080] V i (k+1) is the speed of the i-th robot at the k-th moment, V i (k+1) is the speed of the i-th robot at the k+1th moment, a is the proportional coefficient, a ij is the element corresponding to the i-th row and j-th column of the adjacency matrix, V j is the speed of the jth robot.

[0081] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A multi-robot connectivity control system based on a multimodal interactive interface, characterized in that: It includes an interactive interface module, a communication module and a multi-robot collaborative control module; The multi-robot collaborative control module includes a connectivity maintenance module and an adaptive module. The connectivity maintenance module adopts a connectivity maintenance algorithm based on graph theory to ensure the connection of the robot team. The adaptive module adopts a consistency algorithm to achieve consistent speed between all followers and between followers and leaders. The two together realize autonomous collaborative control of the multi-robot team.

2. The multi-robot connectivity control system based on a multimodal interactive interface according to claim 1, characterized in that: The interactive interface module includes a brain-computer interface module, a sight tracking module, a fusion module, and a hand controller module; the brain-computer interface module includes a visual stimulation presentation module, an electroencephalogram signal acquisition module, and an electroencephalogram signal processing module. The visual stimulation presentation module induces the operator's cerebral cortex to generate SSVEP signals by designing a stimulus source that flashes at a specific frequency on the interactive interface. The electroencephalogram signal acquisition module detects the SSVEP signals generated by the operator and transmits them to the electroencephalogram signal processing module. The electroencephalogram signal processing module recognizes the SSVEP signals, obtains the recognition results, and sends them to the fusion module. The sight tracking module includes an eye movement signal acquisition module. The eye movement signal acquisition module detects the operator's gaze point position on the interactive interface through an eye tracker and transmits it to the eye movement signal processing module. The eye movement signal processing module identifies the gaze point position, obtains the identification result and sends it to the fusion module; the fusion module fuses the output results of the brain-computer interface module and the line of sight tracking module, obtains the target selection instruction according to the fusion result, maps it into a formation connectivity instruction and sends it to the multi-robot collaborative control module; the hand controller module maps the hand controller end position into linear velocity and angular velocity control instructions and sends them to the multi-robot collaborative control module to control the movement of the navigator.

3. The multi-robot connectivity control system based on a multimodal interactive interface according to claim 1, characterized in that: The communication module is used to realize data interaction between the interactive interface module and the multi-robot collaborative control module.

4. The multi-robot connectivity control system based on a multimodal interactive interface according to claim 1, characterized in that: The multi-robot collaborative control module includes a connectivity maintenance module and an adaptive control module; the connectivity maintenance module includes three modules: a sensing module, an obstacle avoidance module and a formation distance module, wherein the sensing module represents a specific inter-robot sensing / communication model, the formation module realizes formation control of a desired distance between robots, the obstacle avoidance module realizes collision avoidance between robots, and the connectivity maintenance module enables the robot team to perform connection maintenance in a flexible manner under the three constraints of sensing distance constraint, formation distance constraint and obstacle avoidance constraint; The adaptive module achieves consistent speed among all followers and between followers and the leader.

5. The multi-robot connectivity control system based on a multimodal interactive interface according to claim 1, characterized in that: The following three constraints / requirements are embedded into a connectivity-preserving action: (1) inter-robot sensing / communication model; (2) formation control; and (3) obstacle avoidance control, so that the robot team can maintain connectivity in a flexible way under the above constraints, allowing interactive links to be established / deleted at any time while maintaining global connectivity.

6. The multi-robot connectivity control system based on a multimodal interactive interface according to claim 1, characterized in that: While the robot team is autonomously controlling the robots through the connectivity maintenance module and the adaptive module, the user uses the multimodal interaction interface to (1) control the linear velocity and angular velocity of the leader with the hand controller and (2) send the formation connectivity command to the fusion module to change the minimum threshold of algebraic connectivity online. Enable the robot team to switch between dispersed and close formations by itself. represents the minimum connectivity measure of the group, The lower it is, the more dispersed the group is, and vice versa. The higher it is, the tighter the group; it enables users to better control the behavior of the robot team, effectively combining the operator's intelligent decision-making ability with the machine's task execution ability.

7. The control method of the multi-robot connectivity control system based on the multimodal interactive interface according to claim 1, characterized in that: The following steps are involved: Step 1: The operator remotely starts the robot team driver through the human-machine interface, and the robot team enters the standby state until receiving the next motion control instruction; Step 2: The operator controls the hand controller to send continuous linear speed and angular speed control instructions to the multi-robot collaborative control module to control the movement of the leader; Step 3: The minimum threshold for increasing or decreasing algebraic connectivity of the operator’s gaze on the human-computer interface The eye tracking module can track the eye movement and gaze movement, realize eye tracking control, and detect the operator's effective gaze events on the interactive interface and send them to the fusion module; Step 4: The operator wears an EEG cap, which collects the gaze-related potential signal generated by the cerebral cortex, detects the SSVEP signal generated by the operator and transmits it to the EEG signal processing module, which recognizes the SSVEP signal, obtains the recognition result and sends it to the fusion module; Step 5: The fusion module can fuse the output results of the brain-computer interface module and the gaze tracking module, obtain the target selection instruction based on the fusion result, and map it to the minimum threshold of increasing or decreasing the algebraic connectivity The second type of control instructions are sent to the multi-robot collaborative control module; Step 6: Send the minimum threshold for increasing or decreasing algebraic connectivity sent by the fusion module through TCP communication The discrete control instructions are sent to the multi-robot collaborative control module; Step 7: After receiving the control command, the multi-robot collaborative control module uses the connectivity maintenance module to switch between the dispersed and close formations of the robot teams and maintain the connection between the robot teams. represents the minimum connectivity measure of the group, The lower it is, the more dispersed the group is, and vice versa. The higher it is, the tighter the group is; at the same time, the adaptive module ensures that the speed is consistent between all followers and between followers and the leader.

Citation Information

Patent Citations

  • Wearable augmented reality robot control system combining eye movement and brain-computer interface

    CN110442232A

  • Rapid formation method for unmanned aerial vehicle fleet

    CN110687926A

  • Multi-mobile robot cooperative control system based on multi-modal interaction interface

    CN111890389A

  • Multi-robot connectivity control system and method based on multi-modal interaction interface

    CN116931562A

  • Brain-computer interface for user's visual focus detection

    US20190307356A1

Cited By

  • Industrial robot multi-machine cooperative interaction method and system

    CN119407799A

  • Distributed control method and system for maintaining global connectivity of teleoperation open multi-robot

    CN120065933A