A human-computer interaction operation system and method based on a BCI and VR fusion robot working environment

Abstract

This invention relates to the field of BCI technology and the integration of VR technology with robotics, and particularly to a human-computer interaction operating system and method for a BCI and VR-integrated robot working environment. The system includes a BCI control system, a physical environment system, a VR assistance system, and a robot equipment system interconnected with the VR assistance system. The BCI control system connects the VR assistance system and the physical environment system via communication devices to form an interactive system. The VR assistance system and the physical environment system are also connected to the robot equipment system via communication devices to form an interactive system. The robot equipment system includes an instruction analysis and judgment module, a motion control module, and a mechanical structure module. The physical environment system includes a worker performing the operation, a virtual worker, on-site hydraulic equipment, an on-site operating robot, and the tasks performed by the robot. The VR assistance system includes a smart wearable device and a smart application terminal device, with the smart application terminal device including an image display module.

Description

Technical Field

[0001] This invention relates to the field of BCI technology and the combination of VR technology with robots, and in particular to a human-computer interaction operating system and method for a robot working environment based on the fusion of BCI and VR. Background Technology

[0002] Currently, most industrial robots are controlled by PLC systems and their mechanical transmissions. The field of BCI systems has gradually become a research hotspot in recent years. Brain-Computer Interface (BCI) enables the control of external devices through the brain's thoughts. At present, BCI technology can be used in two ways: one is by implanting a chip in the brain, and the other is by wearing an EEG helmet. The latter involves placing a flat metal plate in contact with the scalp to measure brain waves. The brain waves acquired by the helmet are wirelessly transmitted to a computer via a small amplifier, thereby enabling the control of robots used in the hydraulic industry. VR, or Virtual Reality, is a new and practical technology developed in the 20th century. Virtual Reality technology encompasses computer science, electronic information, and other fields. Virtual reality (VR) technology integrates simulation technology, and its basic implementation method is to simulate a virtual environment by computer to give people a sense of immersion. However, at present, VR technology is mostly used in the entertainment and gaming industry. If this technology could be applied to the work environment, workers could experience the virtual on-site work environment and state firsthand, adding a game element to their work, thus reducing workplace stress and making work as easy as a game. In the current traditional hydraulic industry, hydraulic presses are heavy industrial equipment for large-scale casting and forging. If the operation is not handled properly by the workers during production or operation, some safety accidents may occur. To address the above-mentioned shortcomings, this invention proposes a human-computer interaction operating system and method based on the integration of BCI and VR fields and the application of large robots in the hydraulic industry. The BCI system collects the movement and imagination intentions of the workers, analyzes them, and sends them to the VR system. The VR system executes the robot to complete the task instructions distributed in the workers' minds, eliminating safety accidents caused by improper operation during production or operation, and also achieving the beneficial effect of reducing workplace stress and making production work as easy as a game. Summary of the Invention

[0003] The purpose of this invention is to solve the problems existing in the prior art, and to propose a human-computer interaction operating system and method based on the working environment of a hydraulic robotic arm that integrates BCI and VR.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] A human-computer interaction operating system based on a BCI and VR fusion robot working environment includes a BCI control system, a physical environment system, a VR assistance system, and a robot equipment system that interacts with the VR assistance system. The BCI control system is used for acquiring source signals, generating signals, monitoring signals, processing signals, and extracting BCI signal features.

[0006] The BCI control system is connected to the VR assistance system and the physical environment system via communication devices to form an interactive system. The VR assistance system and the physical environment system are also connected to the robot equipment system via communication devices to form an interactive system. The robot equipment system includes an instruction analysis and judgment module, a motion control module, and a mechanical structure module. The mechanical structure module includes a traveling device and a robotic arm. The traveling device includes a first traveling device and a corresponding second traveling device. Both the first and second traveling devices are equipped with a traveling controller and a displacement sensor. The robotic arm includes a first degree-of-freedom arm, a second degree-of-freedom arm, and a gripping device. The motion control module includes a traveling controller, a first controller, a second controller, a third controller, a displacement sensor, an angle sensor one, an angle sensor two, and an angle sensor three. The first controller and angle sensor one are mounted on the first degree-of-freedom arm; the second controller and angle sensor two are mounted on the second degree-of-freedom arm; the third controller and angle sensor three are mounted on the gripping device; and a positioning device is also mounted on the gripping device.

[0007] The VR assistance system establishes an information interaction connection with the command analysis and judgment module through a communication device; the command analysis and judgment module includes multiple correct command information output ports and incorrect command information output ports; the multiple correct command information output ports establish information interaction connections with the movement controller, the first controller, the second controller and the third controller respectively.

[0008] Furthermore, the physical environment system includes operational personnel, virtual operational personnel, on-site hydraulic equipment, on-site operating robots, and tasks performed by the robots.

[0009] Furthermore, the BCI control system includes a smart helmet, which houses a signal acquisition system, a signal processing system, and a signal output system. The signal acquisition system collects raw data of the operator's electroencephalogram (EEG) and electromyographic (EMG) signals and sends the raw data to the signal processing system. The signal processing system preprocesses the raw data, extracts features, classifies the information into patterns, and then transmits the data to the signal output system. The EEG and EMG signals are neurophysiological signals from an electroencephalogram. The signal acquisition system is a non-embedded acquisition system that collects the operator's EEG and EMG signals by setting electrode sensors inside the smart helmet.

[0010] Furthermore, the signal output system transmits instruction information to the information module in the VR assistance system. After collecting all the information, the information module establishes different types of target instructions, synthesizes different types of target instructions into virtual operation action demonstration instructions for virtual workers in the virtual scene, and sends the corresponding synthesized instructions to the travel controller, the first controller, the second controller, and the third controller to control their first travel device, the second travel device, the first degree of freedom arm, the second degree of freedom arm, and the clamping device to complete the task action instructions.

[0011] Furthermore, the VR assistance system also includes smart wearable devices and smart application terminal devices, with the smart application terminal devices including an image display module.

[0012] Furthermore, after the image display module and the information module establish an information interaction connection, the smart wearable device is located between the image display module and the information module, giving the operator a visual sense of presence by wearing the smart wearable device; miniature 3D camera one, miniature 3D camera two, and miniature 3D camera three are respectively set on the first degree of freedom arm, the second degree of freedom arm, and the clamping device; the smart wearable device can be any one or more of the following: head-mounted, goggle-style, glove-style, or vest-style; the information module also includes a motion capture unit and a computing unit.

[0013] Furthermore, the motion capture unit is used to capture the movements of different parts of the virtual worker in the image and send them to the computing unit. The sensor in the smart wearable device senses the movement path of the head or body and captures it. The sensor includes a head rotation capture sensor, a body forward and backward or left and right movement capture sensor, as well as a gesture and eye movement capture sensor. The computing unit identifies the current virtual worker's movements through a series of calculations and analyses and sends them to the robot, which then performs the calculations on the travel controller, first controller, second controller, and third controller in the robot equipment system.

[0014] Furthermore, the miniature 3D camera one, miniature 3D camera two, and miniature 3D camera three establish information interaction connections with the information module through communication devices; the information module receives the information fed back after the miniature 3D camera one, miniature 3D camera two, and miniature 3D camera three captures images, constructs the background and object models of the virtual scene, and sends the background and models in the virtual scene to the image display module.

[0015] Furthermore, the angle sensor one, angle sensor two, and angle sensor three perceive the environment around the robot arm through different angles generated by the rotation or movement of the first degree of freedom arm, the second degree of freedom arm, and the clamping device, respectively, and transmit the information to the information module via 3D camera; the displacement sensor perceives the environment around the robot's traveling device through the forward or backward distance of the first traveling device and the second traveling device, and transmits the information to the information module via 3D camera.

[0016] A human-computer interaction operation method for a BCI and VR fusion robot working environment, comprising any of the above-described human-computer interaction operating systems based on a BCI and VR fusion hydraulic robotic arm working environment, the method comprising:

[0017] S1: The operator arranges the process flow that the robot needs to complete on-site according to the process sequence;

[0018] S2: The signal acquisition system collects raw data of electroencephalogram (EEG) and electromyogram (EMG) signals. The raw data is the action instructions in the process flow. After wearing a smart helmet, the operator watches the process flow and concentrates on imagining the action instructions that the robot needs to complete in the process flow.

[0019] S3: The electrode sensors of the signal acquisition system collect the action command information of the operator's mind during the process through the cerebral cortex; the action command information includes information of different wavebands in the subcortical region, and the effective waveband information is collected.

[0020] S4: The signal acquisition system sends valid band information to the signal processing system;

[0021] S5: The signal processing system preprocesses the raw data, extracts features, classifies the information according to patterns, and then transmits it to the signal output system.

[0022] S6: The signal output system distributes information commands to the information module in the VR assistance system through the corresponding interface. After collecting all the information, the information module establishes different types of target commands and synthesizes the different types of target commands into an image to demonstrate the operation actions of the virtual worker in the virtual scene. After the information module identifies and calculates the action commands of the virtual worker through the motion capture unit and the computing unit, it sends the action command information to the robot's command analysis and judgment module. After analyzing the command information, the command analysis and judgment module sends the correct command information to the travel controller, the first controller, the second controller, and the third controller, respectively, to control their travel devices and robotic arms to complete the corresponding action command tasks; and feeds back the incorrect command information to the signal output system, which then feeds it back to the signal processing system for processing.

[0023] S7: Simultaneously, the 3D camera device located on the robot sends the background and object models of the robot's surrounding environment to the information module in the VR assistance system.

[0024] S8: The information module receives information from the cameras of miniature 3D camera one, miniature 3D camera two, and miniature 3D camera three, and constructs a background and object model of the virtual scene. The background and model in the virtual scene are then sent to the image display module, which is a computer screen. After the computer screen establishes an interactive connection with the information module in the VR system, the operator can experience the stereoscopic imaging by wearing a smart wearable device.

[0025] Compared with existing technologies, the advantages of this invention are:

[0026] This invention proposes a human-computer interaction operating system and method based on a BCI and VR integrated robot working environment. The BCI system collects the operator's motion imagination intentions, analyzes them, and sends them to the VR system. The VR system executes the robot to complete the task instructions assigned by the operator, thus eliminating safety accidents caused by improper operation by the operator during the production process or operation. At the same time, it also achieves the beneficial effect of reducing workplace stress and making production work as easy as a game. Attached Figure Description

[0027] Figure 1 This is a system framework diagram of a human-computer interaction operating system and method based on a BCI and VR fusion robot working environment in this invention;

[0028] Figure 2 This is a flowchart illustrating the operation of a robot system based on a human-computer interaction operating system and method for a robot working environment that integrates BCI and VR, as described in this invention.

[0029] Figure 3This is a flowchart of the BCI system data acquisition process for a human-computer interaction operating system and method based on a BCI and VR fusion robot working environment, as described in this invention.

[0030] Figure 4 This is a flowchart of a 3D camera imaging process for a human-computer interaction operating system and method based on a BCI and VR fusion robot working environment in this invention.

[0031] Figure 5 This is a system operation flowchart of a human-computer interaction operating system and method based on a BCI and VR fusion robot working environment in this invention.

[0032] In the diagram: 1-BCI control system; 2-Physical environment system; 3-VR assistance system; 4-Robot equipment system; 5-Command analysis and judgment module; 6-Control motion module; 7-Mechanical structure module; 8-Traveling device; 9-Traveling controller; 10-Displacement sensor; 11-First degree of freedom arm; 12-Second degree of freedom arm; 13-Clamping device; 14-Traveling controller; 15-First controller; 16-Second controller; 17-Third controller; 18-Displacement sensor; 19-Angle sensor one; 20-Angle sensor two; 21-Angle sensor three; 22-Communication device; 23-Correct command information output port; 24-Incorrect command information output port; 25-Signal acquisition system; 26-Signal processing system; 27-Signal output system; 28-Information module; 29-Image display module; 30-Miniature 3D camera one; 31-Miniature 3D camera two; 32-Miniature 3D camera three. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Example 1, please refer to Figure 1 This document describes a human-computer interaction operating system based on a BCI and VR-integrated robot working environment. The operating system includes a BCI control system 1, a physical environment system 2, a VR assistance system 3, and a robot equipment system 4 interconnected with the VR assistance system 3. The BCI control system 1 is used for acquiring source signals, generating signals, monitoring signals, processing signals, and extracting BCI signal features. In this embodiment, please refer to... Figure 3In the BCI system data acquisition flowchart, the BCI control system 1 includes a smart helmet, which houses a signal acquisition system 25, a signal processing system 26, and a signal output system 27. The signal acquisition system 25 collects raw EEG and EMG signals from the operator and sends the raw data to the signal processing system 26. The signal processing system 26 preprocesses the raw data, extracts features, classifies the information into patterns, and then transmits it to the signal output system 27. In this embodiment, the main function of preprocessing is to filter out non-EEG data, i.e., artifacts, from the signal. The main methods include: 1. Filtering and regression (using filters); 2. Blind source separation or signal decomposition (first decomposing the signal, then identifying artifacts in the signal components, and finally assembling non-artifact components). In the feature extraction of this embodiment, the features of the EEG signal are mainly distributed in the time domain, frequency domain, and spatial domain. Using the feature signal as the source signal, various parameters are determined and used as vectors to form a feature vector representing the signal features. Feature parameters include time-domain signals (such as amplitude) and frequency-domain signals (such as frequency). There are two main categories, and the corresponding feature extraction methods are also divided into time-domain methods, frequency-domain methods, and time-frequency-domain methods. Information classification is based on the characteristic that EEG signals can produce different responses to EEG activity according to different movements or consciousness, and determines the relationship between the type of movement or consciousness and the feature signals. The quality of the signal classification results depends on two factors: first, whether the feature signals to be classified have obvious characteristics, that is, the nature of the feature signals; and second, whether the classification method is effective.

[0035] Example 2: Based on the above examples, the electroencephalogram (EEG) and electromyogram (EMG) signals are neurophysiological signals from the EEG. The signal acquisition system 25 is a non-embedded acquisition system, mainly acquiring the EEG and EMG signals of the operator by setting electrode sensors inside the smart helmet. In this example, BCI stands for Brain-Computer Interface. BCI is a signal transmission technology established between the brain and machine devices. Simply put, the human brain can be controlled by the machine through BCI technology, or the machine can provide feedback to the human brain's thoughts. During the signal acquisition process, various brain signals (such as Electroencephalogram (EEG), Magnetoencephalogram (MEG), Functional Magnetic Resonance Imaging (FMRI), Functional Near Infrared Spectroscopy (FIR), Cortical EEG, and Local Field Potentials, etc.) can all be used as signals for the BCI. In this example, the brain signal is the Electroencephalogram (EEG). EEG is an electrical signal, usually acquired using electrodes that contact the scalp.

[0036] In Example 3, based on the above examples, the BCI control system 3 connects to the VR assistance system 1 and the physical environment system 2 via communication device 22 to form an interactive system. The VR assistance system 3 and the physical environment system 2 are connected to the robot equipment system 4 via communication device 22 to form an interactive system. The VR assistance system includes a smart wearable device and a smart application terminal device. The smart application terminal device includes an image display module 29 and an information module 28. After the image display module 29 and the information module 28 establish an information interaction connection, the smart wearable device is located between the image display module 29 and the information module 28. By having the operator wear the smart wearable device, a visual sense of presence is given.

[0037] Example 4, please refer to Figure 2 Combination Figure 4 In the 3D camera recording flowchart, the recording is performed by a 3D camera in the robot equipment system. The robot equipment system 4 includes an instruction analysis and judgment module 5, a motion control module 6, and a mechanical structure module 7. The mechanical structure module 7 includes a traveling device 8 and a robotic arm. The traveling device includes a first traveling device and a corresponding second traveling device, which are the robot's walking and moving devices. Both the first and second traveling devices are equipped with a traveling controller 9 and a displacement sensor 10. The robotic arm includes a first degree-of-freedom arm 11, a second degree-of-freedom arm 12, and a gripping device 13. The motion control module includes a traveling controller 14, a first controller 15, a second controller 16, a third controller 17, a displacement sensor 18, an angle sensor 19, an angle sensor 20, and an angle sensor 31. The first controller 15 and angle sensor 19 are mounted on the first degree-of-freedom arm 11; the second controller 16 and angle sensor 20 are mounted on the second degree-of-freedom arm 12; the third controller 17 and angle sensor 31 are mounted on the clamping device 13; a positioning device is also mounted on the clamping device 13; a miniature 3D camera 30, a miniature 3D camera 31, and a miniature 3D camera 32 are respectively mounted on the first degree-of-freedom arm 11, the second degree-of-freedom arm 12, and the clamping device 13; in this embodiment, the traveling device, the first degree-of-freedom arm 11, the second degree-of-freedom arm 12, and the clamping device 9 are also equipped with freely movable joints. The robot can perform all the action commands that a human can complete, such as rotation, left turn, right turn, moving, lifting, dragging, pulling, clamping, moving, and walking, under the condition of the integration of BCI technology and VR technology, according to the needs of on-site work.

[0038] Example 4 continues based on the above examples. The smart wearable device can be any one or more of the following: headband, goggles, gloves, or vest. In this example, the information module 28 also includes a motion capture unit and a computing unit. The VR assistance system 3 establishes an information interaction connection with the instruction analysis and judgment module 5 via the communication device 22. The instruction analysis and judgment module 5 includes multiple correct instruction information output ports 23 and incorrect instruction information output ports 24. The multiple correct instruction information output ports 23 establish information interaction connections with the movement controller 14, the first controller 15, the second controller 16, and the third controller 17, respectively. In this example, the judgment of correct and incorrect instruction information is performed by the instruction analysis and judgment module 5 in the robot, which parses and filters the action instructions transmitted in the previous step and distributes the instructions that match the robot's ability to complete the action to the movement controller 14, the first controller 15, the second controller 16, and the third controller. Action instructions that the robot cannot complete are fed back to the signal output system.

[0039] Example 5, please refer to Figure 3 In the physical environment system 2, there are performing personnel, virtual personnel, on-site hydraulic equipment, on-site operating robots, and tasks performed by the robots. When the signal acquisition system collects the electroencephalogram (EEG) and electromyogram (EMG) information of the performing personnel in the physical environment system, it sends it to the signal processing system in the BCI. The signal processing system processes the signal and sends it to the signal output system. The signal output system then uses the information module to synthesize the virtual personnel and overlays them onto the on-site operating robot, which then completes the work at the on-site hydraulic equipment.

[0040] In Example 6, based on Implementation 1 above, the signal output system 27 transmits instruction information to the information module 28 in the VR assistance system 3. After collecting all the information, the information module 28 establishes different types of target instructions, synthesizes the different types of target instructions into virtual operation action demonstration instructions for virtual workers in the virtual scene, and sends the corresponding synthesized instructions to the travel controller 14, the first controller 15, the second controller 16, and the third controller 17 respectively to control their first travel device, the second travel device, the first degree of freedom arm 11, the second degree of freedom arm 12, and the clamping device 13 to complete the task action instructions.

[0041] In Example 7, based on Example 4, the motion capture unit is used to capture the movements of different parts of the virtual worker in the image and send them to the computing unit. The motion capture is performed by sensing the movement path of the head or body through sensors in the smart wearable device. The sensors include sensors for head rotation, sensors for body forward, backward or left and right movement, and sensors for gesture and eye movement. The computing unit identifies the current virtual worker's movements through a series of calculations and analyses and sends them to the robot, which then covers the travel controller 9, the first controller 15, the second controller 16 and the third controller 17 in the robot equipment system.

[0042] In Example 8, continuing from Example 4, miniature 3D cameras 30, 31, and 32 establish information interaction connections with the information module 28 via communication device 22. The information module 28 receives the information fed back from the miniature 3D cameras 30, 31, and 32 after they have captured images, constructs background and object models of the virtual scene, and sends the background and models in the virtual scene to the image display module 29.

[0043] In Example 9, continuing from Example 4, angle sensor 19, angle sensor 20, and angle sensor 31 perceive the environment around the robot arm through different angles generated by the rotation or movement of the first degree-of-freedom arm 11, the second degree-of-freedom arm 12, and the clamping device 13, respectively, and transmit the information to the information module 28 via 3D imaging; the displacement sensor 18 perceives the environment around the robot's traveling device through the forward or backward distance of the first and second traveling devices, and transmits the information to the information module 28 via 3D imaging.

[0044] Example 10, please refer to Figure 5 Among them, a human-computer interaction operation method for a BCI and VR integrated robot working environment includes any of the above-mentioned human-computer interaction operating systems based on the BCI and VR integrated hydraulic robotic arm working environment, and the method includes:

[0045] 1: The operators arrange the process flow that the robot needs to complete on-site according to the process sequence;

[0046] 2: The signal acquisition system 25 collects raw data of electroencephalogram (EEG) and electromyogram (EMG) signals. The raw data is the action instructions in the process flow. After wearing the smart helmet, the operator watches the process flow and concentrates his mind to imagine the action instructions that the robot needs to complete in the process flow.

[0047] 3: The electrode sensors of the signal acquisition system 25 collect the action command information of the operator's mind during the process through the cerebral cortex; the action command information includes information of different wavebands in the subcortical region, and the effective waveband information is collected;

[0048] 4: The signal acquisition system 25 sends the valid band information to the signal processing system 26;

[0049] 5: In the signal processing system 26, the raw data is preprocessed, features are extracted, and information is classified into patterns before being transmitted to the signal output system 27;

[0050] 6: The signal output system 27 distributes information commands to the information module 28 in the VR assistance system 3 through the corresponding interface. After collecting all the information, the information module 28 establishes different types of target commands and synthesizes the different types of target commands into an image to demonstrate the operation actions of the virtual worker in the virtual scene. After the information module 28 identifies and calculates the action commands of the virtual worker through the motion capture unit and the computing unit, it sends the action command information to the robot's command analysis and judgment module 5. After analyzing the command information, the command analysis and judgment module 5 sends the correct command information to the travel controller 14, the first controller 15, the second controller 16, and the third controller 17 respectively, which control their travel devices and robotic arms to complete the corresponding action command tasks. The incorrect command information is fed back to the signal output system 27, which then feeds it back to the signal processing system for processing.

[0051] 7: Simultaneously, the 3D camera device located on the robot transmits the background and object models of the robot's surrounding environment to the information module 28 in the VR assistance system 3.

[0052] 8: Information module 28 receives information from miniature 3D camera 1 30, miniature 3D camera 2 31 and miniature 3D camera 32 to construct the background and object models of the virtual scene, and sends the background and models in the virtual scene to image display module 29. Image display module 29 is a computer screen. After the computer screen establishes an interactive connection with information module 28 in the VR system, the operator can experience the virtual working environment in stereoscopic imaging by wearing a smart wearable device.

[0053] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A human-computer interaction operation system based on BCI and VR fusion robot working environment, comprising a BCI control system (1), a physical environment system (2), a VR assistance system (3), and a robot device system (4) connected with the VR assistance system (3), characterized in that: The BCI control system (1) is used for source signal source acquisition, signal generation, signal monitoring, signal processing and BCI signal feature extraction; the BCI control system (1) is connected with a VR assisting system (3) and a physical environment system (2) through a communication device (22) to form an interactive system, the VR assisting system (3) and the physical environment system (2) are connected with a robot equipment system (4) through a communication device (22) to form an interactive system; the robot equipment system (4) comprises an instruction analysis and judgment module (5), a control motion module (6) and a mechanical structure module (7); the mechanical structure module (7) comprises a traveling device (8) and a mechanical arm, the traveling device comprises a first traveling device and a corresponding second traveling device; the mechanical arm comprises a first degree of freedom arm (11), a second degree of freedom arm (12) and a clamping device (13); the control motion module (6) comprises a traveling controller (9), a first controller (15), a second controller (16), a third controller (17), a displacement sensor (10), an angle sensor one (19), an angle sensor two (20) and an angle sensor three (21); the traveling controller (9) and the displacement sensor (10) are arranged in the first traveling device and the second traveling device; the first controller (15) and the angle sensor one (19) are arranged on the first degree of freedom arm (11); the second controller (16) and the angle sensor two (20) are arranged on the second degree of freedom arm (12); the third controller (17) and the angle sensor three (21) are arranged on the clamping device (13); a positioning device is further arranged on the clamping device (13); the VR assisting system (3) is connected with the instruction analysis and judgment module (5) through the communication device (22) to establish information interaction connection; The instruction analysis and judgment module (5) comprises a plurality of correct instruction information output ports (23) and error instruction information output ports (24); the plurality of correct instruction information output ports (23) are connected with the traveling controller (9), the first controller (15), the second controller (16) and the third controller (17) to establish information interaction connection; the physical environment system (2) comprises an execution worker, a virtual execution worker, a field hydraulic device, a field operation robot and a task executed by the robot. The BCI control system (1) comprises an intelligent helmet, a signal acquisition system (25), a signal processing system (26) and a signal output system (27) arranged in the intelligent helmet; the signal acquisition system (25) acquires raw data of brain electrical signals and muscle electrical signals of an execution worker and sends the raw data to the signal processing system (26); the signal processing system (26) pre-processes, extracts features and classifies information modes of the raw data and then transmits the data to the signal output system (27); the brain electrical signals and the muscle electrical signals are neurophysiological signals in an electroencephalogram, and the signal acquisition system (25) is a non-embedded acquisition system which acquires the brain electrical signals and the muscle electrical signals of the execution worker by arranging electrode sensors in the intelligent helmet; The signal output system (27) transmits instruction information to an information module (28) in a VR assistance system (3); the information module (28) collects all information and establishes different types of target instructions; different types of target instructions are synthesized into operation action demonstration instructions of a virtual worker in a virtual scene; corresponding synthesized instructions are respectively sent to a travel controller (9), a first controller (15), a second controller (16) and a third controller (17) to control a first travel device, a second travel device, a first degree of freedom arm (11), a second degree of freedom arm (12) and a clamping device (13) to complete task action instructions; the VR assistance system (3) further comprises intelligent wearable devices and intelligent application terminal devices, and the intelligent application terminal devices comprise an image display module (29); after the image display module (29) and the information module (28) are connected in information interaction, the intelligent wearable devices are located between the image display module (29) and the information module (28), and the execution worker wears the intelligent wearable devices to give a visual sense of presence; Miniature 3D cameras one (30), two (31) and three (32) are arranged on the first degree of freedom arm (11), the second degree of freedom arm (12) and the clamping device (13) respectively; the intelligent wearable devices are any one of a head-mounted type, a glasses type, a glove type and a waistcoat type, and the information module (28) further comprises a motion capture unit and a calculation unit.

2. The human-computer interaction operation system based on the fusion of BCI and VR robot working environment according to claim 1, characterized in that: The motion capture unit is used to capture the motion of different parts of the virtual worker in an image and send the motion to the calculation unit; the motion is captured through sensors in the intelligent wearable devices sensing the motion path of the head or the body; the sensors comprise a capture sensor for head rotation, a capture sensor for forward and backward or left and right motion of the body, and a capture sensor for hand gestures and eye movements; the calculation unit identifies the current motion of the virtual worker through series of calculation and analysis and sends the motion to the robot, covering the travel controller (9), the first controller (15), the second controller (16) and the third controller (17) in the robot device system.

3. The human-machine interaction operation system based on the fusion of BCI and VR robot working environment according to claim 1, characterized in that: The micro 3D camera one (30), micro 3D camera two (31) and micro 3D camera three (32) are respectively connected with the information module (28) through the communication device (22) to establish information interaction connection; the information module (28) receives the information fed back by the micro 3D camera one (30), micro 3D camera two (31) and micro 3D camera three (32) after shooting to construct the background and object model of the virtual scene, and sends the background and model in the virtual scene to the image display module (29).

4. The human-machine interaction operation system based on the fusion of BCI and VR robot working environment according to claim 1, characterized in that: The angle sensor one (19), angle sensor two (20) and angle sensor three (21) respectively perceive the environment around the robot arm through 3D camera transmission to the information module (28) through different angles generated by the rotation or movement of the first degree of freedom arm (11), second degree of freedom arm (12) and clamping device (13); the displacement sensor (10) perceives the environment around the robot advancing device through 3D camera transmission to the information module (28) through the distance of the forward or backward movement of the first advancing device and second advancing device.

5. A method for human-computer interaction operation of a BCI and VR fused robot working environment, characterized in that, The method comprises the following steps: S1: the execution worker arranges the process flow required by the robot to be completed in the field according to the process sequence; S2: the signal acquisition system (25) acquires the original data of the electroencephalogram and electromyogram, and the original data is the action instruction in the process flow; the execution worker wears the smart helmet, watches the process flow and concentrates on the action instruction required by the robot to be completed in the process flow; S3: the electrode sensor of the signal acquisition system (25) acquires the action instruction information of the process flow in the brain of the execution worker through the cerebral cortex; the action instruction information includes the information of different wave bands under the cerebral cortex, and the effective wave band information is acquired; S4: the signal acquisition system (25) sends the effective wave band information to the signal processing system (26); S5: the signal processing system (26) pre-processes, extracts features and classifies the information, and then transmits the information to the signal output system (27); S6: The signal output system (27) distributes the information instructions to the information module (28) in the VR assistance system (3) through the corresponding interface. The information module (28) collects all the information and establishes different types of target instructions. The different types of target instructions are synthesized into the image of the virtual operator's operation action demonstration instruction in the virtual scene. After the information module (28) identifies and calculates the action instruction of the virtual operator through the action capture unit and the operation unit, the action instruction information is sent to the instruction analysis and judgment module (5) of the robot. After the instruction analysis and judgment module (5) analyzes the instruction information, the correct instruction information is transmitted to the travel controller (9), the first controller (15), the second controller (16) and the third controller (17) respectively, which control the travel device and the mechanical arm to complete the corresponding action instruction task. The error instruction information is fed back to the signal output system (27), which is then fed back to the signal processing system (26) for processing. S7: At the same time, the miniature 3D camera one (30), the miniature 3D camera two (31) and the miniature 3D camera three (32) on the robot send the background and object model in the surrounding environment of the camera robot scene to the information module (28) in the VR assistance system (3); S8: The information module (28) receives the feedback information after the miniature 3D camera one (30), the miniature 3D camera two (31) and the miniature 3D camera three (32) take pictures and constructs the background and object model of the virtual scene. The background and model in the virtual scene are sent to the image display module (29). The image display module (29) is a computer display screen. After the computer display screen and the information module (28) in the VR system establish interactive connection, and after the execution operator wears the smart wearable device, the stereoscopic imaging is experienced.

Images