Man-machine cooperation interactive operation system and method based on multi-bci and mr fusion

Abstract

The application discloses a man-machine collaborative interaction operation system and method based on multi-BCI and MR fusion, belongs to the field of man-machine hybrid intelligent interaction control, and selects a task by an intelligent operator controlling an AR-BCI assisting system, judges whether a collaborative robot needs to participate in assistance, when the collaborative robot needs to participate, a VR control center acquires AR glasses and collaborative robot position information, then analyzes a collaborative robot path in unity, the AR glasses present path planning information, the intelligent operator controls the collaborative robot to reach a specified working position according to the AR presented path information, after reaching the specified working position, executes a subsequent collaborative task, and when the robot does not need to participate, directly executes the collaborative task step.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of human-computer hybrid intelligent interaction control, and in particular to a human-computer collaborative interaction operation system and method based on multi-BCI and MR fusion. BACKGROUND

[0002] BCI systems based on SSVEP and MI have gradually become a research hotspot in brain-computer interface technology in recent years. By combining AR with BCI, the human-computer interaction between the user and the surrounding environment or remote device can be significantly enhanced, that is, according to the current field of view or the state of the environment of the user, the presented SSVEP stimulation interface and the MI motor imagination interface are dynamically adjusted, which can provide the user with an adaptive and personalized interaction interface. Moreover, by providing real-time robot actions and other information to the user as interaction state feedback, that is, visual feedback, the user can more intuitively understand the current interaction state, and control the working speed of the collaborative robot in combination with the electromyographic signal, and adjust the output of the next instruction according to the result of the current instruction, thereby improving the efficiency of human-computer interaction.

[0003] The patent for invention with publication number CN 110134243 A discloses a brain-controlled robotic arm sharing control system and method based on augmented reality. The system includes a computer vision device, a steady-state visual evoked potential brain-computer interface device, a sharing control device, an augmented reality device, and a robotic arm. The computer vision device is used to identify the color and position information of objects in the workspace. The augmented reality device is used to generate visual stimulation signals and act on the user. The steady-state visual evoked potential brain-computer interface device acquires the user's electroencephalogram signal and converts it into a visual brain control command. The sharing control device uses a sharing control strategy to control the robotic arm. The robotic arm realizes the grasping of target objects under the control of the sharing control device. The invention combines human brain control commands with computer vision-based autonomous control of the robotic arm to achieve control functions, realizes more natural human-computer interaction, reduces the psychological burden of the user, enhances the applicability of the robotic arm in complex environments, and thus improves the practicality and reliability of the entire system.

[0004] The invention patent with publication number CN 114841215 A discloses a brain-computer interface intelligent device control system and method. In this system, a camera installed on an augmented reality device can collect environmental images in real time and transmit the images to a scene understanding module. This module can understand scene information by using deep learning technology and transmit the understood information to an augmented reality display device. The eye tracker built into the augmented reality display device can obtain the user's gaze position in real time. It combines the SSVEP paradigm to operate on the target in the scene, and then sends control or information acquisition instructions to the target and provides feedback to the augmented reality device. This fully leverages the enhancement of cognitive, visual, and perceptual abilities brought about by the combination of brain-computer interface and augmented reality technology, and realizes a solution for paralyzed patients to intuitively and efficiently control external devices.

[0005] Chinese patent application CN 113101021 A discloses a robotic arm control method based on a MI-SSVEP hybrid brain-computer interface. This method achieves robotic arm control through steps including non-invasive EEG data acquisition, online preprocessing of EEG signals, online processing and decoding of EEG signals, generation of user intent targets, and conversion of those targets into control commands. By employing an experimental paradigm based on the MI-SSVEP hybrid brain-computer interface, it increases the number of degrees of freedom in the brain-computer interface-controlled robotic arm system, enabling motion control of the robotic arm in three-dimensional space. This enhances the flexibility of user control and establishes a novel mapping relationship between brain-computer interface signal decoding results and user intent, making it more intuitive for user control and improving the efficiency of user-controlled robotic arm control.

[0006] The invention patent with publication number CN 112454363 A discloses an AR aid for welding operations.

[0007] The robot control method includes: 1. Using a wireless handheld pointer for the user to define the path and tasks in the robot's work area; 2. The user defines the critical path; 3. Setting up a virtual robot in an AR scene, overlaying the real robot in the workspace, and displaying the motion parameters of the robot for specific planned tasks in real time; pointing along the path to the virtual robot arm, the virtual robot arm follows the handheld pointer, and the corresponding posture of the virtual robot arm's links is calculated using an inverse kinematics solver. This invention overcomes the shortcomings of existing AR-assisted technology applications in the welding field, featuring an AR user interface and a handheld pointer for interaction. Users can move freely within the work unit to truly define the robot's following 3D points and paths, enabling quick and intuitive definition of the robot path.

[0008] The above research is based on brain-computer interface technology or AR technology to achieve robot collaboration, but it has the following drawbacks.

[0009] 1. In the aspect of perception, when the intelligent operator wears AR glasses to perform tasks, the AR glasses present all task information on the AR interface, which may affect the operation of the intelligent operator due to too much information presented by the interface;

[0010] 2. In the aspect of decision-making, during the combination of brain-computer and AR glasses, the calculation ability of the AR glasses is limited, and only simple tasks can be performed by using brain-computer to control AR;

[0011] 3. In the aspect of control, the SSVEP-based steady-state visual evoked stimulation interface can only realize fixed selection of several pictures, and cannot dynamically update option pictures to realize selection of different functions according to task content, and the control instructions that can be executed are less;

[0012] 4. In the aspect of task execution, when the mechanical arm is controlled by using the electroencephalogram signal, the intelligent operator and the machine are only in a simple control and controlled relationship, and there is no good cooperation between the two, and the problem of collision between the mechanical arm and the intelligent operator and the surrounding environment is not considered. SUMMARY

[0013] Based on the defects in the prior art, the purpose of the present application is to provide a man-machine collaborative interaction operation system and method based on multi-BCI and MR fusion, so as to control the AR manipulation system to execute task instructions and adaptively adjust the AR interface by using the electroencephalogram and electromyogram signals, and at the same time, the AR manipulation system sends commands to the VR control center, and the VR control center controls the collaborative robot according to the task instructions.

[0014] To solve the above technical problems, according to one aspect of the present application, a man-machine collaborative interaction operation system based on multi-BCI and MR fusion is provided, which comprises an AI control system, an AR-BCI assistance system and a physical working environment;

[0015] The AI control system comprises a VR control center, a data analysis center and an image management center; the AR-BCI assistance system comprises an AR information module, an electroencephalogram acquisition module and an electromyogram acquisition module; and the physical working environment comprises an intelligent operator, a collaborative robot and a collaborative task.

[0016] The AI control system is used for controlling the collaborative robot, wherein the image management center is used for on-site extraction and storage of images; the data analysis center is used for storing a training model generated by the electroencephalogram and electromyogram signals and planning a path for the collaborative robot; and the VR control center is used for connecting the image management center and the data analysis center, wherein after the VR control center obtains information transmitted by the AR-BCI assistance system, the behavior intention of the intelligent operator is analyzed by calling the image management center and the data analysis center, and the task instructions are transmitted back to the AR information module.

[0017] The AR-BCI assistance system is used for connecting the AI control system in information dimension and the physical working environment in physical dimension, wherein the electroencephalogram signal module is used for collecting and processing the electroencephalogram signal; the electromyography collection module is used for collecting the electromyography signal of the intelligent operator during work; the AR information module is used for presenting the prepared AR interface, observing the task information in the first person perspective and sending the control instruction to the VR control center, and the VR control center presents the decision result to the AR information module according to the task instruction transmitted by the AR-BCI assistance system, so that the intelligent operator makes a selection and assists the intelligent operator to complete the task.

[0018] In the physical working environment, the intelligent operator wears the AR glasses to perform the collaborative task in the working area, and the collaborative robot is located in the standby area and waits for the instruction of the intelligent operator.

[0019] Further, the training model generated by storing the electroencephalogram and electromyography signal is used for comparing the current electroencephalogram and electromyography signal with the training model of the data analysis center in the online experiment process, identifying the current intention of the intelligent operator according to the label made to the model of the data analysis center.

[0020] Further, the image management of the AI control system directly calls the stored picture for the task without changing the task option, extracts the picture on site for the task with the task option needing to be updated, takes a picture to obtain the picture by calling the photo capture class in the VS, saves the picture to the specified file location, changes the picture name at the same time, repeatedly extracts the picture according to the task requirement until the result is met, and then runs the corresponding brain-computer code.

[0021] Further, the electroencephalogram signal module is used for: (1) collecting the electroencephalogram signal generated when the intelligent operator gazes at the corresponding block of the SSVEP stimulation interface, and sending the electroencephalogram signal to the AR information module, the AR information module is used for presenting the SSVEP stimulation interface, and the electroencephalogram signal is preprocessed, then identified by using the CCA algorithm, and the AR-BCI assistance system is controlled to execute the corresponding instruction by analyzing the SSVEP; (2) collecting the offline training electroencephalogram signal generated when the intelligent operator gazes at the MI interaction interface for motor imagination, preprocessing the offline training electroencephalogram signal, and generating a training model by using the CSP algorithm; (3) obtaining the online training electroencephalogram signal generated when the intelligent operator gazes at the MI interaction interface for motor imagination during online training, preprocessing the online training electroencephalogram signal, and classifying and identifying the online training electroencephalogram signal according to the training model and the CSP algorithm, obtaining the motor imagination intention of the intelligent operator, controlling the information to be presented by the AR interface by analyzing the MI, and adaptively adjusting the AR control interface.

[0022] Further, the AR-BCI assistance system collects the electromyographic signals of the intelligent operator during work through the electromyographic acquisition device, selects three variables such as average absolute value, root mean square value, and waveform length as the characteristics of the electromyographic signals, constructs a feature classifier of the electromyographic signals based on SVM, tests the test data by using the constructed classifier, and uses the classifier for collaborative intention recognition.

[0023] Further, the collaborative robot in the physical working environment includes a mechanical arm, a computer vision device, and a laser detector, etc. The computer vision device is used to obtain the poses of the intelligent operator and the task target. The mechanical arm is used for object clamping and carrying. The laser detector is used for scanning the working scene and perceiving the environment around the collaborative robot.

[0024] Further, the computer vision device of the collaborative robot is located at the position of the mechanical arm of the collaborative robot, and a depth camera is used to identify the human behavior characteristics and the pose of the task target. The identification of the human behavior characteristics is used to identify the intention of the worker in the VR control center, to determine the next work of the human based on the labels of the pre-set training model library, and to control the collaborative robot to assist the human to work. The identification of the pose of the task target is used to make the collaborative robot accurately obtain the position and pose of the required object, and to analyze the path planning of the mechanical arm through the VR control center. The laser detector can measure the distance and reflectivity of the surrounding environment in a long range, and can repair the point cloud data in the virtual space after obtaining the point cloud data of the laser detector, and construct a virtual working environment with the same scale as the real environment.

[0025] Further, the intelligent operator wears AR glasses to perform tasks in the physical working space, and the AR-BCI assistance system is controlled by the brain and electromyographic signals of the intelligent operator to adaptively perform tasks. The VR control center plans the task path of the collaborative robot according to the instructions transmitted by the AR-BCI assistance system. The computer vision device transmits the collected information to the VR control center and compares and analyzes the information with the model of the data analysis center, so as to control the collaborative robot and the intelligent operator to work cooperatively.

[0026] Further, the collaborative task in the physical working environment includes a judgment question, a selection question and a dynamic update question; the judgment question is made of two pictures, the code in the SSVEP paradigm is changed to identify the two pictures, the selection question is made of 3-8 pictures, which respectively represent different positions that are wanted to view, and the dynamic update question firstly calls a photo capture object to shoot each part that needs to be clamped, saves to a specified file position, and changes the picture name to realize the acquisition of dynamic pictures. It is needed to judge whether the robot is assisted to participate in the execution of the task, the selection judgment question controls the AR-BCI assistance system through the electroencephalogram signal, identifies the task picture to execute the corresponding task instruction, the dynamic update picture uploads the collected picture to an AI control system, the AI control system identifies the key information of the uploaded picture, and a camera on the mechanical arm clamps the corresponding object by using a deep learning algorithm.

[0027] According to another aspect of the present application, a multi-BCI and MR fusion human-machine collaborative interaction operation method is provided, which is realized according to the multi-BCI and MR fusion human-machine collaborative interaction operation system described above;

[0028] The AR-BCI assistance system selects a task controlled by the intelligent operator, judges whether the collaborative robot needs to participate in assistance, when the collaborative robot needs to participate, the VR control center acquires the position information of the AR glasses and the collaborative robot, then analyzes the path of the collaborative robot in unity, the AR glasses present the path planning information, the intelligent operator controls the collaborative robot to reach the specified working position according to the path information presented by the AR glasses, and after reaching the specified working position, executes the subsequent collaborative task, when the robot does not need to participate, directly executes the collaborative task.

[0029] When the collaborative task is executed, first, the AR glasses present a visual stimulation interface, the intelligent operator selects the collaborative task based on the ssvep electroencephalogram signal, then the AR glasses present an MI interaction interface, the intelligent operator selects the information that is wanted to view according to his own style preference, and controls the AR-BCI assistance system through the acquisition of the electroencephalogram signal, so as to control the mechanical arm of the collaborative robot to execute the task, during the execution of the task by the mechanical arm, the intelligent operator and the collaborative robot are located in the same working area, and the movement speed of the mechanical arm is controlled through the acquisition of the electromyogram signal of the intelligent operator; the intelligent operator judges whether the task is completed according to the execution of the task, if not, returns to the visual stimulation interface to make a new selection, if yes, the task is ended.

[0030] According to the application, the intelligent operator wears AR glasses to perform tasks in a physical workspace, and the AR-BCI assistance system is controlled by collecting the brain and muscle signals of the intelligent operator to adaptively perform tasks, the VR control center plans the task path of the collaborative robot according to the instructions transmitted by the AR-BCI assistance system, and the computer vision device compares and analyzes the collected information with the model of the data analysis center after transmitting the collected information to the VR control center, so as to control the collaborative robot and the intelligent operator to work cooperatively.

[0031] Compared with the prior art, the beneficial technical effects of the application mainly lie in the following aspects,

[0032] In terms of perception, the task information presented by the AR glasses in the application can be adaptively adjusted, the AR task interface is controlled by collecting the MI motor imagery type brain electrical signals to present the information that the user wants to present, so that the excessive AR interface information does not affect the line of sight of the intelligent operator and the work efficiency is reduced.

[0033] In terms of decision-making, the application is based on brain-computer and MR to work, in the traditional brain-computer and AR control mechanical arm process, the AR solving ability is not enough, and only some simple control can be performed on the mechanical arm, while a large amount of complex information can be analyzed through MR, the data obtained by the data analysis center can accurately identify the brain-computer signal and plan the path of the collaborative robot, so that the intelligent operator can more easily control the robot to cooperate.

[0034] In terms of control, the application is provided with an image management center, which can collect pictures in real time, collect the required task pictures on the scene for different task scenes, select the pictures through the steady-state visual evoked stimulation interface of SSVEP, extract and identify the features of the pictures, and control the mechanical arm to grip the object through the vision acquisition device of the collaborative robot, so as to enrich the control instructions that can be realized by the brain-computer.

[0035] In terms of task execution, the application is a man-machine cooperation method that controls the collaborative robot by combining electroencephalogram and electromyogram, the intelligent operator generates brain electrical signals to control the intervention of the collaborative robot to perform corresponding tasks, the electromyogram controls the working speed of the collaborative robot, the collaborative robot obtains the current behavior characteristics of the intelligent operator through the vision acquisition device, and predicts the next work to be done.

[0036] The multi-BCI and MR fusion human-machine collaborative interaction operation system can be applied in many fields, and the application mainly introduces the application in the operation of coal mine underground. Some applications of the system in the operation of coal mine underground are as follows: in the robot operation aspect, the system can realize the control of the robot through the BCI technology, such as controlling the robot to perform tasks such as carrying goods and anchoring; in the environment perception aspect, the system can provide environment perception through the MR technology, including the information of the terrain, obstacles and pipelines in the scene, so that the operator can better understand the operation environment and avoid the collision or jamming of the robot; in the task allocation aspect, the system can realize the allocation of tasks through the MR technology, including the priority and time limit setting of the tasks, so that the operator can better understand the task situation and improve the task completion efficiency; in the fault diagnosis aspect, the system can assist the operator to realize the fault diagnosis of the coal mine underground, including the diagnosis of machine failure, pipeline blockage and the like, which is convenient for the operator to handle the fault in time; in the safety monitoring aspect, the system can monitor the physiological state of the operator, such as fatigue and anxiety, and remind the operator to pay attention to safety. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is the element structure and interface diagram of the application;

[0038] Figure 2 is the perception-decision-control flowchart of the application;

[0039] Figure 3 is the task planning diagram of the application;

[0040] Figure 4 is the specific task flowchart of the application;

[0041] Figure 5 is the brain-machine flowchart of the application. DETAILED DESCRIPTION

[0042] As Figure 1 shown, a typical embodiment of the application provides a human-machine collaborative interaction operation system based on multi-BCI and MR fusion, which includes an AI control system, an AR-BCI assistance system and a physical working environment. The AI control system includes a VR control center, a data analysis center and an image management center; the AR-BCI assistance system includes an AR information module, an electroencephalogram acquisition module and an electromyography acquisition module; the physical working environment includes an intelligent operator, a collaborative robot and a collaborative task.

[0043] (I) AI control system

[0044] The AI control system is mainly used for controlling the collaborative robot. Through the collection and processing of pictures by the image management center, the pictures of the SSVEP stimulation interface can be updated and called to update the task instructions dynamically; through the analysis of the data analysis center, the instructions sent by the electroencephalogram and electromyogram signals can be accurately obtained, and the path planning of the collaborative robot can be performed; the VR control center can connect the data analysis center and the image management center to form a complete AI control system, and the control of the collaborative robot can be realized.

[0045] The VR control center is used for connecting the image management center and the data analysis center. After the VR control center obtains the transmitted information of the AR-BCI assistance system, the behavior intention of the intelligent operator can be analyzed by calling the image management center and the data analysis center, and the operation intention is transmitted back to the AR information module. The intelligent operator can judge whether the intention recognition is accurate according to the instructions presented by the AR information module, and finally make a selection.

[0046] The data analysis center includes two parts: a training model for generating electroencephalogram and electromyogram signals and a collaborative robot path planning. The training model for generating electroencephalogram and electromyogram signals is used to compare the current electroencephalogram and electromyogram signals with the training model of the data analysis center during online experiments, and to identify the current intention of the intelligent operator according to the label of the model of the data analysis center. The collaborative robot path planning mainly includes robot movement path planning and mechanical arm path planning. The robot movement path planning refers to the situation that the intelligent operator wears AR glasses to work, and part of the work needs the cooperation of the collaborative robot. Since the collaborative robot needs to assist in completing tasks at various workstations, the AR-BCI assistance system needs to call the robot to the work position, and a synchronous positioning between the AR glasses and the collaborative robot is needed. After confirming the positional relationship between the two, the path between them is analyzed in the virtual space, and the path information is sent to the AR information module for the intelligent operator to make the best choice. The mechanical arm path planning refers to the situation that the intelligent operator needs the cooperation of the mechanical arm when working at the workstation. The computer vision device identifies the human behavior characteristics and transmits them to the VR control center. The VR control center identifies the next work to be done by the intelligent operator by calling the model of the data analysis center, and controls the mechanical arm to execute the corresponding command.

[0047] The image management center is used for on-site extraction and storage of images. For tasks that do not need to change task options, stored pictures are directly called. For tasks that need to update task options, pictures are extracted on site. First, the photo capture class in VS is called to take pictures and save them to the specified file location. At the same time, the picture name is changed. According to the task requirements, the picture is repeatedly extracted until the result is met, and then the corresponding brain-computer code is run.

[0048] (II) AR-BCI assistance system

[0049] The AR-BCI assistance system is composed of an AR information module, an electromyographic signal module, and an electroencephalographic signal module, and is used to connect the AI control system in the information dimension and the physical working environment in the physical dimension. The AR information module is used to present virtual level information to the intelligent operator. The electroencephalographic signal module and the electromyographic signal module transmit the intention of the intelligent operator through the AR information module, and realize the bidirectional flow of information.

[0050] The AR information module in the AR-BCI assistance system is used to present the AR interface made in unity, observe the task information using the first-person perspective, and send control instructions to the VR control center through the AR-BCI assistance system. The VR control center presents the decision results to the AR information module according to the task instructions transmitted by the AR-BCI assistance system, and the intelligent operator makes a selection to assist the intelligent operator to complete the task.

[0051] The electroencephalographic signal module in the AR-BCI assistance system is used for the acquisition and processing of electroencephalographic signals. The electroencephalographic signals are divided into SSVEP and MI signals. The electroencephalographic signal module is used to collect the electroencephalographic signals generated when the intelligent operator gazes at the corresponding block of the SSVEP stimulation interface, and send the electroencephalographic signals to the AR information module. The AR information module is used to present the SSVEP stimulation interface, and use the CCA algorithm to identify after preprocessing the electroencephalographic signals, and control the AR-BCI assistance system to execute the corresponding instructions by analyzing the SSVEP. The electroencephalographic signal module is used to collect the offline training electroencephalographic signals generated when the intelligent operator gazes at the MI interactive interface for motor imagination, and generate a training model using the CSP algorithm after preprocessing the offline training electroencephalographic signals. The electroencephalographic acquisition device is used to obtain the online training electroencephalographic signals generated when the intelligent operator gazes at the MI interactive interface for motor imagination, and classify and identify the online training electroencephalographic signals according to the training model and the CSP algorithm after preprocessing the online training electroencephalographic signals, to obtain the motor imagination intention of the intelligent operator, control the information to be presented by the AR interface by analyzing the MI, and adaptively adjust the AR control interface.

[0052] The electromyographic acquisition device in the AR-BCI assistance system is used to collect the electromyographic signals of the intelligent operator during work. Three variables such as the average absolute value, the root mean square value, and the waveform length are selected as the characteristics of the electromyographic signals. A feature classifier of the electromyographic signals is constructed based on the SVM. The constructed classifier is used to test the test data and recognize the cooperation intention. The electromyographic signals are mainly used to measure the mental state of the operator. The working state of the operator is adaptively adjusted to adjust the speed of the collaborative robot, so as to avoid accidents.

[0053] The intelligent operator in the physical working environment refers to an operator who has a clear understanding of the work task, has sufficient knowledge of the AR-BCI assistance system and the work task, can perform the task together with the collaborative robot, and can make manual intervention when the task execution process is not appropriate.

[0054] (Three) Physical working environment

[0055] The physical working environment mainly refers to the intelligent operator wearing AR glasses to perform collaborative tasks in the working area, and the collaborative robot is located in the standby area waiting for the instructions of the intelligent operator.

[0056] Among them, the collaborative task in the physical working environment includes judgment questions, selection questions and dynamically updated questions.

[0057] The judgment question (for example: check if the valve is closed) is made of two pictures, and the code is changed to identify two pictures in the SSVEP paradigm. The selection question (for example: view the pose status of the hydraulic support top beam, shield beam, and side shield) is made of 3-8 pictures, each representing a different position to be viewed. The dynamically updated question (for example: picking up objects) first calls the photo capture object to take pictures of each part to be picked up and saves them to a specified file location, while changing the picture name to achieve dynamic picture acquisition. The execution task needs to determine whether to participate in robot assistance. The selection judgment question controls the AR-BCI assistance system through the electroencephalogram signal to identify the task picture and execute the corresponding task instruction. This part of the task does not need robot assistance. The dynamically updated picture question uploads the collected pictures to the AI control system, which identifies the key information of the uploaded pictures. The camera on the mechanical arm uses a deep learning algorithm to pick up the corresponding objects (Yolov5 algorithm).

[0058] Among them, the collaborative robot in the physical working environment includes mechanical arm, computer vision device and laser detector, etc. The computer vision device is used to obtain the pose of the intelligent operator and the task target. The mechanical arm is used for object picking and carrying. The laser detector is used to scan the working scene and perceive the environment around the collaborative robot.

[0059] The mechanical arm of the collaborative robot has good compatibility and can be applied to different application scenarios. Whether it is free to move or complete a set of action sequences. When installing different adaptive clamps, it can pick up objects of different shapes and materials. The mechanical arm can perform highly accurate linear motion and has rich functions, whether it is handheld teaching (through a teach pendant), trajectory reproduction, or application in laboratories, etc. In addition, the flexible design of the mechanical arm needs to be considered to ensure that it does not pose a danger to the user even when direct contact collisions occur.

[0060] The computer vision device of the collaborative robot is located at the position of the mechanical arm of the collaborative robot, adopts a depth camera to identify human behavior features and a pose of a task target, the human behavior features are identified to identify the intention of the staff at the VR control center, a label of a pre-set training model library is used to determine the work to be done by the human next, and the collaborative robot is controlled to assist the human to work; the pose of the task target is identified to enable the collaborative robot to accurately obtain the position and attitude of the required article, and the path planning of the mechanical arm is analyzed through the VR control center.

[0061] The laser detector of the collaborative robot can upload the measurement values of the surrounding distance and reflectivity in real time, has the ability of long-range distance measurement, and constructs a virtual working environment with the same proportion as the real environment by repairing the point cloud data in the virtual space after obtaining the point cloud data of the laser detector.

[0062] As shown in Figure 2 As shown in

[0063] At the decision-making level, the information dimension depends on the collaborative arrangement of VR and AR, the VR control center decides the path planning of the collaborative robot based on the neural network algorithm, and transmits the optimal result analyzed to the AR glasses, at the physical level, the human brain makes the final decision by using the AR-BCI assistance system according to the information presented by the AR and the perception of the physical environment.

[0064] At the control level, the information dimension is realized by the VR control center to control the collaborative robot, the VR control center has a complete program to control the collaborative robot, and at the physical dimension, the intelligent operator can control the collaborative robot through the AR glasses by using the multi-modal interaction modes such as electroencephalogram, electromyogram, voice and gesture according to the current task execution status of the collaborative robot.

[0065] As shown in Figure 3 As shown in

[0066] The collaborative robot movement path planning refers to that when the intelligent worker wears AR glasses and an electroencephalogram and electromyogram signal acquisition device to work, part of the work needs to be cooperated with a collaborative robot. Since the collaborative robot needs to reach each workbench to assist in completing the task, the AR-BCI collaborative system needs to call the collaborative robot to the work position, and thus a synchronous positioning between the two is needed. After confirming the positional relationship between the two, path planning is performed between the two according to the known site environment, and the path information is sent to the AR control system, and the human makes the best choice.

[0067] The collision detection refers to that the intelligent worker wears AR glasses to work in the work area, and the robot perceives the surrounding road conditions in the movement process by relying on the computer vision device worn by the robot itself, avoids the collision of the collaborative robot by using the pre-set parameters, and simultaneously monitors the position information of the AR glasses and the robot in the VR control center in real time. When the distance between the robot and the AR is too close, the robot stops moving.

[0068] The man-machine cooperation refers to that when the intelligent worker works at the workbench, the mechanical arm needs to be cooperated to participate. The computer vision device identifies the human behavior characteristics, and transmits the behavior characteristics to the VR control center. The VR control center judges the next work of the intelligent worker according to the data analysis center, so as to control the mechanical arm to execute the corresponding command. The mechanical arm is affected by the working state of the intelligent worker in the movement process, and the movement speed of the mechanical arm is controlled by collecting the electromyogram of the human body.

[0069] For example, Figure 4As shown, another typical embodiment of the present application provides a human-machine collaborative interaction operation method based on the fusion of multi-BCI and MR. The flow task is that the intelligent operator controls the AR-BCI assisted system to select the task, judges whether the collaborative robot needs to participate in assistance, and when the collaborative robot needs to participate, the VR control center acquires the position information of the AR glasses and the collaborative robot, then analyzes the path of the collaborative robot in unity, the AR glasses present the path planning information, the intelligent operator controls the collaborative robot to reach the specified work position according to the AR presented path information, after reaching the specified work position, the subsequent collaborative task is executed, and when the robot does not need to participate, the step of executing the collaborative task is directly executed. When the collaborative task is executed, first, the AR glasses present a visual stimulation interface, the intelligent operator selects the collaborative task based on the ssvep electroencephalogram signal, then the AR glasses present a MI interaction interface, the intelligent operator selects the information he wants to watch according to his own style preference, after the control of the AR-BCI assisted system through the above-mentioned electroencephalogram signal, the mechanical arm of the collaborative robot is controlled to execute the task, that is, the human-machine collaborative part. During the execution of the task by the mechanical arm, since the intelligent operator and the collaborative robot are located in the same work area, the movement speed of the mechanical arm is controlled by collecting the electromyographic signal of the intelligent operator. The intelligent operator judges whether the task is completed according to the execution of the task, if not, returns to the visual stimulation interface and reselects the task, if yes, the task ends.

[0070] As shown, Figure 5 The brain-machine task execution flow of the present application is divided into two parts. The first part is that the AR glasses present a ssvep stimulation interface to induce the generation of electroencephalogram signals, the generated electroencephalogram signals are preprocessed, then the electroencephalogram signal features are extracted, the intention of the electroencephalogram signal is classified and recognized, so as to control the AR-BCI assisted system to execute the corresponding task instruction. The second part is that the AR glasses present a MI interaction interface, the electroencephalogram signals are generated by motor imagination, the generated electroencephalogram signals are preprocessed, then the offline training model is called, the corresponding instruction of the offline model is recognized online through the recognition of the generated model of offline training, then the AR-BCI assisted system is controlled to execute the corresponding task instruction.

Claims

1. A human-machine collaborative interaction operation system based on multi-BCI and MR fusion, characterized in that: it comprises an AI control system, an AR-BCI assistance system and a physical working environment; the AI control system comprises a VR control center, a data analysis center and an image management center; the AR-BCI assistance system comprises an AR information module, an electroencephalogram signal module and an electromyogram signal module; the physical working environment comprises an intelligent operator, a collaborative robot and a collaborative task; the AI control system is used for controlling the collaborative robot, wherein the image management center is used for on-site extraction and storage of images; the data analysis center is used for storing a training model generated by electroencephalogram and electromyogram signals and path planning of the collaborative robot; the VR control center is used for connecting the image management center and the data analysis center, after the VR control center obtains information transmitted by the AR-BCI assistance system, the behavior intention of the intelligent operator is analyzed by calling the image management center and the data analysis center, and task instructions are transmitted back to the AR information module; the AR-BCI assistance system is used for connecting the AI control system in the information dimension and the physical working environment in the physical dimension, wherein the electroencephalogram signal module is used for collecting and processing electroencephalogram signals; the electromyogram signal module is used for collecting electromyogram signals of the intelligent operator during work; the AR information module is used for presenting a prepared AR interface, observing task information in a first-person perspective and sending control instructions to the VR control center, and the VR control center presents decision results to the AR information module according to task instructions transmitted by the AR-BCI assistance system, so that the intelligent operator makes a selection to assist the intelligent operator to complete the task; the electroencephalogram signal module is used for: (1) collecting electroencephalogram signals generated when the intelligent operator gazes at a corresponding block of an SSVEP stimulation interface, and sending the electroencephalogram signals to the AR information module, the AR information module is used for presenting the SSVEP stimulation interface, and after the electroencephalogram signals are preprocessed, CCA algorithm is used for identification, and the AR-BCI assistance system is controlled by analyzing the SSVEP to execute corresponding instructions; (2) collecting offline training electroencephalogram signals generated when the intelligent operator gazes at a MI interaction interface for motor imagery, generating a training model by preprocessing the offline training electroencephalogram signals and using CSP algorithm; (3) obtaining online training electroencephalogram signals generated when the intelligent operator gazes at the MI interaction interface for motor imagery during online training, preprocessing the online training electroencephalogram signals, classifying and identifying the online training electroencephalogram signals according to the training model and CSP algorithm, obtaining the motor imagery intention of the intelligent operator, controlling information to be presented by the AR interface by analyzing the MI, and adaptively adjusting the AR control interface; the electromyogram signal module collects electromyogram signals of the intelligent operator during work, selects three variables of average absolute value, root mean square value and waveform length as characteristics of the electromyogram signals, constructs a feature classifier of the EMG signals based on SVM, tests the test data by using the constructed classifier, and is used for collaborative intention recognition. ​ ​ ​ ​ ​ ​ The intelligent operator wears AR glasses to perform a collaborative task in a work area, and the collaborative robot is located in a standby area and waits for the instruction of the intelligent operator in the physical working environment; The collaborative robot in the physical working environment comprises a mechanical arm, a computer vision device and a laser detector, etc., the computer vision device is used to acquire the poses of the intelligent operator and the task target, the mechanical arm is used to pick up and carry objects, and the laser detector is used to scan the working scene and perceive the environment around the collaborative robot; The collaborative task in the physical working environment comprises a judgment question, a selection question and a dynamically updated question; the judgment question is made of two pictures, and the code in the SSVEP paradigm is changed to identify the two pictures; the selection question is made of 3-8 pictures, which respectively represent different positions that need to be viewed; the dynamically updated question firstly calls a photo capture object to shoot each part that needs to be picked up, saves the part to a specified file location, and changes the picture name to realize the acquisition of dynamic pictures; the execution of the task needs to judge whether the robot needs to participate in assistance; the selection question is controlled by the electroencephalogram signal to control the AR-BCI assistance system to identify the task picture and execute the corresponding task instruction; the dynamically updated picture uploads the collected pictures to an AI control system, the AI control system identifies the key information of the uploaded pictures, and the camera on the mechanical arm picks up the corresponding object by using a deep learning algorithm; The intelligent operator controls the AR-BCI assistance system to select a task and judge whether the collaborative robot needs to participate in assistance; when the collaborative robot needs to participate in assistance, the VR control center acquires the position information of the AR glasses and the collaborative robot, then analyzes the path of the collaborative robot in unity, the AR glasses present the path planning information, the intelligent operator controls the collaborative robot to reach the specified working position according to the path planning information presented by the AR glasses, and after reaching the specified working position, executes the subsequent collaborative task; when the robot does not need to participate, the intelligent operator directly executes the collaborative task; When the collaborative task is executed, the AR glasses present a visual stimulation interface, the intelligent operator selects the collaborative task based on the ssvep electroencephalogram signal, then the AR glasses present an MI interaction interface, the intelligent operator selects the information that needs to be viewed according to his own style and preference, and manipulates the AR-BCI assistance system by collecting the electroencephalogram signal, so as to control the mechanical arm of the collaborative robot to execute the task; during the execution of the task by the mechanical arm, the intelligent operator and the collaborative robot are located in the same working area, and the movement speed of the mechanical arm is controlled by collecting the electromyogram signal of the intelligent operator; the intelligent operator judges whether the task is completed according to the execution of the task, if not, returns to the visual stimulation interface to make a new selection, and if yes, the task is completed.

2. The system of claim 1, wherein: The training model generated by storing the electroencephalogram and electromyogram signals is used to compare the current electroencephalogram and electromyogram signals with the training model of the data analysis center in the online experiment process, and the current intention of the intelligent operator is identified according to the label of the model of the data analysis center.

3. The system of claim 1 or 2, wherein: The image management of the AI control system directly calls the stored picture for the task that does not need to change the task option, extracts on site for the task that needs to update the task option, first takes a picture to obtain the picture through the photo capture class in the VS, saves to the specified file position, changes the picture name at the same time, repeatedly extracts the picture according to the task requirement until the result is satisfied, and then runs the corresponding brain-machine code.

4. The system of claim 3, wherein: The computer vision device of the collaborative robot is located at the position of the mechanical arm of the collaborative robot, adopts a depth camera to identify the human behavior feature and the pose of the task target, the human behavior feature is used for identifying the intention of the staff in the VR control center, the next work of the human is judged based on the label of the pre-set training model library, the collaborative robot is controlled to assist the human to work; the identification of the pose of the task target is to make the collaborative robot accurately obtain the position and attitude of the required article, and the path planning of the mechanical arm is analyzed through the VR control center; the laser detector uploads the measurement values of the surrounding distance and reflectivity in real time, has the ability of long-range distance measurement, after the point cloud data of the laser detector is obtained, the point cloud data is repaired in the virtual space, and the virtual working environment with the same proportion as the real environment is constructed.

5. The system of claim 4, wherein: The intelligent operator wears AR glasses to perform tasks in the physical working space, the AR-BCI assistance system is controlled by collecting the electroencephalogram and electromyogram signals of the intelligent operator to adaptively perform tasks, the VR control center performs path planning on the collaborative robot according to the instructions transmitted by the AR-BCI assistance system, and the computer vision device compares and analyzes the collected information with the model of the data analysis center after the information is transmitted back to the VR control center, so that the collaborative robot and the intelligent operator work cooperatively.

Images