A method and system for capturing EEG of a typical mental state of an operator

Abstract

The present invention discloses a method and system for capturing EEG signals of a typical mental state of an operator. The EEG signal acquisition module acquires EEG signals of a typical mental state of an operator by means of a capture instruction trigger acquisition mode and a timing acquisition mode, and includes a first computer unit and an EEG acquisition device connected to the first computer unit; a virtual collaborative robot operation module controls a virtual collaborative robot to complete a typical mental state inducing task, and continuously induces the operator to produce a variety of typical mental states, and includes a second computer unit for simulating the movement of the collaborative robot and a control instruction sending device connected to the second computer unit; the first computer unit is connected to the second computer unit through a data transmission network, and realizes the functions of monitoring the dynamic process of the operator operating the virtual collaborative robot and adaptively acquiring EEG signals of a typical mental state of the operator. The present invention improves the flexibility of the EEG acquisition system of the operator's mental state and the validity of EEG data.

Description

Technical Field

[0001] The present invention belongs to the technical field of brain-computer interface, and in particular relates to a method and system for capturing the electroencephalogram (EEG) of a typical mental state of an operator. Background Art

[0002] In human-machine systems, the operator's mental state is easily affected by internal or external factors when performing operation tasks, which may reduce the operator's reaction speed and operation accuracy. Therefore, it is very important to identify the operator's mental state when working. Mental state recognition methods based on physiological signals have attracted widespread attention due to their objectivity. Among them, EEG signals have become an effective means of identifying mental states due to their high temporal resolution.

[0003] Collecting EEG signals under different mental states is the basis for studying the EEG response mechanism of mental state changes and identifying the operator's mental state based on EEG signals. On the one hand, the existing mental state-related EEG signal collection is mainly based on two methods. One is to collect EEG signals under continuous stimulation by pictures or audio and video, which has low ecological validity when applied to human-machine systems; the other is to collect EEG signals of operators during physical operation, which has high system operation costs. On the other hand, in actual human-machine systems, the mental state fluctuations of operators do not always exist, and most of the existing EEG signal collection systems collect and store EEG data for a full time under the stimulation of induced materials. This method lacks a picking mechanism and may mistakenly collect and store EEG signals in non-target mental states, resulting in a decrease in the reliability of the obtained EEG data. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method and system for capturing the EEG of an operator's typical mental state in response to the above-mentioned deficiencies in the prior art, dynamically and adaptively capturing the EEG signals of the operator in a typical mental state during continuous operation, improving the flexibility of EEG signal acquisition and the reliability of the collected and stored EEG signals, and meeting the experimental needs of collecting the EEG signals of the operator in a typical mental state during operation.

[0005] The present invention adopts the following technical solutions:

[0006] An electroencephalogram (EEG) capture system for a typical mental state of an operator, comprising:

[0007] The EEG signal acquisition module is used to acquire EEG signals of the operator in a typical mental state by capturing instructions to trigger the acquisition mode, and includes a first computer unit for acquiring EEG signals and an EEG acquisition device connected to the first computer unit;

[0008] A virtual collaborative robot operation module controls the virtual collaborative robot to complete typical mental state inducing tasks and continuously induces the operator to produce multiple typical mental states, including a second computer unit for simulating the movement of the collaborative robot and a virtual collaborative robot control instruction sending device connected to the second computer unit;

[0009] The first computer unit is connected to the second computer unit via a data transmission network to realize the functions of monitoring the dynamic process of the operator operating the virtual collaborative robot and adaptively collecting the EEG signals of the operator's typical mental state.

[0010] Specifically, the first computer unit includes an EEG signal capture discriminator and a typical mental state subjective evaluation scale; the EEG signal capture discriminator receives the operation log information of the second computer unit through the data transmission network, and sends a capture instruction to the EEG acquisition device to trigger EEG signal acquisition; the typical mental state subjective evaluation scale is used to determine the typical mental state category.

[0011] Furthermore, the typical mental state subjective evaluation scale includes focused state, non-focused state, tense state, fatigue state, positive state and negative state.

[0012] Specifically, the EEG signal acquisition module acquires EEG signals of the operator in a typical mental state in a timed acquisition mode, which acquires EEG signals of the operator in a typical mental state during a period of time when there is no obvious operation log information during the operation process.

[0013] Specifically, the virtual collaborative robot control instruction sending device is connected to the second computer unit via USB wireless transmission, and is used to send virtual collaborative robot control instructions. The virtual collaborative robot receives and executes the virtual collaborative robot control instructions, and generates operation result log information.

[0014] Furthermore, the virtual collaborative robot control instructions include instructions for controlling the robot chassis, and instructions for the movement or action of the robot arm and gripper.

[0015] Specifically, the EEG acquisition device includes an EEG cap, the electrodes of which are arranged in the frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of the brain, and the sampling frequency of the EEG acquisition device is 1000 Hz.

[0016] Another technical solution of the present invention is a method for capturing the EEG of a typical mental state of an operator, using an EEG capturing system of a typical mental state of an operator, comprising the following steps:

[0017] S1, EEG acquisition equipment collects EEG signals from the frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of the cerebral cortex;

[0018] S2. Controlling the movement of the virtual collaborative robot through the game controller to complete the mental state inducing task. The mental state inducing task is a combination of one or more of trajectory tracking, grabbing objects, and transferring objects. In the process of completing the inducing task, each time a control instruction related to the robot arm or the gripper is executed, the corresponding operation log information is generated;

[0019] S3, when the mental state inducing task is to grab or transfer an object, the operation log information generated in step S2 is transmitted from the second computer unit to the first computer unit through the data transmission network, and a capture instruction is used to trigger the collection. When the first computer unit recognizes the operation log information corresponding to the typical mental state, a capture instruction is sent to collect the EEG signal under the typical mental state;

[0020] S4, when the mental state induction task is trajectory tracking, the EEG signal of step S1 is collected and stored at regular intervals;

[0021] S5. Obtain and store the labels of the EEG signals obtained in step S3 and step S4 through the typical mental state subjective evaluation scale of the first computer unit, thereby completing the EEG capture of the operator's typical mental state.

[0022] Specifically, in step S2, the virtual collaborative robot operation module includes five mental state inducing tasks, and the virtual collaborative robot movement is controlled by the game controller as follows:

[0023] The game controller is connected to the second computer unit through a wireless USB receiver. The virtual collaborative robot runs in the robot operating system ROS environment of the second computer unit. The game controller is operated to send control instructions to the virtual collaborative robot in the second computer unit. The virtual collaborative robot includes a chassis, a robotic arm and a gripper. The control instructions for controlling the linear velocity or angular velocity of the chassis movement are sent to the virtual collaborative robot through the joystick of the game controller. The chassis performs forward, backward, left turn and right turn movements according to the control instructions, realizing parallel motion of movement and rotation; control instructions for controlling the opening or closing of the gripper and the movement or rotation of the end of the robotic arm along the X, Y, and Z directions in the Cartesian coordinate system are sent to the virtual collaborative robot through the buttons of the game controller, and the gripper or robotic arm performs stepping motion according to the button control instructions.

[0024] Specifically, step S3 is as follows:

[0025] S301, using a first computer unit running MATLAB as a ROS node, realizing communication connection between nodes in the ROS network running on the first computer unit and the second computer unit under the same network, and realizing two-way information transmission between the first computer unit and the second computer unit;

[0026] S302, in the ROS Master run by the second computer unit, the node / rosout is the publisher of the topic / rosout_agg, and the / Msg information of the topic / rosout_agg is the control instructions received by the virtual collaborative robot, the calculation results generated by the received instructions, and the robot instruction execution results, so that the first computer unit node running MATLAB subscribes to the topic / rosout_agg and cyclically reads the message content to obtain real-time log information of the operation;

[0027] S303. When the EEG signal capture discriminator running on the first computer unit recognizes specific operation log information related to the operator's typical mental state, a capture instruction is generated to wirelessly transmit the EEG data collected by the EEG acquisition device to the first computer unit via the data transmission network and store the data. The EEG acquisition device collects and stores EEG signals for 5 to 60 seconds.

[0028] Compared with the prior art, the present invention has at least the following beneficial effects:

[0029] The present invention discloses an EEG capture system for a typical mental state of an operator, which is intended to obtain EEG data of an operator in a typical mental state during operation, realize dynamic monitoring of the operator's operation process, adaptively collect EEG signals of an operator in the process of continuously operating a virtual collaborative robot to complete a mental state induction task, and obtain the function of a typical mental state label of the EEG data. On the one hand, the virtual collaborative robot operation module provides a virtual interactive operation scene, realizes the collection of EEG data of a subject during the interactive operation process, and compared with the stimulation of collecting pictures or videos, the interactive induced task operation restores the influencing factors of the operator's mental state fluctuation in the actual working environment to a higher degree, and lays a research foundation for the mental state monitoring algorithm to better apply to the actual working conditions; the virtual interactive operation is significantly less expensive than operating a real robot. On the other hand, considering that the operator's mental state fluctuation does not always exist in an actual human-machine system, in addition to the timing collection method, a capture instruction trigger collection method is also designed in the EEG collection module. The capture instruction trigger collection method monitors the operator's operation process, triggers the collection of typical mental state EEG signals according to the actual situation of the mental state induction task, and improves the induction efficiency of the stimulus material and the reliability of the stored EEG data.

[0030] Furthermore, the EEG signal capture discriminator is used to receive and identify the operation log information from the second computer unit and generate an EEG signal capture instruction. The advantage is that according to the actual situation of the operator operating the virtual collaborative robot to complete the mental state induction task, "correct" and "wrong" operation prompts are provided in a timely manner, and the operator is induced to produce a positive state or a negative state in combination with the operation results, and the EEG signal of the operator in a typical mental state after the correct or wrong operation result and the prompt is received is collected. The typical mental state subjective evaluation scale is used to obtain the mental state label of the EEG data. Considering the individual differences caused by factors such as personality, individuality, and experience in the induction of mental state, setting a typical mental state subjective evaluation scale can avoid the EEG data label deviation caused by individual differences and improve the accuracy of the acquired EEG data label. The typical mental state subjective evaluation scale is located in the first computer unit of the EEG acquisition module as a part of human-computer interaction, which improves the automation of the experiment. There are two existing methods for evaluating mental state: subjective evaluation and objective evaluation. Subjective evaluation is to obtain the current mental state of the subject based on subjective evaluation, which can provide a basis for studying the relationship between mental state and EEG response.

[0031] Furthermore, the focused state, non-focused state, nervous state, fatigue state, positive state and negative state match the design of the mental state induced task; more fundamentally, the focused state, non-focused state, nervous state, fatigue state, positive state and negative state often appear in the actual human-machine system, and are closely related to the operator's decision-making and behavioral performance. When the operator's mental state is not good, it is easy to make insufficient or even wrong operations. When the operator is tired, the attention span decreases, and the judgment and operation ability deteriorate; excessive pressure and tension will cause slow reaction. In the non-focused state and negative state, the operator's performance is lower than that in the focused state and positive state; the focused state, non-focused state, nervous state, fatigue state, positive state and negative state describe the mental state from different perspectives, basically covering all possible categories in the operation process of the mental state induced task; in addition, the evaluation method of discrete mental state is simpler and more practical than the evaluation based on dimensional model.

[0032] Furthermore, the timed acquisition method is used as a supplement to the capture command trigger acquisition method to obtain EEG signals of operators in a typical mental state during a period without obvious operation log information. Setting the timed acquisition method helps improve the efficiency of the system and helps obtain more EEG data. The timed acquisition method is mainly used to obtain EEG data in the focused state, non-focused state, nervous state and fatigue state.

[0033] Furthermore, the virtual collaborative robot control instruction sending device is connected to the second computer unit via a USB ultra-micro receiver, which is plug-and-play and uses 2.4G gaming-level wireless technology to achieve high-speed data transmission.

[0034] Furthermore, the virtual collaborative robot consists of three parts: chassis, robotic arm and gripper. In order to control the virtual collaborative robot to complete tasks such as movement and grasping, the instructions for controlling the robot chassis movement, the robotic arm movement instructions and the gripper movement instructions are designed. The movement task is achieved by controlling the movement of the chassis, and the grasping task is achieved by the coordinated movement of the robotic arm and the gripper.

[0035] Furthermore, the electrodes of the EEG cap are arranged in the frontal lobe, occipital lobe, parietal lobe, and temporal lobe areas of the brain. There are 30 EEG electrodes in total, and the sampling frequency is 1000 Hz. The collected EEG signals have high temporal resolution and relatively high spatial resolution.

[0036] A method for capturing the EEG of an operator's typical mental state combines a capture instruction trigger acquisition method and a timing acquisition method to obtain the EEG signal of the operator when the operator controls the movement of a virtual collaborative robot through a game controller to complete a mental state task, and obtains the mental state label corresponding to the stored EEG data.

[0037] Furthermore, in step S2, when the movement of the virtual collaborative robot is controlled by the game controller, the joystick of the controller is used to change the linear velocity and angular velocity of the chassis movement, so as to realize stepless speed regulation; the linear velocity and angular velocity of the chassis can be adjusted at the same time to realize the parallel motion of the robot movement and rotation; the movement form of the robotic arm and the gripper is stepping motion, so as to realize the grasping target by adjusting the posture of the end of the robotic arm and making it close to the gripping target, and finally closing the gripper to complete the grasping.

[0038] Furthermore, step S3 realizes two-way information transmission between the first computer unit and the second computer unit based on ROS-MATLAB communication, so that the first computer unit reads the real-time log information of the operator operating the virtual collaborative robot, and collects the EEG signals of the operator in a typical mental state based on the EEG signal capture discriminator.

[0039] In summary, the present invention realizes the function of dynamically monitoring the operator's operation process, adaptively collecting the EEG signals of the operator in a typical mental state when the operator continuously operates a virtual collaborative robot to complete a mental state induction task in an interactive environment, and obtaining data labels through a subjective evaluation scale.

[0040] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a continuous induction and dynamic adaptive capture system for the EEG signals of the operator's typical mental state;

[0042] Figure 2Schematic diagram of the mental state induction task in the virtual collaborative robot operation module;

[0043] Figure 3 Schematic diagram of the channel location of the EEG acquisition device. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0046] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0047] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0048] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0049] See also Figure 1 The present invention provides an electroencephalogram (EEG) capturing system of a typical mental state of an operator, comprising:

[0050] The EEG signal acquisition module includes a first computer unit for EEG signal acquisition and an EEG acquisition device and a data transmission network connected to the first computer unit;

[0051] The virtual collaborative robot operation module includes a second computer unit for simulating the movement of the collaborative robot and a virtual collaborative robot control instruction sending device connected to the second computer unit.

[0052] The EEG signal acquisition module acquires and stores the EEG signals of the operator in a typical mental state in two ways: one is the capture instruction trigger acquisition method, and the other is the timing acquisition method.

[0053] The timed acquisition method is a supplement to the command-triggered acquisition method. It collects the EEG signals of the operator's typical mental state during the period when there is no obvious operation log information during the operation process, thereby improving the work efficiency of the system.

[0054] In the EEG signal acquisition module, the capture instruction triggers the acquisition method, which is realized by the EEG signal capture discriminator in the first computer unit; the EEG signal capture discriminator receives the operation log information from the second computer unit in the virtual collaborative robot operation module through the data transmission network; when specific operation information related to certain typical mental states is identified, the EEG signal capture discriminator sends a capture instruction to trigger EEG signal acquisition, thereby realizing the functions of monitoring the dynamic process of the operator operating the virtual collaborative robot and adaptively acquiring the EEG signals of the operator's typical mental state.

[0055] The first computer unit is designed with a typical mental state subjective evaluation scale. The labels of the EEG data obtained by the two EEG signal acquisition methods in the EEG signal acquisition module, that is, the categories of the typical mental states, are determined by the typical mental state subjective evaluation scale.

[0056] The subjective evaluation scale is an evaluation scale that includes six typical mental states. The operator completes the subjective evaluation of the mental state based on the content of the scale. The six typical mental states are states that are easy to occur in actual operations and may affect the quality of operation, including concentration state, non-concentration state, tension state, fatigue state, positive state and negative state.

[0057] The EEG acquisition device includes an EEG cap, the electrodes of which are arranged in the frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of the brain, and the sampling frequency of the EEG acquisition device is 1000 Hz.

[0058] In the virtual collaborative robot operation module, the second computer unit and the virtual collaborative robot control instruction sending device are connected via USB wireless transmission. The operator manually operates and controls the virtual collaborative robot instruction sending device to send control instructions to the virtual collaborative robot in the second computer unit to control the virtual collaborative robot to complete typical mental state inducing tasks, and continuously induce it to produce multiple typical mental states.

[0059] The control instructions of the virtual collaborative robot include instructions for controlling the movement or action of the robot chassis, robotic arms and grippers, that is, instructions for controlling the linear velocity and angular velocity of the chassis movement, instructions for adjusting the end position of the robotic arm and instructions for controlling the opening and closing of the grippers.

[0060] The virtual collaborative robot in the second computer unit receives and executes the control instruction, and generates operation result log information at the same time. The operation result log information is the result information generated after the virtual collaborative robot receives and executes the control instruction, such as successful execution of the control instruction, failure to execute the control instruction, etc.

[0061] The present invention provides a method for capturing the EEG of a typical mental state of an operator, which enables the operator to operate a virtual collaborative robot through a game controller according to task requirements under a virtual collaborative robot operation module, and generates real-time operation log information during the operation. An EEG signal acquisition module receives the real-time operation log information from the virtual collaborative robot operation module, determines whether to generate a capture instruction according to the operation log information, and collects and stores the operator's EEG signal after the capture instruction is generated. The method comprises the following steps:

[0062] S1, EEG acquisition equipment collects EEG signals from the frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of the cerebral cortex;

[0063] The operator of the virtual collaborative robot wears an EEG cap and sits in front of the second computer unit, with his head 40 to 100 cm away from the monitor of the second computer unit. A 34-lead EEG cap (including 2 reference electrodes, 2 eye electrodes, and the remaining 30 EEG electrodes arranged according to the international 10-20 standard, with specific electrode positions as shown below) is used. Figure 3 As shown in the figure, the EEG signals of the operator's frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of the cerebral cortex are collected, and the sampling frequency of the EEG amplifier is 1kHz.

[0064] S2. Controlling the movement of the virtual collaborative robot through the game controller to complete the mental state inducing task. The mental state inducing task is a combination of one or more of trajectory tracking, grabbing objects, and transferring objects. In the process of completing the inducing task, each time a control instruction related to the robot arm or the gripper is executed, the corresponding operation log information is generated;

[0065] The virtual collaborative robot operator uses a game controller to control the movement of the virtual collaborative robot in the virtual collaborative robot operation module (the second computer unit and the external operating environment space) to complete the operation task according to the task requirements. The operation task is selected from a combination of one or more of trajectory tracking, grabbing objects, and transferring objects.

[0066] See also Figure 2 There are 5 mental state induction tasks in the virtual collaborative robot operation module. The specific requirements of each task are as follows:

[0067] In Task 0, the operator needs to control the virtual collaborative robot to walk in a circle in a clockwise direction in a rectangular room without restricting the movement trajectory of the virtual collaborative robot.

[0068] In Task 1, the operator needs to control the virtual collaborative robot to walk along two sides of the triangle, namely a and c.

[0069] In Task 2, the operator needs to control the virtual collaborative robot to grab the small blue object on the table and then place the small object in the white tray.

[0070] Task 3 is a time-limited task in which the operator needs to control the virtual collaborative robot to get out of the maze within 1 minute.

[0071] Compared to Task 2, in Task 4, the small object was replaced by a Coke bottle and the tray was placed on another table.

[0072] In the EEG capture system of the operator's typical mental state, the typical mental states selected are the positive state, negative state, focused state, non-focused state, nervous state and fatigue state that often appear in the human-machine system and affect the operation quality. The execution order of the mental state inducing task requirements is Task 1→Task 2→Task 3→Task 0→Task 1→Task 2→Task 3→Task 0→Task 4. Since Task 1 is the first task of the entire experiment and Task 1 has high requirements for the walking trajectory of the virtual collaborative robot, Task 1 is used to induce the operator's focused state. Task 2 is used to induce the operator to produce positive and negative states. When the operator is executing Task 2, once an erroneous operation occurs during the task execution, the experimental guidance interface running in the first computer unit will give the operator an erroneous operation prompt according to the operation log information received from the second computer unit. The erroneous operation prompt includes voice prompts and text prompts. The erroneous operation refers to a fault such as a collision after the virtual collaborative robot performs an action. On the contrary, if Task 2 in the second computer unit is successfully completed, the experimental guidance interface will give the operator correct operation prompts, which include voice prompts and text prompts. Incorrect operation prompts are used to induce negative states, and correct operation prompts are used to induce positive states. Because the trajectory of Task 0 is monotonous, simple and long, and Task 0 has no requirements for the walking trajectory of the virtual collaborative robot, Task 0 is used to induce the operator's non-focused state. Task 3 is used to induce the operator to be in a nervous state, because Task 3 is set as a time-limited task, and the operator needs to control the virtual collaborative robot to walk out of the maze within 1 minute. Task 4 is placed at the end of the mental state inducing task execution order. After the previous two rounds of mental state inducing tasks, Task 4 is used to induce fatigue in the operator.

[0073] To complete the above tasks, the specific implementation method of the operator operating the virtual collaborative robot in the virtual collaborative robot operation module is as follows:

[0074] The game controller is connected to the second computer unit through a wireless USB receiver. The virtual collaborative robot runs in the robot operating system ROS environment of the second computer unit. The operator operates the game controller to send control instructions to the virtual collaborative robot in the second computer unit. The virtual collaborative robot consists of three parts: chassis, robotic arm and gripper. The operator sends the analog value of linear velocity or angular velocity to control the movement of the chassis to the virtual collaborative robot through the joystick of the game controller. The chassis performs forward, backward, left turn and right turn according to the control instructions, and can also realize parallel movement and rotation; the operator sends control instructions to the virtual collaborative robot to control the opening or closing of the gripper and the movement or rotation of the end of the robotic arm along the X, Y and Z directions in the Cartesian coordinate system through the key operation of the game controller. The gripper or robotic arm performs stepping movement according to the key control instructions (switch quantity).

[0075] The virtual collaborative robot is a simulation of a real collaborative robot. In the experimental stage, the movement and actions of the virtual collaborative robot in the virtual collaborative robot operation module are controlled to replace the movement and actions of the real collaborative robot in actual operation, and the typical mental state of the operator is induced by combining the virtual collaborative robot operation environment and operation results to reduce the experimental cost.

[0076] S3, when the mental state inducing task is to grab or transfer an object, the operation log information generated in step S2 is transmitted from the second computer unit to the first computer unit via TCP / IP communication of the same local area network (here the local area network is 5GHz Wi-Fi). In the capture instruction triggering acquisition mode, when the first computer unit recognizes the operation log information related to the typical mental state, it sends a capture instruction to collect and store 30 channels of EEG signals of the operator under the typical mental state;

[0077] The specific steps include:

[0078] S301, the capture of the operator's EEG signal in a typical mental state is implemented based on MATLAB. In order to read the log information of the operation of the virtual collaborative robot in the ROS environment running on the second computer unit in real time, the first computer unit running MATLAB is used as a ROS node, so as to realize the communication connection between the nodes in the ROS network running on the first computer unit and the second computer unit under the same network, and finally realize the two-way information transmission between the first computer unit and the second computer unit;

[0079] S302, in the ROS Master running on the second computer unit, the node / rosout is the publisher of the topic / rosout_agg, and the / Msg information of the topic / rosout_agg is the control instructions received by the virtual collaborative robot, the calculation results generated by the received instructions, and the robot instruction execution results. Make the first computer unit node running MATLAB subscribe to the topic / rosout_agg, and read the message content cyclically to obtain real-time log information of the operation;

[0080] S303, when the EEG signal capture discriminator running on the first computer unit recognizes specific operation log information related to the typical mental state of the operator, a capture instruction is generated, so that the 30 channels of EEG data collected by the EEG acquisition device are wirelessly transmitted to the first computer unit via the data transmission network and stored. In this system, after the EEG signal capture instruction is generated, the EEG acquisition device collects and stores EEG signals for 5 to 60 seconds.

[0081] S4, when the mental state induction task is trajectory tracking, the EEG signal of step S1 is collected and stored at regular intervals;

[0082] In the timed acquisition mode, when the operator operates the virtual collaborative robot to complete the mental state induction task, the EEG signal is collected at a timed interval. The tasks using timed acquisition are Task 0, Task 1, Task 3 and Task 4.

[0083] S5. Obtain and store the labels of the EEG signals obtained in step S3 and step S4 through the typical mental state subjective evaluation scale of the first computer unit, and complete the EEG capture of the operator's typical mental state; the label is stored as the name of the EEG data file, and its mental state label is obtained through the naming method of the EEG data file;

[0084] S6. After the acquisition and storage of the EEG data of the operator in a typical mental state is completed, steps S2 to S5 are looped. As the operator continues to perform the task control, the operation log information is continuously transmitted from the second computer unit to MATLAB on the first computer unit in real time. MATLAB determines whether it is the target operation log information related to the typical mental state. If so, a capture instruction is sent to acquire and store the operator's EEG signal. Otherwise, the EEG data at this time is not stored.

[0085] In another embodiment of the present invention, a terminal device is provided, the terminal device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the EEG capture method of the typical mental state of the operator, including:

[0086] The EEG acquisition device acquires EEG signals from the frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of the cerebral cortex; the movement of the virtual collaborative robot is controlled by a game controller to complete the operation task, the category of the mental state inducing task is a combination of one or more of trajectory tracking, grabbing blocks, and transferring blocks. In the process of completing the induced task, each control instruction related to the robotic arm or the gripper is executed to generate corresponding operation log information; when the mental state inducing task is grabbing blocks or transferring blocks, the generated operation log information is transmitted to the first computer unit through the second computer unit, and the capture instruction is used to trigger the acquisition method. When the first computer unit recognizes the operation log information corresponding to the typical mental state, it sends a capture instruction to acquire the EEG signal in the typical mental state; when the mental state inducing task is trajectory tracking, the EEG signal is collected at a fixed time and stored; the label of the acquired EEG signal is acquired and stored through the typical mental state subjective evaluation scale of the first computer unit to complete the EEG capture of the operator's typical mental state.

[0087] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0088] The processor may load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the method for capturing the EEG of the operator's typical mental state in the above embodiment; the processor may load and execute the following steps:

[0089] The EEG acquisition device acquires EEG signals from the frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of the cerebral cortex; the movement of the virtual collaborative robot is controlled by a game controller to complete the operation task, the category of the mental state inducing task is a combination of one or more of trajectory tracking, grabbing blocks, and transferring blocks. In the process of completing the induced task, each control instruction related to the robotic arm or the gripper is executed to generate corresponding operation log information; when the mental state inducing task is grabbing blocks or transferring blocks, the generated operation log information is transmitted to the first computer unit through the second computer unit, and the capture instruction is used to trigger the acquisition method. When the first computer unit recognizes the operation log information corresponding to the typical mental state, it sends a capture instruction to acquire the EEG signal in the typical mental state; when the mental state inducing task is trajectory tracking, the EEG signal is collected at a fixed time and stored; the label of the acquired EEG signal is acquired and stored through the typical mental state subjective evaluation scale of the first computer unit to complete the EEG capture of the operator's typical mental state.

[0090] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0091] The present invention conducts experiments on multiple operators according to the above steps S1 to S6. In the experiment, the EEG signals of the operators in the typical mental state when they operate the virtual collaborative robot in the virtual collaborative robot operation module through the game controller are recorded. During the experiment, the operators should try to avoid blinking. The results show that the EEG capture system of the operator's typical mental state can dynamically and adaptively collect the EEG data of the operator in different typical mental states, and the EEG data has high reliability, which is manifested in that the subjective evaluation is basically consistent with the design goal of the induced task and the separability of the acquired EEG data is high.

[0092] The beneficial effects of the present invention are embodied in:

[0093] First, the interactive virtual collaborative robot operation task restores the influencing factors of the operator's mental state fluctuations in the actual working environment to a higher degree, with high ecological validity and low cost.

[0094] Second, the capture instruction triggered acquisition method improves the induction efficiency of the stimulus material and the reliability of the stored EEG data, while assisting with the timing acquisition method to improve the efficiency of the system and obtain more EEG data.

[0095] Third, the typical mental state subjective evaluation scale covers several mental states that often appear in human-machine systems and affect the operator's decision-making and behavior, avoiding EEG data label deviations caused by individual differences, improving the accuracy of the acquired EEG data labels and the degree of automation of the test system. It lays a good research foundation for studying the EEG response mechanism of mental state changes, mental state recognition, and the relationship between mental state and operation quality.

[0096] In summary, the present invention provides an EEG capture method and system of an operator's typical mental state, which enables the operator to operate the virtual collaborative robot through a game controller according to a certain operation process under the virtual collaborative robot operation module on the computer, and monitors the operation of the task in real time, and collects and stores EEG data in a typical mental state based on the operation log information. The present invention can realize the dynamic adaptive collection of EEG data in a typical mental state during the operation process, improve the flexibility of the EEG collection system and the effectiveness of EEG data, and can provide a research basis for studying the EEG response mechanism of the operator's mental state changes, mental state recognition, the correlation between mental state and operation quality, and operation compensation strategies.

[0097] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0098] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0101] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A method for capturing the EEG of an operator's typical mental state, characterized in that: The EEG capture system using the operator's typical mental state includes: The EEG signal acquisition module is used to acquire EEG signals of the operator in a typical mental state by capturing instructions to trigger the acquisition mode, and includes a first computer unit for acquiring EEG signals and an EEG acquisition device connected to the first computer unit; A virtual collaborative robot operation module controls the virtual collaborative robot to complete typical mental state inducing tasks and continuously induces the operator to produce multiple typical mental states, including a second computer unit for simulating the movement of the collaborative robot and a virtual collaborative robot control instruction sending device connected to the second computer unit; The first computer unit is connected to the second computer unit through a data transmission network to realize the functions of monitoring the dynamic process of the operator operating the virtual collaborative robot and adaptively collecting the EEG signals of the operator's typical mental state. The first computer unit includes an EEG signal capture discriminator and a typical mental state subjective evaluation scale; the EEG signal capture discriminator receives the operation log information of the second computer unit through the data transmission network, and sends a capture instruction to the EEG acquisition device to trigger EEG signal acquisition; the typical mental state subjective evaluation scale is used to determine the typical mental state category, and the typical mental state subjective evaluation scale includes a focused state, a non-focused state, a tense state, a fatigued state, a positive state, and a negative state; The virtual collaborative robot control instruction sending device is connected to the second computer unit via USB wireless transmission, and is used to send the virtual collaborative robot control instructions. The virtual collaborative robot receives and executes the virtual collaborative robot control instructions, and generates operation result log information at the same time. The virtual collaborative robot control instructions include instructions for controlling the robot chassis, and instructions for the movement or action of the robot arm and the gripper; The EEG signal acquisition module acquires the EEG signals of the operator in a typical mental state in a timing acquisition mode, which acquires the EEG signals of the operator in a typical mental state during a period when there is no obvious operation log information during the timing operation process; The EEG acquisition device includes an EEG cap, the electrodes of which are arranged in the frontal lobe, occipital lobe, parietal lobe and temporal lobe of the brain, and the sampling frequency of the EEG acquisition device is 1000Hz; the specific steps are as follows: S1, EEG acquisition equipment collects EEG signals from the frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of the cerebral cortex; S2. Control the movement of the virtual collaborative robot through the game controller to complete the mental state inducing task. The category of mental state inducing task is a combination of one or more of trajectory tracking, grabbing blocks, and transferring blocks. In the process of completing the inducing task, each time a control instruction related to the robot arm or gripper is executed, the corresponding operation log information is generated. The virtual collaborative robot operation module includes 5 mental state inducing tasks. The specific control of the virtual collaborative robot movement through the game controller is: The game controller is connected to the second computer unit via a wireless USB receiver, and the virtual collaborative robot runs in the robot operating system ROS environment of the second computer unit. The game controller is operated to send control instructions to the virtual collaborative robot in the second computer unit. The virtual collaborative robot includes a chassis, a robotic arm and a gripper. The control instructions for controlling the linear velocity or angular velocity of the chassis movement are sent to the virtual collaborative robot through the joystick of the game controller, and the chassis performs forward, backward, left turn and right turn movements according to the control instructions to realize parallel movement and rotation; control instructions for controlling the opening or closing of the gripper and the movement or rotation of the end of the robotic arm along the X, Y, and Z directions in the Cartesian coordinate system are sent to the virtual collaborative robot through the buttons of the game controller, and the gripper or robotic arm performs stepping motion according to the button control instructions; The specific requirements of the five mental state induction tasks are as follows: In Task 0, the operator needs to control the virtual collaborative robot to walk in a circle in a clockwise direction in a rectangular room without restricting the movement trajectory of the virtual collaborative robot; In Task 1, the operator needs to control the virtual collaborative robot to walk along the two sides of the triangle; In Task 2, the operator needs to control the virtual collaborative robot to grab the small blue object on the table and then place the small object in the white tray; Task 3 is a time-limited task, where the operator needs to control the virtual collaborative robot to get out of the maze within 1 minute; Compared with Task 2, in Task 4, the small object was replaced by a Coke bottle, and the tray was placed on another table; The execution order of the mental state induced task requirements is Task 1→Task 2→Task 3→Task 0→Task 1→Task 2→Task 3→Task 0→Task 4; Task 1 is used to induce the operator's concentration state, and Task 2 is used to induce the operator to have a positive state and a negative state. When the operator is executing Task 2, once an error operation occurs during the task execution, the experiment guidance interface running in the first computer unit will give the operator an error operation prompt according to the operation log information received from the second computer unit. The error operation prompt includes voice prompts and text prompts. The error operation refers to a collision failure after the virtual collaborative robot performs an action; on the contrary, if Task 2 in the second computer unit is successfully completed, the experiment guidance interface will give the operator a correct operation prompt, and the correct operation prompt includes voice prompts and text prompts; the error operation prompt is used to induce a negative state, and the correct operation prompt is used to induce a positive state; Task 0 is used to induce the operator's non-concentration state; Task 3 is used to induce the operator to be in a nervous state; Task 4 is placed at the end of the mental state inducing task execution order. After the previous two rounds of mental state inducing tasks, Task 4 is used to induce the operator to have a fatigue state; S3. When the mental state inducing task is to grab or transfer an object, the operation log information generated in step S2 is transmitted from the second computer unit to the first computer unit through the data transmission network, and the capture instruction is used to trigger the collection. When the first computer unit recognizes the operation log information corresponding to the typical mental state, it sends a capture instruction to collect the EEG signal under the typical mental state, specifically: The first computer unit running MATLAB is used as a ROS node, and under the same network, a communication connection is realized between the first computer unit and the nodes in the ROS network running on the second computer unit, so as to realize two-way information transmission between the first computer unit and the second computer unit; In the ROS Master run by the second computer unit, the node / rosout is the publisher of the topic / rosout_agg. The / Msg information of the topic / rosout_agg is the control instructions received by the virtual collaborative robot, the calculation results generated by the received instructions, and the robot instruction execution results. The first computer unit node running MATLAB subscribes to the topic / rosout_agg and cyclically reads the message content to obtain real-time log information of the operation; When the EEG signal capture discriminator running on the first computer unit recognizes specific operation log information related to the typical mental state of the operator, a capture instruction is generated to wirelessly transmit the EEG data collected by the EEG acquisition device to the first computer unit via the data transmission network and store the data. The EEG acquisition device collects and stores EEG signals for 5 to 60 seconds; S4, when the mental state induction task is trajectory tracking, the EEG signal of step S1 is collected and stored at regular intervals; S5. Obtain and store the labels of the EEG signals obtained in step S3 and step S4 through the typical mental state subjective evaluation scale of the first computer unit, thereby completing the EEG capture of the operator's typical mental state.

Images