A brain-computer interface intelligent device control system and method

Abstract

The present invention relates to a brain-computer interface intelligent device control system and method. In this system, a camera installed on an augmented reality device can capture environmental images in real time and transmit the images to a scene understanding module. This module can use deep learning technology to understand scene information and transmit this understanding information to an augmented reality display device. An eye tracker built into the display device can obtain the user's gaze position in real time. This eye tracker, combined with the SSVEP paradigm, operates on targets in the scene, then sends control or information acquisition instructions to the operated targets and provides feedback to the augmented reality device. This fully leverages the enhanced cognitive, visual, and perceptual capabilities of the combination of brain-computer interface and augmented reality technology, providing a solution for paralyzed patients to intuitively and efficiently control external devices.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface technology, and in particular to a brain-computer interface intelligent device control system and method. Background Art

[0002] A brain-computer interface (BCI) is a technology that establishes a direct connection between the human or animal brain and an external device. As one of the most promising research areas in neuroengineering today, BCI also holds important research significance and enormous application potential in fields such as rehabilitation medicine and intelligent robotics. In particular, over the past decade, BCI technology has made significant progress driven by advances in materials and biological sciences, gradually opening up a wide range of application scenarios. For example, in the medical and rehabilitation fields, it can be of significant help to patients with damaged nervous systems; in the military, it can be used for EEG information fusion and collaborative operations; and in education, it can help students develop concentration skills, among other applications.

[0003] To date, domestic and international laboratories have developed a wide variety of BCI systems for everyday applications. These utilize different stimulation paradigms and achieve diverse functions. Examples include steady-state visual evoked potential (SSVEP)-based BCI typing systems, multi-paradigm BCI VR gaming systems, and P300-based BCI virtual smart home control systems.

[0004] The above applications all achieve control of external devices or virtual devices through brain-computer interfaces and have achieved certain results. However, controlling other devices directly through a computer screen is not intuitive enough and cannot effectively improve the quality of life of paralyzed patients. Therefore, it should be combined with augmented reality (AR) technology. First, it can improve the intuitiveness of brain-computer interface control. Second, it solves the problem that AR devices must rely on hand movements to interact with the outside world, making them difficult to use for people with disabilities. However, the speed of signal recognition is relatively slow during use, and the user experience is not good.

[0005] Eye trackers can already, to a certain extent, provide paralyzed patients with the ability to control external devices. Their high recognition speed can be of great help in helping disabled patients control peripherals. However, their poor recognition accuracy and the need for frequent calibration will greatly reduce the user experience, so their ease of use has always been questioned.

[0006] Therefore, how to improve the control accuracy and recognition rate of brain-computer interface intelligent devices is an urgent problem to be solved. Summary of the Invention

[0007] The purpose of the present invention is to provide a brain-computer interface intelligent device control system and method, which can improve the accuracy and recognition rate of brain-computer interface intelligent device control.

[0008] To achieve the above object, the present invention provides the following solutions:

[0009] A brain-computer interface intelligent device control system, comprising: a camera, an augmented reality stimulation device, an eye tracker, a scene understanding module, an EEG acquisition device, an EEG processing module, a result integration module, a central control module, and a peripheral control system;

[0010] The camera is mounted on the augmented reality stimulation device and sends the real-time collected environment image to the augmented reality stimulation device via a USB interface;

[0011] The augmented reality stimulation device is connected to the scene understanding module; and the real-time collected environment image is sent to the scene understanding module; the scene understanding module uses deep learning technology to classify the real-time collected environment image, and sends the image classification result to the augmented reality display device in the augmented reality stimulation device and the result integration module;

[0012] The eye tracker is arranged inside the augmented reality display device and connected to the EEG processing module; the eye tracker is used to obtain the user's gaze position in real time;

[0013] The EEG acquisition device is connected to the EEG processing module; the EEG processing module uses an asynchronous SSVEP EEG signal judgment algorithm based on the EEG signal matrix collected by the EEG acquisition device to determine the classification result of the real-time EEG signal, and sends the classification result of the real-time EEG signal to the result integration module; the classification result of the real-time EEG signal is the signal frequency induced in the user's occipital lobe area;

[0014] The result integration module determines the target selected by the user based on the image classification result and the real-time EEG signal classification result, and sends the selection result to the central control module;

[0015] The central control module determines an operation instruction according to the selection result and sends the operation instruction to the peripheral control system;

[0016] The peripheral control system controls the smart device according to the operation instructions.

[0017] Optionally, the augmented reality stimulation device converts the real-time collected environment image into a 1280×720×3 image matrix through the OpenCV toolkit.

[0018] Optionally, the PC of the augmented reality display device is AR glasses.

[0019] Optionally, the EEG acquisition device acquires EEG signals at a sampling rate of 1000 Hz according to the 10-20 international standard.

[0020] A method for controlling a brain-computer interface intelligent device, comprising:

[0021] Using a camera to capture the scene, and sending the real-time captured environment image to the augmented reality stimulation device;

[0022] The augmented reality stimulation device sends the real-time collected environment image to the scene understanding module. The scene understanding module uses deep learning technology to classify the real-time collected environment image and sends the image classification result to the augmented reality display device and the result integration module in the augmented reality stimulation device.

[0023] The EEG processing module obtains the user's gaze position in real time from the eye tracker and the EEG signal matrix collected by the EEG acquisition device, adopts the asynchronous SSVEP EEG signal judgment algorithm to determine the classification result of the real-time EEG signal, and sends the classification result of the real-time EEG signal to the result integration module;

[0024] The result integration module determines the user's operation target based on the image classification results and the real-time EEG signal classification results, and sends the determination results to the central control module;

[0025] The central control module determines the operation instruction according to the determination result and sends the operation instruction to the peripheral control system;

[0026] The peripheral control system controls the intelligent device according to the operation instructions.

[0027] Optionally, the augmented reality stimulation device sends the real-time collected environment image to the scene understanding module, and the scene understanding module classifies the real-time collected environment image using deep learning technology and sends the image classification result to the augmented reality display device and the result integration module in the augmented reality stimulation device, specifically including:

[0028] The augmented reality stimulation device transmits the real-time collected environment images to the scene understanding module in the form of video stream;

[0029] The scene understanding module divides the video stream into frames and performs object detection on the scene frame by frame;

[0030] After confirming the existence of the operation target, the scene understanding module assigns stimulus attributes to the target and communicates with the augmented reality display device and the result integration module.

[0031] Optionally, the EEG processing module obtains the user's gaze position in real time from the eye tracker and the EEG signal matrix collected by the EEG acquisition device, adopts an asynchronous SSVEP EEG signal judgment algorithm to determine the classification result of the real-time EEG signal, and sends the classification result of the real-time EEG signal to the result integration module, specifically including:

[0032] Determine whether the user's gaze position is within the judgment target hotspot range;

[0033] If it is outside the judgment target hotspot range, the augmented reality stimulation device will notify the EEG processing module to stop calculating the classification results of the real-time EEG signal.

[0034] Optionally, the EEG acquisition device acquires EEG signals at a sampling rate of 1000 Hz according to the 10-20 international standard.

[0035] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0036] The present invention provides a brain-computer interface intelligent device control system and method. The camera installed on the augmented reality device can collect environmental images in real time and transmit the images to the scene understanding module. The module can understand the scene information by adopting deep learning technology and transmit the understood information to the augmented reality display device. The eye tracker built into the reality display device can obtain the user's gaze position in real time. It combines the SSVEP paradigm to operate the target in the scene, and then sends control or information acquisition instructions to the operation target, and provides feedback to the augmented reality device. It fully utilizes the combination of brain-computer interface and augmented reality technology to bring about enhancements in cognitive, visual, perceptual and other abilities, and realizes a solution for paralyzed patients to intuitively and efficiently control external devices. Furthermore, it can improve the accuracy and recognition rate of brain-computer interface intelligent device control. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is a schematic diagram of the control system structure of a brain-computer interface intelligent device provided by the present invention;

[0039] Figure 2 This is a diagram of the first-level menu that users see through AR glasses;

[0040] Figure 3This is a diagram of the secondary menu that users see through AR glasses. DETAILED DESCRIPTION

[0041] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0042] The purpose of the present invention is to provide a brain-computer interface intelligent device control system and method, which can improve the accuracy and recognition rate of brain-computer interface intelligent device control.

[0043] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] Figure 1 This is a schematic diagram of the control system structure of a brain-computer interface intelligent device provided by the present invention, such as Figure 1 As shown, the present invention provides a brain-computer interface intelligent device control system, including: a camera, an augmented reality stimulation device, an eye tracker, a scene understanding module, an EEG acquisition device, an EEG processing module, a result integration module, a central control module, and a peripheral control system. The camera is a 720P camera.

[0045] The camera is mounted on the augmented reality stimulation device and sends the real-time collected environment image to the augmented reality stimulation device via a USB interface.

[0046] The augmented reality stimulation device is connected to the scene understanding module; and sends the real-time collected environmental images to the scene understanding module; the scene understanding module uses deep learning technology to classify the real-time collected environmental images, and sends the image classification results to the augmented reality display device in the augmented reality stimulation device and the result integration module.

[0047] The eye tracker is arranged inside the augmented reality display device and connected to the electroencephalogram processing module; the eye tracker is used to obtain the user's gaze position in real time.

[0048] The EEG acquisition device is connected to the EEG processing module; the EEG processing module uses an asynchronous SSVEP EEG signal judgment algorithm based on the EEG signal matrix collected in the EEG acquisition device to determine the classification result of the real-time EEG signal, and sends the classification result of the real-time EEG signal to the result integration module; the classification result of the real-time EEG signal is the signal frequency induced by the user's occipital lobe area.

[0049] The result integration module jointly determines the user-selected target based on the image classification result and the real-time EEG signal classification result, and sends the selection result to the central control module.

[0050] The central control module determines an operation instruction according to the selection result and sends the operation instruction to the peripheral control system.

[0051] The peripheral control system controls the smart device according to the operating instructions, that is, operates through Bluetooth or wireless local area network.

[0052] The central control module verifies the input information: it verifies the operability of the selected object and queries the operations that can be performed on it. Its internal program database stores the names, operation methods, and current status of all operable devices in the scene. After processing by the central control module, if the operation object and operation instructions are valid, it will communicate with the peripheral control system and control the valid device through the peripheral control system.

[0053] Figure 1 In the system, a circular data buffer is used to achieve isolation between the EEG processing module and the EEG acquisition device, between the augmented reality stimulation device and the scene understanding module, and between the result integration module and other modules. This ensures that each endpoint can flexibly read and process data or instructions, and improves the stability of the program. In addition, since the system uses an asynchronous EEG processing algorithm and combines eye gaze data, there is no need to connect a parallel port trigger between the AR stimulation and the EEG acquisition device for clock synchronization. It can also ensure that fewer judgment results appear when the user is not looking at the stimulation interface, reducing the false alarm rate; when the user is looking at the stimulation target, the target can be confirmed quickly, improving the judgment speed.

[0054] like Figure 2 and Figure 3 As shown in the figure, the user sees the first-level menu through the AR glasses, which displays the devices that can be operated in the scene. Stimulus targets of different frequencies or phases are superimposed in the field of view through the AR glasses; if the user looks at the air-conditioning target block, the system enters the corresponding second-level menu, and the available operations are displayed in the stimulation interface.

[0055] The present invention only uses the SSVEP paradigm instead of using multiple paradigms such as P300, RSVP, and MI. Firstly, it can greatly reduce the computational complexity of each computing unit and reduce energy consumption. Secondly, the method of using the system by only looking at the operable target is more in line with the logic of general interaction, which reduces the threshold for use and the difficulty of learning. In addition, the number of operations is much smaller than that of P300. 2Or the order of magnitude of n is greatly reduced, and only two operations and four EEG signal judgments are needed at most to complete the control of the device.

[0056] When the subsystems interact, the information that needs to be transmitted includes video streams, string instructions, etc. In addition, considering the flexibility and performance of the deployment, the Kafka message middleware technology with good performance and deployment convenience is adopted to transmit messages. The Kafka message middleware has high throughput, load balancing, pull system, dynamic expansion capabilities, and can realize the distributed deployment and operation of the various systems in the invention, greatly reducing the maintenance cost of users.

[0057] As a specific embodiment, the scene understanding module uses deep learning technology to classify the image matrix captured by the camera. This patent uses the Yolo_v5 neural network for this operation. This includes training the network using a dataset for identifying targets and using the trained network to recognize the video stream captured by the camera. The recognition results include the name of the object and the object's position matrix in the image.

[0058] Train the Yolo_v5 network before the system runs:

[0059] First, take clear, 1280x720, jpg images of the objects in the scene. For each object, take 10 images from four different angles, front, back, left, and right, with varying lighting conditions. Then, use the yolo_mark software to annotate the objects in the images, including their names and coordinates. Once the annotations are complete, the software generates a binary file containing the image data and the annotations. This training set can then be used to train the Yolo_5 network.

[0060] During system operation, the steps for using the Yolo_v5 network to understand the scene of the data collected by the camera are as follows:

[0061] Grayscale conversion: In the RGB model, R=G=B becomes a grayscale value. We only need to use one byte for each pixel to store the grayscale value of 0-255. This patent uses the weighted average method for grayscale conversion, and the formula is:

[0062] f(x,y)=0.299R(x,y)+0.587G(x,y)+0.114B(x,y)

[0063] Geometric transformation: This step, also known as image space transformation, involves geometrically transforming the image using translation, transposition, mirroring, scaling, and rotation. This can correct for systematic errors in image acquisition and sensor-induced deformations. A bilinear interpolation algorithm is used to map the transformed coordinates to the integer coordinates of the input image. The formula is:

[0064]

[0065]

[0066] Among them, Q 11 =(x1, y1), Q 12 =(x1,y2)Q 21 =(x2, y1)Q 22 =(x², y²). f is the value at a certain point after the transformation. For each image, there is a 5% and independent probability of performing the above-mentioned translation, transposition, mirroring, scaling, and rotation transformations.

[0067] After the above preprocessing, the Yolo_v5 neural network can be used for target recognition.

[0068] When the Yolo_v5 neural network processes each frame, the network will give a bounding box of each target: [x1, y1; x2, y2] and the target label content and the judgment probability [label, p]. First, find the midpoint of the bounding box for each target. Area S target =|(x1-x2)(y1-y2)|. The network will perform dynamic window processing on each decision result: ①: Calculate the average value of the midpoint of the corresponding object from the current frame to 90 frames (i.e. 3 seconds for the camera). If the value exceeds the large-scale motion decision threshold ∈ m , it is judged that the user is still moving at high speed relative to the device;②: calculate the average area of ​​the corresponding object from the current frame to 90 frames ago, if it exceeds the long-distance target judgment threshold ∈ dist , it is judged that the user is far away from the device and is not suitable for operation, and the number of non-target stimuli in the field of view can be reduced; ③: the average judgment probability of the corresponding object from the current frame to 90 frames ago is calculated (if an object existing in the previous frame is detected in a frame, its judgment probability is set to 0), if the average judgment probability of a target is less than ∈ ext , then the object is judged to be a false alarm of the neural network.

[0069] If a target does not meet the above three elimination strategies, the scene understanding module in step 1 will assign stimulus attributes to each operable object, specifically:

[0070] The SSVEP paradigm is used to present and select stimulus targets. The frequency of the human brain's response to the SSVEP paradigm is generally between 8Hz and 16Hz, and a 0.2Hz interval can obtain better results. The lower the frequency, the more obvious the EEG response. First, the distance from the midpoint of each target to the midpoint of the visual field is calculated and sorted to obtain d i′ , the smaller the distance, the lower the frequency is assigned (unit: Hz):

[0071] f i′ =8.0+(i-1)*0.2, and a phase difference of 0.5π: φ i′ =imod4×0.5π. In addition, the area of ​​each point is set to the average area in the above process.

[0072] As a specific embodiment, the augmented reality stimulation device converts the real-time collected environment image into a 1280×720×3 image matrix through the OpenCV toolkit.

[0073] As a specific embodiment, the PC of the augmented reality display device is AR glasses.

[0074] As a specific embodiment, the EEG acquisition device acquires EEG signals at a sampling rate of 1000 Hz according to the 10-20 international standard.

[0075] The present invention provides a method for controlling a brain-computer interface intelligent device, comprising:

[0076] S101, using a camera to capture a scene, and sending the real-time captured environment image to the augmented reality stimulation device;

[0077] S102, the augmented reality stimulation device sends the real-time collected environment image to the scene understanding module, the scene understanding module uses deep learning technology to classify the real-time collected environment image, and sends the image classification result to the augmented reality display device and the result integration module in the augmented reality stimulation device;

[0078] S102 specifically includes:

[0079] The augmented reality stimulation device transmits the real-time collected environment images to the scene understanding module in the form of video stream;

[0080] The scene understanding module divides the video stream into frames and performs object detection on the scene frame by frame;

[0081] After confirming the existence of the operation target, the scene understanding module assigns stimulus attributes to the target and communicates with the augmented reality display device and the result integration module.

[0082] S103: The EEG processing module obtains the user's gaze position in real time from the eye tracker and the EEG signal matrix collected by the EEG acquisition device, uses the asynchronous SSVEP EEG signal judgment algorithm to determine the classification result of the real-time EEG signal, and sends the classification result of the real-time EEG signal to the result integration module;

[0083] S103 specifically includes:

[0084] Determine whether the user's gaze position is within the judgment target hotspot range;

[0085] If it is outside the judgment target hotspot range, the augmented reality stimulation device will notify the EEG processing module to stop calculating the classification results of the real-time EEG signal.

[0086] S104, the result integration module determines the user operation target based on the image classification result and the real-time EEG signal classification result, and sends the determination result to the central control module;

[0087] S105, the central control module determines an operation instruction according to the determination result, and sends the operation instruction to the peripheral control system;

[0088] S106: The peripheral control system controls the smart device according to the operation instruction.

[0089] The method provided by the present invention is further described below through specific examples:

[0090] Step 1: The camera on the augmented reality device captures the scene and transmits it to the scene understanding module in the form of a video stream. The scene understanding module divides the video stream into frames and performs target detection on the scene frame by frame. After confirming the existence of a valid operation target, the scene understanding module assigns stimulus attributes to the operable object and communicates with the augmented reality device and result integration module to change its operating state to the first-level menu state.

[0091] Step 2: The AR display device receives the target attributes and renders the target. After entering the first-level menu, the central control module is notified to enable EEG processing mode. Simultaneously, the eye tracker begins providing gaze data to the AR display device to assist with EEG signal recognition in step 3. The EEG processing system begins pulling EEG data from the EEG acquisition device into a ring buffer. Once the AR display device begins stable stimulation, the central control module notifies the EEG processing system to retrieve data from the ring buffer and begin calculations.

[0092] Step 3: If the user's gaze falls outside the judgment target hotspot, the stimulation device notifies the central control module to stop EEG signal calculation and determine that the user has mistakenly entered the stimulation scene. If the user's gaze is close to a judgment target, the central control module increases the confidence level of the stimulation target, increasing its likelihood of being judged. If the user selects a device, the EEG processing module notifies the result integration module, which queries the corresponding device, its available operations, and the corresponding stimulation attributes based on the judgment result. The result integration module simultaneously communicates with the augmented reality module and the central control module to transmit the results. At the same time, the built-in program of the eye tracker uses the position of the stimulation target for calibration.

[0093] Step 4: After receiving the results, the stimulation module renders the stimulus according to the new stimulus attributes. Entering the secondary menu corresponding to the device selected in Step 3 and commencing SSVEP stimulation, the EEG processing module continues to pull EEG data from the EEG acquisition device. Simultaneously, the central control module establishes a connection with the peripheral device through the peripheral control module and obtains the current device's attribute information.

[0094] Step 5: When the user selects an action, the result integration module queries the action corresponding to the stimulus, similar to step 3. It then communicates with the central control module, which then controls the smart device through the peripheral control module. After the action is completed, the camera continues capturing the scene, and the process returns to step 1.

[0095] In step 3, the eye tracker can output the coordinates of the user's gaze point on the AR glasses at a frequency of 100Hz, represented by p = (x, y). The stimulation device will perform a dynamic window processing on this data: the weighted average of the user's gaze point within 2 seconds (200 sampling points) is calculated, and it is determined whether this point is within the stimulation target range obtained in step 1. If more than 80 gaze points (80% of the time) within 1 second are not within the stimulation target range, the stimulation system will communicate with the central control system and the EEG processing system, terminate the EEG recognition task, return to the previous interface, and return to step 1.

[0096] If the user has more than 80 gaze points within the range of a stimulation target within 1 second, the stimulation system will communicate with the EEG processing system and send the frequency of the stimulation target. The EEG processing system will lower the judgment threshold of the target, that is, increase the possibility of the target being recognized, that is, it can speed up the recognition of the target. After the user selects a target, the result integration module will process the judgment result and update the subsequent system logic. Specifically:

[0097] During the EEG judgment process, the system only determines that the user has performed a specific action if they continuously select a button. This secondary confirmation improves fault tolerance. After calculating the judgment result, the EEG processing module sends the judgment frequency to the result integration module. The result integration module then queries the corresponding device based on this frequency and its corresponding secondary menu operation. In this example, the "air conditioner" device is selected. The stimulus attributes corresponding to the secondary menu are no longer automatically generated by the system, but are stored in a predefined database.

[0098] The eye tracker can be calibrated using this coordinate to ensure a shorter calibration interval, which can greatly enhance the auxiliary role of the eye tracker in EEG recognition.

[0099] In step 4, the central control module establishes a connection with the peripheral through the peripheral control module and obtains attribute information, specifically:

[0100] The IP addresses of various devices are pre-stored in the peripheral control module, and they interact through port 11451.

[0101] This invention achieves brain-computer interface stimulation by collecting user EEG signals and using augmented reality to identify and display peripheral devices in the scene. This further enables the coordinated application of brain-computer interfaces, augmented reality, and brain-computer interface systems. This enables the brain-computer interface to integrate environmental information and intuitively and efficiently control external devices. This improves the ability of paralyzed patients to care for themselves and significantly enhances their quality of life.

[0102] Based on the steady-state visual evoked potential (SSVEP) brain-computer interface system developed in the early stage, this invention combines augmented reality technology, eye movement signal capture technology, and artificial intelligence technology to realize an augmented reality-based brain-computer interface smart home control system, allowing paralyzed patients to directly and intuitively control smart home devices through the SSVEP paradigm.

[0103] This invention implements an intelligent brain-computer interface (BCI)-controlled smart home system platform, dedicated to enhancing the self-care abilities and quality of life of paralyzed patients. Leveraging advanced BCI and augmented reality technologies, the system integrates the human brain with a computer based on scene information, enabling simple and easy-to-use smart home control. This breakthrough transcends the limitations of BCI technology, enhances human-computer integration, and revolutionizes the operational model of practical BCI systems.

[0104] The present invention only adopts the asynchronous SSVEP paradigm and does not require the use of a Trigger device to synchronize various subsystems, which greatly simplifies the difficulty of system deployment and reduces deployment costs; the single SSVEP paradigm mode can only use the 8-channel EEG data of the occipital lobe, reducing the amount of data in the system; and the present invention uses a camera to collect information about the scene, and can detect operable peripherals in the field of view in real time through artificial intelligence technology, which is more intuitive to operate than other inventions; and the two-confirmation strategy in the present invention is more efficient and simple to operate than the "confirm"-"cancel" logic in other inventions; the present invention additionally utilizes eye movement capture technology, which can speed up the speed of EEG signal recognition, and the technology will not be perceived by the user, that is, the user does not need to learn, adapt and convert multiple paradigms like the paradigms provided in other related inventions. Eye movement technology is introduced into the system in a way that will not be perceived by the user, while achieving the purpose of improving user experience.

[0105] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0106] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A brain-computer interface intelligent device control system, characterized in that: include: Camera, augmented reality stimulation device, eye tracker, scene understanding module, EEG acquisition device, EEG processing module, result integration module, central control module and peripheral control system; The camera is mounted on the augmented reality stimulation device and sends the real-time collected environment image to the augmented reality stimulation device via a USB interface; The augmented reality stimulation device is connected to the scene understanding module; and sending the real-time collected environment image to the scene understanding module; The scene understanding module uses deep learning technology to classify the real-time collected environmental images, and sends the image classification results to the augmented reality display device in the augmented reality stimulation device and the result integration module; The eye tracker is disposed inside the augmented reality display device and connected to the central control module; the eye tracker is used to send the acquired user's gaze position to the central control module, and the central control module determines whether the user's gaze position is within the judgment target hotspot range; and when the user's gaze position is outside the judgment target hotspot range, notifies the EEG processing module to stop calculating the classification result of the real-time computer signal; The EEG acquisition device is connected to the EEG processing module; the EEG processing module obtains the user's gaze position and the EEG signal matrix collected by the EEG acquisition device in real time based on the eye tracker, adopts the asynchronous SSVEP EEG signal judgment algorithm to determine the classification result of the real-time EEG signal, and sends the classification result of the real-time EEG signal to the result integration module; The classification result of the real-time EEG signal is the signal frequency induced in the user's occipital lobe area; The result integration module determines the target selected by the user based on the image classification result and the real-time EEG signal classification result, and sends the selection result to the central control module; The central control module determines an operation instruction according to the selection result and sends the operation instruction to the peripheral control system; The peripheral control system controls the smart device according to the operation instructions.

2. A brain-computer interface intelligent device control system according to claim 1, characterized in that: The augmented reality stimulation device converts the real-time collected environment image into a 1280×720×3 image matrix through the OpenCV toolkit.

3. A brain-computer interface intelligent device control system according to claim 1, characterized in that: The PC of the augmented reality display device is AR glasses.

4. A brain-computer interface intelligent device control system according to claim 1, characterized in that: The EEG acquisition device collects EEG signals at a sampling rate of 1000 Hz according to the 10-20 international standard.

5. A method for controlling a brain-computer interface intelligent device, characterized in that: include: The camera is used to capture the scene and the real-time captured environment image is sent to the augmented reality stimulation device; The augmented reality stimulation device sends the real-time collected environment image to the scene understanding module. The scene understanding module uses deep learning technology to classify the real-time collected environment image and sends the image classification result to the augmented reality display device and the result integration module in the augmented reality stimulation device. The EEG processing module obtains the user's gaze position in real time from the eye tracker and the EEG signal matrix collected by the EEG acquisition device, adopts the asynchronous SSVEP EEG signal judgment algorithm to determine the classification result of the real-time EEG signal, and sends the classification result of the real-time EEG signal to the result integration module; The result integration module determines the user's operation target based on the image classification results and the real-time EEG signal classification results, and sends the determination results to the central control module; The central control module determines the operation instruction according to the determination result and sends the operation instruction to the peripheral control system; The peripheral control system controls the smart device according to the operating instructions; The central control module determines whether the user's gaze position is within the judgment target hotspot range; If it is outside the judgment target hotspot range, the EEG processing module is notified to stop calculating the classification results of the real-time EEG signal.

6. A method for controlling a brain-computer interface intelligent device according to claim 5, characterized in that: The augmented reality stimulation device sends the real-time collected environment image to the scene understanding module. The scene understanding module uses deep learning technology to classify the real-time collected environment image and sends the image classification results to the augmented reality display device and result integration module in the augmented reality stimulation device, specifically including: The augmented reality stimulation device transmits the real-time collected environment images to the scene understanding module in the form of video stream; The scene understanding module divides the video stream into frames and performs object detection on the scene frame by frame; After confirming the existence of the operation target, the scene understanding module assigns stimulus attributes to the target and communicates with the augmented reality display device and the result integration module.

7. A method for controlling a brain-computer interface intelligent device according to claim 5, characterized in that: The EEG acquisition device collects EEG signals at a sampling rate of 1000 Hz according to the 10-20 international standard.

Images