VR fusion online brain-controlled flight simulation driving system

Abstract

The present invention discloses a VR fusion online brain-controlled flight simulation driving system, the system comprising: a flight instruction VR excitation module, an electroencephalogram signal acquisition and processing module, an electroencephalogram signal classification module and a VR flight simulation driving module. The flight instruction VR excitation module provides VR visual excitation and stereo auditory excitation to excite users to imagine flight instructions; the electroencephalogram acquisition and processing module collects electroencephalogram signals when users imagine flight instructions and processes them; the electroencephalogram signal classification module constructs an electroencephalogram signal classifier by extracting the features of the electroencephalogram signal data set in the training phase, and uses the classifier to classify the electroencephalogram signals of users in the online brain control phase, and sends the classification results to the VR flight simulation driving module; the VR flight simulation driving module converts the received classification results into flight instructions for VR flight simulation, and sends the flight attitude to the flight instruction VR excitation module.

Description

Technical Field

[0001] The present invention relates to the technical field of brain-controlled flight simulation driving, and in particular to a VR-fused online brain-controlled flight simulation driving system. Background Art

[0002] A brain-computer interface is a communication system that does not rely on the normal output pathways of the brain's peripheral nerves and muscles. It can directly connect the brain to external devices and control external devices by collecting and processing EEG signals, such as prosthetic limbs, wheelchairs, and aircraft.

[0003] The flight driving simulation system is an important tool for pilot training and an effective means of integrated aircraft design and experimentation. It can significantly reduce training and design costs, improve efficiency and safety, and expand the scope of application. Virtual reality technology, with its inherent immersiveness, interactivity, and multi-sensory features, can not only enhance the visual and interactive realism of flight simulation driving, but also effectively solve the problems of high cost and easy site constraints.

[0004] In the existing brain-controlled flight technology, for example, the brain-controlled drone method based on steady-state visual evoked potential brain-computer interface disclosed in Chinese patent application CN111487988A stimulates the user to generate EEG signals by constructing an SSVEP stimulation interface, processes and classifies them, maps the classification results to different UAV flight control command vectors, fuses the control command vectors output by m+1 EEG data modules and transmits them to the UAV to achieve UAV flight control. Another example is an aircraft control system based on a hybrid brain-computer interface and its control method disclosed in Chinese patent application CN110716578A, which is characterized by comprising a main control module, a wireless communication module and an aircraft module, the aircraft module is equipped with an image acquisition submodule, the main control module includes a signal acquisition submodule, a data processing submodule and an image display submodule; the image display submodule obtains the real-time picture taken by the aircraft through the wireless communication module and displays it on a PC. For example, Chinese patent application CN108762303A discloses a portable brain-controlled drone system and control method based on motor imagery. It uses a high-precision and highly integrated analog front end and AD conversion chip to reduce the power consumption of the EEG signal acquisition module and improve the signal-to-noise ratio, and combines the EEG signal acquisition part with the electrode cap to improve the user's comfort and flexibility.

[0005] The problems with the above methods include: 1) The use of planar visual stimulation is inconsistent with the actual 3D visual ability of the human eye, resulting in a decrease in the classification performance of EEG signals; 2) The object controlled by the brain-computer interface is a physical object, which is easily restricted by the site, has poor flexibility and safety, high cost, and cannot meet the actual application needs of brain-controlled large aircraft. Summary of the invention

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to propose a VR fusion online brain-controlled flight simulation driving system with the characteristics of low cost and high performance.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] A VR-integrated online brain-controlled flight simulation driving system comprises a flight instruction VR excitation module, an electroencephalogram signal acquisition and processing module, an electroencephalogram signal classification module and a VR flight simulation driving module; the flight instruction VR excitation module provides VR visual excitation and stereo auditory excitation to excite users to imagine flight instructions; the electroencephalogram signal acquisition and processing module collects and processes electroencephalogram signals of users when imagining flight instructions; the electroencephalogram signal classification module constructs an electroencephalogram signal classifier by extracting features of electroencephalogram signals in a training phase, and uses the electroencephalogram signal classifier to classify electroencephalogram signals of users in an online brain control phase, and sends the classification results to the VR flight simulation driving module; the VR flight simulation driving module receives the classification results sent by the electroencephalogram signal classification module, converts the classification results into flight instructions for VR flight simulation, and sends the flight attitude to the flight instruction VR excitation module.

[0009] Furthermore, the flight instruction VR excitation module includes a VR visual excitation unit and a stereo auditory excitation unit; wherein the VR visual excitation unit provides VR flight instruction graphic symbols and VR aircraft posture, and the stereo auditory excitation unit provides stereo flight instruction pronunciation.

[0010] Furthermore, the EEG acquisition and processing module includes non-invasive and invasive EEG signal acquisition and processing modules.

[0011] Furthermore, the EEG acquisition and processing module acquires EEG signals of the user's imagination of flight instructions, and processes the EEG signals to eliminate interference and noise.

[0012] Furthermore, the training phase includes an idle phase and an imagination phase under audio-visual stimulation, and the audio-visual stimulation of different flight instructions is performed alternately.

[0013] Furthermore, the construction of the EEG signal classifier includes the following steps:

[0014] S1: The VR excitation module alternately displays various flight instructions on the user screen with different graphic symbols and VR aircraft postures, and simultaneously performs flight instruction pronunciation excitation;

[0015] S2: The EEG acquisition and processing module is connected to the flight instruction VR excitation module to collect EEG signals under different flight instruction conditions during the user training phase, and process and store them to form an EEG signal training data set;

[0016] S3: extract the features of the EEG signal training data set to form a feature vector as a training sample;

[0017] S4: Construct a multi-classifier model for EEG signal classification of flight commands;

[0018] S5: Use the training samples to train the multi-classifier model to obtain the EEG signal multi-classifier.

[0019] Furthermore, the VR flight simulation driving module includes a motion tracking unit and a stereoscopic display. The VR flight simulation driving module converts the received EEG signal classification results into flight instructions, and performs virtual reality flight simulation on the stereoscopic display according to the flight instructions. The flight scene changes with the user's head movement.

[0020] Furthermore, the user uses polarized 3D glasses to observe VR visual stimulation scenes and VR flight simulation driving scenes.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] The present invention integrates virtual reality (VR), stereo hearing, flight simulation driving and brain-computer interface technology. The VR scene is presented to the user through technologies such as a motion tracking system, a stereo display, and polarized 3D glasses, thereby avoiding the problems of heavy and dizzy helmet-type virtual reality. The present invention is particularly suitable for brain-controlled flight simulation driving application scenarios, can provide users with more motivating audio-visual incentives, generate higher quality electroencephalogram signals, and construct a more robust electroencephalogram signal classifier, so that users can realize online and accurate control of the VR flight simulation driving system through the brain-computer interface. The present invention has the characteristics of more realistic VR flight simulation driving scenes, higher safety and efficiency, no site restrictions, wide application range, strong robustness, low cost, high cost performance, etc. The present invention is suitable for controlling various simulated aircraft, provides an effective virtual training and testing environment for online brain-controlled flight simulation driving, and provides a new interactive mode for virtual reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the system structure of the present invention.

[0024] Figure 2 The schematic diagram of the multi-classification support vector machine (SVM) based on the pairwise classification method.

[0025] Figure 3 An example diagram showing the conversion of classification results into flight instructions. DETAILED DESCRIPTION

[0026] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0027] As attached Figure 1 As shown, the present invention discloses a VR fusion online brain-controlled flight simulation driving system, which includes a flight instruction VR excitation module, an electroencephalogram signal acquisition and processing module, an electroencephalogram signal classification module and a VR flight simulation driving module, wherein the flight instruction VR excitation module provides VR visual excitation and stereo auditory excitation to encourage users to imagine flight instructions; the electroencephalogram signal acquisition and processing module collects and processes the electroencephalogram signals of users when imagining flight instructions; the electroencephalogram signal classification module constructs an electroencephalogram signal classifier by extracting the features of electroencephalogram signals in a training phase, and uses the classifier to classify the electroencephalogram signals of users in an online brain control phase, and sends the classification results to the VR flight simulation driving module; the VR flight simulation driving module receives the classification results sent by the electroencephalogram signal classification module, converts them into flight instructions for VR flight simulation, and sends the flight attitude to the flight instruction VR excitation module.

[0028] Specifically, in this embodiment, the flight instruction VR excitation module includes a VR visual excitation unit and a stereo auditory excitation unit, wherein the VR visual excitation unit provides VR flight instruction graphic symbols and VR aircraft postures, and the stereo auditory excitation unit provides stereo flight instruction pronunciations, and the user wears polarized 3D glasses to observe the flight instruction VR visual excitation scene.

[0029] The EEG acquisition and processing module uses a non-invasive EEG signal acquisition and processing module to collect the EEG signals of the user when imagining flight instructions and process them to eliminate interference and noise. Among them, EEG signal processing includes baseline correction, eye electroencephalogram processing, artifact removal processing and bandpass filtering processing of EEG signal data.

[0030] The EEG signal classification module constructs an EEG signal classifier by extracting the features of the EEG signals in the training phase. The construction of the EEG signal classifier includes the following steps:

[0031] S1: The VR visual excitation unit displays six flight instructions, namely pitch, elevation, left roll, right roll, left yaw and right yaw, and the corresponding VR aircraft postures on the user screen in alternating manner with different graphic symbols. At the same time, the stereo auditory excitation unit performs corresponding flight instruction pronunciation excitation.

[0032] S2: The EEG acquisition and processing module acquires EEG signals under different flight command conditions during the user training phase, processes and stores them to form an EEG signal training data set.

[0033] S3: Extract the common spatial pattern features of the EEG signal training data set to form a feature vector as a training sample.

[0034] S4: A six-classification support vector machine model is constructed using the one-against-one (pairwise classification) method. The pairwise classification method constructs a binary support vector machine between every two classes, so it is necessary to train 6 (6-1) / 2 = 15 binary support vector machines (SVM1, SVM2, ..., SVM15). The final classification result is generated by voting, as shown in the attached figure. Figure 2 As shown in the figure, class A, class B, class C, class D, class E, and class F correspond to the six flight instructions of pitch, pitch, left roll, right roll, left yaw, and right yaw, respectively. Figure 3 shown.

[0035] S5: Use the training samples to train a six-class support vector machine model.

[0036] VR fusion online brain-controlled flight simulation driving includes the following steps:

[0037] S1: The VR visual excitation unit alternately displays the pitch, roll, yaw flight instructions and VR aircraft on the user screen with different graphic symbols, and at the same time the stereo auditory excitation unit performs the corresponding flight instruction pronunciation excitation;

[0038] S2: The non-invasive EEG signal acquisition and processing module acquires and processes the EEG signals of the user when he / she is imagining flight instructions;

[0039] S3: The EEG signal classification module uses the above-mentioned multi-classification support vector machine to classify the EEG signals of the user in the online brain control stage, and the classification result is one of the categories A, B, C, D, E, and F, and the classification result is sent to the VR flight simulation driving module;

[0040] S4: The VR flight simulation driving module receives the classification results sent by the EEG signal classification module through the TCP / IP protocol, and Figure 3 The classification results are converted into flight instructions for VR flight simulation, and the real-time flight attitude parameters are sent to the flight instruction VR excitation module through the TCP / IP protocol;

[0041] S5: The flight instruction VR excitation module changes the attitude of the VR aircraft in the flight instruction VR excitation module according to the received flight attitude parameters.

[0042] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A VR-integrated online brain-controlled flight simulation driving system, characterized by: It includes a flight instruction VR excitation module, an EEG signal acquisition and processing module, an EEG signal classification module and a VR flight simulation driving module; the flight instruction VR excitation module provides VR visual excitation and stereo auditory excitation to encourage users to imagine flight instructions; the EEG signal acquisition and processing module collects and processes EEG signals of users when imagining flight instructions; the EEG signal classification module constructs an EEG signal classifier by extracting the features of EEG signals in the training phase, and uses the EEG signal classifier to classify the EEG signals of users in the online brain control phase, and sends the classification results to the VR flight simulation driving module; the VR flight simulation driving module receives the classification results sent by the EEG signal classification module, converts the classification results into flight instructions for VR flight simulation, and sends the flight posture to the flight instruction VR excitation module; The construction of the EEG signal classifier comprises the following steps: S1: The VR excitation module alternately displays various flight instructions on the user screen with different graphic symbols and VR aircraft attitudes, and simultaneously performs flight instruction pronunciation excitation; specifically, the VR visual excitation unit alternately displays six flight instructions of pitch, elevation, left roll, right roll, left yaw, and right yaw and the corresponding VR aircraft attitudes on the user screen with different graphic symbols; S2: The EEG acquisition and processing module is connected to the flight command VR excitation module to collect EEG signals under different flight command conditions during the user training phase, and process and store them to form an EEG signal training data set; S3: extract the features of the EEG signal training data set to form a feature vector as a training sample; S4: Construct a multi-classifier model for EEG signal classification of flight instructions; specifically, a six-classification support vector machine model is constructed using a pairwise classification method. The pairwise classification method is to construct a two-classification support vector machine between every two classes. A total of 15 two-classification support vector machines are trained. The final classification result is generated by voting, where class A, class B, class C, class D, class E, and class F correspond to the six flight instructions of pitch, pitch, left roll, right roll, left yaw, and right yaw respectively; S5: Use the training samples to train the multi-classifier model to obtain the EEG signal multi-classifier; When VR is integrated with online brain-controlled flight simulation driving, the EEG signal classification module uses a classification support vector machine to classify the user's EEG signals in the online brain control stage. The classification result is one of A, B, C, D, E, and F, and the classification result is sent to the VR flight simulation driving module; the VR flight simulation driving module accepts the classification result sent by the EEG signal classification module, converts the classification result into flight instructions for VR flight simulation, and sends the real-time flight attitude parameters to the flight instruction VR excitation module; the flight instruction VR excitation module changes the attitude of the VR aircraft in the flight instruction VR excitation module according to the received flight attitude parameters.

2. The VR fusion online brain-controlled flight simulation driving system according to claim 1 is characterized by: The flight instruction VR excitation module includes a VR visual excitation unit and a stereo auditory excitation unit; wherein the VR visual excitation unit provides VR flight instruction graphic symbols and VR aircraft postures, and the stereo auditory excitation unit provides stereo flight instruction pronunciations.

3. The VR fusion online brain-controlled flight simulation driving system according to claim 1 is characterized by: The EEG acquisition and processing module includes non-invasive and invasive EEG signal acquisition and processing modules.

4. The VR fusion online brain-controlled flight simulation driving system according to claim 3 is characterized by: The EEG acquisition and processing module acquires the EEG signals of the user's imagination of flight instructions, and processes the EEG signals to eliminate interference and noise.

5. The VR fusion online brain-controlled flight simulation driving system according to claim 1 is characterized by: The training phase includes an idle phase and an imagination phase under audio-visual stimulation, and the audio-visual stimulation of different flight instructions is performed alternately.

6. The VR fusion online brain-controlled flight simulation driving system according to claim 1 is characterized by: The VR flight simulation driving module includes a motion tracking unit and a stereoscopic display. The VR flight simulation driving module converts the received EEG signal classification results into flight instructions, and performs virtual reality flight simulation on the stereoscopic display according to the flight instructions. The flight scene changes with the user's head movement.

7. The VR fusion online brain-controlled flight simulation driving system according to claim 1 is characterized by: Users use polarized 3D glasses to observe VR visual stimulation scenes and VR flight simulation driving scenes.

8. A computer device, characterized in that: The computer device is programmed or configured to execute the VR-integrated online brain-controlled flight simulation driving system as described in any one of claims 1 to 7, or the memory of the computer device stores computer programs and data that are programmed or configured to execute the VR-integrated online brain-controlled flight simulation driving system as described in any one of claims 1 to 7.

Images