A vehicle-human interaction control method and device based on SSVEP-MI fusion and a vehicle

Abstract

The application relates to a kind of vehicle interaction control methods based on SSVEP-MI fusion, comprising: obtaining visual stimulation signal induced by user in one of a plurality of predetermined regions on the central control display and / or the brain potential signal generated by user based on motor imagination;The brain potential signal is decoded, and the user intention is determined based on the decoding result;Based on user intention, the corresponding control is executed to the vehicle.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and brain science fusion, specifically to a human-vehicle interaction control method, device, and automobile based on SSVEP-MI fusion. Background Technology

[0002] Voice interaction is not easy to achieve in silent or noisy environments, while brain-computer interaction has advantages in these special conditions.

[0003] Brain-computer interface (BCI) refers to the communication and control between the human brain and computers or other electronic devices based on brain electrical activity (BED) signals. It is a novel human-computer interaction method. Its principle is to acquire changes in the electrical activity of the nervous system through BED signal detection technology, classify and identify these BED signals to determine the intentional actions that trigger these changes, and then use a computer to convert human thought processes into command signals to drive external devices. This allows the human brain to directly control the external environment without the direct involvement of muscles and peripheral nerves.

[0004] Voice interaction is difficult to achieve in noisy environments, while brain-computer interface (BCI) has advantages in these special conditions. Of course, in most cases, voice and gesture technologies have inherent advantages, but EEG is being considered a forward-looking technology because it has greater potential. Its information encoding potential far exceeds that of voice and gesture, and its current technological development is still limited, but its future potential is enormous.

[0005] Studies have shown that when a person is exposed to a visual stimulus of a fixed frequency, the visual cortex of the brain produces a continuous response related to the stimulus frequency (at the fundamental or harmonic frequency). This response is known as Steady-State Visual Evoked Potentials (SSVEP).

[0006] Motor imagery (MI), as the name suggests, involves the activation of specific brain regions even when a person imagines their limbs (or muscles) moving without actual physical movement. By analyzing brain electrical signals and detecting the activation effects of different brain regions, the user's intention can be determined, thereby enabling direct communication and control between the brain and external devices. Currently, the most common brain regions involved in motor imagery are: left and right, right hand, both feet, and tongue.

[0007] BCI is an abbreviation for Brain Computer Interface, which is a direct connection pathway established between the human or animal brain (or a culture of brain cells) and an external device.

[0008] Reference document 1: Patent CN 108459714 B discloses a few-channel asynchronous control brain-computer interface system based on the dual paradigms of MI and SSVEP. It is mainly aimed at BCI systems with few channels of brain potential signals. It adopts the motor imagery paradigm as the switching module of the BCI system and the steady-state visual evoked potential paradigm as the multi-select module of the BCI system. The two modules are connected in series to form an asynchronous control BCI system. The multivariate empirical pattern decomposition algorithm is used to decompose the few-channel brain potential signals into multiple intrinsic mode functions, thereby improving the control effect and classification accuracy of the few-channel BCI system.

[0009] Comparative document 2: Patent CN 113101021 A discloses a robotic arm control method based on a MI-SSVEP hybrid brain-computer interface. This method achieves robotic arm control through steps such as non-invasive EEG data acquisition, online preprocessing of EEG signals, online processing and decoding of EEG signals, generation of user intent targets, and conversion of those targets into control commands. By using the MI-SSVEP hybrid brain-computer interface experimental paradigm, the number of degrees of freedom in the brain-computer interface-controlled robotic arm system is increased, enabling motion control of the robotic arm in three-dimensional space.

[0010] Both patents mentioned above use the MI-SSVEP hybrid paradigm brain-computer interface to control different operating systems. Comparative document 1 uses the motor imagery paradigm as the system's switch, and the SSVEP paradigm as the main paradigm for controlling the system, undertaking the task of selecting all functional modules. It requires the user to maintain visual stimulation after the system is turned on in order to control the functions, which cannot solve the problem of keeping eyes on the road while driving. Comparative document 2 defines nine robotic arm action commands corresponding to each frequency of SSVEP and each state of the MI paradigm. SSVEP and the MI paradigm can each control four action commands, but about 30% of people are naturally unable to train motor imagery, that is, they cannot control the corresponding four commands through the MI paradigm. This system does not provide a solution to this problem. Summary of the Invention

[0011] This invention provides a human-vehicle interaction control method, device, and vehicle based on SSEVP-MI fusion, which solves the problem that drivers cannot operate in-vehicle applications when their hands and feet are occupied during driving, and replaces the traditional human-machine interaction mode.

[0012] The technical solution of this invention is as follows:

[0013] This invention provides a human-vehicle interaction control method based on SSVEP-MI fusion, comprising:

[0014] Periodically acquire visual stimulation signals generated by the user due to one of the multiple predetermined areas on the central control display screen and / or brain potential signals generated by the user based on motor imagination;

[0015] The brain potential signals acquired during the sampling period are decoded, and the user's intent is determined based on the decoding results;

[0016] Based on the user's intent, corresponding controls are executed on the vehicle.

[0017] Preferably, when multiple predetermined areas on the central control display screen generate visual stimulus signals, the visual stimulus frequency corresponding to each predetermined area is different;

[0018] The same predetermined area may represent different functions on different display interfaces, but the same predetermined area may correspond to the same frequency of visual stimuli on different display interfaces.

[0019] Preferably, the central control display screen shows a main function area and an auxiliary function area. The functions indicated by multiple predetermined areas in the main function area can only be triggered by the user based on visual stimulus signals. The functions indicated by multiple predetermined areas in the auxiliary function area can be triggered by the user based on visual stimulus signals, or by the user based on motion imagination.

[0020] Preferably, the step of decoding the brain potential signals acquired within the sampling period and determining the user's intent based on the decoding results includes:

[0021] The brain potential signals acquired within the sampling period are amplified, filtered, and artifact removed.

[0022] The brain potential signals are input into the SSVEP potential signal feature extraction channel and the MI potential signal feature extraction channel, respectively, to determine whether the driver generates brain potential signals based on visual stimulus signals during the sampling period.

[0023] If the driver generates brain potential signals based on visual stimuli, the Fisher linear recognition method is used to classify the extracted SSVEP potential signal features, and the user intent is obtained based on the classification results.

[0024] If the driver does not generate brain potential signals based on visual stimuli, the LDA classifier is used to classify the extracted MI potential signal features, and the user intent is obtained based on the classification results.

[0025] Preferably, the user generates a conductive bit signal corresponding to the function of the left predetermined area in the auxiliary function area by generating the idea of ​​clenching the left fist, and generates a conductive bit signal corresponding to the function of the right predetermined area in the auxiliary function area by generating the idea of ​​clenching the right fist.

[0026] The present invention also provides a human-vehicle interaction control device based on SSVEP-MI fusion, comprising:

[0027] The acquisition module is used to periodically induce visual stimulation signals generated by the user due to one of the multiple predetermined areas on the central control display screen and / or brain potential signals generated by the user based on motor imagination.

[0028] The decoding module is used to decode the brain potential signals acquired within the sampling period and determine the user's intent based on the decoding results.

[0029] The control module is used to perform corresponding controls on the vehicle based on user intent.

[0030] This invention also provides an automobile, comprising: a visual stimulation system, a signal acquisition system, and a control system. The visual stimulation system generates a visual stimulation signal in one of a plurality of predetermined areas on a central control display screen. The signal acquisition system periodically acquires brain electrical signals induced by the visual stimulation signal generated by the user in one of the predetermined areas on the central control display screen and / or generated by the user based on motor imagery. The system decodes the brain electrical signals acquired by the signal acquisition system within the sampling period and determines the user's intention based on the decoding result. The control system performs corresponding control on the vehicle based on the user's intention. The beneficial effects of this invention are:

[0031] (1) The two paradigms can be used interchangeably to control in-vehicle applications at any time, which is more flexible; (2) The motion imagination paradigm provides two virtual shortcut buttons, so that users do not need to look at the central control display screen when controlling certain high-frequency functions. Users can keep their eyes on the road while driving, which improves driving safety; (3) SSVEP is used as the main paradigm and motion imagination is used as the auxiliary paradigm. SSVEP covers all control functions, and motion imagination includes some control functions, which makes up for the shortcomings of other SSVEP-MI hybrid paradigm systems where some people cannot use some functions through the motion imagination paradigm. Attached Figure Description

[0032] Figure 1 This is a structural diagram of the car in this embodiment;

[0033] Figure 2 This is a schematic diagram showing the distribution of multiple predetermined areas of the central control display screen in this embodiment;

[0034] Figure 3 This is a logic diagram for the acquisition and control of brain potential signals in this embodiment. Detailed Implementation

[0035] like Figure 1 As shown, the vehicle in this embodiment includes a visual stimulation system, a signal acquisition system, and a control system.

[0036] 1. The visual stimulation system generates visual stimulation signals, providing a sufficiently bright trigger with a specific flashing frequency to induce visual potential signals in the cerebral cortex. It also provides feedback to guide the driver through multiple interactions. The visual stimulation system is displayed to the user via the vehicle's central control screen.

[0037] The central control display shows 12 visually stimulating square flashing blocks (i.e., 12 predetermined areas), with each flashing block having a different flashing frequency using a sine wave. Each flashing block on the central control display has multiple preset functions, and the same flashing block refers to different functions on different display interfaces. The functions of the flashing blocks can be switched by turning pages or entering the next level of a certain function.

[0038] like Figure 2 The central control display screen shows two areas (such as...) Figure 2 As shown), the main function area and the auxiliary function area are respectively. The main function area has 10 flashing blocks evenly distributed in a 2*5 pattern. Each flashing block is preset to a function, such as music, navigation, telephone, etc., and can only be driven by the SSVEP paradigm. The auxiliary function area has 2 flashing blocks, distributed on the left and right sides of the area. These two flashing blocks are preset to frequently used auxiliary functions such as previous page, next page, previous track, and next track. They can be triggered by the SSVEP paradigm or controlled by the motion imagination paradigm.

[0039] 2. The signal acquisition system consists of two parts: a signal acquisition module and a decoding module. The specific working logic is as follows: Figure 3 As shown.

[0040] (1) Signal acquisition module: The subject wears an EEG acquisition device (EEG cap) to record the EEG signals on the scalp through electrodes; the EEG cap uses 32 electrodes, which are attached to the user's occipital and parietal lobes according to international standards.

[0041] (2) Decoding module:

[0042] 1) Transmit the brain potential signals collected during the sampling period to the decoding module via Wi-Fi;

[0043] 2) The decoding module amplifies, filters, and removes artifacts from the raw brain potential signals collected during the sampling period.

[0044] 3) Input the brain potential signals into the SSVEP potential signal feature extraction channel and the MI potential signal feature extraction channel respectively to determine whether the driver generated brain potential signals based on visual stimulus signals during the sampling period; wherein, the SSVEP potential signal feature extraction channel uses the classical correlation analysis method (CCA) to extract SSVEP potential signal features, and the MI potential signal feature extraction channel uses the common spatial pattern algorithm (CSP) to extract MI potential signal features.

[0045] First, the SSVEP potential signal feature extraction channel can only extract SSVEP potential signal features. If the input brain potential signal is a motor imagery signal, the SSVEP potential signal feature extraction channel cannot extract features. Similarly, the MI potential signal feature extraction channel can only extract MI potential signal features. If the input brain potential signal is an SSVEP potential signal, the MI potential signal feature extraction channel cannot extract features.

[0046] Furthermore, the CCA method analyzes SSVEP signals by calculating the canonical correlation coefficient between two sets of signals: one set of signals consists of brain potential signals acquired by all electrodes, and the other set consists of reference signals corresponding to the flashing frequencies of each scintillation block. For a detailed explanation of the calculation principle, please refer to the section on "CCA Algorithm Applied to SSVEP EEG Signal Recognition." In other words, by calculating the canonical correlation coefficient between the brain potential signals and the reference signals corresponding to the flashing frequencies of each scintillation block, it is determined which scintillation block's visual stimulus induced the SSVEP conductive potential signal.

[0047] The principle of CSP algorithm for extracting MI potential signal features is described in the article "Feature Extraction and Recognition of Motor Imagery EEG Signals Based on CSP".

[0048] 4) If the driver generates brain potential signals based on visual stimuli, the Fisher linear recognition method is used to classify the extracted SSVEP potential signal features, and the user intention is obtained based on the classification results; if the driver does not generate brain potential signals based on visual stimuli, the LDA classifier is used to classify the extracted MI potential signal features, and the user intention is obtained based on the classification results.

[0049] Since the CCA method has already established the association between the SSVER signal and the flashing block, the Fisher linear recognition method can further compare the classification result with the function represented by the current interface of the associated flashing block to determine the specific function corresponding to the classification result, thus realizing the recognition of user intent. Similarly, it can realize the user intent corresponding to motion imagery signals.

[0050] 5) Output instructions.

[0051] 3. The vehicle control system includes in-vehicle controllable facilities such as the vehicle infotainment system, air conditioning, windows, seats, and ambient lighting.

[0052] The vehicle's infotainment system is responsible for receiving control commands from the signal acquisition system. These commands are then transmitted via cables to the actuators for execution. The control commands can also be fed back to the visual stimulation system via cables to provide interactive feedback.

[0053] In this embodiment, the user focuses on the Xth flashing block on the central control display screen. The EEG cap collects the user's brain potential signal and transmits it to the decoding module. The vehicle system receives and executes the command to activate function X and provides feedback to the user through the central control display screen. The user can then stop focusing on the main control screen and control the left-side auxiliary function of the function interface by having the thought of clenching their left fist. Alternatively, the user can continue to focus on the screen and control the left-side auxiliary function of the function interface by focusing on the flashing left-side auxiliary function block.

[0054] The above embodiments of the present invention can solve the problem that drivers cannot operate in-vehicle applications when their hands and feet are occupied during driving, and replace the traditional human-computer interaction mode.

Claims

1. A human-vehicle interaction control method based on SSVEP-MI fusion, characterized in that, include: Periodically acquire visual stimulation signals generated by the user due to one of the multiple predetermined areas on the central control display screen and / or brain potential signals generated by the user based on motor imagination; The brain potential signals acquired during the sampling period are decoded, and the user's intent is determined based on the decoding results; Based on the user's intent, execute corresponding controls on the vehicle; The central control display screen shows a main function area and an auxiliary function area. The functions indicated by the multiple predetermined areas in the main function area can only be triggered by the user based on visual stimulus signals. The functions indicated by the multiple predetermined areas in the auxiliary function area can be triggered by the user based on visual stimulus signals, or by the user based on motion imagination. The user generates a conductive bit signal to perform the function corresponding to the predetermined area on the left side of the auxiliary function area by generating the thought of clenching the left fist, and generates a conductive bit signal to perform the function corresponding to the predetermined area on the right side of the auxiliary function area by generating the thought of clenching the right fist.

2. The human-vehicle interaction control method based on SSVEP-MI fusion according to claim 1, characterized in that, When multiple predetermined areas on the central control display screen generate visual stimulus signals, the visual stimulus frequency corresponding to each predetermined area is different; The same predetermined area may represent different functions on different display interfaces, but the same predetermined area may correspond to the same frequency of visual stimuli on different display interfaces.

3. The human-vehicle interaction control method based on SSVEP-MI fusion according to claim 1, characterized in that, The steps of decoding the brain electrical activity signals acquired during the sampling period and determining the user's intent based on the decoding results include: The brain potential signals acquired within the sampling period are amplified, filtered, and artifact removed. The brain potential signals are input into the SSVEP potential signal feature extraction channel and the MI potential signal feature extraction channel, respectively, to determine whether the driver generates brain potential signals based on visual stimulus signals during the sampling period. If the driver generates brain potential signals based on visual stimuli, the Fisher linear recognition method is used to classify the extracted SSVEP potential signal features, and the user intent is obtained based on the classification results. If the driver does not generate brain potential signals based on visual stimuli, the LDA classifier is used to classify the extracted MI potential signal features, and the user intent is obtained based on the classification results.

4. A human-vehicle interaction control device based on SSVEP-MI fusion, characterized in that, include: The acquisition module is used to periodically induce visual stimulation signals generated by the user due to one of the multiple predetermined areas on the central control display screen and / or brain potential signals generated by the user based on motor imagination. The decoding module is used to decode the brain potential signals acquired within the sampling period and determine the user's intent based on the decoding results. The control module is used to perform corresponding controls on the vehicle based on user intent; The central control display screen shows a main function area and an auxiliary function area. The functions indicated by the multiple predetermined areas in the main function area can only be triggered by the user based on visual stimulus signals. The functions indicated by the multiple predetermined areas in the auxiliary function area can be triggered by the user based on visual stimulus signals, or by the user based on motion imagination. The user generates a conductive bit signal to perform the function corresponding to the predetermined area on the left side of the auxiliary function area by generating the thought of clenching the left fist, and generates a conductive bit signal to perform the function corresponding to the predetermined area on the right side of the auxiliary function area by generating the thought of clenching the right fist.

5. A car, characterized in that, include: The system includes a visual stimulation system, a signal acquisition system, and a control system. The visual stimulation system generates a visual stimulation signal in one of a plurality of predetermined areas on a central control display screen. The signal acquisition system periodically acquires brain electrical signals induced by the visual stimulation signal generated in one of the plurality of predetermined areas on the central control display screen and / or generated by the user based on motor imagery. The system decodes the brain electrical signals acquired by the signal acquisition system within the sampling period and determines the user's intention based on the decoding results. The control system performs corresponding control on the vehicle based on the user's intention. The central control display screen shows a main function area and an auxiliary function area. The functions indicated by the multiple predetermined areas in the main function area can only be triggered by the user based on visual stimulus signals. The functions indicated by the multiple predetermined areas in the auxiliary function area can be triggered by the user based on visual stimulus signals, or by the user based on motion imagination. The user generates a conductive bit signal to perform the function corresponding to the predetermined area on the left side of the auxiliary function area by generating the thought of clenching the left fist, and generates a conductive bit signal to perform the function corresponding to the predetermined area on the right side of the auxiliary function area by generating the thought of clenching the right fist.

Images