A method, device and equipment for human-computer interaction based on brain wave signals

Abstract

The application discloses a method, device and equipment for human-computer interaction based on brain wave signals, and the method comprises the following steps: acquiring original brain wave signals through an electroencephalogram acquisition device and performing pretreatment; based on a current opened human-computer interaction mode, performing feature extraction on the pretreated brain wave signals according to at least one state index required in each human-computer interaction mode; determining state values of the current state indexes according to brain wave feature information corresponding to each state index obtained after feature extraction; and when a control opening condition in any human-computer interaction mode is met, executing a control function corresponding to the human-computer interaction mode. The application monitors the real mental state of a user based on brain wave signals and converts the recognized real intention of the user into a corresponding control signal, so that the control of a vehicle based on the brain wave signals of the user in each human-computer interaction mode is realized.

Description

Technical Field

[0001] This application belongs to the field of vehicle control, and in particular relates to a method, device and equipment for human-computer interaction based on brainwave signals. Background Technology

[0002] The intelligentization of automobiles is bringing new impacts to the relationship between people, vehicles, and the environment, with human-machine interaction becoming a core element in the development and innovation of intelligent vehicles. A series of functions, including intelligent autonomous driving systems, in-vehicle infotainment systems, intelligent in-vehicle voice interaction systems, and road condition information systems, will establish a unique ecosystem using automobiles as a carrier, enabling the scenario-based application of intelligent vehicles.

[0003] Brain-computer interfaces (BCIs) establish a novel communication and control channel between the brain and the external environment, independent of peripheral nerves and muscles, enabling direct interaction between the brain and external devices. With advancements in neuroscience, biocompatible materials, sensors, big data, and artificial intelligence, BCI technology has entered a phase of rapid development, achieving significant progress in key areas such as signal acquisition and processing, decoding algorithms, and system implementation. This allows for communication between the human brain and the external environment to control devices.

[0004] In existing smart car applications, human-computer interaction is based on research into facial features, body movements, and voice. However, these external features are easily faked and cannot reflect the user's true emotions. For example, when detecting whether a user is in a state of fatigued driving, if the judgment is based on pupil changes and eye size, pupil changes may be caused by other reasons, and some users may have small eyes in their natural state. Therefore, in some cases, these two external features cannot reflect the true state of human fatigue like brainwave signals inside the human body. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, and device for human-computer interaction based on brainwave signals, which can monitor the user's true mental state based on brainwave signals and convert the identified user's true intentions into corresponding control signals, so as to realize the control of the vehicle based on the user's brainwave signals in various human-computer interaction modes.

[0006] In a first aspect, this application provides a method for human-computer interaction based on brainwave signals, applied to intelligent vehicles, the method comprising:

[0007] Raw brainwave signals are acquired and preprocessed using EEG acquisition equipment;

[0008] Based on the currently enabled human-computer interaction mode, feature extraction is performed on the preprocessed brainwave signal according to at least one state indicator required in each human-computer interaction mode.

[0009] Based on the brainwave feature information corresponding to each state index obtained after feature extraction, the current state value of each state index is determined.

[0010] Based on the status values ​​of each status indicator, when the control activation conditions for any human-computer interaction mode are met, the control function corresponding to that human-computer interaction mode is executed.

[0011] In one possible implementation, raw brainwave signals are acquired and preprocessed using an EEG acquisition device, including:

[0012] Raw brainwave signals are acquired using an EEG acquisition device, and then amplified and filtered to obtain noise-filtered brainwave signals.

[0013] Based on a preset time-frequency transformation algorithm, the noise-filtered brainwave signal is converted into a time-frequency signal to obtain the spectrum signal corresponding to different frequency bands.

[0014] In one possible implementation, the human-computer interaction mode includes at least one of fatigue detection mode, emotion recognition mode, immersive space mode, and remote window control mode;

[0015] The state indicators corresponding to the fatigue detection mode include fatigue level; the state indicators corresponding to the emotion recognition mode include emotional stability; the state indicators corresponding to the immersive space mode include relaxation level and concentration level; and the state indicators corresponding to the car window remote control mode include blinking frequency.

[0016] Based on at least one state indicator required for each human-computer interaction mode, feature extraction is performed on the preprocessed brainwave signal, including at least one of the following steps:

[0017] When the fatigue detection mode is currently enabled, based on the spectrum signals corresponding to different frequency bands obtained after preprocessing, the relaxation state features are extracted from the α waveform related to the relaxation state, the subconscious-related features are extracted from the θ waveform at the subconscious level, and the blinking waveform features are extracted from the spectrum signals of the frequency band related to blinking.

[0018] When it is determined that the emotion recognition mode is currently enabled, tension features are extracted from the β waveform related to the tension state based on the spectrum signals corresponding to different frequency bands obtained after preprocessing.

[0019] When the immersive space mode is currently enabled, the relaxation index and focus index are extracted from the α waveform related to the relaxation state based on the spectrum signals corresponding to different frequency bands obtained after preprocessing.

[0020] When the current window remote control mode is determined, blink waveform features are extracted from the frequency spectrum signals of the frequency bands related to blinking, based on the obtained spectrum signals corresponding to different frequency bands.

[0021] In one possible implementation, the conditions for activating the fatigue detection mode include the vehicle being in a high-speed driving state, or the vehicle driving continuously for a longer period of time than a preset duration, or receiving a control instruction to activate the fatigue detection mode.

[0022] Based on the state values ​​of each state indicator, when the control activation conditions for the fatigue detection mode are met, the control function corresponding to that human-machine interaction mode is executed, including:

[0023] When it is determined that the user is in a state of fatigued driving based on the current fatigue level value, the control activation conditions of the fatigue detection mode are met, and at least one of the following steps is executed:

[0024] The system controls the vehicle to provide voice reminders for fatigued driving, play relevant relaxing music, remind drivers to rest using a UI interface, and release stimulating scents.

[0025] The nearest parking spot is determined based on the vehicle's location, and the destination is changed based on the user's confirmation information.

[0026] The user's relevant information is uploaded to the backend, and the user receives reminders from the backend via voice or phone.

[0027] In one possible implementation, the activation condition of the emotion recognition mode includes receiving a control instruction to activate the emotion recognition mode.

[0028] Based on the state values ​​of each state indicator, when the control activation conditions for the emotion recognition mode are met, the control function corresponding to the emotion recognition mode is executed, including:

[0029] When it is determined that the user is in an emotionally unstable state based on the current emotional stability value, the control activation conditions under the emotion recognition mode are met, and at least one of the following steps is executed:

[0030] The vehicle can provide voice prompts to those experiencing emotional instability, play soothing music, and control the release of calming scents.

[0031] The system monitors the user's driving behavior and changes the vehicle's driving mode to a preset safe driving mode.

[0032] In one possible implementation, the conditions for activating the immersive space mode include receiving a control instruction to activate the immersive space mode when the vehicle speed is zero, and determining that the user is in a relaxed state based on the preprocessed brainwave signal.

[0033] Based on the status values ​​of each status indicator, when the control activation conditions for the immersive space mode are met, the corresponding control function for the immersive space mode is executed, including:

[0034] When the user is determined to be in a relaxed state based on the current relaxation index, determine if the control activation conditions for the immersive space mode are met and perform at least one of the following steps:

[0035] Control the vehicle to play audio and video corresponding to the preset relaxation state and release relaxing scents according to the preset relaxation state configuration;

[0036] The vehicle controls the user's seat and adjusts it according to the preset configuration for a relaxed state.

[0037] When the user is determined to be focused based on the current focus index, exit the immersive space mode and perform at least one of the following steps:

[0038] Before the vehicle returns to immersive space mode, it plays audio and releases invigorating scents.

[0039] Control the vehicle to restore the user's seat to the state it was in before entering the immersive space mode.

[0040] In one possible implementation, the conditions for activating the remote window mode include receiving a control instruction to activate the remote window mode and detecting that the user's blinking frequency reaches a set value.

[0041] Based on the status values ​​of each status indicator, when it is determined that the control opening conditions in the remote window control mode are met, the control function corresponding to the remote window control mode is executed, including:

[0042] Based on the current blink frequency value, when the number of blinks by the user within the set time interval reaches the set value, the control opening conditions in the remote window mode are met, and the following steps are executed:

[0043] The vehicle control system monitors the window the user is currently looking at and issues open / close commands to that window.

[0044] Secondly, this application provides a device for human-computer interaction based on brainwave signals, comprising:

[0045] The data acquisition and processing module is used to acquire raw brainwave signals through EEG acquisition equipment and perform preprocessing.

[0046] The feature extraction module is used to extract features from the preprocessed brainwave signal based on the currently enabled human-computer interaction mode and according to at least one state indicator required for each human-computer interaction mode.

[0047] The state value determination module is used to determine the current state value of each state indicator based on the brainwave feature information corresponding to each state indicator obtained after feature extraction.

[0048] The human-computer interaction control module is used to determine, based on the state values ​​of various state indicators, the control activation conditions for any human-computer interaction mode are met, and then execute the control function corresponding to that human-computer interaction mode.

[0049] Thirdly, embodiments of this application provide a device for human-computer interaction based on brainwave signals, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method for human-computer interaction based on brainwave signals as provided in any of the first aspects of this application.

[0050] Fourthly, embodiments of this application also provide a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by the processor of a terminal device, enables the terminal device to perform the human-computer interaction method based on brainwave signals as described in any of the first aspects of this application.

[0051] The technical solutions provided by the embodiments of this application have at least the following beneficial effects:

[0052] This application provides a method, apparatus, and device for human-computer interaction based on brainwave signals. It analyzes the user's true mental state based on brainwave signals to perform corresponding fatigue detection and emotion recognition, and provides reminders from multiple modalities such as vision, smell, and hearing. It can also convert the recognized user's true intentions into corresponding control signals to enable the user to remotely control various modules of the vehicle. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating a method for human-computer interaction based on brainwave signals provided in this application embodiment;

[0055] Figure 2 This is a schematic diagram of the brain-computer interface wearing process provided in an embodiment of this application;

[0056] Figure 3 This is a schematic diagram of the signal acquisition and processing module provided in an embodiment of this application;

[0057] Figure 4 This is a schematic diagram of brain-computer interface interaction with a vehicle provided in an embodiment of this application;

[0058] Figure 5 This is a schematic diagram of module interaction under the fatigue detection mode provided in the embodiments of this application;

[0059] Figure 6 This is a schematic diagram of module interaction in the emotion recognition mode provided in the embodiments of this application;

[0060] Figure 7 This is a schematic diagram of module interaction in the immersive space mode provided in the embodiments of this application;

[0061] Figure 8 This is a schematic diagram of module interaction in the remote window control mode provided in an embodiment of this application;

[0062] Figure 9 A schematic diagram of a device for human-computer interaction based on brainwave signals provided in an embodiment of this application;

[0063] Figure 10 This is a schematic diagram of a device for human-computer interaction based on brainwave signals, provided in an embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0065] Electroencephalography (EEG) is a method that uses electrophysiological indicators to record electrical signals generated in the human scalp. EEG measures the potential fluctuations caused by ion currents within neurons in the cerebral cortex or scalp surface, reflecting changes in brain waves during activity. Monitoring brainwave signals is fundamental to theoretical research in neuroscience and is widely used in clinical practice.

[0066] Studies have shown a strong correlation between human cognitive behavior and psychological activities and brainwave signals, which can directly reflect an individual's emotional activity. With advancements in neuroscience, biocompatible materials, sensors, big data, and artificial intelligence, brainwave information decoding is widely used in brain-computer interface (BCI) systems. A BCI system is an electroencephalogram (EEG) acquisition device that can acquire and process brainwave signals. By converting different frequency bands of brainwave signals output by the BCI system into corresponding physical control signals, direct interaction between the brain and external devices can be achieved, thus realizing human-computer interaction using brainwave signals.

[0067] In existing smart car applications, human-computer interaction is based on research into facial features, body movements, and voice. However, these external features are easily faked and cannot reflect the user's true emotions. For example, when detecting whether a user is in a state of fatigued driving, if the judgment is based on pupil changes and eye size, pupil changes may be caused by other reasons, and some users may have naturally small eyes. Therefore, in some cases, these two external features cannot reflect the true state of human fatigue like brainwave signals inside the human body.

[0068] In view of the above problems, this application provides a method for human-computer interaction based on brainwave signals, applicable to intelligent vehicles, such as... Figure 1 As shown, the method includes the following steps.

[0069] S101, acquires raw brainwave signals through an EEG acquisition device and performs preprocessing;

[0070] In this embodiment, when the brain-computer interface is enabled, it is necessary to detect whether the user is wearing the EEG acquisition device correctly, and remind the user when it is detected that the user is not wearing the EEG acquisition device correctly, until the user wears the EEG acquisition device correctly and then reminds the user that the brain-computer interface has entered normal working state.

[0071] As a feasible implementation method, before acquiring raw brainwave signals through an EEG acquisition device, such as... Figure 2 The following is a description of the brain-computer interface wearing process provided in an embodiment of this application:

[0072] S201, Receive the initial raw brainwave signal sent by the EEG acquisition device, and perform feature matching between the received initial raw brainwave signal and the corresponding raw brainwave signal under correct wearing conditions;

[0073] S202, Based on the matching result, determine whether the EEG acquisition device is worn correctly. If yes, proceed to S203; otherwise, proceed to S204.

[0074] S203, issues a message indicating correct wearing;

[0075] S204, issue a failure message indicating that the device was not worn and execute S201.

[0076] As a feasible implementation method, raw brainwave signals are acquired and preprocessed using an EEG acquisition device, including:

[0077] Raw brainwave signals are acquired using an EEG acquisition device, and then amplified and filtered to obtain noise-filtered brainwave signals.

[0078] Based on a preset time-frequency transformation algorithm, the noise-filtered brainwave signal is converted into a time-frequency signal to obtain the spectrum signal corresponding to different frequency bands.

[0079] In this embodiment, a non-invasive brain-computer interface is used to acquire raw brainwave signals and perform preprocessing. Raw brainwave signals can be acquired from the scalp using non-contact dry electrodes on a head-mounted EEG acquisition device. Since the acquired raw brainwave signals are relatively weak and contain artifacts, preprocessing of the acquired raw brainwave signals is necessary.

[0080] Preprocessing involves amplifying the raw brainwave signals and then filtering out artifacts. The artifacts to be removed primarily include electromyography (EMG), electrocardiography (ECG), power line interference, electromagnetic interference, and other task-irrelevant brainwave signals, such as... Figure 3 The diagram shown is a schematic of the signal acquisition and processing module provided in an embodiment of this application. This application processes the amplified brainwave signal through a high-pass filter circuit, a low-pass filter circuit, and a power frequency limiting circuit to filter out frequency bands prone to interference, retaining only the valid frequency bands of the brainwave signal. For artifacts that are difficult to remove through filtering, the valid frequency band brainwave signal is amplified again by the main amplification circuit, and then time-domain filtering, spatial-domain filtering, and other methods are used to identify interference signals and separate the remaining brainwave signals.

[0081] After the raw brainwave signals are amplified and artifact-filtered as described above, they still need to undergo frequency domain transformation preprocessing. By preprocessing the raw brainwave signals, we obtain amplified and artifact-filtered brainwave signals. Then, we transform the amplified and artifact-filtered brainwave signals from the time domain to the frequency domain, from which relevant frequency domain characteristics can be extracted as brainwave feature information.

[0082] In this embodiment, based on a preset time-frequency conversion algorithm, the amplified and artifact-filtered brainwave signal is converted to a frequency spectrum signal corresponding to different frequency bands. For example, the time-frequency conversion algorithm includes algorithms such as Discrete Fourier Transform (DFT) and Fast Fourier Transform (FFT).

[0083] In this embodiment of the application, the spectral signal converted from the amplified and artifact-filtered brainwave signal is divided into five frequency bands according to different frequency bands: delta wave (1-4Hz), theta wave (4-7Hz), alpha wave (8-13Hz), beta wave (13-30Hz), and gamma wave (30-47Hz).

[0084] The spectral signal converted from the amplified and artifact-filtered brainwave signal is divided into five frequency bands according to different frequency bands, and their corresponding brain feature information is as follows:

[0085] Delta waves (1-4Hz): Studies have shown that brainwave signals in this range typically appear in infancy or early childhood when intellectual development is immature, or in adults during periods of fatigue and deep sleep.

[0086] Theta waves (4-7Hz): This range of brainwave signals appears more frequently in adolescents (10 to 17 years old). This frequency band signal may also appear when adults are in a state of mental relaxation or light sleep, and it often appears in the frontal lobe. In adults with depressive mood or mental illness, this frequency wave is also very obvious, and it often appears in the temporoparietal lobe.

[0087] Alpha waves (8-13Hz): Signals in this range usually appear in the occipital lobe when a person is quiet, awake, and resting with their eyes closed. When the eyes are open or when other external stimuli are received, alpha waves will disappear.

[0088] Beta waves (13-30Hz): These are the main brainwave components involved in human logical analysis, primarily concentrated in the frontal lobe of the brain. The brain is more prone to generating signals in this frequency band when a person is awake, under mental stress, or emotionally excited.

[0089] Gamma waves (30-47Hz): These are the highest frequency components in brainwave signals. Gamma waves are easily generated when a person is focused on something or is alert.

[0090] Based on the above description, the brain-computer interface in this embodiment functions as follows: it acquires raw brainwave signals through an EEG acquisition device and converts the raw brainwave signals into spectral signals of the above five frequency bands after preprocessing.

[0091] S102, based on the currently activated human-computer interaction mode, extract features from the preprocessed brainwave signal according to at least one state indicator required under each human-computer interaction mode.

[0092] like Figure 4 The diagram shown is a schematic diagram of brain-computer interface and vehicle interaction provided in an embodiment of this application. In this embodiment, human-computer interaction is realized based on the interaction between the brainwave reading system and the vehicle control system. The brainwave reading system acquires raw brainwave signals through an EEG acquisition device, processes the brainwave signals through a filtering and amplification circuit and a pre-set frequency domain conversion algorithm, and finally obtains the spectrum signals of the aforementioned five frequency bands. Then, the spectrum signals of different frequency bands are sent to the vehicle control system.

[0093] After receiving spectrum signals from different frequency bands, the vehicle control system determines the current human-machine interaction mode and at least one required state indicator, and extracts features from the preprocessed brainwave signals. Based on the brain feature information extracted from each frequency band, the vehicle control system can analyze the intentions of the user wearing the EEG acquisition device in real time, and convert the logical control signals represented by the decoded user intentions into semantic control signals according to specific communication or control application requirements, and then convert the semantic control signals into corresponding physical control signals. Therefore, in the currently active human-machine interaction mode, the vehicle control system can control the relevant drive circuits to complete the corresponding control commands in the active human-machine interaction mode through the physical control signals corresponding to the decoded user intentions.

[0094] It should be noted that, in one or more embodiments, the communication method between the brain-computer interface and the vehicle control system in this application can be Bluetooth, or other wireless or wired communication methods, and this application does not limit this.

[0095] In this embodiment, the human-computer interaction mode includes at least one of a fatigue detection mode, an emotion recognition mode, an immersive space mode, and a remote window control mode. The fatigue detection mode is used to detect whether the user is fatigued. When fatigue is detected, the vehicle control system controls various vehicle modules to perform corresponding operations according to pre-configured settings. The state indicators corresponding to the fatigue detection mode include the degree of fatigue.

[0096] The emotion recognition mode is used to detect whether the user is in an emotionally stable state. When the system detects that the user is in an emotionally unstable state, such as excitement or anxiety, the vehicle control system controls the vehicle's various modules to perform corresponding operations according to the pre-configured settings. The state indicators corresponding to the emotion recognition mode include emotional stability.

[0097] The immersive space mode is used to detect whether the user is in a relaxed state. When the user is detected to be in a relaxed state, the vehicle control system controls the vehicle modules to perform corresponding operations according to the pre-configured settings. The state indicators corresponding to the immersive space mode include relaxation level and focus level.

[0098] The remote window control mode is used to detect whether the user's blinking frequency reaches a set value, and when the user's blinking frequency reaches the set value, it performs corresponding control on the window that the user needs to control. The status indicators corresponding to the remote window control mode include blinking frequency.

[0099] It should be noted that the human-computer interaction modes based on brainwave signals proposed in this application are not limited to the above four human-computer interaction modes. They may also include other human-computer interaction modes based on brainwave signals to realize multimodal interaction between user intentions and vehicles according to actual needs. This application does not limit these modes.

[0100] S103, Determine the current state value of each state indicator based on the brainwave feature information corresponding to each state indicator obtained after feature extraction;

[0101] Based on the above S102, according to the brainwave feature information extracted from the state indicators of each human-computer interaction mode, the current state value of each state indicator can be determined, and then it can be determined whether the current state value of each state indicator in the current human-computer interaction mode meets the corresponding control activation conditions.

[0102] S104. Based on the state values ​​of each state indicator, when it is determined that the control activation condition for any human-machine interaction mode is met, the control function corresponding to that human-machine interaction mode is executed.

[0103] The vehicle control system analyzes the brainwave feature information corresponding to the received brainwave signal based on the currently activated human-machine interaction mode, and determines the state value of each state indicator corresponding to the currently activated human-machine interaction mode. Based on the state value of each state indicator, when the control activation condition of any human-machine interaction mode is met, the control function corresponding to that human-machine interaction mode is executed.

[0104] The activation conditions and corresponding control functions of the aforementioned four human-computer interaction modes are explained below.

[0105] (1) Fatigue detection mode

[0106] like Figure 5 The diagram shown is a schematic diagram of module interaction in the fatigue detection mode provided in the embodiment of this application.

[0107] Control activation conditions: The vehicle is traveling at high speed, or the vehicle has been traveling continuously for more than a preset time, or a control instruction to activate the fatigue detection mode is received.

[0108] Feature extraction: When the fatigue detection mode is currently enabled, based on the spectrum signals corresponding to different frequency bands obtained after preprocessing, extract relaxation state features from the α waveform related to the relaxation state, extract subconscious-related features from the θ waveform at the subconscious level, and extract blinking waveform features from the spectrum signals of the frequency band related to blinking.

[0109] Among them, the blink waveform feature refers to the interference of the muscle current generated by blinking on the brainwave signal, so that the brainwave signal is converted into a continuous spectral signal that presents the corresponding blink waveform feature.

[0110] Control function: When it is determined that the user is in a state of fatigued driving based on the current fatigue level value, execute at least one of the following steps:

[0111] ① Control the vehicle to provide voice reminders for fatigued driving, play relevant relaxing music, use UI interface to remind drivers to rest, and release stimulating scents;

[0112] Among them, the voice reminder to control the vehicle's fatigue driving state can be "You have entered a fatigue state, please drive safely"; the UI interface screen reminder to take a break can be to control the vehicle's system and instrument panel to pop up a UI screen indicating that a break is needed, for example, a coffee cup pattern can be used.

[0113] ② Determine the nearest parking location based on the vehicle's location, and determine whether to change the destination based on the user's confirmation information;

[0114] The nearest parking location includes the nearest service area or rest area calculated by the car navigation software based on the vehicle's location. The car navigation software can automatically add service area or rest area stops along the way and require the driver to take a rest. The driver can choose whether to take a rest based on their personal situation.

[0115] ③ Upload the user's relevant information to the backend and receive reminders from the backend via voice or phone call.

[0116] In one or more embodiments, after the fatigue detection mode is enabled, if it is determined that the user is not in a state of fatigued driving based on the current fatigue level status value, the user is again determined to be in a state of fatigued driving based on the current fatigue level status value after a set time interval.

[0117] It should be noted that, in the embodiments of this application, the control methods executed when the user is detected to be in a state of fatigue are not limited to the above three types, and may also include other control methods formulated to reduce the negative impact of user fatigue driving. This application does not limit these methods.

[0118] (2) Emotion Recognition Model

[0119] like Figure 6 The diagram shown is a schematic diagram of module interaction in the emotion recognition mode provided in the embodiment of this application.

[0120] Control activation condition: Receive a control instruction to activate emotion recognition mode.

[0121] Feature extraction: When the emotion recognition mode is currently enabled, extract the tension state features from the β waveform related to the tension state based on the spectrum signals corresponding to different frequency bands obtained after preprocessing.

[0122] Control function: When it is determined that the user is in an emotionally unstable state based on the current emotional stability value, execute at least one of the following steps:

[0123] ① Control the vehicle to provide voice reminders for emotionally unstable states, play soothing music, and control the release of scents with a calming effect;

[0124] One example is the voice prompt that controls the vehicle to respond to emotionally unstable states, such as "You are a little agitated, please drive safely."

[0125] ② Monitor the user's driving behavior and change the vehicle's driving mode to a preset safe driving mode.

[0126] Among these features, the vehicle controls the monitoring of the user's driving behavior and changes the vehicle's driving mode to a preset safe driving mode, such as switching from Sport mode or other driving modes to ECO mode (the economy mode for automatic transmission cars), to prevent the user from making extreme driving behaviors due to emotional excitement.

[0127] In one or more embodiments, after the emotion recognition mode is enabled, when it is determined that the user is in an emotionally stable state based on the current emotional stability value, the user is again determined to be in an emotionally unstable state based on the current emotional stability value after a set time interval.

[0128] It should be noted that, in the embodiments of this application, the control method executed when the user is detected to be in an emotionally unstable state is not limited to the control methods given above, and may also include other control methods formulated to reduce the negative impact of the user being in an emotionally unstable state. This application does not limit this.

[0129] (3) Immersive Space Mode

[0130] like Figure 7 The diagram shown is a schematic diagram of module interaction in the immersive space mode provided in the embodiment of this application.

[0131] Control activation conditions: When the vehicle speed is zero, a control instruction to activate the immersive space mode is received, and the user is determined to be in a relaxed state based on the pre-processed brainwave signals.

[0132] Among them, a vehicle speed of 0 can indicate that the engine speed is 0 or the electric vehicle's power status is not in the ready position.

[0133] As a possible implementation, the control activation condition also includes using millimeter-wave radar to determine that the user wearing the brain-computer interface is inside the vehicle.

[0134] Feature extraction: When the immersive space mode is currently enabled, the relaxation index and focus index are extracted from the alpha waveform related to the relaxation state based on the spectrum signals corresponding to different frequency bands obtained after preprocessing.

[0135] Control function: When the user is determined to be in a relaxed state based on the current relaxation index, execute at least one of the following steps:

[0136] ① Control the vehicle to play audio and video corresponding to the preset relaxation state and release relaxing scents according to the preset relaxation state configuration;

[0137] ② Control the vehicle to locate the user's seat and adjust the user's seat according to the preset configuration for a relaxed state;

[0138] Among them, when controlling the vehicle to play the corresponding audio and video in the preset relaxation state configuration, and adjusting the user's seat, it can play the set white noise and switch the UI interface screen to the set relaxation screen, and put the occupied seat into the queen seat mode.

[0139] When the user is determined to be focused based on the current focus index, exit the immersive space mode and perform at least one of the following steps:

[0140] ① Before the vehicle resumes immersive space mode, it plays audio and releases invigorating scents;

[0141] ② Control the vehicle to restore the user's seat to the state it was in before entering the immersive space mode.

[0142] It should be noted that the relaxation index and focus index mentioned above are values ​​calculated based on the existing eSense algorithm for brainwave signals. The specific calculation method will not be detailed here.

[0143] It should be noted that, in the embodiments of this application, the control method executed when the user is detected to be in a relaxed or focused state is not limited to the control methods given above, and may also include other control methods designed to improve the user's relaxation or focus. This application does not limit this.

[0144] (4) Remote window control mode

[0145] like Figure 8 The diagram shown is a schematic diagram of module interaction in the remote window control mode provided in the embodiment of this application.

[0146] Control opening conditions: Receives a control instruction to open the remote window mode and detects that the user's blinking frequency reaches the set value.

[0147] As a possible implementation, the control activation conditions also include using millimeter-wave radar to determine that the user wearing the brain-computer interface is inside the vehicle, or that the electric vehicle's power status is in the Ready or ON position.

[0148] Feature extraction: When the current window remote control mode is determined, blink waveform features are extracted from the frequency spectrum signals of the frequency bands related to blinking, based on the obtained spectrum signals of different frequency bands.

[0149] Control function: Based on the current blink frequency, when the number of blinks by the user within a set time interval reaches a set value, the vehicle controls the monitoring window that the user is currently looking at and issues an open / close command to the window.

[0150] As one feasible implementation method, the user's eye is tracked using an in-vehicle camera to monitor the window the user is currently looking at.

[0151] It should be noted that, in the embodiments of this application, the control method executed when the user's blinking frequency reaches the set value is not limited to the control method given above, and may also include other control methods formulated according to actual needs. This application does not limit this.

[0152] The present application provides a method for human-computer interaction based on brainwave signals in intelligent vehicles. This method can analyze the user's true mental state based on brainwave signals to perform corresponding fatigue detection and emotion recognition, and provide reminders from multiple modalities such as vision, smell, and hearing. It can also convert the recognized user's true intentions into corresponding control signals to enable the user to remotely control various modules of the vehicle.

[0153] Based on the same inventive concept, embodiments of this application also provide a device for human-computer interaction based on brainwave signals, such as... Figure 9 As shown, the device includes:

[0154] The data acquisition and processing module 901 is used to acquire raw brainwave signals through the EEG acquisition device and perform preprocessing.

[0155] In one possible implementation, before acquiring the raw brainwave signals via the EEG acquisition device, the data acquisition and processing module is further configured to:

[0156] Receive the initial raw brainwave signal sent by the EEG acquisition device, and perform feature matching between the received initial raw brainwave signal and the corresponding raw brainwave signal under correct wearing conditions;

[0157] Based on the matching results, it is determined whether the EEG acquisition device is worn correctly, and a prompt message is issued when it is determined that it is not worn correctly, until it is determined that the EEG acquisition device is worn correctly.

[0158] In one possible implementation, the data acquisition and processing module acquires raw brainwave signals through an EEG acquisition device and performs preprocessing, including:

[0159] Raw brainwave signals are acquired using an EEG acquisition device, and then amplified and filtered to obtain noise-filtered brainwave signals.

[0160] Based on a preset time-frequency transformation algorithm, the noise-filtered brainwave signal is converted into a time-frequency signal to obtain the spectrum signal corresponding to different frequency bands.

[0161] The feature extraction module 902 is used to extract features from the preprocessed brainwave signal based on the currently enabled human-computer interaction mode and according to at least one state indicator required in each human-computer interaction mode.

[0162] In one possible implementation, the human-computer interaction mode includes at least one of fatigue detection mode, emotion recognition mode, immersive space mode, and remote window control mode;

[0163] The state indicators corresponding to the fatigue detection mode include fatigue level; the state indicators corresponding to the emotion recognition mode include emotional stability; the state indicators corresponding to the immersive space mode include relaxation level and concentration level; and the state indicators corresponding to the car window remote control mode include blinking frequency.

[0164] The feature extraction module extracts features from the preprocessed brainwave signal based on at least one state indicator required for each human-computer interaction mode, including at least one of the following steps:

[0165] When the fatigue detection mode is currently enabled, based on the spectrum signals corresponding to different frequency bands obtained after preprocessing, the relaxation state features are extracted from the α waveform related to the relaxation state, the subconscious-related features are extracted from the θ waveform at the subconscious level, and the blinking waveform features are extracted from the spectrum signals of the frequency band related to blinking.

[0166] When it is determined that the emotion recognition mode is currently enabled, tension features are extracted from the β waveform related to the tension state based on the spectrum signals corresponding to different frequency bands obtained after preprocessing.

[0167] When the immersive space mode is currently enabled, the relaxation index and focus index are extracted from the α waveform related to the relaxation state based on the spectrum signals corresponding to different frequency bands obtained after preprocessing.

[0168] When the current window remote control mode is determined, blink waveform features are extracted from the frequency spectrum signals of the frequency bands related to blinking, based on the obtained spectrum signals corresponding to different frequency bands.

[0169] The state value determination module 903 is used to determine the current state value of each state indicator based on the brainwave feature information corresponding to each state indicator obtained after feature extraction.

[0170] The human-computer interaction control module 904 is used to determine, based on the state values ​​of each state indicator, when the control activation conditions for any human-computer interaction mode are met, to execute the control function corresponding to that human-computer interaction mode.

[0171] In one possible implementation, the conditions for activating the fatigue detection mode include the vehicle being in a high-speed driving state, or the vehicle driving continuously for a longer period of time than a preset duration, or receiving a control instruction to activate the fatigue detection mode.

[0172] The human-machine interaction control module determines, based on the state values ​​of each state indicator, that the control activation conditions for the fatigue detection mode are met, and then executes the control function corresponding to that human-machine interaction mode, including:

[0173] When it is determined that the user is in a state of fatigued driving based on the current fatigue level value, the control activation conditions of the fatigue detection mode are met, and at least one of the following steps is executed:

[0174] The system controls the vehicle to provide voice reminders for fatigued driving, play relevant relaxing music, remind drivers to rest using a UI interface, and release stimulating scents.

[0175] The nearest parking spot is determined based on the vehicle's location, and the destination is changed based on the user's confirmation information.

[0176] The user's relevant information is uploaded to the backend, and the user receives reminders from the backend via voice or phone.

[0177] In one possible implementation, the activation condition of the emotion recognition mode includes receiving a control instruction to activate the emotion recognition mode.

[0178] The human-computer interaction control module determines, based on the state values ​​of each state indicator, that the control activation conditions under the emotion recognition mode are met, and then executes the control function corresponding to the emotion recognition mode, including:

[0179] When it is determined that the user is in an emotionally unstable state based on the current emotional stability value, the control activation conditions under the emotion recognition mode are met, and at least one of the following steps is executed:

[0180] The vehicle can provide voice prompts to those experiencing emotional instability, play soothing music, and control the release of calming scents.

[0181] The system monitors the user's driving behavior and changes the vehicle's driving mode to a preset safe driving mode.

[0182] In one possible implementation, the conditions for activating the immersive space mode include receiving a control instruction to activate the immersive space mode when the vehicle speed is zero, and determining that the user is in a relaxed state based on the preprocessed brainwave signal.

[0183] The human-computer interaction control module determines, based on the state values ​​of various state indicators, that when the control activation conditions for the immersive space mode are met, it executes the control function corresponding to the immersive space mode, including:

[0184] When the user is determined to be in a relaxed state based on the current relaxation index, determine if the control activation conditions for the immersive space mode are met and perform at least one of the following steps:

[0185] Control the vehicle to play audio and video corresponding to the preset relaxation state and release relaxing scents according to the preset relaxation state configuration;

[0186] The vehicle controls the user's seat and adjusts it according to the preset configuration for a relaxed state.

[0187] When the user is determined to be focused based on the current focus index, exit the immersive space mode and perform at least one of the following steps:

[0188] Before the vehicle returns to immersive space mode, it plays audio and releases invigorating scents.

[0189] Control the vehicle to restore the user's seat to the state it was in before entering the immersive space mode.

[0190] In one possible implementation, the conditions for activating the remote window mode include receiving a control instruction to activate the remote window mode and detecting that the user's blinking frequency reaches a set value.

[0191] The human-machine interaction control module determines, based on the state values ​​of various state indicators, that the control opening conditions under the remote window control mode are met, and then executes the control function corresponding to the remote window control mode, including:

[0192] Based on the current blink frequency value, when the number of blinks by the user within the set time interval reaches the set value, the control opening conditions in the remote window mode are met, and the following steps are executed:

[0193] The vehicle control system monitors the window the user is currently looking at and issues open / close commands to that window.

[0194] The specific implementation methods of each module are the same as those in the foregoing embodiments, and will not be described in detail here.

[0195] Based on the same inventive concept, this application also provides a device 1000 for human-computer interaction based on brainwave signals, such as... Figure 10 As shown, it includes at least one processor 1002; and a memory 1001 communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method for human-computer interaction based on brainwave signals.

[0196] The memory 1001 is used to store programs. Specifically, the program may include program code, which includes computer operation instructions. The memory 1001 may be volatile memory, such as random-access memory (RAM); it may also be non-volatile memory, such as flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it may be any one or a combination of the above-mentioned volatile and non-volatile memory types.

[0197] Processor 1002 can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. It can also be a hardware chip. The aforementioned hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0198] Based on the same inventive concept, this application provides a computer program medium storing a computer program, which is used to enable a computer to execute the human-computer interaction method based on brainwave signals described in the above embodiments.

[0199] The aforementioned storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0200] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0201] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0202] The technical solutions provided in this application have been described in detail above. Specific examples have been used in this application to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0203] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0204] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood by those skilled in the art from the computer program instructions that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of one or more computer-usable storage media containing computer-usable program code (implementing each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams). These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0205] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0206] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0207] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for human-computer interaction based on brainwave signals, applied to intelligent vehicles, characterized in that, The method includes: Raw brainwave signals are acquired and preprocessed using EEG acquisition equipment; Based on the currently activated human-computer interaction mode, feature extraction is performed on the preprocessed brainwave signal according to at least one state indicator required under each human-computer interaction mode; the human-computer interaction mode includes at least one of fatigue detection mode, emotion recognition mode, immersive space mode and car window remote control mode. Based on the brainwave feature information corresponding to each state indicator obtained after feature extraction, determine the state value of each state indicator corresponding to the currently activated human-computer interaction mode. Based on the state values ​​of each state indicator, when the control activation conditions for any human-computer interaction mode are met, the control function corresponding to that human-computer interaction mode is executed; the control activation conditions for the fatigue detection mode include the vehicle being in a high-speed driving state, or the vehicle continuously driving for a longer period of time than a preset duration, or receiving a control instruction to activate the fatigue detection mode; and / or the control activation conditions for the emotion recognition mode include receiving a control instruction to activate the emotion recognition mode; and / or the control activation conditions for the immersive space mode include receiving a control instruction to activate the immersive space mode when the vehicle speed is zero, and determining that the user is in a relaxed state based on preprocessed brainwave signals; and / or the control activation conditions for the window remote control mode include receiving a control instruction to activate the window remote control mode and detecting that the user's blinking frequency reaches a set value.

2. The method according to claim 1, characterized in that, Raw brainwave signals are acquired using EEG acquisition equipment and preprocessed, including: Raw brainwave signals are acquired using an EEG acquisition device, and then amplified and filtered to obtain noise-filtered brainwave signals. Based on a preset time-frequency transformation algorithm, the noise-filtered brainwave signal is converted into a time-frequency signal to obtain the spectrum signal corresponding to different frequency bands.

3. The method according to any one of claims 1 to 2, characterized in that, The state indicators corresponding to the fatigue detection mode include fatigue level; the state indicators corresponding to the emotion recognition mode include emotional stability; the state indicators corresponding to the immersive space mode include relaxation level and focus level; and the state indicators corresponding to the car window remote control mode include blinking frequency. Based on at least one state indicator required for each human-computer interaction mode, feature extraction is performed on the preprocessed brainwave signal, including at least one of the following steps: When the fatigue detection mode is currently enabled, based on the spectrum signals corresponding to different frequency bands obtained after preprocessing, the relaxation state features are extracted from the α waveform related to the relaxation state, the subconscious-related features are extracted from the θ waveform at the subconscious level, and the blinking waveform features are extracted from the spectrum signals of the frequency band related to blinking. When it is determined that the emotion recognition mode is currently enabled, tension features are extracted from the β waveform related to the tension state based on the spectrum signals corresponding to different frequency bands obtained after preprocessing. When the immersive space mode is currently enabled, the relaxation index and focus index are extracted from the α waveform related to the relaxation state based on the spectrum signals corresponding to different frequency bands obtained after preprocessing. When the current window remote control mode is determined, blink waveform features are extracted from the frequency spectrum signals of the frequency bands related to blinking, based on the obtained spectrum signals corresponding to different frequency bands.

4. The method according to claim 3, characterized in that, Based on the state values ​​of each state indicator, when the control activation conditions for the fatigue detection mode are met, the control function corresponding to that human-machine interaction mode is executed, including: When it is determined that the user is in a state of fatigued driving based on the current fatigue level value, the control activation conditions of the fatigue detection mode are met, and at least one of the following steps is executed: The system controls the vehicle to provide voice reminders for fatigued driving, play relevant relaxing music, remind drivers to rest using a UI interface, and release stimulating scents. The nearest parking spot is determined based on the vehicle's location, and the destination is changed based on the user's confirmation information. The user's relevant information is uploaded to the backend, and the user receives reminders from the backend via voice or phone.

5. The method according to claim 3, characterized in that, Based on the state values ​​of each state indicator, when the control activation conditions for the emotion recognition mode are met, the control function corresponding to the emotion recognition mode is executed, including: When it is determined that the user is in an emotionally unstable state based on the current emotional stability value, the control activation conditions under the emotion recognition mode are met, and at least one of the following steps is executed: The vehicle can provide voice prompts to those experiencing emotional instability, play soothing music, and control the release of calming scents. The system monitors the user's driving behavior and changes the vehicle's driving mode to a preset safe driving mode.

6. The method according to claim 3, characterized in that, Based on the status values ​​of each status indicator, when the control activation conditions for the immersive space mode are met, the corresponding control function for the immersive space mode is executed, including: When the user is determined to be in a relaxed state based on the current relaxation index, determine if the control activation conditions for the immersive space mode are met and perform at least one of the following steps: Control the vehicle to play audio and video corresponding to the preset relaxation state and release relaxing scents according to the preset relaxation state configuration; The vehicle controls the user's seat and adjusts it according to the preset configuration for a relaxed state. When the user is determined to be focused based on the current focus index, exit the immersive space mode and perform at least one of the following steps: Before the vehicle returns to immersive space mode, it plays audio and releases invigorating scents. Control the vehicle to restore the user's seat to the state it was in before entering the immersive space mode.

7. The method according to claim 4, characterized in that, Based on the status values ​​of each status indicator, when it is determined that the control opening conditions in the remote window control mode are met, the control function corresponding to the remote window control mode is executed, including: Based on the current blink frequency value, when the number of blinks by the user within the set time interval reaches the set value, the control opening conditions in the remote window mode are met, and the following steps are executed: The vehicle control system monitors the window the user is currently looking at and issues open / close commands to that window.

8. A device for human-computer interaction based on brainwave signals, characterized in that, include: The data acquisition and processing module is used to acquire raw brainwave signals through EEG acquisition equipment and perform preprocessing. The feature extraction module is used to extract features from the preprocessed brainwave signal based on the currently activated human-computer interaction mode and according to at least one state indicator required under each human-computer interaction mode; the human-computer interaction mode includes at least one of fatigue detection mode, emotion recognition mode, immersive space mode and car window remote control mode. The state value determination module is used to determine the state value of each state indicator corresponding to the currently activated human-computer interaction mode based on the brainwave feature information corresponding to each state indicator obtained after feature extraction. The human-computer interaction control module is used to determine, based on the state values ​​of various state indicators, the control activation conditions for any human-computer interaction mode are met, and then execute the control function corresponding to that human-computer interaction mode. The control activation conditions for the fatigue detection mode include the vehicle being in a high-speed driving state, or the vehicle continuously driving for a longer period of time than a preset duration, or receiving a control instruction to activate the fatigue detection mode; and / or the control activation conditions for the emotion recognition mode include receiving a control instruction to activate the emotion recognition mode; and / or the control activation conditions for the immersive space mode include receiving a control instruction to activate the immersive space mode when the vehicle speed is zero, and determining that the user is in a relaxed state based on preprocessed brainwave signals; and / or the control activation conditions for the window remote control mode include receiving a control instruction to activate the window remote control mode and detecting that the user's blinking frequency reaches a set value.

9. A device for human-computer interaction based on brainwave signals, characterized in that, The method includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-7.

Images