Real-time SSVEP-based head-up display using deep learning to control closed-loop features in a vehicle

Abstract

The present application relates to closed-loop real-time SSVEP-based head-up display using deep learning to control features in a vehicle. A vehicle system includes a controller programmed to display a plurality of icons on a HUD of the vehicle, receive EEG data from a driver of the vehicle, perform a fast Fourier transform of the EEG data to obtain an EEG spectrum, input the EEG spectrum into a trained machine learning model, determine which icon of the plurality of icons the driver is viewing based on an output of the trained machine learning model, and perform one or more vehicle operations based on the output of the trained machine learning model.

Description

Technical Field

[0001] This specification relates to vehicle systems, and more particularly to a closed-loop, real-time SSVEP-based head-up display that uses deep learning to control features in a vehicle. Background Technology

[0002] Many car accidents are caused by distracted driving. In many cases, drivers become distracted by engaging with or operating features within the vehicle (e.g., using vehicle-integrated components or controls). Therefore, technologies that facilitate driver use of vehicle features without causing driver distraction may be desirable. Summary of the Invention

[0003] In one embodiment, the vehicle system may include a controller. The controller may be programmed to display multiple icons on the vehicle's HUD, receive EEG data from the vehicle's driver, perform an FFT on the EEG data to obtain an EEG spectrum, input the EEG spectrum into a trained machine learning model, determine which of the multiple icons the driver is viewing based on the output of the trained machine learning model, and perform one or more vehicle operations in real time based on the output of the trained machine learning model.

[0004] In another embodiment, the method may include: displaying multiple icons on a vehicle's HUD, receiving EEG data from the vehicle's driver, performing an FFT on the EEG data to obtain an EEG spectrum, inputting the EEG spectrum into a trained machine learning model, determining which of the multiple icons the driver is viewing based on the output of the trained machine learning model, and performing one or more vehicle operations based on the input of the trained machine learning model.

[0005] In another embodiment, the method may include: receiving training data, which includes EEG data collected from a plurality of individual subjects while each subject is viewing a specific icon; performing an FFT on the training data to obtain EEG spectral data; and training a machine learning model to predict which icon a subject is viewing based on the EEG spectral data. Attached Figure Description

[0006] The embodiments illustrated in the accompanying drawings are illustrative and exemplary in nature and are not intended to limit this disclosure. The detailed description of the following illustrative embodiments can be understood when read in conjunction with the following drawings, wherein the same structures are indicated by the same reference numerals, and wherein:

[0007] Figure 1 An example vehicle interior is depicted according to one or more embodiments shown and described herein;

[0008] Figure 2A schematic diagram of a vehicle system according to one or more embodiments shown and described herein is depicted;

[0009] Figure 3 One or more embodiments according to the examples shown and described herein are depicted. Figure 2 A schematic diagram of one or more memory modules of a vehicle system;

[0010] Figure 4 The invention describes one or more embodiments shown and described herein. Figure 2 A schematic diagram of an example machine learning model for vehicle system maintenance;

[0011] Figure 5 One or more embodiments according to the examples shown and described herein are depicted. Figure 4 A schematic diagram of the SE block of the machine learning model;

[0012] Figure 6 One or more embodiments according to the examples shown and described herein are depicted. Figure 4 A graph showing the relationship between the accuracy of a machine learning model and the input size;

[0013] Figure 7 One or more embodiments according to the examples shown and described herein are depicted. Figure 4 The graph shows the relationship between the accuracy of the machine learning model and three other models and the signal length.

[0014] Figure 8 Operation according to one or more embodiments shown and described herein is described. Figure 2 The vehicle system is used for training Figure 4 The flowchart of the model's method; and

[0015] Figure 9 The provision according to one or more embodiments shown and described herein is described. Figure 2 The vehicle system to utilize Figure 4 The flowchart shows the method for training a model. Detailed Implementation

[0016] The embodiments disclosed herein provide systems and methods for deep learning-based SSVEP detection in vehicles. Steady-state visual evoked potentials (SSVEPs) are signals that represent a natural human response to visual stimuli of specific frequencies. When an individual's retina is subjected to visual stimuli in the range of approximately 3.5 Hz to 75 Hz, the individual's brain generates electrical activity at the same frequency or multiples of the frequency of the visual stimulus. This can thus be used as a method for monitoring brain activity.

[0017] In embodiments of this disclosure, a series of icons are presented to the driver of the vehicle via a head-up display (HUD). The driver can focus their gaze on a specific icon, and then their brainwaves are collected via an electroencephalogram (EEG) using wearable headphones. Machine learning algorithms, as disclosed herein, are used to decode the EEG signals and predict which icon the driver is looking at based on the SSVEP signals generated in the driver's brain. Thus, the driver can easily operate various vehicle functions (e.g., climate control, audio, navigation, vehicle settings, etc.) by looking at icons on the HUD without taking their eyes off the road.

[0018] Now turn to the diagrams. Figure 1 The interior of vehicle 100 is depicted. Driver 102 sits in the driver's seat of vehicle 100 and drives vehicle 100. As disclosed herein, a head-mounted device 104 for reading the brainwaves of driver 102 may be worn by driver 102. A HUD 108 is displayed on the windshield 106 or other surface of vehicle 100. In an embodiment, HUD 108 is a transparent display that can display text, images, or other information that driver 102 can see without taking their eyes off the windshield 106 or other surface, thereby avoiding taking their eyes off the road.

[0019] exist Figure 1 In the example, HUD 108 displays icons 110, 112, 114, and 116. Although in Figure 1 In the example shown, four icons are displayed on HUD 108, but it should be understood that in other examples, HUD 108 can display any number of icons in any orientation. Icons 110, 112, 114, and 116 each have different shapes and / or different colors. Thus, when driver 102 views one of icons 110, 112, 114, or 116, a specific SSVEP response will be triggered in the driver's brain, depending on which icon driver 102 is viewing. In the example shown, green or black-and-white icons exhibit the best performance. However, any color can be used for icons. As disclosed herein, the generated specific SSVEP response can be detected by head-mounted device 104, thereby indicating which icon driver 102 is viewing.

[0020] By detecting which icon the driver 102 is viewing, displayed on the HUD 108, the driver 102 can control certain functions of the vehicle 100 without taking their eyes off the road or manually pressing any buttons or switches. Specifically, when the driver 102 views a particular icon, the vehicle system of the vehicle 100 can recognize which icon is being viewed and can cause the HUD 108 to display additional icons (e.g., submenus) or to operate specific vehicle functions. For example, icon 110 may lead to a submenu related to audio options, icon 112 to a submenu for navigation options, icon 114 to a settings submenu, and icon 116 to a temperature options submenu. The submenu can display additional icons on the HUD 108 that the driver 102 can activate by viewing them. For example, when the driver 102 views icon 116, the submenu can display an additional icon that allows the driver 102 to set the vehicle temperature for the heating or cooling system of the vehicle 100.

[0021] As described above, the head-mounted device 104 can detect brainwaves generated by the driver 102. In an embodiment, the head-mounted device 104 may include multiple electrodes for detecting electrical signals generated by the driver 102's brain. In an embodiment, the head-mounted device 104 uses electroencephalography (EEG) to measure electrical activity in the driver's brain. However, in other examples, the head-mounted device 104 may detect the driver 102's brainwaves in other ways. The brainwaves detected by the head-mounted device 104 can be transmitted to the vehicle system of the vehicle 100, as discussed in further detail below. In one example, the head-mounted device 104 is *G. Nautilus*. TM Headphone device. However, in other examples, head-mounted device 104 may include other devices that perform the functions disclosed herein.

[0022] Figure 2 Describing what can be included in Figure 1 Vehicle system 200 in vehicle 100. Figure 2 In the example, vehicle system 200 includes one or more processors 202, communication path 204, one or more memory modules 206, network interface hardware 208, and data storage component 210, the details of which will be described in the following paragraphs.

[0023] Each of the one or more processors 202 can be any device capable of executing machine-readable and executable instructions. Therefore, each of the one or more processors 202 can be a controller, integrated circuit, microchip, computer, or any other computing device. In some examples, the one or more processors 202 may include a graphics processing unit (GPU).

[0024] One or more processors 202 are coupled to a communication path 204, which provides signal interconnectivity between the various modules of the vehicle system 200. Thus, the communication path 204 can communicatively couple any number of processors 202 to each other, and allows modules coupled to the communication path 204 to operate in a distributed computing environment. Specifically, each module can operate as a node capable of sending and / or receiving data. As used herein, the term "communically coupled" means that the coupled components are able to exchange data signals with each other, such as, for example, electrical signals via a conductive medium, electromagnetic signals via air, optical signals via an optical waveguide, etc.

[0025] Therefore, the communication path 204 can be formed of any medium capable of transmitting signals, such as, for example, wires, conductive traces, optical waveguides, etc. In some embodiments, the communication path 204 can facilitate the transmission of wireless signals, such as Wi-Fi, Near Field Communication (NFC), etc. Furthermore, the communication path 204 can be formed by a combination of media capable of transmitting signals. In one embodiment, the communication path 204 includes a combination of conductive traces, wires, connectors, and buses that cooperate to allow the transmission of electrical data signals to components such as processors, memory, sensors, input devices, output devices, and communication devices. Thus, the communication path 204 can include vehicle buses, such as, for example, LIN buses, CAN buses, VAN buses, etc. Additionally, it should be noted that the term "signal" refers to a waveform (e.g., electrical, optical, magnetic, mechanical, or electromagnetic) capable of propagating through a medium, such as DC, AC, sine waves, triangle waves, square waves, vibrations, etc.

[0026] Vehicle system 200 includes one or more memory modules 206 coupled to communication path 204. The one or more memory modules 206 may include RAM, ROM, flash memory, hard disk drive, or any device capable of storing machine-readable and executable instructions that are accessible by one or more processors 202. Machine-readable and executable instructions may include logic or one or more algorithms written in any programming language of any generation (e.g., 1GL, 2GL, 3GL, 4GL, or 5GL), such as machine language that can be directly executed by a processor, or assembly language, object-oriented programming (OOP), scripting languages, microcode, etc., that can be compiled or assembled into machine-readable and executable instructions and stored on one or more memory modules 206. Alternatively, machine-readable and executable instructions may be written in a hardware description language (HDL), such as logic implemented via a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC) or equivalent. Thus, the methods described herein can be implemented in any conventional computer programming language, as pre-programmed hardware elements, or as a combination of hardware and software components.

[0027] Still referencing Figure 2 The vehicle system 200 includes network interface hardware 208 for communicatively coupling the vehicle system 200 to the head-mounted device 104. In some examples, the network interface hardware 208 may also couple the vehicle system 200 to one or more other external devices, including remote computing devices (e.g., cloud servers or edge servers) or other vehicles. The network interface hardware 208 may be communicatively coupled to communication path 204 and may be any device capable of transmitting and / or receiving data via a network or via a hardwired connection (e.g., cable connection) to the head-mounted device 104. In some examples, the network interface hardware 208 may include a communication transceiver for transmitting and / or receiving any wired or wireless communication. For example, the network interface hardware 208 may include an antenna, modem, LAN port, Wi-Fi card, WiMax card, mobile communication hardware, near-field communication hardware, satellite communication hardware, and / or any wired or wireless hardware for communicating with the head-mounted device 104 and / or other networks and / or devices. In one embodiment, the network interface hardware 208 includes components configured to... Hardware for operating wireless communication protocols. In the example shown, the network interface hardware 208 of the vehicle system 200 can receive brainwave data collected by the head-mounted device 104, as disclosed in further detail below.

[0028] Still referencing Figure 2 The vehicle system 200 includes a data storage component 210. The data storage component 210 can store data used by various components of the vehicle system 200. Specifically, the data storage component 210 can store brainwave data received from the head-mounted device 104. The data storage component 210 can also store parameters of a model maintained by the vehicle system 200 for analyzing brainwave data from the head-mounted device 104, as disclosed herein.

[0029] Now for reference Figure 3The vehicle system 200 includes one or more memory modules 206 comprising a training data receiving module 300, an EEG data receiving module 302, a data filtering module 304, a data segmentation module 306, a Fast Fourier Transform (FFT) module 308, a model training module 310, an icon detection module 312, a vehicle system operation module 314, and an icon color adjustment module 316. Each of the training data receiving module 300, EEG data receiving module 302, data filtering module 304, data segmentation module 306, FFT module 308, model training module 310, icon detection module 312, vehicle system operation module 314, and icon color adjustment module 316 can be a program module in the form of an operating system, application module, or other program module stored in one or more memory modules 206. Such program modules may include, but are not limited to, routines, subroutines, programs, objects, components, data structures, etc., for performing specific tasks or performing specific data types, as described below.

[0030] In some examples, program modules may be stored in a remote storage device that can communicate with vehicle system 200. In some examples, the functionality of one or more of the memory modules 206 may be performed by a remote computing device (e.g., an edge server or cloud computing device) communicatively coupled to vehicle system 200. For example, vehicle system 200 may transmit data received from head-mounted device 104 to a cloud server for processing.

[0031] The training data receiving module 300 may include programming instructions for receiving training data that can be used to train a model maintained by the vehicle system 200, as disclosed herein. After training the model, the model can be manipulated in real time to determine which icons the driver 102 is viewing on the HUD 108, as disclosed herein. Furthermore, after training the model, it can be periodically updated and retrained using additional training data. In some examples, the training data receiving module 300 may include programming instructions for receiving training data from the head-mounted device 104. In other examples, the training data receiving module 300 may include programming instructions for receiving training data from another source (e.g., a computing device that collects training data).

[0032] In this embodiment, the training data received by the training data receiving module 300 includes EEG data collected from multiple individual subjects during a training session while each of the multiple individual subjects is viewing a specific icon while wearing the head-mounted device 104. In some examples, the training session is conducted while the subject is inside vehicle 100. In other examples, the training session can be conducted while the subject is in a simulated environment inside simulated vehicle 100.

[0033] When collecting data during training sessions, a conductive paste can be applied between the electrodes of the head-mounted device 104 and the subject's skin to reduce the impedance to below 5 kΩ, thereby reducing vulnerability to electrical artifacts and / or movement. Although signals from the parietal lobe have lower SSVEP potentials, channels from the visual cortex in the parietal and occipital regions are ideal for recording SSVEP.

[0034] In this embodiment, based on the International 10-20 system for describing scalp electrode locations, sixteen electrodes of the head-mounted device 104 are placed at Fz, Cz, CPz, P1, Pz, P2, PO3, POz, PO4, PO7, PO8, O1, Oz, O2, TP9, and TP10. A reference electrode is connected to the mastoid process, and a ground electrode is connected to AFz. However, in other examples, the electrodes of the head-mounted device 104 may be placed at other locations. In the illustrated example, the head-mounted device 104 collects data at a resolution of 500 Hz. However, in other examples, the head-mounted device 104 may collect data at any other resolution.

[0035] In this embodiment, the training data collected from each subject includes a calibration dataset. That is, a specific set of icons is presented to the subject via HUD 108 at predetermined intervals during a training session, and the individual is instructed to view a specific icon from that set in a predetermined pattern during the training session. Thus, it is known what icon the individual is viewing at each point in time during the training session. Consequently, EEG data can be collected from the head-mounted device 104 during the training session, and the icons viewed by the subject during the training session can be used as ground truth to train the model.

[0036] In the example shown, each set of icons is displayed on HUD 108 for 3 seconds during the training session, and the subject is instructed to look at a specific icon during these 3 seconds. Multiple different sets of icons can be displayed to the subject in a predetermined or random pattern. Each icon is designed to trigger a different SSVEP response in the subject. Thus, in the example shown, training data is collected at 3-second intervals while a specific set of icons is displayed to the subject. However, in other examples, the set of icons can be displayed for longer or shorter periods than 3 seconds, and data can be collected at the corresponding time intervals. After the training data receiving module 300 receives the training data, the training data can be used to train the model maintained by the vehicle system 200, as discussed in further detail below.

[0037] Still referencing Figure 3The brainwave data receiving module 302 may include programming instructions for receiving brainwave data from the head-mounted device 104 during real-time operation, as disclosed herein. Specifically, once the model is trained, brainwave data from the head-mounted device 104 can be received when the driver 102 is viewing a specific icon on the HUD 108. The brainwave data received by the brainwave data receiving module 302 can be fed into the trained model to determine which icon the driver 102 is viewing, as disclosed herein. A specific vehicle function of the vehicle 100 can then be operated, as disclosed herein, depending on which icon the driver 102 is viewing.

[0038] Still referencing Figure 3 The data filtering module 304, data segmentation module 306, and FFT module 308 may include programming instructions for performing data preprocessing before the data is input into the model, as disclosed herein. Specifically, the data filtering module 304 may include programming instructions for filtering brainwave data received by the training data receiving module 300 or the brainwave data receiving module 302. The data filtering module 304 may include programming instructions for applying filters to the received brainwave data to remove ambient noise and preserve data relevant to the SSVEP frequency. In the illustrated example, the data filtering module 304 applies a 4th-order Butterworth bandpass filter between 3 and 40 Hz. However, in other examples, other types of filters may be used.

[0039] Still referencing Figure 3 The data segmentation module 306 may include programming instructions for segmenting the brainwave data received by the training data receiving module 300 or the brainwave data receiving module 302. In the illustrated example, the data segmentation module 306 includes programming instructions for segmenting the received brainwave data into 0.25-second segments. However, in other examples, other data segment lengths may be used.

[0040] Still referencing Figure 3 The FFT module 308 can perform a Fast Fourier Transform (FFT) on the data filtered by the data filtering module 304 and segmented by the data segmentation module 306. Specifically, the FFT module 308 includes programming instructions for obtaining the complex spectrum of each data segment generated by the data segmentation module 306. The complex spectrum obtained by the FFT module 308 can be represented as:

[0041] X comp =Concatenate(Re{FFT(input)},Im{FFT(input)});

[0042] Where Re{FFT(input)} is the real part of the time-segment input, and Im{FFT(input)} is the imaginary part of the time-segment input. That is, as disclosed in this paper, amplitude and phase information are combined and provided as input to the model.

[0043] In the example shown, FFT module 308 uses a start frequency of 3 Hz and an end frequency of 35 Hz. However, in other examples, other start and end frequencies can be used. The length of the FFT vector can be obtained using the following equation:

[0044]

[0045] Still referencing Figure 3 The model training module 310 may include programming instructions for training a model maintained by the vehicle system 200, as disclosed herein. Specifically, the model training module 310 may include programming instructions for receiving training data preprocessed by the data filtering module 304, the data segmentation module 306, and the FFT module 308, as described above, and training the model to predict which icon on the HUD 108 the driver 102 is viewing based on EEG data.

[0046] Figure 4 The example architecture of model 400, maintained by vehicle system 200, is shown. Figure 4 In the example, model 400 includes a convolutional neural network (CNN). Model 400 operates as an end-to-end system, receiving multi-channel EEG signals as input and classifying the signals into one of four different categories, indicating one of the four icons that driver 102 is viewing. Model 400 includes a residual neural network (ResNet) architecture and squeeze and excitation (SE) blocks. The ResNet architecture demonstrates remarkable ability to extract features from the input data. Figure 4 In Model 400, the ResNet architecture has been combined with SE blocks to further enhance the model's feature extraction capabilities.

[0047] exist Figure 4 In the example, the SE block is responsible for improving channel interdependencies and highlighting key features. The SE block allows the model to treat each feature map differently based on its importance by assigning weights to each feature map. The SE block can significantly improve model performance in various applications.

[0048] exist Figure 4In the example, model 400 includes a convolutional block 402, a first SE-Res block 404, a second SE-Res block 406, and a classifier block 408. Convolutional block 402 includes a 2D convolutional layer 410, a batch normalization layer 412, and a Corrected Linear Unit (ReLU) activation layer 414. The first SE-Res block 404 includes a 2D convolutional layer 416, a batch normalization layer 418, a ReLU activation layer 420, and an SE block 422. The second SE-Res block 406 includes a 2D convolutional layer 424, a batch normalization layer 426, a ReLU activation layer 428, and an SE block 430. Classifier block 408 includes a dropout layer 432, a fully connected layer with four units, and a softmax classifier 436. Figure 4 As shown, the input of the first SE-res block 404 is summed to the output of the first SE-res block 404. Similarly, the input of the second SE-res block 406 is summed to the output of the second SE-res block 406.

[0049] Figure 5 The architecture of SE block 422 of the first SE-Res block 404 is shown. SE block 430 of the second SE-Res block 406 is similarly constructed. SE block 422 includes an input layer 500, a global max-pooling layer 502, a first fully connected layer 504, a ReLU activation layer 506, a second fully connected layer 508, a Sigmoid activation layer 510, and a reshaping layer 512. The first fully connected layer 504 has half the number of units as the input, and the second fully connected layer 508 has the same number of units as the input. Fully connected layers 504 and 508 are responsible for scoring each feature map based on its importance in the final label. The scores are then applied to the input features via multiplication.

[0050] Return to reference Figure 4 Dropout layer 432 helps prevent overfitting to the specific training data used to train model 400. In the example shown, a dropout value of 0.25 is used. However, other dropout values ​​can be used in other examples. Figure 4 In the example, the input to model 400 is a tensor of shape (C, F, 1), where C is the input channel and F is the output of the FFT algorithm. The output of model 400 is a vector of size 4 showing the category to which the input data belongs, indicating which icon driver 102 was looking at when the data was collected.

[0051] In the example shown, model 400 is implemented in the Keras framework. However, other frameworks may be used in other examples. In the example shown, a learning rate of 0.001 and a batch size of 32 are used. However, other learning rates and / or batch sizes may be used in other examples.

[0052] In the example shown, such as Figure 6 The diagram shows the accuracy of Model 400 tested using different input sizes ranging from 0.25 seconds to 3 seconds (with a step size of 0.25 seconds). Figure 6 As shown, the highest accuracy is achieved with an input size of 1 second. Therefore, an input size of 1 second is used in the example shown. However, different input sizes can be used in other examples.

[0053] Return to reference Figure 3 The model training module 310 may include tools for training Figure 4 The model training module 310 may include programming instructions for training the model 400, which is maintained by the vehicle system 200. Specifically, the model training module 310 may include programming instructions for training the model 400 based on training data received by the training data receiving module 300 and a benchmark true value, after the data has been preprocessed by the data filtering module 304, the data segmentation module 306, and the FFT module 308. In the example shown, the model training module 310 includes programming instructions for training the model 400 end-to-end for up to 50 epochs using an early stopping strategy with the Adam optimization algorithm and binary cross-entropy as the loss function. However, in other examples, the model training module 310 may include programming instructions for using different optimization algorithms, different loss functions, and / or different numbers of epochs. After the model training module 310 trains the model 400, the learned model parameters may be stored in the data storage component 210.

[0054] To test the disclosed Model 400, its accuracy was compared with several other models. Specifically, the same training data was used for the disclosed Model 400 and several known methods, including Canonical Correlation (CCA), Extended Canonical Correlation (eCCA), and User Independent Complex Spectral Features (UI-C-CNN). Figure 7 The average accuracy of each of these models is shown for various signal lengths. For example... Figure 7 As shown, the disclosed model outperforms these known models for any signal length.

[0055] Return to reference Figure 3The icon detection module 312 includes programming instructions for using the model 400 to detect in real time which icon the driver 102 is looking at on the HUD 108 after the model 400 has been trained by the model training module 310. Specifically, when the driver 102 is looking at an icon on the HUD 108, the head-mounted device 104 can capture EEG signals from the driver and transmit these signals to the EEG data receiving module 302. The received signals can be preprocessed by the data filtering module 304, the data segmentation module 306, and the FFT module 308. The icon detection module 312 can then input the preprocessed signals into the trained model 400, and the model can output a prediction of which icon the driver 102 is looking at based on the input signals.

[0056] Still referencing Figure 3 The vehicle system operation module 314 may include programmed instructions for performing one or more vehicle operations based on the icon determined by the icon detection module 312. In one example, the vehicle system operation module 314 may cause the HUD 108 to display different sets of icons (e.g., icon submenus) based on the icon being viewed by the driver 102. In another example, the vehicle system operation module 314 may cause the functions of the vehicle 100 to be stopped, started, or modified based on the icon being viewed by the driver 102.

[0057] exist Figure 1 In the example, the icons in the HUD 108 that driver 102 is looking at can cause the vehicle system operation module 314 to display submenus for those icons. For example, if driver 102 looks at icon 110, the vehicle system operation module 314 can display a submenu related to audio options; if driver 102 looks at icon 112, the vehicle system operation module 314 can display a submenu related to navigation options; if driver 102 looks at icon 114, the vehicle system operation module 314 can display a submenu related to settings; and if driver 102 looks at icon 116, the vehicle system operation module 314 can display a submenu related to temperature options.

[0058] Submenus may include icons for adjusting different options or settings related to the vehicle. For example, after driver 102 views icon 116, HUD 108 may display a submenu with different icons related to temperature settings. In particular, one of the icons in the submenu may be related to air conditioning, such that when driver 102 views that icon, vehicle system operation module 314 activates the air conditioning in vehicle 100.

[0059] Return to reference Figure 3The icon color adjustment module 316 may include programming instructions, as disclosed herein, for adjusting one or more colors of icons displayed by the HUD 108. Studies have shown that some people are better able to recognize certain colors. Therefore, the icon color adjustment module 316 can adjust the colors of icons displayed by the HUD 108 to be better recognized by the driver 102.

[0060] In one example, when HUD 108 displays a submenu of icons, one of the icons can be used to return to the previous menu. This allows the driver 102 to return to the previous menu and select a different submenu if they accidentally select the wrong menu. If the driver 102 uses this feature frequently, it could indicate that the driver 102 has difficulty recognizing the colors of the icons displayed by HUD 108, leading the driver 102 to select the wrong icon.

[0061] Therefore, in this embodiment, the icon color adjustment module 316 may include programming instructions for monitoring the number of times the driver 102 views and selects an icon to return to a previous menu. If the icon color adjustment module 316 determines that the driver 102 uses one of these icons more than a predetermined threshold number of times during a predetermined time period, then the icon color adjustment module 316 may adjust the color of one or more of these icons displayed on the HUD 108 to help the driver 102 better identify the icons and reduce the frequency of the driver 102 selecting the wrong icon. In some examples, when this determination is made, the icon color adjustment module 316 may adjust the color of all the icons displayed by the HUD 108. In other examples, the icon color adjustment module 316 may identify the color of the icon that the driver 102 most frequently misselects and may adjust only the color of that icon.

[0062] Figure 8 A flowchart depicts an example method of operating vehicle system 200 to train model 400. At step 800, training data receiving module 300 receives training data. As described above, the training data may include EEG data collected from multiple drivers wearing head-mounted devices 104 while viewing specific icons on HUD 108. The training data may include ground truth values ​​indicating which icon each driver is viewing during each time step.

[0063] At step 802, the data filtering module 304 filters the training data received by the training data receiving module 300. In the example shown, the data filtering module 304 applies a Butterworth bandpass filter in the 3-40Hz range. However, in other examples, the data filtering module 304 may apply other types of filters with other frequency ranges.

[0064] At step 804, the data segmentation module 306 performs data segmentation on the training data received by the training data receiving module 300. In the example shown, the data segmentation module 306 segments the training data into intervals of 0.25 seconds. However, in other examples, the data segmentation module 306 may segment the training data into intervals of other lengths.

[0065] At step 806, FFT module 308 performs an FFT on the training data received by training data receiving module 300. In the example shown, FFT module 308 uses a start frequency of 3 Hz and an end frequency of 35 Hz. However, in other examples, different start and / or end frequencies may be used.

[0066] At step 808, model training module 310 trains model 400 using the techniques described above based on the training data received by training data receiving module 300. In the example shown, model training module 310 uses the Adam optimization algorithm and binary cross-entropy as the loss function, a learning rate of 0.001, and a batch size of 32 to train model 400. However, in other examples, model training module 310 may use other optimization algorithms, loss functions, learning rates, and batch sizes. After model training module 310 trains model 400, at step 810, model training module 310 stores the learned parameters in data storage component 210.

[0067] Figure 9 A flowchart illustrating an example method for operating vehicle system 200 after model 400 has been trained is provided. At step 900, while the driver is viewing one of the icons displayed on HUD 108, brainwave data receiving module 302 receives brainwave data (e.g., EEG data) from head-mounted device 104 worn by driver 102. At step 902, data filtering module 304 filters the brainwave data received by brainwave data receiving module 302 as described above. At step 904, data segmentation module 306 performs data segmentation on the brainwave data received by brainwave data receiving module 302 as described above. At step 906, FFT module 308 performs an FFT on the brainwave data received by brainwave data receiving module 302 as described above.

[0068] At step 908, the icon detection module 312 inputs the preprocessed EEG data into the trained model 400. The model 400 then outputs a prediction about which icon the driver 102 is looking at. Then, at step 910, the vehicle system operation module 314 executes operations of the vehicle system 200 based on the output of the model 400.

[0069] It should now be understood that the embodiments described herein relate to using deep learning to control a closed-loop, real-time SSVEP-based head-up display (HUD) in a vehicle. As disclosed herein, a model maintained by the vehicle system can be trained to predict which icon the driver is viewing on the HUD based on EEG data. During real-time operation, EEG data can be received from the driver wearing a head-up device and fed into the trained model. The trained model can predict which icon the driver is viewing on the vehicle's HUD. The vehicle system can then perform one or more vehicle operations based on the model's output. The machine learning architecture and training methods of the embodiments disclosed herein can predict which icon the driver is viewing faster and more accurately than known methods, thereby enabling real-time operation while driving a vehicle.

[0070] It should be noted that the terms “substantially” and “approximately” are used herein to indicate the degree of inherent uncertainty attributable to any quantitative comparison, value, measurement, or other representation. These terms are also used herein to indicate the extent to which a quantitative representation may differ from the stated reference without causing a change in the fundamental function of the subject matter under discussion.

[0071] While specific embodiments have been illustrated and described herein, it should be understood that various other changes and modifications can be made without departing from the spirit and scope of the claimed subject matter. Furthermore, although various aspects of the claimed subject matter have been described herein, these aspects need not be used in combination. Therefore, the appended claims are intended to cover all such changes and modifications within the scope of the claimed subject matter.

Claims

1. A vehicle system comprising a controller programmed to: display a plurality of icons on a head-up display (HUD) of a vehicle; receive electroencephalogram (EEG) data from a driver of the vehicle; perform a fast Fourier transform (FFT) of the EEG data to obtain an EEG spectrum; input the EEG spectrum into a trained machine learning model that outputs a prediction of which icon the driver is looking at based on the EEG spectrum; determine which icon of the plurality of icons the driver is viewing based on the output of the trained machine learning model; and perform one or more vehicle operations based on the output of the trained machine learning model.

2. The vehicle system of claim 1, wherein each icon of the plurality of icons has a different color.

3. The vehicle system of claim 1, wherein each icon of the plurality of icons has a different shape.

4. The vehicle system of claim 1, wherein the controller is further programmed to: apply a bandpass filter to the EEG data to obtain filtered EEG data; and perform an FFT of the filtered EEG data.

5. The vehicle system of claim 1, wherein the controller is further programmed to: perform data segmentation of the EEG data to obtain segmented EEG data; and perform an FFT of the segmented EEG data.

6. The vehicle system of claim 1, wherein the controller is further programmed to: receive training data comprising EEG data collected from a plurality of individual subjects while each subject of the plurality of individual subjects was viewing a particular icon; and train a machine learning model to predict which icon the individual subjects were viewing based on the training data to implement the trained machine learning model.

7. The vehicle system of claim 1, wherein the trained machine learning model comprises a convolutional neural network.

8. The vehicle system of claim 7, wherein the convolutional neural network comprises a residual neural network architecture.

9. The vehicle system of claim 1, wherein the trained machine learning model comprises one or more squeeze-and-excitation (SE) blocks.

10. The vehicle system of claim 9, wherein at least one of the SE blocks comprises a global max pooling layer, a first fully connected layer with a rectified linear unit activation function, and a second fully connected layer with a sigmoid activation function.

11. The vehicle system of claim 1, wherein the trained machine learning model comprises two SE-Res blocks, wherein each SE-Res block comprises: a two-dimensional convolutional layer; a batch normalization layer; an activation layer; and an SE block.

12. The vehicle system of claim 11, wherein an input of each SE-Res block is summed with an output of that SE-Res block.

13. The vehicle system of claim 11, wherein the trained machine learning model further comprises: a dropout layer; and a Softmax classification layer.

14. A method comprising: displaying a plurality of icons on a head-up display (HUD) of a vehicle; receiving electroencephalogram (EEG) data from a driver of the vehicle; performing a fast Fourier transform (FFT) of the EEG data to obtain an EEG spectrum; inputting the EEG spectrum into a trained machine learning model that outputs a prediction of which icon the driver is looking at based on the EEG spectrum; determining which icon of the plurality of icons the driver is viewing based on the output of the trained machine learning model; and performing one or more vehicle operations based on the output of the trained machine learning model. receiving electroencephalogram (EEG) data from a driver of a vehicle; performing a fast Fourier transform (FFT) of the EEG data to obtain an EEG spectrum; inputting the EEG spectrum into a trained machine learning model that outputs a prediction of an icon that the driver is looking at based on the EEG spectrum; determining which icon of the plurality of icons the driver is viewing based on the output of the trained machine learning model; and performing one or more vehicle operations based on the output of the trained machine learning model.

15. The method of claim 14, further comprising: applying a bandpass filter to the EEG data to obtain filtered EEG data; performing a data segmentation of the filtered EEG data to obtain segmented EEG data; and performing an FFT of the segmented EEG data.

16. The method of claim 14, wherein the trained machine learning model comprises a convolutional neural network comprising: two SE-Res blocks, wherein each SE-Res block comprises: a two-dimensional convolutional layer; a batch normalization layer; an activation layer; and an SE block.

17. The method of claim 16, wherein the SE block comprises a global max pooling layer, a first fully connected layer with a rectified linear unit activation function, and a second fully connected layer with a sigmoid activation function.

18. The method of claim 16, wherein the trained machine learning model further comprises: a dropout layer; and a Softmax classification layer.

19. A method comprising: receiving training data comprising EEG data collected from a plurality of individual subjects while each subject of the plurality of individual subjects was viewing a particular icon; performing an FFT of the training data to obtain EEG spectral data; and training a machine learning model to predict which icon the individual subjects were viewing based on the EEG spectral data.

20. The method of claim 19, wherein the machine learning model comprises: two SE-Res blocks, wherein each SE-Res block comprises: a two-dimensional convolutional layer; a batch normalization layer; an activation layer; an SE block; and wherein the SE block comprises a global max pooling layer, a first fully connected layer with a rectified linear unit activation function, and a second fully connected layer with a sigmoid activation function. ​ ​ ​ ​

Images