Visual-control intelligent driving method and system based on brain-like intelligence

Abstract

The application provides a visual-control intelligent driving method and system based on brain-like intelligence, which comprises the following steps: S1, collecting data information of a driver; the data information comprises eye movement tracking data, electroencephalogram signals and electromyogram signals of hands and feet; S2, pre-processing different data information; S3, extracting features in time and frequency domains of different data information; S4, fusing features of different data information for multiple times and identifying a fusion result; and S5, constructing a visual-control intelligent driving model based on brain-like intelligence according to the fusion result and an artificial neural network. The application fuses comprehensive features of eye movement tracking data, electroencephalogram signals and electromyogram signals of hands and feet of the driver, effectively solves the problem of incomplete single signal information, combines the artificial neural network, enables the autonomous driving vehicle to continuously observe and learn, gradually improves the human-like driving capability, and improves the driving safety and efficiency of the intelligent driving vehicle.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving, and more specifically, to a vision-control intelligent driving method and system based on brain-like intelligence. Background Technology

[0002] With the development of artificial intelligence, big data, and internet technologies, intelligent driving vehicles have gradually become an important research direction leading the rapid development of the automotive industry. Intelligent driving vehicles can autonomously perceive their surroundings, analyze traffic conditions and driver status in real time, and thus make safer, more accurate, and more efficient driving decisions.

[0003] Intelligent driving systems utilize environmental and vehicle information from internal sensors (such as in-vehicle cameras) and external sensors (such as radar and video) to help drivers make decisions and take actions in potential hazardous situations. Existing intelligent driving systems propose various methods to detect driving intentions (lane change, lane keeping, braking, etc.), including methods based on vehicle information, methods based on environmental information, methods based on driving behavior, and methods based on biosignals. Vehicle information includes vehicle speed, acceleration, brake pedal deflection, etc.; environmental information includes road information, pedestrian information, and road traffic information. Driver behavior information includes foot position, arm position, head position, facial expressions, and blinking behavior, etc.; biosignal information includes electroencephalogram (EEG) signals and electromyogram (EMG) signals.

[0004] Driving is a complex task that requires various cognitive processes, including attention, perception, judgment, and decision-making. However, traditional intelligent driving assistance systems do not take into account the driver's intentions and mental state. Therefore, technologies that detect and identify the driver's driving intentions using biosignal information methods are receiving increasing attention.

[0005] Existing research typically relies on single electroencephalogram (EEG) signals or electromyogram (EMG) signals from the hands and feet to detect a driver's intentions, but this method has poor accuracy and robustness.

[0006] Chinese patent document CN113034951A discloses an intelligent driving system and method. The intelligent driving system includes a traffic information collection module, a cloud platform, and at least one vehicle-mounted terminal. The vehicle-mounted terminal includes a vehicle startup unit, a vehicle management unit, and an autonomous driving unit. The traffic information collection module collects and sends vehicle traffic-related information to the cloud platform in real time. The vehicle startup unit generates and sends vehicle startup information to the cloud platform. The cloud platform calculates and generates cloud information based on the vehicle traffic-related information and the vehicle startup information, and sends the cloud information to at least the vehicle management unit. The vehicle management unit determines traffic resources and traffic dynamics based on at least the cloud information, and sends the traffic resources and traffic dynamics to the autonomous driving unit. The autonomous driving unit generates autonomous driving commands to direct vehicle traffic based on the traffic resources and traffic dynamics. However, this patent document only improves road traffic efficiency through cloud-based control of the vehicle-mounted terminal, without considering the driver's intentions and mental state, thus failing to solve the aforementioned problems. Summary of the Invention

[0007] In view of the deficiencies in the prior art, the purpose of this invention is to provide a vision-controlled intelligent driving method and system based on brain-like intelligence.

[0008] A vision-controlled intelligent driving method based on brain-like intelligence, provided by the present invention, includes:

[0009] Step S1: Collect driver data; the data includes eye-tracking data, electroencephalogram (EEG) signals, and electromyography (EMG) signals from the hands and feet.

[0010] Step S2: Preprocess the different data separately;

[0011] Step S3: Extract time-frequency domain features from different data information respectively;

[0012] Step S4: Merge features from different data multiple times and identify the fusion result;

[0013] Step S5: Based on the fusion results and artificial neural networks, construct a vision-control intelligent driving model based on brain-like intelligence.

[0014] Preferably, the multiple fusions include fusing eye-tracking data features with EEG signal features and electromyography (EMG) signal features from the hands and feet to obtain two time-frequency domain fusion features, and then fusing the two time-frequency domain fusion features again to obtain the final features of the multi-source signal.

[0015] Preferably, the vision-control intelligent driving model based on brain-like intelligence continuously corrects the lateral steering data and longitudinal braking data output by the intelligent driving model through the BP algorithm, so that they are consistent with the expected values.

[0016] Preferably, the eye-tracking data is recorded using an eye tracker; the electroencephalogram (EEG) signals and the electromyogram (EMG) signals of the hands and feet are collected using a data acquisition device at a fixed location on the driver's body; the data acquisition device includes a 32-channel data acquisition device.

[0017] Preferably, the preprocessing includes eye-tracking data preprocessing, electroencephalogram (EEG) signal preprocessing, and electromyography (EMG) signal preprocessing of the hands and feet; the eye-tracking data preprocessing includes an eye-tracking analysis module; the eye-tracking analysis module includes a data storage module; the data storage module extracts the subject's eye-tracking image data, performs model construction and ability assessment on the eye-tracking parameters, and stores the eye-tracking parameter analysis results; the EEG signal preprocessing and the EMG signal preprocessing of the hands and feet include using bandpass filters to filter the acquired EEG signals and EMG signals of the hands and feet respectively; and using downsampling processing to process the sampling frequency of the filtered EEG signals and EMG signals of the hands and feet to obtain sample data of EEG signals and EMG signals.

[0018] Preferably, the time-frequency domain feature extraction includes selecting an appropriate number of channels using a co-space mode, calculating the envelopes of the EEG signal and the electromyography signals of the hands and feet as time-domain features; calculating the power spectral density function of each channel using a fast Fourier transform as frequency-domain features; and sequentially concatenating the time-domain features and frequency-domain features as time-frequency domain features.

[0019] Preferably, the vision-control intelligent driving model based on brain-like intelligence takes multi-source information fusion feature data of eye tracking data, electroencephalogram (EEG) signals and electromyography (EMG) signals of the hands and feet as input, processes it through a long short-term memory network, and outputs steering data of the autonomous vehicle; the steering data includes the vehicle's direction and angle.

[0020] A vision-control intelligent driving system based on brain-like intelligence, according to the present invention, includes:

[0021] Module M1: Collects driver data; the data includes eye-tracking data, electroencephalogram (EEG) signals, and electromyographic (EMG) signals from the hands and feet.

[0022] Module M2: Performs preprocessing on different types of data.

[0023] Module M3: Performs time-frequency domain feature extraction on different types of data;

[0024] Module M4: Integrates features from multiple data sources to identify the fusion results;

[0025] Module M5: Based on the fusion results and artificial neural networks, construct a vision-control intelligent driving model based on brain-like intelligence.

[0026] Preferably, the multiple fusions include fusing eye-tracking data features with EEG signal features and electromyography (EMG) signal features from the hands and feet to obtain two time-frequency domain fusion features, and then fusing the two time-frequency domain fusion features again to obtain the final features of the multi-source signal.

[0027] Preferably, the vision-control intelligent driving model based on brain-like intelligence continuously corrects the lateral steering data and longitudinal braking data output by the intelligent driving model through the BP algorithm, so that they are consistent with the expected values.

[0028] Preferably, the eye-tracking data is recorded using an eye tracker; the electroencephalogram (EEG) signals and the electromyogram (EMG) signals of the hands and feet are collected using a data acquisition device at a fixed location on the driver's body; the data acquisition device includes a 32-channel data acquisition device.

[0029] Preferably, the preprocessing includes eye-tracking data preprocessing, electroencephalogram (EEG) signal preprocessing, and electromyography (EMG) signal preprocessing of the hands and feet; the eye-tracking data preprocessing includes an eye-tracking analysis module; the eye-tracking analysis module includes a data storage module; the data storage module extracts the subject's eye-tracking image data, performs model construction and ability assessment on the eye-tracking parameters, and stores the eye-tracking parameter analysis results; the EEG signal preprocessing and the EMG signal preprocessing of the hands and feet include using bandpass filters to filter the acquired EEG signals and EMG signals of the hands and feet respectively; and using downsampling processing to process the sampling frequency of the filtered EEG signals and EMG signals of the hands and feet to obtain sample data of EEG signals and EMG signals.

[0030] Preferably, the time-frequency domain feature extraction includes selecting an appropriate number of channels using a co-space mode, calculating the envelopes of the EEG signal and the electromyography signals of the hands and feet as time-domain features; calculating the power spectral density function of each channel using a fast Fourier transform as frequency-domain features; and sequentially concatenating the time-domain features and frequency-domain features as time-frequency domain features.

[0031] Preferably, the vision-control intelligent driving model based on brain-like intelligence takes multi-source information fusion feature data of eye tracking data, electroencephalogram (EEG) signals and electromyography (EMG) signals of the hands and feet as input, processes it through a long short-term memory network, and outputs steering data of the autonomous vehicle; the steering data includes the vehicle's direction and angle.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. This invention proposes a "vision-control" intelligent driving system and method based on brain-like intelligence, which integrates the comprehensive features of the driver's eye-tracking data, electroencephalogram (EEG) signals, and electromyographic (EMG) signals from the hands and feet, effectively solving the problem of incomplete information from a single signal. Combined with artificial neural networks, it enables autonomous vehicles to continuously observe and learn, gradually improving human-like driving capabilities and enhancing the driving safety and efficiency of intelligent driving vehicles.

[0034] 2. This invention establishes a "vision-control" intelligent driving system based on brain-like intelligence, which can continuously reduce the gap between the intelligent driving system model and the expected value through iterative training strategies, thereby effectively improving the robustness of the model.

[0035] 3. This invention, by fusing features at the feature level and selecting features from eye tracking, EEG signals, and EMG signals respectively, weakens the influence of a single channel on the entire detection process, thereby solving the problem of partial channel defects or incomplete single information in EEG signals and EMG signals from the hands and feet.

[0036] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0037] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0038] Figure 1 This is a flowchart illustrating the "vision-control" intelligent driving system based on brain-like intelligence in an embodiment of the present invention.

[0039] Figure 2 This is a schematic diagram illustrating the feature fusion based on driver eye-tracking data, electroencephalogram (EEG) signals, and electromyography (EMG) signals from the hands and feet in an embodiment of the present invention.

[0040] Figure 3 This is a schematic diagram of the internal structure of the LSTMN cell unit in an embodiment of the present invention.

[0041] Figure 4 This is a schematic diagram of error feedback for a "vision-control" intelligent driving model based on brain-like intelligence in an embodiment of the present invention.

[0042] Figure 5 This is a schematic diagram of the BP algorithm model in an embodiment of the present invention. Detailed Implementation

[0043] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0044] Reference Figure 1 As shown, a vision-controlled intelligent driving method based on brain-like intelligence includes:

[0045] First, data on the driver's eye-tracking, electroencephalogram (EEG), and electromyography (EMG) signals from the hands and feet are collected.

[0046] Eye-tracking data was recorded using an eye tracker; electroencephalogram (EEG) signals and electromyographic (EMG) signals from the hands and feet were collected using a 32-channel acquisition device at fixed locations on the driver's body. After acquisition, the different data were preprocessed.

[0047] The specific steps are as follows:

[0048] Eye-tracking data preprocessing: The eye-tracking analysis module contains a data storage module. The data storage module extracts the eye-tracking image data of the subjects, builds models and assesses the ability of the eye-tracking parameters, and then stores the eye-tracking parameter analysis results.

[0049] Preprocessing of EEG signals and EMG signals from the hands and feet: Bandpass filters were used to filter the acquired EEG signals and EMG signals from the hands and feet respectively; downsampling was used to process the sampling frequency of the filtered EEG signals and EMG signals from the hands and feet to obtain sample data of EEG signals and EMG signals.

[0050] Next, using the co-space mode to select an appropriate number of channels, the envelopes of EEG signals and EMG signals from the hands and feet are calculated as time-domain features. The power spectral density function of each channel is calculated using fast Fourier transform as the frequency-domain feature. The time-domain features and frequency-domain features are sequentially concatenated as time-frequency domain features, thus completing the time-frequency domain feature extraction for different data information.

[0051] Reference Figure 2 As shown, eye-tracking data features are fused with EEG signal features and electromyography (EMG) signal features from the hands and feet to obtain two time-frequency domain fusion features. These two time-frequency domain fusion features are then fused again to obtain the final features of the multi-source signal.

[0052] Finally, based on the fusion results and artificial neural networks, a vision-control intelligent driving model based on brain-like intelligence is constructed. (Refer to...) Figure 5As shown, this model continuously corrects the lateral steering data and longitudinal braking data output by the intelligent driving model through the BP algorithm to make them consistent with the expected values. It takes multi-source information fusion feature data of eye tracking data, EEG signals and electromyography signals of the hands and feet as input, processes it through a long short-term memory network, and outputs the steering data of the autonomous vehicle in terms of direction and angle.

[0053] The above are basic embodiments of the present invention. The technical solution of the present invention will be further described below through a preferred embodiment.

[0054] Example 1

[0055] Reference Figure 1 As shown, this embodiment discloses a "vision-control" intelligent driving method based on brain-like intelligence, including:

[0056] First, eye-tracking data, electroencephalogram (EEG) signals, and electromyographic (EMG) signals from the driver's hands and feet were collected. The collection process is as follows:

[0057] The experiment used an eye tracker to track and record the driver's eye movement data. The eye tracker consisted of a head-mounted glasses and a portable recording device, with a sampling rate of 50Hz. The glasses module supported lens replacement and was equipped with a scene camera, which could record the subject's visual field in real time while tracking eye movements.

[0058] According to the location and distribution rules of the international standard 10 / 20 system, channels F3, F4, Fz, Cz, Pz, C3, C4, T7, T8, P7, P3, P4, P8, 01, 02 and 0z are selected as EEG signal acquisition locations;

[0059] The rectus femoris, vastus medialis, vastus lateralis, tibialis anterior, biceps brachii, medial gastrocnemius, lateral gastrocnemius, and soleus muscles were selected as the sites for electromyography signal acquisition.

[0060] Electroencephalogram (EEG) signals and electromyogram (EMG) signals from the hands and feet were acquired using a 32-channel acquisition device.

[0061] Next, the eye-tracking data, electroencephalogram (EEG) signals, and electromyography (EMG) signals from the driver's hands and feet were preprocessed.

[0062] Eye-tracking data preprocessing includes: time synchronization preprocessing, driving behavior data preprocessing, and eye-tracking data preprocessing.

[0063] A second-order Butterworth bandpass filter was used to filter the acquired EEG signals and EMG signals from the hands and feet, respectively, to obtain EEG signals with a target frequency band of 0.53-60Hz and EMG signals with a target frequency band of 20-120Hz.

[0064] The sampling frequency of the filtered EEG signal and the electromyography (EMG) signal of the hands and feet was reduced to below 200Hz by using downsampling processing. The EEG signal and the EMG signal of the hands and feet before the current time were extracted as sample data with a window width of 1 second.

[0065] Baseline correction was performed using the first 100ms of sample data as a reference. The average value of the superimposed data of each channel of the EEG signal was subtracted from the data of each channel of the EMG signal. At the same time, the average value of the superimposed data of each channel of the EMG signal was subtracted from the data of each channel of the EMG signal to obtain the EEG signal data and the EMG signal data of the hands and feet after removing common noise.

[0066] Then, time-frequency domain features were extracted from the preprocessed eye-tracking data, EEG signals, and electromyography signals from the hands and feet.

[0067] Due to driver activity interference and occlusion during the experiment, the raw data collection suffered from eye movement point loss and data anomalies. Linear interpolation was used to compensate for signal loss and replace outliers, followed by noise reduction using moving average filtering. Furthermore, gaze and saccade behavior data were separated and extracted based on the Velocity-Threshold Identification (I-VT) algorithm. Blink segments were identified based on time thresholds, obtaining time-series eye movement data labeled with gaze, saccade, blink behavior, and pupil diameter information. Indicators were extracted within the study time window for eye movement characteristic analysis.

[0068] By selecting an appropriate number of channels using the co-space model, the envelopes of EEG signals and EMG signals from the hands and feet are calculated separately as time-domain features. The power spectral density function of each channel is calculated using fast Fourier transform as a frequency-domain feature. The time-domain features and frequency-domain features are sequentially concatenated as time-frequency domain features. Distance correlation analysis is used to select features, and time-domain features, frequency-domain features, and time-frequency domain features with specified scaling factors are obtained.

[0069] Reference Figure 2 As shown, the eye-tracking data features are then fused with EEG signal features and electromyography (EMG) signal features from the hands and feet, respectively. The newly obtained eye-EEG and eye-EMG features are then subjected to the above operations to obtain the final features of the multi-source signals.

[0070] The standard distance between each row of data in the two sample data sets, eye-tracking data and EEG signal, is calculated using the following formula;

[0071] a ij =∣∣M i -M j ||, i, j = 1, 2, ..., n;

[0072] b ij =∣∣Ni -N j ||, i, j = 1, 2, ..., n;

[0073] In the above formula, M and N represent the preliminary feature calculation matrices of the two samples, eye-tracking data and EEG signal, respectively; a and b represent the standard distance of the corresponding sample feature matrices, respectively; i and j represent the rows and columns of the matrix, respectively; and n represents the dimension of the matrix.

[0074] The standard distance matrix between two sample data points is centered using the following formula:

[0075]

[0076]

[0077] In the above formula, and Let represent the average values ​​of the norm matrix of the i-th row, respectively; and Let represent the average values ​​of the norm matrix in the j-th row, respectively; and Let A and B represent the average values ​​of the distance matrices of the two samples, respectively; and let B and B represent the centered distance matrices of the two samples, respectively.

[0078] Calculate matrix vector A using the following formula i B j The covariance matrix;

[0079]

[0080] Similarly, calculate the standard distance between each row of data in two samples of eye-tracking data and electromyography signals from the hands and feet:

[0081] a ij =∣∣M i -M j ||, i, j = 1, 2, ..., n;

[0082] c ij =∣∣O i -O j ||, i, j = 1, 2, ..., n;

[0083] In the above formula, M and O represent the preliminary feature calculation matrices of the two samples, eye-tracking data and EEG signal, respectively; a and c represent the standard distance of the corresponding sample feature matrices, respectively; i and j represent the rows and columns of the matrix, respectively; and n represents the dimension of the matrix.

[0084] The standard distance matrix between two sample data points is centered using the following formula:

[0085]

[0086]

[0087] In the above formula, and and represent the average values ​​of the norm matrix of the i-th row, respectively; and Let represent the average values ​​of the norm matrix in the j-th row, respectively; and Let A and C represent the average values ​​of the distance matrices of the two samples, respectively; let A and C represent the distance matrices of the two samples after centering, respectively.

[0088] The covariance matrix of matrices A and C is calculated using the following formula;

[0089] Calculate matrix vector A using the following formula i C j The covariance matrix;

[0090]

[0091] The two fusion features are then fused to obtain the final fusion feature:

[0092]

[0093] Finally, based on multi-source information fusion and artificial neural networks, a "vision-control" intelligent driving system and model based on brain-like intelligence were built.

[0094] The input to the intelligent driving model is multi-source information fusion feature data, including eye-tracking data, electroencephalogram (EEG) signals, and electromyography (EMG) signals from the hands and feet. This data is processed through a long short-term memory network to output lateral steering data and longitudinal braking data.

[0095] It should be noted that Long Short-Term Memory Network (LSTMNN) is developed based on Recurrent Neural Network (RNN). In a standard RNN, the cell unit has only a simple tanh layer, while each repeating module of an LTMNN has four fully connected layers that interact with each other in a special way.

[0096] Reference Figure 3As shown, a repeating unit A is called a cell. Each line with an arrow transmits a data vector from one node to another. Pink circles represent point-by-point operations, such as multiplication or addition of data as shown in the diagram. Boxed areas represent neural network layers, such as tanh or sigmoid layers. Merging lines combine data from two lines; conversely, separating lines represent copying data from one line and transmitting it to a different location. It can be observed that LTMNs are composed of four interacting yellow components.

[0097] In LSTMN, the cell state—the horizontal line passing through the topmost cell unit—allows information to be deleted or added through "gate" structures. LSTMN has three gates: the forget gate... t Input gate i t Output gate O t These gates are used to regulate cell states. The forget gate determines the proportion of information from previous cells that the current cell discards, the input gate controls the proportion of information from candidate states added to the current cell, and the output gate determines the proportion of information from the current cell's state that flows to subsequent cells. Each gate uses a sigmoid activation function to control its output value between 0 and 1, where 0 represents the weight of retaining the corresponding information, and 1 represents complete retention.

[0098] The calculation method for LTSMN cells at time t is as follows:

[0099] f t =σ(W f •[h t-1 ,x t ]+b f );

[0100] i t =σ(W i ·[h t-1 ,x t ]+b i );

[0101]

[0102]

[0103] o t =σ(W o ·[h t-1 ,x t ]+b o );

[0104] h t =o t *tanh(C t);

[0105] Among them, f t This refers to the output value of the forget gate, σ(·) is the Sigmoid function, and W f It is the weight matrix of the forget gate, h t-1 and x t These are the output value of the LTMN at the previous time step and the input value of the network at the current time step, respectively. t-1 ,x t The symbol ] represents concatenating two vectors into a longer vector, b f It is the bias term of the forget gate, i t W refers to the output value of the input gate. i It is the weight matrix of the forget gate, b i It is the bias term of the input gate. This refers to the current memory location, tanh(·) is the tanh function, and W C It is the weight matrix in the current memory, C t This refers to the current state of the cell, b c It is the bias term of the cell state at the current moment, C t-1 It refers to long-term memory, o t This refers to the output value of the output gate, W. o It is the weight matrix of the output gate, b o It is the bias term of the output gate, h t This is the output value of LSTMN at the current time.

[0106] Reference Figure 4 and Figure 5 As shown, the desired lateral steering data and longitudinal braking data are input, compared with the lateral steering data and longitudinal braking data output by the intelligent driving model, and the error between the two is obtained and input into the intelligent driving model. The back-propagation (BP) algorithm is used to continuously correct the lateral steering data and longitudinal braking data output by the intelligent driving model so that they are consistent with the expected values.

[0107] Specifically, the backpropagation algorithm updates model parameters, and the main steps are as follows:

[0108] Forward propagation: Training samples are input into the model, and the output value of the model is obtained through forward computation. The input of each layer undergoes a linear transformation of weights and biases, and the output is calculated through an activation function.

[0109] Loss calculation: Compare the model's output value with the expected value and calculate the value of the cross-entropy loss function.

[0110] Backpropagation: Starting from the output layer, calculate the partial derivative of each parameter with respect to the loss function. Using the chain rule, propagate the error backward from the output layer, calculating the gradient of each layer.

[0111] Parameter Update: Based on gradient information, the model parameters are updated using an optimization algorithm. The parameter values ​​are adjusted according to the direction and magnitude of the gradient to reduce the loss function value.

[0112] Repeated iterations: Repeat the steps above using different training samples or batches until the stopping condition is met. Through continuous iteration and parameter updates, the error backpropagation algorithm enables the model to gradually converge to a state that minimizes the error, allowing the model to better fit the training data and make accurate predictions on unseen data.

[0113] This invention proposes a brain-inspired "vision-control" intelligent driving system based on autonomous vehicles as a platform. Based on the fusion characteristics of driver eye-tracking data, electroencephalogram (EEG) signals, and electromyographic (EMG) signals from the hands and feet, a brain-inspired "vision-control" intelligent driving model is established. An iterative training strategy continuously reduces the gap between the intelligent driving model and the expected value, thereby improving the robustness of the model, enhancing the human-like driving skills and reaction capabilities of autonomous vehicles, and improving the driving safety and efficiency of autonomous vehicles.

[0114] The present invention also provides a vision-control intelligent driving system based on brain-inspired intelligence. The vision-control intelligent driving system based on brain-inspired intelligence can be implemented by executing the process steps of the vision-control intelligent driving method based on brain-inspired intelligence. That is, those skilled in the art can understand the vision-control intelligent driving method based on brain-inspired intelligence as a preferred embodiment of the vision-control intelligent driving system based on brain-inspired intelligence.

[0115] Specifically, a vision-control intelligent driving system based on brain-like intelligence includes:

[0116] Module M1: Collects driver data; the data includes eye-tracking data, electroencephalogram (EEG) signals, and electromyographic (EMG) signals from the hands and feet.

[0117] Module M2: Performs preprocessing on different types of data.

[0118] Module M3: Performs time-frequency domain feature extraction on different types of data;

[0119] Module M4: Integrates features from multiple data sources to identify the fusion results;

[0120] Module M5: Based on the fusion results and artificial neural networks, construct a vision-control intelligent driving model based on brain-like intelligence.

[0121] The multiple fusions include fusing eye-tracking data features with EEG signal features and electromyography (EMG) signal features from the hands and feet to obtain two time-frequency domain fusion features, and then fusing the two time-frequency domain fusion features again to obtain the final features of the multi-source signal.

[0122] The vision-control intelligent driving model based on brain-like intelligence continuously corrects the lateral steering data and longitudinal braking data output by the intelligent driving model through the BP algorithm, so that they are consistent with the expected values.

[0123] The eye-tracking data is recorded using an eye tracker; the electroencephalogram (EEG) signals and the electromyogram (EMG) signals of the hands and feet are collected at fixed locations on the driver using an acquisition device; the acquisition device includes a 32-channel acquisition device.

[0124] The preprocessing includes eye-tracking data preprocessing, EEG signal preprocessing, and electromyography (EMG) signal preprocessing of the hands and feet. The eye-tracking data preprocessing includes an eye-tracking analysis module, which includes a data storage module. The data storage module extracts the subject's eye-tracking image data, constructs a model for the eye-tracking parameters, assesses abilities, and stores the eye-tracking parameter analysis results. The EEG signal preprocessing and EMG signal preprocessing of the hands and feet include using bandpass filters to filter the acquired EEG signals and EMG signals from the hands and feet, respectively. The sampling frequency of the filtered EEG signals and EMG signals from the hands and feet is processed using downsampling to obtain sample data of the EEG and EMG signals.

[0125] The time-frequency domain feature extraction includes selecting an appropriate number of channels using a co-space pattern, calculating the envelopes of EEG signals and EMG signals from the hands and feet as time-domain features, calculating the power spectral density function of each channel using a fast Fourier transform as frequency-domain features, and sequentially concatenating the time-domain features and frequency-domain features as time-frequency domain features.

[0126] The vision-control intelligent driving model based on brain-like intelligence takes multi-source information fusion feature data, including eye-tracking data, electroencephalogram (EEG) signals, and electromyography (EMG) signals from the hands and feet, as input, processes it through a long short-term memory network, and outputs steering data for the autonomous vehicle; the steering data includes the vehicle's direction and angle.

[0127] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0128] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A vision-controlled intelligent driving method based on brain-like intelligence, characterized in that, include: Step S1: Collect driver data; The data includes eye-tracking data, electroencephalogram (EEG) signals, and electromyographic (EMG) signals from the hands and feet. Step S2: Preprocess the different data separately; Step S3: Extract time-frequency domain features from different data information respectively; Step S4: Merge features from different data multiple times and identify the fusion result; Step S5: Based on the fusion results and artificial neural networks, construct a vision-control intelligent driving model based on brain-like intelligence; The multiple fusions include fusing eye-tracking data features with EEG signal features and electromyography (EMG) signal features from the hands and feet to obtain two time-frequency domain fusion features, and then fusing the two time-frequency domain fusion features again to obtain the final features of the multi-source signal. The vision-control intelligent driving model based on brain-like intelligence takes multi-source information fusion feature data, including eye-tracking data, electroencephalogram (EEG) signals, and electromyography (EMG) signals from the hands and feet, as input, processes it through a long short-term memory network, and outputs steering data for the autonomous vehicle; the steering data includes the vehicle's direction and angle.

2. The vision-control intelligent driving method based on brain-like intelligence according to claim 1, characterized in that, The vision-control intelligent driving model based on brain-like intelligence continuously corrects the lateral steering data and longitudinal braking data output by the intelligent driving model through the BP algorithm, so that they are consistent with the expected values.

3. The vision-control intelligent driving method based on brain-like intelligence according to claim 1, characterized in that, The eye-tracking data is recorded using an eye tracker; the electroencephalogram (EEG) signals and the electromyogram (EMG) signals of the hands and feet are collected using a data acquisition device at a fixed location on the driver's body; the data acquisition device is a 32-channel data acquisition device.

4. The vision-control intelligent driving method based on brain-like intelligence according to claim 1, characterized in that, The preprocessing includes eye-tracking data preprocessing, electroencephalogram (EEG) signal preprocessing, and electromyography (EMG) signal preprocessing of the hands and feet. An eye-tracking data preprocessing module is used; the eye-tracking analysis module includes a data storage module; the data storage module extracts the subject's eye-tracking image data, constructs a model for the eye-tracking parameters and assesses the ability, and then stores the eye-tracking parameter analysis results; the preprocessing of EEG signals and the preprocessing of EMG signals from the hands and feet include using bandpass filters to filter the acquired EEG signals and EMG signals from the hands and feet, respectively. The sampling frequency of the filtered EEG signal and the electromyography (EMG) signal from the hands and feet was processed using downsampling to obtain sample data of the EEG and EMG signals.

5. The vision-control intelligent driving method based on brain-like intelligence according to claim 1, characterized in that, The time-frequency domain feature extraction includes selecting an appropriate number of channels using a co-space pattern, calculating the envelopes of EEG signals and EMG signals from the hands and feet as time-domain features, calculating the power spectral density function of each channel using a fast Fourier transform as frequency-domain features, and sequentially concatenating the time-domain features and frequency-domain features as time-frequency domain features.

6. A vision-control intelligent driving system based on brain-like intelligence, characterized in that, include: Module M1: Collects driver data; The data includes eye-tracking data, electroencephalogram (EEG) signals, and electromyographic (EMG) signals from the hands and feet. Module M2: Performs preprocessing on different types of data. Module M3: Performs time-frequency domain feature extraction on different types of data; Module M4: Integrates features from multiple data sources to identify the fusion results; Module M5: Based on the fusion results and artificial neural networks, construct a vision-control intelligent driving model based on brain-like intelligence; The multiple fusions include fusing eye-tracking data features with EEG signal features and electromyography (EMG) signal features from the hands and feet to obtain two time-frequency domain fusion features, and then fusing the two time-frequency domain fusion features again to obtain the final features of the multi-source signal. The vision-control intelligent driving model based on brain-like intelligence takes multi-source information fusion feature data, including eye-tracking data, electroencephalogram (EEG) signals, and electromyography (EMG) signals from the hands and feet, as input, processes it through a long short-term memory network, and outputs steering data for the autonomous vehicle; the steering data includes the vehicle's direction and angle.

7. A vision-control intelligent driving system based on brain-like intelligence according to claim 6, characterized in that, The vision-control intelligent driving model based on brain-like intelligence continuously corrects the lateral steering data and longitudinal braking data output by the intelligent driving model through the BP algorithm, so that they are consistent with the expected values.

Images