Method, device and equipment for predicting charging intention of driver and storage medium

Abstract

The invention discloses a driver charging intention prediction method, device and equipment and a storage medium, and relates to the technical field of electric automobiles, and the method comprises the steps: collecting a face image, an electrocardiosignal and an electroencephalogram signal of a driver; wavelet packet decomposition denoising is carried out on the electroencephalogram signals, heart interval feature extraction is carried out on the electrocardiosignals, and alignment processing is carried out on the face images; constructing a three-level fusion architecture comprising a time sequence alignment layer, a feature cross layer and a decision integration layer, and fusing the multi-modal signals; multi-modal signal time sequence reference alignment is realized by using a time sequence alignment layer, cross-modal feature association enhancement is performed through three-dimensional convolution of a feature cross layer and a multi-head self-attention mechanism, emotion category judgment and confidence coefficient weighting are performed through a decision integration layer, and to-be-processed information including an emotion state, a real-time position and a charging pile state is obtained; the charging intention prediction result is obtained through the spatio-temporal joint prediction model to determine the target charging pile and path planning, and the prediction efficiency of the charging intention is improved.

Description

Technical Field

[0001] This invention relates to the field of electric vehicle technology, and in particular to a method, apparatus, device, and storage medium for predicting a driver's charging intention. Background Technology

[0002] Currently, with the large-scale development of the electric vehicle industry, the power grid system faces a dual challenge: on the one hand, the large-scale integration of power batteries poses a severe challenge to grid stability; on the other hand, charging infrastructure exhibits a significant supply-demand imbalance in the spatiotemporal dimensions, revealing a structural contradiction between local resource idleness and demand aggregation. Against this backdrop, accurately analyzing drivers' charging intentions has become a key research topic for optimizing energy dispatch and infrastructure layout. Current technical approaches have room for improvement in three main dimensions: first, while single-modal recognition paradigms (such as independent facial micro-expression analysis or EEG signal decoding) are theoretically feasible, they carry the risk of misjudgment due to their limited feature dimensions; second, while cross-modal fusion methods attempt to integrate physiological signals and visual data, they face the challenge of temporal alignment of multi-source heterogeneous data; and third, the complex environment in dynamic driving scenarios significantly interferes with sensor data. These technical bottlenecks severely restrict the generalization ability and practical application value of charging intention prediction models.

[0003] As can be seen from the above, how to improve the efficiency of predicting drivers' charging intentions is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for predicting driver charging intentions, which can improve the efficiency of predicting driver charging intentions during the process of predicting driver charging intentions, thereby improving the stability of the power grid. The specific solution is as follows:

[0005] Firstly, this application provides a method for predicting a driver's charging intention, including:

[0006] The system collects facial image data, electrocardiogram (ECG) signals, and electroencephalogram (EEG) signals corresponding to the driver by using a target camera and a multi-channel physiological signal acquisition device deployed in the cockpit. The EEG signals are then decomposed into wavelet packets to denoise the data, and the signal processing results are obtained. The ECG signals are then subjected to intercardiac interval feature extraction, and the feature extraction results are obtained. Finally, the facial image data is aligned to obtain an aligned image.

[0007] A three-level fusion architecture is constructed using a temporal alignment layer based on dynamic time warping, a feature crossover layer based on multi-head attention mechanism, and a decision integration layer based on confidence weighting. The signal processing results, the feature extraction results, and the aligned image are then fused to obtain the feature fusion result.

[0008] The feature fusion result is aligned with the multimodal signal temporal reference using the temporal alignment layer in the three-level fusion architecture to obtain the alignment result. Then, the alignment result is enhanced by cross-modal feature association using the three-dimensional convolution and multi-head self-attention mechanism in the feature cross-layer to obtain the enhancement result. Finally, the enhancement result is processed by emotion category judgment and confidence weighting using the decision integration layer to obtain unprocessed information including emotion state, real-time location, and charging pile status.

[0009] The information to be processed is processed using a spatiotemporal joint prediction model to obtain a charging intention prediction result, and the target charging pile and route planning result corresponding to the driver are determined based on the charging intention prediction result.

[0010] Optionally, the acquisition of facial image data, electrocardiogram signals, and electroencephalogram signals corresponding to the driver via a target camera and multi-channel physiological signal acquisition device deployed in the cockpit includes:

[0011] A preset recording camera is used to record the driver's visual representation data to meet the preset ultra-high-definition resolution conditions, thereby obtaining facial image data corresponding to the driver; the preset recording camera is placed inside the vehicle at a preset distance from the driver;

[0012] The electrocardiogram (ECG) signal corresponding to the driver is acquired using a pre-set disposable patch electrode and a chest lead method with a pre-set sampling rate. Then, the electroencephalogram (EEG) signal corresponding to the driver is acquired using a pre-set standard electrode placed in a specific channel position and flexible dry electrode technology.

[0013] The facial image data, the electrocardiogram (ECG) signal, and the electroencephalogram (EEG) signal include data corresponding to the driver in a resting state and data corresponding to the driver in a stimulated state.

[0014] Optionally, the step of performing wavelet packet decomposition denoising on the EEG signal to obtain the signal processing result, then performing intercardiac interval feature extraction on the ECG signal to obtain the feature extraction result, and finally performing alignment processing on the face image data to obtain the aligned image, includes:

[0015] The high-frequency noise and baseline drift in the EEG signal are eliminated using a preset digital filter to obtain the eliminated EEG signal. The eliminated EEG signal is then centered and the mean is removed to obtain the EEG signal to be processed. Then the covariance matrix corresponding to the EEG signal to be processed is determined.

[0016] The covariance matrix is ​​decomposed into eigenvalues ​​to obtain the corresponding eigenvalues. Then, the covariance matrix is ​​whitened using the whitening transformation rule and based on the eigenvalues ​​to obtain the whitening result, thereby determining the initial weight matrix corresponding to the whitening result.

[0017] The initial weight matrix is ​​normalized to obtain the target weight matrix. The whitening result is then processed using the target weight matrix and independent component analysis to obtain the signal processing result.

[0018] The electrocardiogram signal is sequentially processed by a preset electrocardiogram signal processing algorithm, including bandpass filtering, differential processing to enhance the slope, signal squaring, sliding integral window processing, and dynamic threshold update processing, to obtain an initial processing result. The initial processing result is then subjected to cardiac interval feature extraction to obtain the feature extraction result.

[0019] The video captured by the target camera is cropped using a preset sliding window to obtain initial images. The initial images are then normalized to obtain standardized images. The standardized images are then input into a preset facial key point detection model to obtain the coordinates of the key points.

[0020] Based on the key point coordinates and the preset template reference coordinate system, the affine matrix corresponding to the standardized image is determined, and the standardized image is aligned based on the affine matrix and the preset image tensor dimension to obtain the aligned image.

[0021] Optionally, a three-level fusion architecture is constructed using a temporal alignment layer based on dynamic time warping, a feature cross-interaction layer based on multi-head attention, and a decision ensemble layer based on confidence weighting. The signal processing result, the feature extraction result, and the aligned image are then fused to obtain a feature fusion result, including:

[0022] The temporal alignment layer is determined based on dynamic time warping, and the feature cross-layer is constructed based on multi-head attention mechanism. Then, the decision integration layer is constructed based on confidence weighting, so as to build a three-level fusion architecture based on the temporal alignment layer, the feature cross-layer and the decision integration layer.

[0023] The signal processing results and the feature extraction results are converted into signal features in tabular form, and the signal features are mapped into embedding vectors using a linear mapping layer; the number of channels in the embedding vectors is the same as the number of channels in the aligned image.

[0024] The aligned image is input into a preset three-dimensional residual network, and the channel attention mechanism in the preset three-dimensional residual network is used to perform global average pooling and global max pooling on the aligned image to obtain corresponding channel descriptors. The channel descriptors are then fused with the embedding vector to obtain channel weight coefficients. The aligned image is then weighted and adjusted in the channel dimension based on the channel weight coefficients to obtain a channel-enhanced feature map.

[0025] The channel enhancement feature map is subjected to average pooling and max pooling along the channel dimension using the spatial attention mechanism in the preset three-dimensional residual network to obtain corresponding spatial descriptors. The spatial descriptors are then fused with the embedding vector to obtain spatial weight coefficients. These spatial weight coefficients are used to adjust the channel enhancement feature map in the spatial dimension to obtain a spatial enhancement feature map.

[0026] The spatial augmented feature map is pooled along the spatial dimension using the temporal attention mechanism in the preset three-dimensional residual network to obtain a temporal descriptor. This temporal descriptor is then fused with the embedding vector to obtain temporal weight coefficients. These temporal weight coefficients are then used to weight and adjust the spatial augmented feature map along the temporal dimension to obtain a multi-dimensional feature map. The preset three-dimensional residual network includes several sequentially connected residual blocks. Each residual block includes two three-dimensional convolutional layers, a batch normalization unit, a non-linear activation function, and an attention enhancement unit.

[0027] The multi-dimensional feature map is subjected to multi-scale temporal pooling to obtain feature information at different time scales. The presence of facial expression action units is detected by the feature information using a first preset task branch and a nonlinear classification function, resulting in a binary classification presence detection result. The presence detection result is used to determine whether a specific facial expression action unit exists at each time step.

[0028] The feature information is subjected to facial expression action unit intensity regression using the second preset task branch to obtain the corresponding intensity regression result; the intensity regression result is used to characterize the activation degree of facial expression action units at each time step; the first preset task branch and the second preset task branch are parallel processing branches;

[0029] Several intensity regression results are concatenated into an intensity time series matrix, and a sliding window smoothing filter based on least squares fitting is used to perform a smoothing filter on the intensity time series corresponding to each facial expression action unit in the intensity time series matrix to obtain the filtering result.

[0030] The filter weight coefficients corresponding to the filtering result are determined based on the preset window length, and the intensity time series is convolved based on the filter weight coefficients to obtain the smoothed unit intensity change curve. The feature fusion result is determined based on the unit intensity change curve, the multidimensional feature map and the existence detection result.

[0031] Optionally, the feature fusion result is aligned with the multimodal signal temporal reference using the temporal alignment layer in the three-level fusion architecture to obtain an alignment result. Then, the alignment result is enhanced by cross-modal feature association using the 3D convolution and multi-head self-attention mechanism in the feature cross-layer to obtain an enhanced result. Finally, the enhanced result is processed by emotion category judgment and confidence weighting using the decision ensemble layer to obtain unprocessed information including emotional state, real-time location, and charging pile status, including:

[0032] The time alignment layer in the three-level fusion architecture is used to construct a cumulative cost matrix based on a preset dynamic time warping algorithm and the feature fusion result. Then, the cumulative cost matrix is ​​processed using a backtracking method to obtain the minimum cost path, and the alignment result is determined based on the minimum cost path.

[0033] The alignment result is stacked by channel using the feature cross-layer in the three-level fusion architecture to obtain a three-dimensional tensor, and the feature cross-layer is performed on the three-dimensional tensor using a preset three-dimensional convolution kernel to obtain the feature cross-layer result.

[0034] A multi-head self-attention mechanism is used to determine the cross-modal feature association weights corresponding to the feature cross-results. Then, a fully connected layer is used to determine the category scores corresponding to each modality based on the cross-modal feature association weights. The category scores are then converted into classification probabilities using several modality classifiers and activation functions.

[0035] The decision integration layer in the three-level fusion architecture converts each classification probability into a probability allocation function, and uses the probability allocation function to define a conflict metric between modalities. Based on the conflict metric, a conflict factor is determined, and then a corresponding combination strategy is determined based on the conflict factor. The combination strategy is used to determine the emotion category and the corresponding confidence score. Based on the emotion category and the confidence score, unprocessed information including emotional state, real-time location, and charging pile status is determined.

[0036] Optionally, the step of processing the information to be processed using a spatiotemporal joint prediction model to obtain a charging intention prediction result, and determining the target charging station and route planning result corresponding to the driver based on the charging intention prediction result, includes:

[0037] The location and real-time availability of each charging pile in the current area are obtained, and a set of charging pile locations is determined based on the location of each charging pile. The Euclidean distance between the driver and each charging pile is determined based on the current real-time location of the vehicle and the set of charging pile locations.

[0038] The information to be processed, the driver's real-time location, the charging pile location, and the real-time availability status are integrated according to a time window to obtain an input tensor. The input tensor is then subjected to moving average filtering using a preset temporal mixture model to obtain a long-term trend. The input tensor is then subjected to Fourier transform to obtain periodic features. Finally, a multi-head attention mechanism is used to capture cross-modal associations of the input tensor to obtain the charging pile selection probability distribution.

[0039] The target charging pile is determined based on the charging pile selection probability distribution and the real-time availability status. The location of the charging pile corresponding to the target charging pile is determined based on the charging pile number of the target charging pile. Then, the path planning result between the vehicle and the target charging pile is determined based on the charging pile location and the current real-time location of the vehicle.

[0040] Optionally, after determining the target charging station and route planning result corresponding to the driver based on the charging intention prediction result, the method further includes:

[0041] The cross-entropy loss function is used to calculate the loss of the charging intention prediction result, and the model parameters corresponding to the spatiotemporal joint prediction model are adjusted based on the loss calculation result to obtain the first model to be processed.

[0042] The first model to be processed is trained online using historical charging behavior data and a preset learning rate to obtain a second model to be processed. Decision weights and distance decay coefficients are determined for the emotion categories. The obtained decision weights and distance decay coefficients are then used to process the second model to obtain a new spatiotemporal joint prediction model.

[0043] Secondly, this application provides a driver charging intention prediction device, comprising:

[0044] The feature extraction result determination module is used to collect facial image data, electrocardiogram (ECG) signals, and electroencephalogram (EEG) signals corresponding to the driver through a target camera and a multi-channel physiological signal acquisition device deployed in the cockpit. The module performs wavelet packet decomposition and denoising on the EEG signals to obtain signal processing results. Then, it performs intercardiac interval feature extraction on the ECG signals to obtain feature extraction results. Finally, it performs alignment processing on the facial image data to obtain aligned images.

[0045] The feature fusion module is used to construct a three-level fusion architecture using a temporal alignment layer based on dynamic time warping, a feature cross-layer based on multi-head attention mechanism, and a decision integration layer based on confidence weighting, and to perform feature fusion on the signal processing results, the feature extraction results, and the aligned image to obtain the feature fusion result;

[0046] The pending information determination module is used to align the feature fusion result with the multimodal signal temporal reference using the temporal alignment layer in the three-level fusion architecture to obtain the alignment result, and to perform cross-modal feature association enhancement processing on the alignment result using the three-dimensional convolution and multi-head self-attention mechanism in the feature cross layer to obtain the enhancement result. Then, the decision integration layer is used to perform emotion category judgment and confidence weighting processing on the enhancement result to obtain pending information including emotion state, real-time location and charging pile status.

[0047] The charging intention prediction result generation module is used to process the information to be processed using a spatiotemporal joint prediction model to obtain the charging intention prediction result, so as to determine the target charging pile and route planning result corresponding to the driver based on the charging intention prediction result.

[0048] Thirdly, this application provides an electronic device, comprising:

[0049] Memory, used to store computer programs;

[0050] A processor is used to execute the computer program to implement the aforementioned driver charging intention prediction method.

[0051] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned driver charging intention prediction method.

[0052] As can be seen from the above, before predicting the driver's charging intention, this application needs to collect facial image data, electrocardiogram (ECG) signals, and electroencephalogram (EEG) signals corresponding to the driver through a target camera and a multi-channel physiological signal acquisition device deployed in the cockpit. Wavelet packet decomposition is used to denoise the EEG signals to obtain signal processing results. Then, the ECG signals undergo intercardiac interval feature extraction to obtain feature extraction results. The facial image data is then aligned to obtain an aligned image. A three-level fusion architecture is constructed using a temporal alignment layer based on dynamic time warping, a feature cross-layer based on multi-head attention mechanism, and a decision ensemble layer based on confidence weighting. The signal processing results, feature extraction results, and aligned images are then further processed. Feature fusion is performed to obtain the feature fusion result. The temporal alignment layer in the three-level fusion architecture is used to align the feature fusion result with the temporal reference of the multimodal signal to obtain the alignment result. Then, the three-dimensional convolution and multi-head self-attention mechanism in the feature cross layer are used to perform cross-modal feature association enhancement processing on the alignment result to obtain the enhancement result. Then, the decision integration layer is used to perform emotion category judgment and confidence weighting processing on the enhancement result to obtain the information to be processed, including emotion state, real-time location and charging pile status. The spatiotemporal joint prediction model is used to process the information to be processed to obtain the charging intention prediction result. Based on the charging intention prediction result, the target charging pile and route planning result corresponding to the driver are determined.

[0053] Therefore, this application first needs to acquire facial image data, electrocardiogram (ECG) signals, and electroencephalogram (EEG) signals corresponding to the driver through a target camera and a multi-channel physiological signal acquisition device deployed in the cockpit. Wavelet packet decomposition is then performed on the EEG signals to denoise them, yielding signal processing results. Next, intercardiac interval features are extracted from the ECG signals to obtain feature extraction results. Finally, the facial image data is aligned to obtain an aligned image. Second, a three-level fusion architecture is constructed using a temporal alignment layer based on dynamic time warping, a feature cross-interaction layer based on multi-head attention, and a decision integration layer based on confidence weighting. The signal processing results, feature extraction results, and aligned image are then fused to obtain... The feature fusion results are then used. The temporal alignment layer in the three-level fusion architecture aligns the feature fusion results with the multimodal signal temporal reference, yielding an aligned result. The 3D convolution and multi-head self-attention mechanism in the feature cross-layer enhance the aligned result through cross-modal feature association, resulting in an enhanced result. The decision ensemble layer then performs emotion category judgment and confidence weighting on the enhanced result, obtaining unprocessed information including emotional state, real-time location, and charging pile status. Finally, a spatiotemporal joint prediction model processes the unprocessed information to obtain a charging intention prediction result. Based on this prediction, the target charging pile and route planning result corresponding to the driver are determined. This improves the efficiency of predicting the driver's charging intention, thereby enhancing the stability of the power grid. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0055] Figure 1 Here is a flowchart of a method for predicting a driver's charging intention disclosed in this application;

[0056] Figure 2 A flowchart of a specific driver charging intention prediction method disclosed in this application;

[0057] Figure 3 This is a schematic diagram of a driver charging intention prediction device disclosed in this application;

[0058] Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Currently, with the large-scale development of the electric vehicle industry, accurately analyzing drivers' charging intentions has become a key research topic for optimizing energy dispatch and infrastructure layout. Current technical approaches have room for improvement in three main dimensions: First, while single-modal recognition paradigms are theoretically feasible, they suffer from the risk of misjudgment due to the limited feature dimensions; second, while cross-modal fusion methods attempt to integrate physiological signals and visual data, they face the challenge of temporal alignment of multi-source heterogeneous data; and third, the complex environment in dynamic driving scenarios significantly interferes with sensor data. To address these issues, this application provides a method for predicting drivers' charging intentions, which improves the efficiency of predicting drivers' charging intentions and thus enhances the stability of the power grid.

[0061] See Figure 1 As shown, this embodiment of the invention discloses a method for predicting a driver's charging intention, including:

[0062] Step S11: Collect facial image data, electrocardiogram (ECG) signal, and electroencephalogram (EEG) signal corresponding to the driver using a target camera and a multi-channel physiological signal acquisition device deployed in the cockpit. Perform wavelet packet decomposition and denoising on the EEG signal to obtain the signal processing result. Then, extract the cardiac interval feature from the ECG signal to obtain the feature extraction result. Finally, perform alignment processing on the facial image data to obtain the aligned image.

[0063] In this embodiment, the flowchart for predicting the driver's charging intention is as follows: Figure 2 As shown, this embodiment first requires the acquisition of the driver's facial image data, EEG (electroencephalogram), and ECG (electrocardiogram) signals using a high-definition camera and a multi-channel physiological signal acquisition device deployed in the cockpit. Notably, this embodiment uses 3840×2160 ultra-high-definition video recording to capture the driver's visual representation data. The camera is placed inside the vehicle, 0.6m in front of the subject, with a sampling rate of 30fps. ECG signal acquisition uses disposable patch electrodes, V1-V6 unipolar chest leads, with a sampling rate of 1000Hz. In this embodiment, considering the need to use hands and feet for driving operations during the experiment, the electrodermal signal electrodes are placed on the right shoulder to ensure moderate sweat gland density and low interference, with a sampling frequency of 100Hz. EEG signal acquisition uses a 10-20 system electrode placement method, placed at the positions of ten channels: Fp1, Fp2, F3, F4, T7, T8, P3, P4, Cz, and A1 / A2, employing flexible dry electrode technology with an electrode impedance <5kΩ.

[0064] It is worth mentioning that, according to the above-mentioned multi-source data acquisition method, the baseline data of the driver's four modalities in the resting state can be obtained by detecting for 5 minutes. Then, when the test driver's electric vehicle is prompted by the voice that the remaining battery is too low, the data of the driver's four modalities within 10 minutes after being stimulated are recorded.

[0065] Specifically, facial image data, electrocardiogram (ECG) signals, and electroencephalogram (EEG) signals corresponding to the driver are collected through target cameras and multi-channel physiological signal acquisition devices deployed in the cockpit. This can include: using a preset recording camera to record video of the driver's visual representation data at a preset ultra-high-definition resolution to obtain facial image data corresponding to the driver; placing the preset recording camera inside the vehicle at a preset distance from the driver; using preset disposable patch electrodes and employing a chest lead method with a preset sampling rate to collect ECG signals corresponding to the driver; and then using preset standard electrodes placed in specific channel positions and employing flexible dry electrode technology to collect EEG signals corresponding to the driver. Among these, the facial image data, ECG signals, and EEG signals include data corresponding to the driver in a resting state and data corresponding to the driver in a stimulated state.

[0066] In this embodiment, during the signal preprocessing stage, the acquired EEG signal undergoes wavelet packet decomposition for noise reduction, the ECG signal undergoes RR interval feature extraction, and the facial image undergoes micro-expression action unit encoding. Specifically, the signal preprocessing module utilizes the Pan-Tompkins algorithm R-wave detection subunit to sequentially perform: bandpass filtering: a 5th-order Butterworth filter is used to bandpass filter the EEG signal, and the transfer function of the 5th-order Butterworth filter is shown below:

[0067] ;

[0068] in, The transfer function of the digital filter; Let be a complex variable in the Z-transform.

[0069] Subsequently, in this embodiment of the application, the EEG signal needs to be differentiated to enhance the QRS slope:

[0070] ;

[0071] Then, the signal is squared:

[0072] ;

[0073] in, The result is the electrocardiogram signal after squaring.

[0074] The signal is then processed using a sliding integral window (150ms in length):

[0075] ;

[0076] in, The output signal is an integral signal; The window length of the sliding window.

[0077] Furthermore, dynamic threshold updates are used to distinguish between QRS groups and noise.

[0078] Finally, a 1-50Hz digital filter was used to eliminate high-frequency noise and baseline drift in the EEG signal.

[0079] In this embodiment, to address the issue of EEG signal contamination caused by eye movement (EOG) and muscle activity (EMG) in the driving environment, this application embodiment utilizes independent component analysis (ICA) for EEG artifact removal. Let the EEG observation signal be... (N is the number of electrodes, T is the number of sampling points), the ICA model is expressed as:

[0080] ;

[0081] in, The observed signal is obtained after filtering. , It is a mixed matrix. These are independent source signals, including neural activity and artifacts. In practical applications, the mixing matrix... Unknown, the goal is to... To estimate the source signal and mixture matrix .

[0082] It is worth mentioning that before using ICA to process the signal, the embodiments of this application first need to center the observed signal and remove the mean:

[0083] ;

[0084] Subsequently, by performing a whitening transformation on the signal, the covariance matrix of the signal is transformed into an identity matrix, thereby eliminating the correlation between signals. The signal whitening process first requires calculating the signal's covariance matrix:

[0085] ;

[0086] in, The observed signal after centralization; For expectations; for The covariance matrix.

[0087] Secondly, eigenvalue decomposition is performed on the covariance matrix to obtain eigenvalues ​​and eigenvectors:

[0088] ;

[0089] in, It is a diagonal matrix containing the eigenvalues ​​of the covariance matrix. It is the eigenvector matrix.

[0090] Finally, the signal is whitened using the reciprocal of the square root of the eigenvalues:

[0091] ;

[0092] in, This is the observed signal after whitening.

[0093] It is worth mentioning that after whitening the signal, this embodiment of the application needs to maximize the non-Gaussianity of the source signal using FastICA: firstly, a weight matrix needs to be initialized. Iteratively update this weight matrix until convergence:

[0094] ;

[0095] in, This is the updated separation matrix; The separation matrix; It is a nonlinear function; nonlinear function The derivative of, and .

[0096] Then, the updated weight matrix is ​​normalized to ensure that the norm of its row vectors is 1:

[0097] ;

[0098] in, It is a norm.

[0099] Then repeat the above steps until convergence.

[0100] Subsequently, embodiments of this application require the use of the obtained weight matrix. The independent components are extracted, and the final independent component signal can be represented as:

[0101] ;

[0102] in, It is an estimated independent source signal. It is an observation signal that has been whitened.

[0103] In this embodiment, for the acquired video, this application embodiment needs to use a sliding window to capture continuous T frames (default T=16, covering approximately 0.27 seconds of dynamic changes), and standardize the image size to... Pixels, input image tensor dimensions Using the pre-trained 68-point facial landmark detection model Dlib, the keypoint coordinates are output. Calculate the affine matrix between the reference coordinate system (frontal face template) and the current frame. :

[0104] ;

[0105] in, This is the affine transformation matrix for the current frame; For the first detected in the current frame image Coordinates of key facial features; Using the frontal face template as a reference coordinate system, It is an L2 norm.

[0106] Subsequently, an alignment operation is performed on the affine transformation matrix to obtain an aligned facial image sequence; this aligned facial image sequence is represented as follows:

[0107] ;

[0108] in, The aligned facial image sequence; The input is a facial image; This is an affine distortion transformation;

[0109] Specifically, wavelet packet decomposition is performed on the EEG signal to denoise it, yielding the signal processing result. Then, the ECG signal is subjected to intercardiac interval feature extraction, yielding the feature extraction result. Finally, the face image data is aligned to obtain the aligned image. This process may include: using a preset digital filter to eliminate high-frequency noise and baseline drift in the EEG signal, obtaining the de-calibrated EEG signal; centering and removing the mean from the de-calibrated EEG signal to obtain the EEG signal to be processed; determining the covariance matrix corresponding to the EEG signal to be processed; performing eigenvalue decomposition on the covariance matrix to obtain the corresponding eigenvalues; then applying a whitening transformation rule and based on the eigenvalues ​​to whiten the covariance matrix, obtaining the whitening result, and determining the initial weight matrix corresponding to the whitening result; normalizing the initial weight matrix to obtain the target weight matrix, and then using the target weight matrix and the unique... The whitening process is processed using component analysis to obtain signal processing results. A pre-defined ECG signal processing algorithm is used to sequentially perform bandpass filtering, differential processing to enhance slope, signal squaring, sliding integral window processing, and dynamic threshold updating to obtain initial processing results. The initial processing results are then used to extract cardiac interval features to obtain feature extraction results. A pre-defined sliding window is used to crop images from the video captured by the target camera to obtain initial images. Each initial image is then normalized to obtain standardized images. These standardized images are then input into a pre-defined facial keypoint detection model to obtain keypoint coordinates. Based on the keypoint coordinates and a pre-defined template reference coordinate system, an affine matrix corresponding to the standardized images is determined. The standardized images are then aligned based on the affine matrix and the pre-defined image tensor dimension to obtain aligned images.

[0110] Step S12: Construct a three-level fusion architecture using a temporal alignment layer based on dynamic time warping, a feature crossover layer based on multi-head attention mechanism, and a decision integration layer based on confidence weighting. Then, perform feature fusion on the signal processing results, the feature extraction results, and the aligned image to obtain the feature fusion result.

[0111] In this embodiment, the physiological signals (EEG, ECG) need to be converted into tabular features. Mapped to embedding vectors through a linear layer (Aligned with the number of image feature channels). Subsequently, feature extraction is performed on the aligned image, mainly including a 3D ResNet backbone network and a TabAttention module. The 3DResNet backbone network structure consists of: each residual block containing two 3D convolutional layers (kernel size 3×3×3), batch normalization (BN), ReLU activation, and a TabAttention module.

[0112] ;

[0113] in, ; This is the output feature map after processing by the TabAttention module.

[0114] Furthermore, the TabAttention module includes Channel Attention (CAM): for the feature map Perform global average pooling and max pooling to generate channel descriptors. Then merge the table embedding:

[0115] ;

[0116] Finally, channel weighting is performed:

[0117] ;

[0118] in, This is element-wise multiplication.

[0119] Subsequently, spatial attention (SAM) is used to process the signal:

[0120] First of all, Average pooling and max pooling along the channel dimension generate spatial descriptors. Embedded tables:

[0121] ;

[0122] Finally, spatial weighting is performed:

[0123] ;

[0124] Furthermore, time attention (TAM) is used to process the signal:

[0125] Subsequently, Pooling along the spatial dimension generates a temporal descriptor. Embedded tables:

[0126] ;

[0127] Finally, time-weighted calculation is performed:

[0128] ;

[0129] in, The input feature map is the aligned sequence of facial images. ; This is the channel attention weight vector; This refers to the channel descriptors obtained by global average pooling of the input feature map; This refers to the channel descriptors obtained by global max pooling of the input feature map; Embed vectors for tables; It is a multilayer perceptron; For activation functions; This is element-wise multiplication; This is the channel attention feature map after channel attention weighting; This is the spatial attention weight vector; It is a convolutional layer; is the spatial descriptor obtained by average pooling along the channel dimension; is the spatial descriptor obtained by max pooling along the channel dimension; For splicing operations; For reshaping operation; This is the spatial attention feature map after spatial attention weighting; This is the temporal attention weight vector; This is a multi-head self-attention mechanism; This is the time descriptor obtained by average pooling along the spatial dimension; This is the time descriptor obtained by max pooling along the spatial dimension; This is a time attention feature map after time attention weighting.

[0130] It is worth mentioning that the 3DResNet backbone network layer configuration is as follows:

[0131] Input layer: 3D convolution (kernel 7×7×7), output dimension 56×56×8×64.

[0132] Residual block 1: 3×3×3 convolution + TabAttention, output dimension 56×56×8×64.

[0133] Residual block 2: 3×3×3 convolution + TabAttention, output dimension 28×28×4×128.

[0134] Residual block 3: 3×3×3 convolution + TabAttention, output dimension 14×14×2×256.

[0135] Residual block 4: 3×3×3 convolution + TabAttention, output dimension 7×7×1×512.

[0136] In this embodiment of the application, the output layer can perform dual-task AU intensity regression for the final features. Perform multi-scale temporal pooling:

[0137] ;

[0138] in, The feature descriptor is obtained after multi-scale temporal pooling; For splicing operations; For global average pooling; This is for global max pooling.

[0139] It's worth noting that task branch 1 is used for AU (Auction Unit) existence detection, a binary classification task that determines whether a certain AU (Auction Unit) exists at the current moment. Subsequently, a sigmoid activation function is used for prediction, mapping the output to 0 or 1 to indicate the presence or absence of the AU.

[0140] ;

[0141] in, This is the existence prediction vector for AU; and These are the weight matrix and bias vector used for the classification task, respectively; This is the Sigmoid activation function.

[0142] Task branch 2 is used to perform AU strength regression:

[0143] ;

[0144] in, This is the AU intensity prediction vector; It is the ReLU activation function; and The weight matrix and bias vector are used for the regression task, respectively.

[0145] Subsequently, the output results are spliced ​​using a sliding window prediction method, with consecutive time points... The prediction result is , indicating the first An AU in time If the intensity value is given, then the AU intensity time series matrix is ​​defined as:

[0146] ;

[0147] in, For the first Each action unit at time point The predicted AU intensity; This is the AU intensity time series matrix.

[0148] Then, in this embodiment of the application, a Savitzky-Golay filter needs to be applied to the time series of each AU. The processing is performed, and its mathematical form is:

[0149] ;

[0150] in, For the first A smoothed AU intensity temporal feature; These are the filter weighting coefficients.

[0151] In one specific implementation, the window length polynomial order Weighting coefficient Determined by the least squares method, it satisfies:

[0152] ;

[0153] in, The relative index within the window; Let the order be the order of the polynomial. This is the Kronecker delta function.

[0154] In one specific implementation, the specific coefficient is: , , , , .

[0155] Subsequently, in this embodiment of the application, the intensity time series of AUs obtained by sliding window and physiological signal embedding vector are combined with the result smoothed by Savitzky-Golay filter to finally obtain the intensity change of each AU in the time series. This intensity data will be used as part of the feature fusion input to aid in subsequent charging intention prediction.

[0156] Furthermore, in this embodiment of the application, the features need to be fused: First, the signal is processed using a timing alignment layer and a dynamic time warping (DTW) algorithm.

[0157] In one specific implementation, let the set of multimodal signals be... , of which The temporal characteristics of each mode are:

[0158] ;

[0159] in, For feature dimension, This refers to the duration of the time.

[0160] Subsequently, embodiments of this application require consideration of any two modes. and Construct the cumulative cost matrix :

[0161] ;

[0162] in, For the cumulative cost matrix at position The value; and These are any two modes; Let the Euclidean distance be initialized. .

[0163] Then, the embodiments of this application can use backtracking to find the minimum cost path. Thus making Minimize and output aligned temporal features (Unified Time Base) );

[0164] Furthermore, a feature cross-layer is used to stack the aligned multimodal features into a three-dimensional tensor by channel. ,in: ; ; ;

[0165] Subsequently, three-dimensional convolution kernels can be applied in the embodiments of this application. Perform feature crossing:

[0166] ;

[0167] in, The feature map after feature crossing; Kernel size (typical value: ), This refers to the circular index within the three dimensions of the convolution kernel; To input tensor The current position coordinates in three dimensions.

[0168] Furthermore, a multi-head self-attention mechanism is introduced to calculate the cross-modal feature association weights:

[0169] ;

[0170] in, These are query, key, and value matrices, respectively. Depend on Generated by linear transformation, The scaling factor. The tensor after the crossover. ,Will Feature dimensions mapped to number of categories Above, we obtain a length of The vector, representing the score for each category, can be obtained after passing through a fully connected layer:

[0171] ;

[0172] in, For each modality output, a category score is given; It is a fully connected layer.

[0173] Finally, in this embodiment, the obtained score can be standardized using a softmax activation function to convert it into a final classification probability. .

[0174] Specifically, a three-level fusion architecture is constructed using a temporal alignment layer based on dynamic time warping, a feature cross-layer based on multi-head attention, and a decision ensemble layer based on confidence weighting. The signal processing results, feature extraction results, and aligned image are then fused to obtain the feature fusion result. This can include: determining the temporal alignment layer based on dynamic time warping, constructing a feature cross-layer based on multi-head attention, and then constructing a decision ensemble layer based on confidence weighting, thus constructing a three-level fusion architecture based on the temporal alignment layer, feature cross-layer, and decision ensemble layer; converting the signal processing results and feature extraction results into tabular signal features, and using a linear mapping layer to map the signal features into embedding vectors; the number of channels in the embedding vectors is the same as the number of channels in the aligned image; and then... The input is fed into a preset 3D residual network. The channel attention mechanism in the preset 3D residual network is used to perform global average pooling and global max pooling on the aligned image to obtain corresponding channel descriptors. Each channel descriptor is then fused with the embedding vector to obtain channel weight coefficients. Based on the channel weight coefficients, the aligned image is then weighted along the channel dimension to obtain a channel-enhanced feature map. The spatial attention mechanism in the preset 3D residual network is used to perform average pooling and max pooling along the channel dimension on the channel-enhanced feature map to obtain corresponding spatial descriptors. Each spatial descriptor is then fused with the embedding vector to obtain spatial weight coefficients. These spatial weight coefficients are then used to weight the channel-enhanced feature map along the spatial dimension to obtain a spatial-enhanced feature map.

[0175] Subsequently, in this embodiment, the temporal attention mechanism in a preset three-dimensional residual network is used to perform pooling operations on the spatial augmented feature map along the spatial dimension to obtain a temporal descriptor. The temporal descriptor is then fused with the embedding vector to obtain temporal weight coefficients. These temporal weight coefficients are then used to weight and adjust the spatial augmented feature map along the temporal dimension to obtain a multidimensional feature map. The preset three-dimensional residual network includes several sequentially connected residual blocks. Each residual block includes two three-dimensional convolutional layers, a batch normalization unit, a nonlinear activation function, and an attention enhancement unit. Multi-scale temporal pooling is performed on the multidimensional feature map to obtain feature information at different time scales. The presence of facial expression / action units is detected using a first preset task branch and a nonlinear classification function, resulting in a binary classification presence detection result. The presence detection result is used to determine the presence of each time step. Does a specific facial expression action unit exist? The second preset task branch is used to perform facial expression action unit intensity regression on the feature information to obtain the corresponding intensity regression result. The intensity regression result is used to characterize the activation level of the facial expression action unit at each time step. The first preset task branch and the second preset task branch are parallel processing branches. Several intensity regression results are concatenated into an intensity time series matrix, and a sliding window smoothing filter based on least squares fitting is used to smooth the intensity time series corresponding to each facial expression action unit in the intensity time series matrix to obtain the filtering result. The filter weight coefficients corresponding to the filtering result are determined based on the preset window length, and the intensity time series is convolved based on the filter weight coefficients to obtain the smoothed unit intensity change curve. The feature fusion result is determined based on the unit intensity change curve, the multidimensional feature map, and the existence detection result.

[0176] Step S13: Align the feature fusion result with a multimodal signal temporal reference using the temporal alignment layer in the three-level fusion architecture to obtain an alignment result. Then, perform cross-modal feature association enhancement processing on the alignment result using the three-dimensional convolution and multi-head self-attention mechanism in the feature cross-layer to obtain an enhancement result. Finally, perform emotion category judgment and confidence weighting processing on the enhancement result using the decision integration layer to obtain unprocessed information including emotion state, real-time location, and charging pile status.

[0177] In this embodiment, the present application requires utilizing the probability distribution of emotion categories output by each modality classifier in the decision ensemble layer. And convert it to a BPA function:

[0178] ; ;

[0179] in, For the first The mode is assigned to the first Basic probability assignments for each emotion category; This is the sharpening factor (default value 2.5). Indicates the first Emotion-related propositions;

[0180] Subsequently, the modality is defined. and Conflict measurement between:

[0181] ;

[0182] Overall conflict factors: ;

[0183] in, For modality and Conflict measurement between them; It is the overall conflict factor.

[0184] It is worth mentioning that, if Using traditional Dempster rules:

[0185] ;

[0186] like Activate the weighted combination strategy and introduce modal reliability weights. :

[0187] ;

[0188] in, , For modality Average conflict with other modalities.

[0189] Finally, the resulting emotion category is: It also outputs a confidence score.

[0190] Specifically, the temporal alignment layer in the three-level fusion architecture aligns the feature fusion results with the multimodal signal temporal reference, obtaining the alignment result. Then, the 3D convolution and multi-head self-attention mechanism in the feature cross-layer perform cross-modal feature association enhancement processing on the alignment result, obtaining the enhanced result. Finally, the decision ensemble layer performs emotion category judgment and confidence weighting on the enhanced result, obtaining unprocessed information including emotional state, real-time location, and charging pile status. This can include: constructing a cumulative cost matrix using the temporal alignment layer in the three-level fusion architecture and based on a preset dynamic time warping algorithm and the feature fusion results; then processing the cumulative cost matrix using a backtracking method to obtain the minimum cost path, and determining the alignment result based on the minimum cost path; and finally, performing channel-wise stacking operation on the alignment results using the feature cross-layer in the three-level fusion architecture. A three-dimensional tensor is obtained, and a pre-defined three-dimensional convolutional kernel is used to perform feature crossing on the three-dimensional tensor to obtain the feature crossing result. A multi-head self-attention mechanism is used to determine the cross-modal feature association weights corresponding to the feature crossing result. Then, a fully connected layer is used to determine the category score corresponding to each modality based on the cross-modal feature association weights. Several modality classifiers and activation functions are used to convert the category scores into classification probabilities. The decision integration layer in the three-level fusion architecture is used to convert each classification probability into a probability allocation function. The probability allocation function is used to define the conflict metric between modalities to determine the conflict factor. Then, the corresponding combination strategy is determined based on the conflict factor. The combination strategy is used to determine the emotion category and the corresponding confidence score. Based on the emotion category and confidence score, the information to be processed, including emotional state, real-time location, and charging pile status, is determined.

[0191] Step S14: Process the information to be processed using a spatiotemporal joint prediction model to obtain a charging intention prediction result, and determine the target charging pile and route planning result corresponding to the driver based on the charging intention prediction result.

[0192] In this embodiment, it is assumed that the current region exists. The locations of the charging stations are set as follows: , of which The coordinates of the charging piles are And obtain its availability status in real time. (0 indicates unavailable, 1 indicates available). Combined with real-time vehicle location. (Acquired via vehicle GPS), calculate the Euclidean distance between the driver and each charging station:

[0193] ;

[0194] Subsequently, the Euclidean distances of each charging station were obtained. and status information And the temporal features of the emotional state output by the multi-level feature fusion module. spliced ​​as Feature vector at time step And the expression is as follows:

[0195] ;

[0196] Then, Feature vectors at consecutive time points Consolidate into input tensors based on time windows ;

[0197] in, This includes emotional characteristics, the distance to each charging station, and its status;

[0198] Furthermore, spatiotemporal joint prediction is performed using the TimeMixer++ model: First, the input is decomposed into trend terms. Periodic terms and residuals :

[0199] ;

[0200] in, Long-term trends are extracted using moving average filtering. Periodic features are extracted using Fourier transform. Then, a multi-head attention mechanism introduced by S74 is used to capture cross-modal correlations, finally outputting the charging station selection probability distribution.

[0201] ;

[0202] in, For the number of attention heads, These are learnable weights; For the first The output features of each attention head.

[0203] Subsequently, based on the predicted probability and real-time charging station status Filter available charging stations and optimize routes:

[0204] ;

[0205] in, For emotion categories Decision weights, This is the distance attenuation coefficient (default value 0.1); the final output is the target charging station number. and its coordinates .

[0206] Specifically, a spatiotemporal joint prediction model is used to process the information to be processed to obtain the charging intention prediction result. Based on the charging intention prediction result, the target charging pile and the path planning result corresponding to the driver are determined. This can include: obtaining the location and real-time availability status of each charging pile in the current area, and determining the charging pile location set based on each charging pile location, and determining the Euclidean distance between the driver and each charging pile based on the current real-time vehicle location and the charging pile location set; integrating the information to be processed, the driver's real-time location, the charging pile location and the real-time availability status according to the time window to obtain the input tensor, and using a preset temporal mixture model to perform moving average filtering on the input tensor to obtain the long-term trend, and performing Fourier transform on the input tensor to obtain periodic features, and then using a multi-head attention mechanism to capture cross-modal associations of the input tensor to obtain the charging pile selection probability distribution result; determining the target charging pile based on the charging pile selection probability distribution result and the real-time availability status, and determining the location of the charging pile corresponding to the target charging pile based on the charging pile number of the target charging pile, and then determining the path planning result between the vehicle and the target charging pile based on the charging pile location and the current real-time vehicle location.

[0207] In this embodiment, the historical charging behavior data needs to be introduced. (Dimension:) Update model parameters through online learning:

[0208] ;

[0209] in, This is the set of all trainable parameters in the spatiotemporal joint prediction model; For parameter update operations; Learning rate (default value) ); For gradient operators; The cross-entropy loss function; Choose a probability distribution for charging stations; For the first The true situation of each training sample.

[0210] Specifically, after determining the target charging station and route planning result corresponding to the driver based on the charging intention prediction result, the process may further include: calculating the loss of the charging intention prediction result using the cross-entropy loss function, obtaining the loss calculation result, and adjusting the model parameters corresponding to the spatiotemporal joint prediction model based on the loss calculation result to obtain the first model to be processed; using historical charging behavior data and a preset learning rate to perform online learning on the first model to be processed to obtain the second model to be processed, and determining the decision weight and distance decay coefficient for the emotion category, so as to use the obtained decision weight and distance decay coefficient to process the second model to be processed to obtain a new spatiotemporal joint prediction model.

[0211] As can be seen from the above, the embodiments of this application first need to collect facial image data, electrocardiogram (ECG) signals, and electroencephalogram (EEG) signals corresponding to the driver through a target camera and a multi-channel physiological signal acquisition device deployed in the cockpit. Wavelet packet decomposition is then performed on the EEG signals to denoise them, yielding signal processing results. Next, the ECG signals undergo intercardiac interval feature extraction to obtain feature extraction results. Finally, the facial image data is aligned to obtain an aligned image. Secondly, a three-level fusion architecture is constructed using a temporal alignment layer based on dynamic time warping, a feature cross-interaction layer based on multi-head attention, and a decision integration layer based on confidence weighting. Feature fusion is then performed on the signal processing results, feature extraction results, and the aligned image. The process involves obtaining feature fusion results. Then, the temporal alignment layer in the three-level fusion architecture aligns the feature fusion results with the multimodal signal temporal reference, yielding an alignment result. Next, the 3D convolution and multi-head self-attention mechanism in the feature cross-layer enhance the alignment result through cross-modal feature association, resulting in an enhanced result. Finally, the decision ensemble layer performs emotion category judgment and confidence weighting on the enhanced result, obtaining unprocessed information including emotional state, real-time location, and charging pile status. Finally, a spatiotemporal joint prediction model processes the unprocessed information to obtain charging intention prediction results. Based on these predictions, the target charging pile and route planning results corresponding to the driver are determined. This improves the efficiency of predicting driver charging intentions, thereby enhancing the stability of the power grid.

[0212] Accordingly, see Figure 3 As shown, this application also provides a driver charging intention prediction device, comprising:

[0213] The feature extraction result determination module 11 is used to collect facial image data, electrocardiogram (ECG) signals and electroencephalogram (EEG) signals corresponding to the driver through a target camera and a multi-channel physiological signal acquisition device deployed in the cockpit, and to perform wavelet packet decomposition and noise reduction on the EEG signals to obtain signal processing results. Then, the ECG signals are subjected to cardiac interval feature extraction to obtain feature extraction results, and the facial image data is aligned to obtain an aligned image.

[0214] The feature fusion module 12 is used to construct a three-level fusion architecture using a temporal alignment layer based on dynamic time warping, a feature cross-layer based on multi-head attention mechanism, and a decision integration layer based on confidence weighting, and to perform feature fusion on the signal processing result, the feature extraction result, and the aligned image to obtain the feature fusion result;

[0215] The pending information determination module 13 is used to align the feature fusion result with the multimodal signal temporal reference using the temporal alignment layer in the three-level fusion architecture to obtain the alignment result, and to perform cross-modal feature association enhancement processing on the alignment result using the three-dimensional convolution and multi-head self-attention mechanism in the feature cross layer to obtain the enhancement result. Then, the decision integration layer is used to perform emotion category judgment and confidence weighting processing on the enhancement result to obtain pending information including emotional state, real-time location and charging pile status.

[0216] The charging intention prediction result generation module 14 is used to process the information to be processed using a spatiotemporal joint prediction model to obtain the charging intention prediction result, so as to determine the target charging pile and route planning result corresponding to the driver based on the charging intention prediction result.

[0217] In some specific embodiments, the feature extraction result determination module 11 may specifically include:

[0218] A facial image data acquisition unit is used to record video of the driver's visual representation data using a preset recording camera, meeting preset ultra-high definition resolution conditions, to obtain facial image data corresponding to the driver; the preset recording camera is placed inside the vehicle at a preset distance from the driver;

[0219] The electrocardiogram (ECG) signal acquisition unit is used to acquire ECG signals corresponding to the driver using a preset disposable patch electrode and a chest lead method with a preset sampling rate. Then, it acquires EEG signals corresponding to the driver using a preset standard electrode placed in a specific channel position and flexible dry electrode technology. The facial image data, ECG signals, and EEG signals include data corresponding to the driver in a resting state and data corresponding to the driver in a stimulated state.

[0220] In some specific embodiments, the feature extraction result determination module 11 may specifically include:

[0221] The covariance matrix determination unit is used to use a preset digital filter to eliminate high-frequency noise and baseline drift in the EEG signal to obtain the eliminated EEG signal, and to perform a centering and mean removal operation on the eliminated EEG signal to obtain the EEG signal to be processed, and then determine the covariance matrix corresponding to the EEG signal to be processed.

[0222] The whitening processing result determination unit is used to perform eigenvalue decomposition on the covariance matrix to obtain the corresponding eigenvalues, and then use the whitening transformation rule and the eigenvalues ​​to perform whitening processing on the covariance matrix to obtain the whitening processing result, so as to determine the initial weight matrix corresponding to the whitening processing result.

[0223] The signal processing result determination unit is used to normalize the initial weight matrix to obtain a target weight matrix, and then use the target weight matrix and independent component analysis technology to process the whitening result to obtain the signal processing result.

[0224] The feature extraction result determination unit is used to perform bandpass filtering, differential processing to enhance the slope, signal squaring, sliding integral window processing and dynamic threshold update processing on the electrocardiogram signal in sequence using a preset electrocardiogram signal processing algorithm to obtain an initial processing result, and to extract the cardiac interval feature from the initial processing result to obtain the feature extraction result.

[0225] The key point coordinate determination unit is used to extract images from the video captured by the target camera using a preset sliding window to obtain initial images, and to perform image size standardization processing on each initial image to obtain standardized images. Then, each standardized image is input into a preset facial key point detection model to obtain key point coordinates.

[0226] The aligned image determination unit is used to determine the affine matrix corresponding to the standardized image based on the key point coordinates and the preset template reference coordinate system, and to align the standardized image based on the affine matrix and the preset image tensor dimension to obtain the aligned image.

[0227] In some specific embodiments, the feature fusion module 12 may specifically include:

[0228] The three-level fusion architecture building unit is used to determine the temporal alignment layer based on dynamic time warping, construct the feature cross layer based on multi-head attention mechanism, and then construct the decision integration layer based on confidence weighting, so as to construct the three-level fusion architecture based on the temporal alignment layer, the feature cross layer and the decision integration layer.

[0229] The signal feature mapping unit is used to convert the signal processing result and the feature extraction result into signal features in tabular form, and to map the signal features into embedding vectors using a linear mapping layer; the number of channels of the embedding vectors is the same as the number of channels corresponding to the aligned image;

[0230] The channel descriptor determination unit is used to input the aligned image into a preset three-dimensional residual network, and to use the channel attention mechanism in the preset three-dimensional residual network to perform global average pooling and global max pooling on the aligned image to obtain corresponding channel descriptors. The channel descriptors are then fused with the embedding vector to obtain channel weight coefficients. Finally, the aligned image is weighted and adjusted in the channel dimension based on the channel weight coefficients to obtain a channel-enhanced feature map.

[0231] The spatial augmented feature map determination unit is used to perform average pooling and max pooling on the channel augmented feature map along the channel dimension using the spatial attention mechanism in the preset three-dimensional residual network to obtain corresponding spatial descriptors, and to fuse each spatial descriptor with the embedding vector to obtain spatial weight coefficients, so as to use the spatial weight coefficients to perform weighted adjustment on the channel augmented feature map in the spatial dimension to obtain the spatial augmented feature map.

[0232] The temporal weight coefficient determination unit is used to perform pooling operations on the spatial augmented feature map along the spatial dimension using the temporal attention mechanism in the preset three-dimensional residual network to obtain a temporal descriptor, and then fuse the temporal descriptor with the embedding vector to obtain temporal weight coefficients. The temporal weight coefficients are then used to weight and adjust the spatial augmented feature map along the temporal dimension to obtain a multi-dimensional feature map. The preset three-dimensional residual network includes several sequentially connected residual blocks. Each residual block includes two three-dimensional convolutional layers, a batch normalization processing unit, a nonlinear activation function, and an attention enhancement unit.

[0233] An existence detection unit is used to perform multi-scale temporal pooling processing on the multi-dimensional feature map to obtain feature information including different time scales, and to perform existence detection of facial expression action units on the feature information using a first preset task branch and a nonlinear classification function to obtain an existence detection result as a binary classification result; the existence detection result is used to determine whether a specific facial expression action unit exists at each time step.

[0234] The intensity regression result determination unit is used to perform intensity regression of facial expression action units on the feature information using the second preset task branch to obtain the corresponding intensity regression result; the intensity regression result is used to characterize the activation degree of facial expression action units at each moment; the first preset task branch and the second preset task branch are parallel processing branches;

[0235] The filtering result determination unit is used to concatenate several intensity regression results into an intensity time series matrix, and use a polynomial smoothing filter based on least squares fitting to perform sliding window smoothing filtering on the intensity time series corresponding to each facial expression action unit in the intensity time series matrix to obtain the filtering result.

[0236] The unit intensity change curve determination unit is used to determine the filter weight coefficients corresponding to the filtering result based on a preset window length, and to perform convolution operation on the intensity time series based on the filter weight coefficients to obtain a smoothed unit intensity change curve. The unit intensity change curve, the multidimensional feature map, and the existence detection result are used to determine the feature fusion result.

[0237] In some specific embodiments, the information to be processed determination module 13 may specifically include:

[0238] The minimum cost path determination unit is used to construct a cumulative cost matrix by utilizing the temporal alignment layer in the three-level fusion architecture and based on a preset dynamic time warping algorithm and the feature fusion result, and then process the cumulative cost matrix using a backtracking method to obtain the minimum cost path, so as to determine the alignment result based on the minimum cost path.

[0239] The feature cross result determination unit is used to perform channel-by-channel stacking operation on the alignment result using the feature cross layer in the three-level fusion architecture to obtain a three-dimensional tensor, and to perform feature cross on the three-dimensional tensor using a preset three-dimensional convolution kernel to obtain the feature cross result;

[0240] The category score determination unit is used to determine the cross-modal feature association weights corresponding to the feature cross-results using a multi-head self-attention mechanism, and then use a fully connected layer to determine the category scores corresponding to each modality based on the cross-modal feature association weights, so as to convert the category scores into classification probabilities using several modality classifiers and activation functions;

[0241] The conflict factor determination unit is used to convert each classification probability into a probability allocation function using the decision integration layer in the three-level fusion architecture, and to define a conflict metric between modalities using the probability allocation function, so as to determine a conflict factor based on the conflict metric, and then to determine a corresponding combination strategy based on the conflict factor, so as to determine the emotion category and the corresponding confidence score using the combination strategy, and to determine the unprocessed information including emotional state, real-time location and charging pile status based on the emotion category and the confidence score.

[0242] In some specific embodiments, the charging intention prediction result generation module 14 may specifically include:

[0243] The Euclidean distance determination unit is used to obtain the location and real-time availability of each charging pile in the current area, and to determine a set of charging pile locations based on the location of each charging pile, so as to determine the Euclidean distance between the driver and each charging pile based on the current real-time location of the vehicle and the set of charging pile locations.

[0244] The periodic feature determination unit is used to integrate the information to be processed, the driver's real-time location, the charging pile location, and the real-time availability status according to a time window to obtain an input tensor. The input tensor is then subjected to moving average filtering using a preset temporal mixture model to obtain a long-term trend. The input tensor is then subjected to Fourier transform to obtain periodic features. Finally, a multi-head attention mechanism is used to capture cross-modal associations of the input tensor to obtain the charging pile selection probability distribution result.

[0245] The path planning result determination unit is used to determine the target charging pile based on the charging pile selection probability distribution result and the real-time availability status, and to determine the location of the charging pile corresponding to the target charging pile based on the charging pile number corresponding to the target charging pile. Then, it determines the path planning result between the vehicle and the target charging pile based on the charging pile location and the current real-time location of the vehicle.

[0246] In some specific embodiments, the driver charging intention prediction device may further include:

[0247] The first model to be processed determination unit is used to perform loss calculation on the charging intention prediction result using the cross-entropy loss function, obtain the loss calculation result, and adjust the model parameters corresponding to the spatiotemporal joint prediction model based on the loss calculation result to obtain the first model to be processed.

[0248] The distance decay coefficient determination unit is used to perform online learning on the first model to be processed using historical charging behavior data and a preset learning rate to obtain a second model to be processed, and to determine the decision weight and distance decay coefficient for the emotion category, so as to process the second model to be processed using the obtained decision weight and distance decay coefficient to obtain a new spatiotemporal joint prediction model.

[0249] Furthermore, embodiments of this application also disclose an electronic device, Figure 4This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the driver charging intention prediction method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0250] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0251] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0252] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the driver charging intention prediction method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0253] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned driver charging intention prediction method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0254] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0255] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0256] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0257] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0258] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for predicting a driver's charging intention, characterized in that, include: The system collects facial image data, electrocardiogram (ECG) signals, and electroencephalogram (EEG) signals corresponding to the driver by using a target camera and a multi-channel physiological signal acquisition device deployed in the cockpit. The EEG signals are then decomposed into wavelet packets to denoise the data, and the signal processing results are obtained. The ECG signals are then subjected to intercardiac interval feature extraction, and the feature extraction results are obtained. Finally, the facial image data is aligned to obtain an aligned image. A three-level fusion architecture is constructed using a temporal alignment layer based on dynamic time warping, a feature crossover layer based on multi-head attention mechanism, and a decision integration layer based on confidence weighting. The signal processing results, the feature extraction results, and the aligned image are then fused to obtain the feature fusion result. The feature fusion result is aligned with the multimodal signal temporal reference using the temporal alignment layer in the three-level fusion architecture to obtain the alignment result. Then, the alignment result is enhanced by cross-modal feature association using the three-dimensional convolution and multi-head self-attention mechanism in the feature cross-layer to obtain the enhancement result. Finally, the enhancement result is processed by emotion category judgment and confidence weighting using the decision integration layer to obtain unprocessed information including emotion state, real-time location, and charging pile status. The information to be processed is processed using a spatiotemporal joint prediction model to obtain a charging intention prediction result, and the target charging pile and route planning result corresponding to the driver are determined based on the charging intention prediction result.

2. The driver charging intention prediction method according to claim 1, characterized in that, The process involves acquiring facial image data, electrocardiogram (ECG) signals, and electroencephalogram (EEG) signals corresponding to the driver through a target camera and multi-channel physiological signal acquisition device deployed in the cockpit, including: A preset recording camera is used to record the driver's visual representation data to meet the preset ultra-high-definition resolution conditions, thereby obtaining facial image data corresponding to the driver; the preset recording camera is placed inside the vehicle at a preset distance from the driver; The electrocardiogram (ECG) signal corresponding to the driver is acquired using a pre-set disposable patch electrode and a chest lead method with a pre-set sampling rate. Then, the electroencephalogram (EEG) signal corresponding to the driver is acquired using a pre-set standard electrode placed in a specific channel position and flexible dry electrode technology. The facial image data, the electrocardiogram (ECG) signal, and the electroencephalogram (EEG) signal include data corresponding to the driver in a resting state and data corresponding to the driver in a stimulated state.

3. The driver charging intention prediction method according to claim 1, characterized in that, The process involves performing wavelet packet decomposition to denoise the EEG signal, obtaining a signal processing result, then extracting intercardia features from the ECG signal, obtaining a feature extraction result, and finally aligning the facial image data to obtain an aligned image, including: The high-frequency noise and baseline drift in the EEG signal are eliminated using a preset digital filter to obtain the eliminated EEG signal. The eliminated EEG signal is then centered and the mean is removed to obtain the EEG signal to be processed. Then the covariance matrix corresponding to the EEG signal to be processed is determined. The covariance matrix is ​​decomposed into eigenvalues ​​to obtain the corresponding eigenvalues. Then, the covariance matrix is ​​whitened using the whitening transformation rule and based on the eigenvalues ​​to obtain the whitening result, thereby determining the initial weight matrix corresponding to the whitening result. The initial weight matrix is ​​normalized to obtain the target weight matrix. The whitening result is then processed using the target weight matrix and independent component analysis to obtain the signal processing result. The electrocardiogram signal is sequentially processed by a preset electrocardiogram signal processing algorithm, including bandpass filtering, differential processing to enhance the slope, signal squaring, sliding integral window processing, and dynamic threshold update processing, to obtain an initial processing result. The initial processing result is then subjected to cardiac interval feature extraction to obtain the feature extraction result. The video captured by the target camera is cropped using a preset sliding window to obtain initial images. The initial images are then normalized to obtain standardized images. The standardized images are then input into a preset facial key point detection model to obtain the coordinates of the key points. Based on the key point coordinates and the preset template reference coordinate system, the affine matrix corresponding to the standardized image is determined, and the standardized image is aligned based on the affine matrix and the preset image tensor dimension to obtain the aligned image.

4. The driver charging intention prediction method according to claim 1, characterized in that, The three-level fusion architecture is constructed using a temporal alignment layer based on dynamic time warping, a feature crossover layer based on multi-head attention mechanism, and a decision ensemble layer based on confidence weighting. The signal processing result, the feature extraction result, and the aligned image are then fused to obtain the feature fusion result, including: The temporal alignment layer is determined based on dynamic time warping, and the feature cross-layer is constructed based on multi-head attention mechanism. Then, the decision integration layer is constructed based on confidence weighting, so as to build a three-level fusion architecture based on the temporal alignment layer, the feature cross-layer and the decision integration layer. The signal processing results and the feature extraction results are converted into signal features in tabular form, and the signal features are mapped into embedding vectors using a linear mapping layer; the number of channels in the embedding vectors is the same as the number of channels in the aligned image. The aligned image is input into a preset three-dimensional residual network, and the channel attention mechanism in the preset three-dimensional residual network is used to perform global average pooling and global max pooling on the aligned image to obtain corresponding channel descriptors. The channel descriptors are then fused with the embedding vector to obtain channel weight coefficients. The aligned image is then weighted and adjusted in the channel dimension based on the channel weight coefficients to obtain a channel-enhanced feature map. The channel enhancement feature map is subjected to average pooling and max pooling along the channel dimension using the spatial attention mechanism in the preset three-dimensional residual network to obtain corresponding spatial descriptors. The spatial descriptors are then fused with the embedding vector to obtain spatial weight coefficients. These spatial weight coefficients are used to adjust the channel enhancement feature map in the spatial dimension to obtain a spatial enhancement feature map. The spatial augmented feature map is pooled along the spatial dimension using the temporal attention mechanism in the preset three-dimensional residual network to obtain a temporal descriptor. This temporal descriptor is then fused with the embedding vector to obtain temporal weight coefficients. These temporal weight coefficients are then used to weight and adjust the spatial augmented feature map along the temporal dimension to obtain a multi-dimensional feature map. The preset three-dimensional residual network includes several sequentially connected residual blocks. Each residual block includes two three-dimensional convolutional layers, a batch normalization unit, a non-linear activation function, and an attention enhancement unit. The multi-dimensional feature map is subjected to multi-scale temporal pooling to obtain feature information at different time scales. The presence of facial expression action units is detected by the feature information using a first preset task branch and a nonlinear classification function, resulting in a binary classification presence detection result. The presence detection result is used to determine whether a specific facial expression action unit exists at each time step. The feature information is subjected to facial expression action unit intensity regression using the second preset task branch to obtain the corresponding intensity regression result; the intensity regression result is used to characterize the activation degree of facial expression action units at each time step; the first preset task branch and the second preset task branch are parallel processing branches; Several intensity regression results are concatenated into an intensity time series matrix, and a sliding window smoothing filter based on least squares fitting is used to perform a smoothing filter on the intensity time series corresponding to each facial expression action unit in the intensity time series matrix to obtain the filtering result. The filter weight coefficients corresponding to the filtering result are determined based on the preset window length, and the intensity time series is convolved based on the filter weight coefficients to obtain the smoothed unit intensity change curve. The feature fusion result is determined based on the unit intensity change curve, the multidimensional feature map and the existence detection result.

5. The driver charging intention prediction method according to claim 1, characterized in that, The three-level fusion architecture utilizes a temporal alignment layer to align the feature fusion results with multimodal signal temporal references, obtaining an alignment result. Then, the feature cross-layer employs 3D convolution and multi-head self-attention mechanisms to enhance the alignment result through cross-modal feature association, yielding an enhanced result. Finally, the decision ensemble layer performs emotion category judgment and confidence weighting on the enhanced result, resulting in unprocessed information including emotional state, real-time location, and charging pile status, including: The time alignment layer in the three-level fusion architecture is used to construct a cumulative cost matrix based on a preset dynamic time warping algorithm and the feature fusion result. Then, the cumulative cost matrix is ​​processed using a backtracking method to obtain the minimum cost path, and the alignment result is determined based on the minimum cost path. The alignment result is stacked by channel using the feature cross-layer in the three-level fusion architecture to obtain a three-dimensional tensor, and the feature cross-layer is performed on the three-dimensional tensor using a preset three-dimensional convolution kernel to obtain the feature cross-layer result. A multi-head self-attention mechanism is used to determine the cross-modal feature association weights corresponding to the feature cross-results. Then, a fully connected layer is used to determine the category scores corresponding to each modality based on the cross-modal feature association weights. The category scores are then converted into classification probabilities using several modality classifiers and activation functions. The decision integration layer in the three-level fusion architecture converts each classification probability into a probability allocation function, and uses the probability allocation function to define a conflict metric between modalities. Based on the conflict metric, a conflict factor is determined, and then a corresponding combination strategy is determined based on the conflict factor. The combination strategy is used to determine the emotion category and the corresponding confidence score. Based on the emotion category and the confidence score, unprocessed information including emotional state, real-time location, and charging pile status is determined.

6. The driver charging intention prediction method according to claim 1, characterized in that, The process of using a spatiotemporal joint prediction model to process the information to be processed to obtain a charging intention prediction result, and then determining the target charging station and route planning result corresponding to the driver based on the charging intention prediction result, includes: The location and real-time availability of each charging pile in the current area are obtained, and a set of charging pile locations is determined based on the location of each charging pile. The Euclidean distance between the driver and each charging pile is determined based on the current real-time location of the vehicle and the set of charging pile locations. The information to be processed, the driver's real-time location, the charging pile location, and the real-time availability status are integrated according to a time window to obtain an input tensor. The input tensor is then subjected to moving average filtering using a preset temporal mixture model to obtain a long-term trend. The input tensor is then subjected to Fourier transform to obtain periodic features. Finally, a multi-head attention mechanism is used to capture cross-modal associations of the input tensor to obtain the charging pile selection probability distribution. The target charging pile is determined based on the charging pile selection probability distribution and the real-time availability status. The location of the charging pile corresponding to the target charging pile is determined based on the charging pile number of the target charging pile. Then, the path planning result between the vehicle and the target charging pile is determined based on the charging pile location and the current real-time location of the vehicle.

7. The driver charging intention prediction method according to any one of claims 1 to 6, characterized in that, After determining the target charging station and route planning result corresponding to the driver based on the charging intention prediction result, the method further includes: The cross-entropy loss function is used to calculate the loss of the charging intention prediction result, and the model parameters corresponding to the spatiotemporal joint prediction model are adjusted based on the loss calculation result to obtain the first model to be processed. The first model to be processed is trained online using historical charging behavior data and a preset learning rate to obtain a second model to be processed. Decision weights and distance decay coefficients are determined for the emotion categories. The obtained decision weights and distance decay coefficients are then used to process the second model to obtain a new spatiotemporal joint prediction model.

8. A driver charging intention prediction device, characterized in that, include: The feature extraction result determination module is used to collect facial image data, electrocardiogram (ECG) signals, and electroencephalogram (EEG) signals corresponding to the driver through a target camera and a multi-channel physiological signal acquisition device deployed in the cockpit. The module performs wavelet packet decomposition and denoising on the EEG signals to obtain signal processing results. Then, it performs intercardiac interval feature extraction on the ECG signals to obtain feature extraction results. Finally, it performs alignment processing on the facial image data to obtain aligned images. The feature fusion module is used to construct a three-level fusion architecture using a temporal alignment layer based on dynamic time warping, a feature cross-layer based on multi-head attention mechanism, and a decision integration layer based on confidence weighting, and to perform feature fusion on the signal processing results, the feature extraction results, and the aligned image to obtain the feature fusion result; The pending information determination module is used to align the feature fusion result with the multimodal signal temporal reference using the temporal alignment layer in the three-level fusion architecture to obtain the alignment result, and to perform cross-modal feature association enhancement processing on the alignment result using the three-dimensional convolution and multi-head self-attention mechanism in the feature cross layer to obtain the enhancement result. Then, the decision integration layer is used to perform emotion category judgment and confidence weighting processing on the enhancement result to obtain pending information including emotion state, real-time location and charging pile status. The charging intention prediction result generation module is used to process the information to be processed using a spatiotemporal joint prediction model to obtain the charging intention prediction result, so as to determine the target charging pile and route planning result corresponding to the driver based on the charging intention prediction result.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the driver charging intention prediction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the driver charging intention prediction method as described in any one of claims 1 to 7.

Images