Flying detection method of multi-feature fusion network based on electroencephalogram signals

Abstract

The invention discloses a lie detection method of a multi-feature fusion network based on electroencephalogram signals. The lie detection method comprises the following steps: acquiring original electroencephalogram signal data of multiple subjects in different states by adopting an improved double-probe complex test; original electroencephalogram signal artifacts are separated and removed through an ICA algorithm, decomposition and reconstruction are carried out through wavelet transform, and filtering and denoising of the electroencephalogram signals are completed; frequency domain features, time-frequency domain features and spatial features are extracted through power spectrum estimation, wavelet transformation and an FBCSP algorithm; training an MFNet network to obtain a fusion feature classification model; extracting original electroencephalogram signal data of a to-be-detected lie, performing data preprocessing and multi-domain sign extraction in sequence to obtain frequency domain features, time-frequency domain features and spatial features corresponding to electroencephalogram signals of the to-be-detected lie, and inputting the frequency domain features, the time-frequency domain features and the spatial features into the trained fusion feature classification model to obtain classification tags of the to-be-detected lie. The features of multiple domains in the electroencephalogram are extracted through the improved electroencephalogram classification algorithm to complete the recognition task of lie detection of the subject.

Description

Technical Field

[0001] The present invention relates to brain neuroscience, neural networks, signal feature extraction, and classification and recognition technology, and belongs to the field of signal processing and deep learning. More specifically, it relates to a lie detection method based on a multi-feature fusion network of electroencephalogram signals. Background Art

[0002] Polygraph analysis is of great significance in criminal investigations and legal trials. In recent years, with the in-depth study and mastery of brain cognitive science, EEG detection has become an important means of modern polygraph technology due to its subjective uncontrollability. Therefore, EEG detection has become an important means of modern polygraph technology due to its subjective uncontrollability.

[0003] The EEG signal processing process generally includes preprocessing, feature extraction and classification [1] Preprocessing includes removing EEG artifacts such as noise, ECG artifacts, and EOG artifacts. The initial method to remove eye artifacts is regression method. [2] , followed by independent component analysis (ICA) [3] and wavelet transform [4] Methods such as these adaptively adjust the window according to the frequency and have multi-resolution. Wavelet transform has multi-resolution function and can adaptively adjust the window according to the frequency. Wavelet denoising can eliminate sharp noise interference, while independent component denoising can not only eliminate noise but also retain small details of EEG signals.

[0004] In the feature extraction process, most studies only focus on time domain features. [5] Or frequency domain characteristics [6] . However, EEG signals are non-stationary random signals, so in some cases, using only the features of a single domain may not fully capture the signal differences. Common classifiers include support vector machines (SVM), k-nearest neighbors, random forests, and linear discriminant analysis (LDA). However, due to the large amount of EEG data, a large number of features will be left after feature extraction. Faced with these features, traditional classifiers may find it difficult to obtain satisfactory classification results.

[0005] Deep learning can improve feature representation and classification accuracy through multi-layer linear combination. Therefore, more and more studies use deep learning models to classify EEG signals. At present, a variety of feature extraction and recognition methods based on deep learning have been proposed. [7][8] Although these methods show good performance, they often lack interpretability. In addition, although deep learning models extract features directly from raw EEG data, they do not always guarantee accurate localization of valuable features in EEG signals. Therefore, techniques that combine classical feature extraction methods with deep learning techniques are particularly important. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies in the prior art and propose a lie detection method based on a multi-feature fusion network of EEG signals. The features of multiple domains in the EEG are extracted through an improved EEG classification algorithm to complete the lie detection recognition task for the subject.

[0007] The objectives of the present invention are achieved through the following technical solutions.

[0008] The lie detection method based on the multi-feature fusion network of EEG signals of the present invention comprises the following steps:

[0009] S1: EEG signal acquisition

[0010] The improved double-probe complex test was used to obtain the original EEG signal data of multiple subjects in different states, and the EEG lie detection data set was constructed. The original EEG signal data of some subjects were selected as the training set.

[0011] S2: EEG signal data preprocessing

[0012] The ICA algorithm is used to separate and remove artifacts from the original EEG signals in the training set, and the EEG signals after decomposition and reconstruction are performed through wavelet transform to complete the filtering and denoising of the EEG signals.

[0013] S3: Multi-domain feature extraction

[0014] The frequency domain features, time-frequency domain features, and spatial features of the preprocessed EEG signals are extracted through power spectrum estimation, wavelet transform, and FBCSP algorithm.

[0015] S4: Fusion feature classification training

[0016] The frequency domain features, time-frequency domain features, and spatial features extracted in step S3 are used to train the MFNet network to obtain a fusion feature classification model;

[0017] S5: Extract the original EEG signal data of the polygraph subject, perform data preprocessing and multi-domain sign extraction in sequence according to the methods of steps S2 and S3, obtain the frequency domain features, time-frequency domain features, and spatial features corresponding to the EEG signals of the polygraph subject, input them into the fusion feature classification model trained in step S4, and obtain the classification label of the polygraph subject, that is, lying or honest.

[0018] Furthermore, the specific process of the improved dual-probe complex test in step S1 is as follows: 24 subjects are divided into a lying group and an honest group. The lying group participates in the EEG test after the same scenario simulation, and the honest group is divided into two situations. In the first situation, the subjects participate in the EEG test after listening to the simulated case description with the same content as the lying group, and in the second situation, the subjects directly participate in the EEG test; thereafter, each subject is placed in the same soundproof electromagnetic shielding room, and the same text test stimulus sequence is used to obtain the original EEG signal data of each subject.

[0019] Furthermore, the specific process of multi-domain feature extraction in step S3 is as follows:

[0020] S3.1 Extraction of frequency domain features

[0021] Welch power spectrum estimation is used to perform frequency domain analysis on the EEG signal after EEG signal data preprocessing, the EEG signal of any channel of the EEG signal after EEG signal data preprocessing is segmented and windowed, the power spectrum of each segment after windowing is calculated, and then the average power spectrum is obtained, and the power values ​​corresponding to t specific frequencies are selected to form the eigenvector of the channel. The eigenvectors of each channel are obtained by the same method as above, and the eigenvectors are spliced ​​to form a frequency domain feature matrix;

[0022] S3.2 Extraction of time-frequency domain features

[0023] The EEG signal after the EEG signal data preprocessing is analyzed in the time-frequency domain by using wavelet transform, the EEG signal of any channel after the EEG signal data preprocessing is layered, the approximate coefficients corresponding to the approximate components of the corresponding frequency band are obtained by decomposing each layer, and the characteristic vector of the channel is obtained. The time-frequency domain characteristic vectors of each channel are obtained according to the above method and process, and the time-frequency domain characteristic matrix is ​​formed by splicing;

[0024] S3.3 Extraction of spatial features

[0025] The FBCSP algorithm is used to perform spatial domain analysis on the EEG signals after EEG signal data preprocessing to extract spatial features. First, the preprocessed EEG signals are filtered using a filter bank. The EEG signals of each channel are filtered into multiple frequency bands; then the CSP algorithm is used to filter each frequency band of each channel, and the eigenvector of each frequency band after CSP filtering is extracted. The eigenvectors of all frequency bands of the same channel are normalized by variance to form the eigenvector of the channel; finally, the eigenvectors of all channels are concatenated to obtain the final spatial feature matrix.

[0026] Furthermore, the MFNet network in step S4 includes block 1, block 2, block 3, a connection layer, a flat layer, a dense layer, and a softmax classification layer, wherein block 1 includes a two-dimensional convolution layer, a batch normalization layer 1, an activation layer 1, and an average pooling layer 1 arranged in sequence along the data transmission direction, and block 2 includes a separable convolution layer, a batch normalization layer 2, an activation layer 2, and an average pooling layer 2 arranged in sequence along the data transmission direction, and block 3 includes a deep convolution layer, a batch normalization layer 3, an activation layer 3, and an average pooling layer 3 arranged in sequence along the data transmission direction; frequency domain features, time-frequency domain features, and spatial features are respectively input into block 1, block 2, and block 3, and the output features of the three blocks are connected through a feature connection layer, and the connected multidimensional feature map is flattened through a flat layer, and after converting the multidimensional feature map into a one-dimensional vector, feature fusion is performed through a dense layer, and finally the category label of the electroencephalogram signal data, i.e., lying or honest, is obtained through a softmax classification layer.

[0027] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0028] (1) The lie detection method based on the multi-feature fusion network of EEG signals proposed in the present invention. First, a dual-probe composite test scheme was adopted in the lie detection experiment, and the experimental scheme was adjusted to more accurately reflect the real scene and more accurately simulate the real lie detection scene. By adjusting the experimental process, possible interference factors were reduced, making the collection of EEG signals closer to the actual application scene, and further improving the credibility of the experimental results. Secondly, in terms of feature extraction of EEG signals, the present invention breaks through the traditional single feature extraction method and comprehensively captures EEG information from three dimensions: frequency, time-frequency and space. This fusion of multi-dimensional features enables the present invention to prove the effectiveness of the method through a large number of experiments. By comparing with traditional lie detection methods (such as skin electrical response, heart rate changes, etc.), the lie detection method based on EEG signals shows higher accuracy and stronger individual adaptability. Especially in a variety of situations, this method can capture EEG activity characteristics related to lies in a short time, and has broad application potential.

[0029] (2) The present invention has demonstrated the effectiveness of the method through a large number of experiments. By comparing with the traditional EEG lie detection method, the lie detection method based on EEG signals showed a higher accuracy rate. Especially in a variety of situations, this method can capture the EEG activity characteristics related to lies in a short time and has a wide range of application potential. The features are extracted using traditional signal processing methods, and then enhanced using neural networks to improve classification accuracy and feature representativeness. This method solves the limitations of previous studies, that is, single-domain feature extraction cannot fully capture EEG features, and using only neural networks lacks interpretability. This lie detection method based on a multi-feature fusion network of EEG signals can effectively use EEG to analyze the emotions of the subject to achieve analysis of whether the subject is lying. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The invention provides a lie detection method based on a multi-feature fusion network of EEG signals;

[0031] Figure 2 Figure 1 is an example of a stimulus sequence;

[0032] Figure 3 Schematic diagram of data preprocessing;

[0033] Figure 4 It is a schematic diagram of multiple feature extraction;

[0034] Figure 5 Schematic diagram of fusion feature classification model training. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the invention more clearly explained, the embodiments of the invention are further described in detail below with reference to the accompanying drawings.

[0036] Hidden information testing is commonly used in lie detection and includes 3 stimulus protocols [9] , Composite test plan

[10] and dual-probe composite test scheme

[11] .

[0037] The two-probe compound protocol is a significant extension and improvement of the P300 compound protocol. One advantage of the compound protocol is that the target or non-target is presented with a delay during the trial, which allows the implicit detection and independent recognition tasks to be separated in time from the explicit target and non-target recognition tasks. This separation is intended to reduce cognitive resource competition. The two-probe compound protocol doubles the probe content in each trial block while retaining the countermeasure resistance and sensitivity of the simple compound protocol, especially in detecting information unintentionally acquired during lie detection. In addition, subsequent experiments have improved the two-probe compound protocol to improve its effectiveness.

[0038] P300 is an important indicator used in psychological tests to determine the significance of stimuli to subjects. Initially, the amplitude and latency of P300 were used as identification indicators, but their accuracy was found to be low. Subsequent studies used the guided amplitude difference method and the guided correlation difference method, which are closely related to amplitude. However, the signal peak alone cannot fully represent the various characteristics of ERP responses during lie detection and is easily affected by artifacts.

[0039] Traditional preprocessing methods include regression, which can remove artifacts and correlated EEG signals. On the other hand, subsequent methods such as ICA and wavelet transform have been shown to be more effective. Wavelet transform has multi-resolution capabilities and can adaptively adjust the window according to the frequency. Wavelet denoising can remove sharp noise interference, while independent component denoising can not only remove noise but also preserve small details of EEG signals.

[0040] For feature extraction, most existing studies are conducted from the perspective of time domain and frequency domain. For example, Gao used the F-score algorithm to sort the time domain features and found that the amplitude and peak value had the best classification effect.

[12] It is proposed to use static wavelet transform (SWT) and discrete wavelet transform (DWT) for feature extraction.

[13] A new EMD-based P300 spoofing detection method is proposed. These methods are mainly based on single-domain feature extraction. Although EEG has discriminative power in each domain, the features obtained by single-domain feature extraction may not be as sufficient as those obtained by multi-domain feature extraction.

[0041] Deep learning methods make up for the shortcomings of statistical methods. For example, wavelet packet transform (WPT) is used to extract features and deep neural network (DNN) is used for classification. After extracting multiple features of the EEG signal, the signal will be classified through an ensemble framework that integrates multiple classifiers. Saini obtained the intrinsic mode function (IMF) after performing empirical mode decomposition (EMD) on the data. The first three IMFs were selected to extract statistical features in the time and frequency domains, and SVM was used for classification. In addition, a time-spectrum-based squeezing and excitation feature fusion network (TS-SEFFNet) was proposed for MI-EEG decoding. A practical hybrid unsupervised deep convolutional recursive generative strategy network model for EEG feature description and fusion (EEGFuseNet) was proposed

[14] . A multi-branch deep learning fusion model (MBFM) is used to identify epileptogenic signals in the epileptogenic zone. Although deep learning models extract features directly from raw EEG data, they do not always guarantee accurate location of valuable features in EEG signals. Therefore, the present invention combines classical feature extraction methods with deep learning techniques.

[0042] Aiming at the adaptability of EEG signals in lie detection, the present invention proposes a lie detection method based on a multi-feature fusion network of EEG signals to solve the problem of complexity in the recognition of EEG signals. Figure 1 As shown, the following process is included:

[0043] S1: EEG signal acquisition

[0044] An improved dual-probe complex experiment was used to obtain the original EEG signal data of multiple subjects in different states, and an EEG lie detection dataset was constructed. The original EEG signal data of some subjects were selected as the training set.

[0045] The specific process of the modified double-probe complex test: 24 subjects were divided into a lying group and an honest group. The experiment was conducted in a quiet environment. All experiments were approved by the Institutional Review Board of the Institute of Psychology, Chinese Academy of Sciences, and were conducted in accordance with the principles set forth in the Declaration of Helsinki.

[0046] All subjects in the lying group took the EEG test after the same scenario simulation. For example, each subject was asked to enter a pre-arranged room and steal the most valuable item in the room. However, there was actually nothing valuable in the room except a check in the drawer. After claiming to have taken something, the subjects in the lying group were asked to take the EEG test.

[0047] The subjects in the honest group were divided into two conditions: the subjects in the first condition were asked to listen to the description of the simulated case (the same as the scenario simulation of the lying group mentioned above) read by the chief tester, which could include four test questions: 1. What was the stolen object? 2. What was inside the stolen object? 3. How much was the stolen object worth? 4. Where was the stolen object taken from? The tester then asked the subjects to ensure that each subject had indeed received the relevant information, and asked each subject to give the correct answer to the four questions, and finally each subject participated in the EEG test. The subjects in the second condition directly participated in the EEG test.

[0048] At the beginning of the EEG test, each subject is placed in the same soundproof electromagnetic shielding room in turn. The examiner reads the standardized experimental instructions and uses the same text test stimulus sequence to obtain the original EEG signal data of each subject. Among them, each subject can undergo multiple rounds of EEG tests, all using Neuroscan's 64-lead ERP recording system. The subject wears 64 electrodes on the head, and the electrode distribution position refers to the international 10-20 system. Preferably, the test stimulus sequence is displayed in the center of the screen, with black text on a white background, the font is bold, and the size is 80. During the entire EEG test, the subject is required to focus on the central area of ​​the screen, and the distance between the entire screen is about 80 cm.

[0049] The test stimulus sequence refers to the DPCTP paradigm. In order to be consistent with the actual situation, the visual stimulus of stimulus 1 in the original paradigm is replaced by text. For example, the test stimulus sequence of the entire EEG test is as follows: Figure 2 As shown, the stimuli in the test trial included checks, envelopes, check amounts, and drawers. In order to facilitate the subjects to distinguish each trial, a fixation point was introduced before the trial. In a single trial, a fixation point in the form of a "+" sign first appeared and remained visible for 500 milliseconds. Subsequently, word 1 appeared for a duration of 300 milliseconds. After perceiving word 1, the subjects were asked to press the left mouse button immediately. Subsequently, the subjects had a reaction window of 1100 milliseconds to 1550 milliseconds, and the reaction time was randomly determined. After the reaction time ended, word 2 was displayed for 300 milliseconds. According to the instructions issued by the examiner before the test, the subjects were required to press the left button when the designated irrelevant words (target stimuli) appeared, and press the right button in other cases. The reaction window duration of this task was 2000 milliseconds, and the total test duration was about 50 minutes. A series of practice sessions were conducted before starting the formal test trials.

[0050] According to the above test method, the original EEG signal data of 24 subjects can be obtained to construct an EEG lie detection data set, and then the original EEG signal data of 9 subjects can be randomly selected, of which the original EEG signal data of 8 subjects (preferably including subjects in both the lying group and the honest group) are used as training sets, and their labeled data are subsequently used to train the MFNet network to obtain the trained model parameters. The original EEG signal data of 1 subject is used as a test set to evaluate the model performance, and the trained MFNet model is used to predict the classification label corresponding to the input data.

[0051] S2: EEG signal data preprocessing

[0052] The ICA algorithm (i.e., independent component analysis algorithm) is used to separate and remove artifacts from the original EEG signals in the training set, and the EEG signals after decomposition and reconstruction are performed through wavelet transform to complete the filtering and denoising of the EEG signals. Figure 3 shown.

[0053] In order to minimize redundant data, the EEG signal data collected by some electrodes can be selected, such as electrodes FP1, FP2, FZ, FCZ, CZ, C4, P3, PZ, P4, P6, PO3, and OZ. According to the MDPCTP mode, the first step of preprocessing uses the ICA algorithm to remove pseudo-differences, and the second step uses wavelet transform to filter and denoise the EEG signal through decomposition and reconstruction.

[0054] The first step of the preprocessing algorithm is to separate and remove the artifact components in the EEG signal from the mixed signal according to the ICA algorithm to achieve the purpose of removing artifacts.

[15] First, we obtain the unmixed matrix B that separates the independent signals, and then find the artifacts in the signal. We set the rows representing the artifact components in the output signal Y to zero, and correct Y to Observation signal After correction, a relatively pure EEG signal is obtained.

[0055] After removing the artifacts, the EEG signal still contains noise. The second step of the preprocessing algorithm is to decompose and reconstruct the EEG signal after the artifact removal through wavelet transform to complete the filtering and denoising of the EEG signal. Wavelet transform is based on the principle of signal decomposition, which decomposes the signal into a set of added wavelet functions. Wavelet analysis can simultaneously complete the time-frequency analysis of EEG signals, and can also complete the "automatic scaling" of frequency, which is suitable for non-steady-state signal processing of EEG signals.

[0056] Assume that the relatively pure EEG signal obtained after the above-mentioned anti-aliasing processing The EEG signal of one channel is EEG The following decomposition and reconstruction are performed, φ(·) is the generator of multi-scale analysis, and the following results can be obtained from the two-scale equation:

[0057]

[0058] Among them, j is the number of EEG signal decomposition layers, k is the time shift coefficient, t is the discrete time sequence number, φ j,k (t) is the multi-scale analysis generator of the EEG signal when the time-shift coefficient is k at the jth layer, Z represents the set of all integer values, φ j-1,r (t) is the multi-scale analysis generator of the EEG signal at the j-1 layer when the time shift coefficient is r, h(·) is the expansion coefficient of the high-pass filter,

[0059] h(r-2k)=<φ j,r (t),φ j+1,r (t)> (2)

[0060] Among them, φ j,r (t) is the multi-scale analysis generator of the EEG signal at layer j when the time shift coefficient is r, φ j+1,r (t) is the multi-scale analysis generator of the EEG signal at layer j+1 when the time shift coefficient is r.

[0061] c j,k represents the scale factor of the EEG signal with a time shift coefficient of k at layer j, where is the input signal EEG signal of one channel, EEG signal The scale coefficient c of the EEG signal at layer j when the time shift coefficient is k j,k Defined as:

[0062] c j,k =∫ R f(t)φ j,k (t)dt=∑ r∈Z h(r-2k)c j-1,r (3)

[0063] Among them, f(t) is the signal to be processed, that is, φ j,k The inner product of (t). The scaling factor c of the current layer j j,k The expansion coefficient h(r-2k) of the high-pass filter and the scale coefficient c when the time shift coefficient of the previous layer j-1 is r j-1,r Similarly, the detail coefficient d of the EEG signal at layer j when the time shift coefficient is k j,k It can be introduced like this:

[0064] d j,k =∑ r∈z g(r-2k)c j-1,r (4)

[0065] Among them, g(·) is the expansion coefficient of the low-pass filter. After decomposing into the corresponding frequency band, the coefficient c in the high-frequency noise frequency range is j,k and d j,k Zero it and then reconstruct it.

[0066] according to Through multi-level signal reconstruction, we can finally get a channel EEG signal The corresponding reconstructed signal The wavelet transform reconstruction formula is:

[0067] c j-1,r =∑ r∈Z h(k-2r)c j,r +∑ r∈Z g(k-2r)d j,r (5)

[0068] EEG All channel signals of the EEG signal are preprocessed through the above two steps. Can be reconstructed into a relatively pure signal

[0069] S3: Multi-domain feature extraction

[0070] The frequency domain features, time-frequency domain features, and spatial features of the EEG signals after preprocessing are extracted through power spectrum estimation, wavelet transform, and FBCSP algorithm, such as Figure 4 shown.

[0071] S3.1 Extraction of frequency domain features

[0072] Power spectrum estimation is a method to estimate the relationship between the power and frequency of a received signal by analyzing the signal correlation. The Welch algorithm based on the periodogram method is a simple algorithm that minimizes the variance of spectrum estimation while maintaining resolution by segmenting and windowing the data.

[0073] For EEG signals, Welch power spectrum estimation is used for frequency domain analysis. The general procedure involves segmenting and windowing the signal (adding a window function), calculating the power spectrum of each segment, and then obtaining the average power spectrum.

[0074] Assume that the EEG signal after EEG signal data preprocessing The EEG signal of one channel is Will is divided into μ segments, each segment has a length of The data of segment 1 is represented as:

[0075]

[0076] Then, add the window function w(i) to each data segment to obtain the periodogram of the data segment. Calculate the power spectrum of each windowed data segment:

[0077]

[0078] Where E is the normalized coefficient of the window function, The periodogram of each segment is approximately considered to be unrelated, and the power spectra of all data segments are averaged to obtain the average power spectrum, which is the final power spectral density estimate of the channel:

[0079]

[0080] Formula (9) represents the power distribution of the signal at different frequencies. The power values ​​corresponding to t specific frequencies are selected within a certain frequency range to form the characteristic vector (p ω1 ,p ω2 …p ωt ), p ωt Represents the power value corresponding to the selected tth frequency ω. According to the above method, the feature vector of each channel is obtained and concatenated to form the frequency domain feature matrix F1. The dimension of F1 is σ×t, where σ represents the EEG signal The number of channels, each row corresponds to the feature vector of a channel.

[0081] S3.2 Extraction of time-frequency domain features

[0082] Since the EEG has good time-frequency domain characteristics, only the time domain or frequency domain cannot fully reflect its characteristics. Therefore, the present invention considers analyzing the signal from the time-frequency domain. Wavelet transform is usually used for time-frequency analysis. Assume that the EEG signal after EEG signal data preprocessing The EEG signal of one channel is The definition of j-level decomposition is as follows:

[0083]

[0084] Among them, c j is the approximate component of the signal in the jth layer decomposition, representing the low-frequency part, d j is the detail component of the signal in the jth layer decomposition, representing the high-frequency part. Their corresponding approximate coefficients (i.e., scale coefficients) {c j,k} k∈z and detail coefficient {d j,k} k∈Z They are defined by the above formulas (3) and (4) respectively. Using the wavelet coefficients of each layer (including approximate coefficients and detail coefficients), the reconstruction of the EEG signal of the channel is completed.

[0085] The detail coefficient captures high-frequency information, while the approximate coefficient captures low-frequency information, which can be used as time-frequency domain features. Wavelet transform can decompose each layer to obtain the approximate coefficients corresponding to the approximate components of the corresponding frequency band, and the characteristic vector of the channel is {c j,k} k∈Z According to the above method, the time-frequency domain feature vector of each channel is obtained and concatenated to form the time-frequency domain feature matrix F2. The number of rows in F2 is σ, where σ represents the EEG signal The number of channels, each row corresponds to the feature vector of a channel.

[0086] S3.3 Extraction of spatial features

[0087] The FBCSP algorithm is used to perform spatial domain analysis on the EEG signal after EEG signal data preprocessing to extract spatial features. The FBCSP algorithm is generally divided into three stages: First, the preprocessed EEG signal is filtered using a filter bank. The EEG signals of each channel are filtered into multiple frequency bands; then the common spatial pattern (CSP) algorithm is used to filter each frequency band of each channel, and the eigenvector of each frequency band after CSP filtering is extracted. The eigenvectors of all frequency bands of the same channel are normalized by variance to form the eigenvector of the channel; finally, the eigenvectors of all channels are concatenated to obtain the final spatial feature matrix.

[0088] CSP is a spatial filtering method for binary classification, which can reflect various spatial distributions through multi-channel EEG signals.

[16] The basic principle of the algorithm is to obtain a set of optimal filters through matrix diagonal projection to maximize the variance between the two signals and obtain the eigenvector with the highest discriminative power.

[0089] First, the EEG signal data is preprocessed The EEG signal of a channel is passed through a q-order bandpass IIR filter to obtain the filtered EEG data of μ frequency bands of the channel. The filtered EEG data of the i-th frequency band is used Represented by, where i = 1, 2, …, μ, assuming that the two types of original EEG signals in this frequency band are represented by two matrices respectively: and They represent the EEG signals of liars and honest people in this frequency band respectively.

[0090] Step 1: We need the mixed space covariance matrix of the two types of data. and The covariance matrices obtained after normalization are Q1 and Q2 respectively.

[0091]

[0092] In the above two formulas, is a matrix The transpose of . trace represents the sum of all elements on the diagonal. Then calculate the covariance matrix Q of the frequency band, which is the average covariance matrix of the two data: and They represent the average of the normalized covariance matrices of the two classes (liars and honest people) of signals.

[0093] Step 2: Use principal component analysis to calculate the feature matrix. First, perform feature decomposition on the covariance matrix Q of the frequency band according to the following formula:

[0094] Q=VλV T (11)

[0095] Where V is the eigenvector matrix of matrix Q, and λ is a diagonal matrix composed of features. The whitening matrix can be obtained by descending the features, as follows:

[0096]

[0097] Among them, Q1 and Q2 can be converted into G1=JQ1J T ,G2=JQ2J T G1 and G2 have a common eigenvector D, G1 = Dλ1D T,G2=Dλ2D T . Among them, λ1+λ2=I, I is the unit matrix.

[0098] Obviously, if the sum of the features of the two types of matrices is always one, then the vector with the largest feature in G1 can be minimized to G2. The features in λ1 and λ2 are arranged in descending and ascending order respectively. Thus, the optimal variance for separating the two EEG signals is obtained. The projection matrix η is η=D T J.

[0099] The third step is feature extraction. Assume that the test data is Then the feature Z can be obtained through the filter η i,j .

[0100]

[0101] The eigenvectors of are as follows:

[0102]

[0103] Among them, VAR(·) is the variance operator, F cspi is the eigenvector of the ith frequency band of the channel. Following the same method as above, the eigenvector of each frequency band of the channel (μ frequency bands in total) can be obtained, and then the eigenvectors of all frequency bands are normalized by variance to form the eigenvector of the channel {F cspi} i＝1…μ Finally, according to the EEG signal The number of channels σ is calculated, and the feature vectors of all channels are concatenated to obtain the final spatial feature matrix F3. The number of rows in the matrix F3 is σ, and each row corresponds to the feature vector of a channel.

[0104] S4: Fusion feature classification training

[0105] Through the above three feature extraction methods, EEG features F1, F2, and F3 in the frequency domain, time-frequency domain, and spatial domain can be obtained. Next, the present invention uses the MFNet network to further improve the feature representation capability, and uses several common convolution methods: two-dimensional convolution, deep convolution, and separable convolution.

[0106] Based on these three types of convolution, the MFNet network is divided into three branches, corresponding to block 1, block 2, and block 3. Figure 5As shown, block 1 includes a two-dimensional convolution layer, a batch normalization layer 1, an activation layer 1, and an average pooling layer 1 arranged in sequence along the data transmission direction. Block 1 is based on a two-dimensional convolution of size (1,6) and further integrates the frequency domain feature F1. Block 2 includes a separable convolution layer, a batch normalization layer 2, an activation layer 2, and an average pooling layer 2 arranged in sequence along the data transmission direction. Block 2 uses separable convolution to clearly decouple the relationship between the feature map and the feature map to optimize the extracted time-frequency domain feature F2. The separable convolution first applies convolution (depth convolution) to each input channel, and then integrates the information through a point-by-point convolution (1x1 convolution). Block 3 includes a deep convolution layer, a batch normalization layer 3, an activation layer 3, and an average pooling layer 3 arranged in sequence along the data transmission direction. Block 3 uses deep convolution to learn spatial features to further process the spatial feature F3. Deep convolution can focus on the spatial information of each input channel, helping the model to extract features related to spatial layout, and further helping the model understand the activity patterns of different brain regions. The output features of the three blocks are then connected through the feature connection layer, which means that the output feature maps of the three blocks are spliced ​​in the feature dimension. The connected multi-dimensional feature map usually needs to be flattened through a flat layer to convert the multi-dimensional feature map into a one-dimensional vector, and then the dense layer is used to fuse the three types of features to form the final high-dimensional feature representation. Finally, the category label of the EEG signal data is obtained through the softmax classification layer, that is, lying or honest.

[0107] The frequency domain features, time-frequency domain features, and spatial features extracted in step S3 are used to train the above MFNet network, and a fusion feature classification model based on deep learning is obtained after training. Overall training process: data input → F1 enters block 1 (two-dimensional convolution) to further integrate and extract frequency domain features; F2 enters block 2 (separable convolution) to further optimize and extract time-frequency domain features; F3 enters block 3 (deep convolution) to further process and extract spatial features → feature connection → flattening → dense layer fusion → Softmax classification layer → output category label.

[0108] S5: Extract the original EEG signal data of the polygraph subject, perform data preprocessing and multi-domain physical sign extraction in accordance with the methods of steps S2 and S3, obtain the frequency domain features, time-frequency domain features, and spatial features corresponding to the EEG signal of the polygraph subject, input them into the MFNet network (i.e., fusion feature classification model) trained in step S4, and obtain the classification label of the polygraph subject, i.e., lying or honest. Here, the original EEG signal data of the polygraph subject can be extracted, and the 64-lead ERP recording system of Neuroscan can also be used. The polygraph subject wears 64 electrodes on his head, and the distribution position of the electrodes refers to the international 10-20 system.

[0109] Following the MDPCTP model, we first used ICA to remove pseudo-differences, and then used wavelet transform to denoise the signal through decomposition and reconstruction. The sampling rate was reduced to 500Hz. According to the sampling theorem, the highest frequency of the EEG signal is 250Hz, while the typical frequency range of P300 is 1-10Hz. In order to obtain EEG data within this frequency band, the original data was decomposed into four layers. According to the approximate coefficients of the fourth layer, the signal was reconstructed in the time domain. Previous studies have shown that specific electrodes can detect prominent P300 components. In order to minimize redundant data, the following electrodes were specially selected in this study: FP1, FP2, FZ, FCZ, CZ, C4, P3, PZ, P4, P6, PO3, and OZ.

[0110] Welch power spectrum estimation was performed on the EEG signals of the above channels. In the low-frequency band, the power spectrum estimation value of the liar was slightly higher than that of the honest condition. Therefore, the power spectrum density values ​​in the low-frequency range were extracted as frequency domain features. The typical frequency range of P300 is 1-10Hz, which corresponds to the approximate coefficient band of the fourth layer of wavelet decomposition. In the preprocessing stage, the EEG signals were transformed by four layers of wavelet. Therefore, the fourth layer approximate coefficients can be directly extracted and averaged for each time point. Subsequently, FBCSP was used to extract spatial features.

[0111] The present invention designs a comparative experiment to compare five classic methods and four deep learning models. Accuracy (T), accuracy (L) and accuracy (H) represent the accuracy of all subjects, liars and honest people, respectively. The results are shown in Table 1. Traditional methods for processing EEG data usually involve single-domain feature extraction and classification using machine learning classifiers. However, due to the correlation between multi-domain EEG signals, the analysis of EEG signals by single-domain methods is limited. On the other hand, multi-domain feature extraction encounters challenges in feature dimensions, making it difficult for traditional classifiers to achieve high-precision classification. In contrast, this paper proposes a model that combines traditional multi-domain feature extraction technology with deep learning to achieve enhanced feature integration. This method not only retains the interpretability of traditional methods, but also improves classification accuracy by utilizing deep learning.

[0112] Table 1

[0113]

[0114] The classification accuracy of the proposed multi-feature algorithm is compared with that of the single-domain feature extraction algorithm, and the results are shown in Table 2. This table clearly shows that the classification accuracy of any single-domain feature extraction method is lower than that of the multi-domain feature extraction method, thus highlighting the superiority of the proposed model.

[0115] Table 2

[0116]

[0117]

[0118] To further demonstrate the advantages of the proposed model in the classification stage, the proposed MFNet is compared with traditional classifiers and EEGNet. The comparison results are shown in Table 3, indicating that MFNet enhances feature representation and achieves higher classification accuracy. In particular, the results show that MFNet outperforms the random forest (RF) and adaboost (AB) classifiers.

[0119] Table 3

[0120] method Accuracy method Accuracy SVM 79.17 AB 87.23 KNN 83.78 LDA 86.07 RF 68.92 EEGNet 90.89 ELM 69.47 MFNet 92.04

[0121] The present invention proposes a deception detection method based on multiple features of EEG, which adopts an improved dual-probe complex test scheme. In the preprocessing, the present invention uses a series of methods to denoise and improve the signal-to-noise ratio. We also propose a multi-feature extraction method. In the frequency domain, power spectrum estimation is used as the frequency domain feature; in the time-frequency domain, the approximate coefficients obtained by wavelet decomposition are used as the time-frequency domain feature; in the spatial domain, FBCSP is used. This method can fully extract the features of P300 and has a certain degree of interpretability, while reducing the amount of data. The experimental results show that compared with other competing models, our model obtains better or similar performance, which proves the effectiveness of the proposed model.

[0122] The multi-feature deception detection model proposed in the present invention utilizes traditional signal processing methods to extract features, and then utilizes neural networks to enhance them in order to improve classification accuracy and feature representativeness. This method solves the limitations of previous studies, namely that single-domain feature extraction cannot fully capture EEG features, and the use of neural networks alone lacks interpretability. Despite the high classification accuracy, the multi-domain model developed in this study still has certain limitations. First, future studies should consider the changes in P300 intensity at the same scalp location for different subjects. In addition, due to the limitations of experimental conditions, available data is limited, and it is necessary to expand the scope of data collection in future studies. Under the inspiration of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which are all within the protection of the present invention.

[0123] Although the functions and working processes of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific functions and working processes. The above-mentioned specific implementation methods are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which are all within the protection of the present invention.

[0124] References

[0125] [1]Mustapha Moufassih, Oussama Tarahi, Soukaina Hamou, and etal. Spectral feature extraction from EEG based motor imagery using commonspatial patterns. In2022 2nd International Conference on Innovative Researchin Applied Science, Engineering and Technology(IRASET), pages 1–6, 2022.

[0126] [2]Ping He, M Kahle, G Wilson, and C Russell. Removal of ocular artifactsfrom EEG: a comparison of adaptive filtering method and regression methodusing simulated data. In 2005IEEE Engineering in Medicine and Biology 27thAnnual Conference, pages 1110–1113, 2006.

[0127] [3]Junfeng Gao, Liang Lu, Yong Yang, and et al. Anovel concealedinformation test method based on independent component analysis and supportvector machine. Clinical EEG and neuroscience, 43(1):54–63, 2012.

[0128] [4]Souvik Phadikar,Nidul Sinha,and Rajdeep Ghosh.Automatic eyeblinkartifact removal from EEG signal using wavelet transform with heuris-ticallyoptimized threshold.IEEE Journal of Biomedical and Health Informatics,25(2):475–484,2020. [5]Robert Jenke,Angelika Peer,and Martin Buss.Feature extraction andselection for emotion recognition from EEG.IEEE Transactions on AffectiveComputing,5(3):327–339,2014.

[0129] [6]Mesut S,eker,Mehmet Akin,and Mehmet Sirac,¨Ozerdem.Spectrum es-timation of odor EEG responses with parametric-nonparametric spectralanalysis methods.In2018 26th Signal Processing and CommunicationsApplications Conference,pages 1–4,2018.

[0130] [7]Yang Li,Lianghui Guo,Yu Liu,and et al.Atemporal-spectral-basedsqueeze-and-excitation feature fusion network for motor imagery EEGdecoding.IEEE Transactionson Neural Systems and Rehabilitation Engineering,29:1534–1545,2021.

[0131] [8]Zhen Liang,Rushuang Zhou,Li Zhang,and et al.Eegfusenet:Hybridunsupervised deep feature characterization and fusion for high-dimensional EEG withan application to emotion recognition.IEEE Transactionson Neural Systems andRehabilitation Engineering,29:1913–1925,2021.

[0132] [9]Samuel Sutton.P300–thirteen years later.In Evoked brain potentialsand behavior,pages 107–126.1979.

[0133]

[10] J Peter Rosenfeld,Elena Labkovsky,Michael Winograd,and et al.Thecomplextrial protocol(ctp):Anew,countermeasure-resistant,accurate,P300-basedmethod fordetection of concealed information.Psychophysiology,45(6):906–919,2008.,

[0134]

[11] Elena Labkovsky and J Peter Rosenfeld.Anovel dual probe complextrialprotocol for detection of concealed information.Psychophysiology,51(11):1122–1130,2014.

[0136]

[12] Handayani Tjandrasa and Supeno Djanali.Classification of P300event-relatedpotentials using wavelet transform,mlp,and soft margin svm.In2018 TenthInternational Conference on Advanced Computational Intelligence,pages 343–347,

[0137]

[13] Abdollah Arasteh,Mohammad Hassan Moradi,and AminJanghorbani.Anovelmethod based on empirical mode decomposition for P300-baseddetection ofdeception.IEEE Transactions on Information Forensics andSecurity,11(11):2584–2593,2016.

[0139]

[14] Zhen Liang,Rushuang Zhou,Li Zhang,and et al.Eegfusenet:Hybridunsupervised deep feature characterization and fusion for high-dimensional EEG withan application to emotion recognition.IEEE Transactionson Neural Systems andRehabilitation Engineering,29:1913–1925,2021.

[0140]

[15] Pei-Yuan Zhou and Keith C.C.Chan.Fuzzy feature extraction formultichannelEEG classification.IEEE Transactions on Cognitive andDevelopmental Systems,10(2,SI):267–279,2018.

[0141]

[16] Wang Tao,Wu Linyan,Li Yanping,and et al.Learning advanced braincomputerinterface technology:Comparing csp algorithm and wpa algorithm forEEG featureextraction.International Journal of Technology and HumanInteraction,15(3):14–27,2019.

[0142]

[17] Xiaoke Fang,Xiaobin Li,Jiacan Xu,and et al.Research onclassification ofmotor imagination EEG signals based on multi-domainfeatures.In 2021 40th ChineseControl Conference,pages 3225–3230,2021.

[0143]

[18] Navjot Saini,Saurabh Bhardwaj,and Ravinder Agarwal.Classificationof EEGsignals using hybrid combination of features for lie detection.NeuralComputing&Applications,32(8):3777–3787,2020.

[0144]

[19] Damodar Reddy Edla,Shubham Dodia,Annushree Bablani,and etal.Anefficient deep learning paradigm for deceit identification test on eegsignals.ACMTransactions on Management Information Systems,12(3),SEP 2021.

[0145]

[20] Yiping Wang,Yang Dai,Zimo Liu,and et al.Computer-aided intracra-nial EEGsignal identification method based on a multi-branch deep learningfusion model andclinical validation.Brain sciences,11(5):615,2021.

[0146]

[21] Yang Li,Lianghui Guo,Yu Liu,and et al.Atemporal-spectral-basedsqueeze-and-excitation feature fusion network for motor imagery EEGdecoding.IEEETransactions on Neural Systems and Rehabilitation Engineering,29:1534–1545,2021.

Claims

1. A lie detection method based on a multi-feature fusion network of EEG signals, characterized in that: The following steps are involved: S1: EEG signal acquisition The improved double-probe complex test was used to obtain the original EEG signal data of multiple subjects in different states, and the EEG lie detection data set was constructed. The original EEG signal data of some subjects were selected as the training set. S2: EEG signal data preprocessing The ICA algorithm is used to separate and remove artifacts from the original EEG signals in the training set, and the EEG signals after decomposition and reconstruction are performed through wavelet transform to complete the filtering and denoising of the EEG signals. S3: Multi-domain feature extraction The frequency domain features, time-frequency domain features, and spatial features of the preprocessed EEG signals are extracted through power spectrum estimation, wavelet transform, and FBCSP algorithm. S4: Fusion feature classification training The frequency domain features, time-frequency domain features, and spatial features extracted in step S3 are used to train the MFNet network to obtain a fusion feature classification model; S5: Extract the original EEG signal data of the polygraph subject, perform data preprocessing and multi-domain sign extraction in sequence according to the methods of steps S2 and S3, obtain the frequency domain features, time-frequency domain features, and spatial features corresponding to the EEG signals of the polygraph subject, input them into the fusion feature classification model trained in step S4, and obtain the classification label of the polygraph subject, that is, lying or honest.

2. The lie detection method based on the multi-feature fusion network of EEG signals according to claim 1 is characterized in that: The specific process of the improved dual-probe complex test in step S1 is as follows: 24 subjects are divided into a lying group and an honest group. The lying group takes the EEG test after the same scenario simulation, and the honest group is divided into two situations. In the first situation, the subjects take the EEG test after listening to the simulated case description with the same content as the lying group, and in the second situation, the subjects take the EEG test directly. After that, each subject is placed in the same soundproof electromagnetic shielding room, and the same text test stimulus sequence is used to obtain the original EEG signal data of each subject.

3. The lie detection method based on the multi-feature fusion network of EEG signals according to claim 1 is characterized in that: The specific process of multi-domain feature extraction in step S3 is as follows: S3.1 Extraction of frequency domain features Welch power spectrum estimation is used to perform frequency domain analysis on the EEG signal after EEG signal data preprocessing, the EEG signal of any channel of the EEG signal after EEG signal data preprocessing is segmented and windowed, the power spectrum of each segment after windowing is calculated, and then the average power spectrum is obtained, and the power values ​​corresponding to t specific frequencies are selected to form the eigenvector of the channel. The eigenvectors of each channel are obtained by the same method as above, and the eigenvectors are spliced ​​to form a frequency domain feature matrix; S3.2 Extraction of time-frequency domain features The EEG signal after the EEG signal data preprocessing is analyzed in the time-frequency domain by using wavelet transform, the EEG signal of any channel after the EEG signal data preprocessing is layered, the approximate coefficients corresponding to the approximate components of the corresponding frequency band are obtained by decomposing each layer, and the characteristic vector of the channel is obtained. The time-frequency domain characteristic vectors of each channel are obtained according to the above method and process, and the time-frequency domain characteristic matrix is ​​formed by splicing; S3.3 Extraction of spatial features The FBCSP algorithm is used to perform spatial domain analysis on the EEG signals after EEG signal data preprocessing to extract spatial features. First, the preprocessed EEG signals are filtered using a filter bank. The EEG signals of each channel are filtered into multiple frequency bands; then the CSP algorithm is used to filter each frequency band of each channel, and the eigenvector of each frequency band after CSP filtering is extracted. The eigenvectors of all frequency bands of the same channel are normalized by variance to form the eigenvector of the channel; finally, the eigenvectors of all channels are concatenated to obtain the final spatial feature matrix.

4. The lie detection method based on the multi-feature fusion network of EEG signals according to claim 1 is characterized in that: The MFNet network in step S4 includes block 1, block 2, block 3, a connection layer, a flat layer, a dense layer, and a softmax classification layer. The block 1 includes a two-dimensional convolution layer, a batch normalization layer 1, an activation layer 1, and an average pooling layer 1 arranged in sequence along the data transmission direction. The block 2 includes a separable convolution layer, a batch normalization layer 2, an activation layer 2, and an average pooling layer 2 arranged in sequence along the data transmission direction. The block 3 includes a deep convolution layer, a batch normalization layer 3, an activation layer 3, and an average pooling layer 3 arranged in sequence along the data transmission direction; The frequency domain features, time-frequency domain features, and spatial features are input into block 1, block 2, and block 3 respectively. The output features of the three blocks are connected through the feature connection layer. The connected multidimensional feature map is flattened through the flat layer. After the multidimensional feature map is converted into a one-dimensional vector, it is fused through the dense layer. Finally, the category label of the EEG signal data, i.e., lying or honest, is obtained through the softmax classification layer.

Images