Lie detection method based on multi-feature fusion network of electroencephalogram signals

Abstract

The application discloses a lie detection method based on a multi-feature fusion network of an electroencephalogram signal, and relates to the technical field of lie detection, and aims to improve the accuracy of lie detection.The improved double-probe complex test is used to obtain original electroencephalogram signal data of multiple subjects in different states; the ICA algorithm is used to separate and remove the artifacts of the original electroencephalogram signal, wavelet transform is used for decomposition and reconstruction, and the filtering and denoising of the electroencephalogram signal are completed; power spectrum estimation, wavelet transform and the FBCSP algorithm are used to extract frequency domain features, time-frequency domain features and spatial features respectively; the MFNet network is trained to obtain a fusion feature classification model; the original electroencephalogram signal data of a to-be-detected liar are extracted, data preprocessing and multi-domain feature extraction are sequentially performed, the frequency domain features, the time-frequency domain features and the spatial features corresponding to the electroencephalogram signal of the to-be-detected liar are obtained, and the classification label of the to-be-detected liar is obtained by inputting the features into the trained fusion feature classification model.

Description

TECHNICAL FIELD

[0001] The present application relates to brain neuroscience, neural networks, signal feature extraction, classification and recognition technology, belongs to the field of signal processing and deep learning, more specifically, it relates to a lie detection method based on a multi-feature fusion network of electroencephalogram signals. BACKGROUND

[0002] Polygraph analysis is of great significance in criminal investigation and legal trials. In recent years, with the in-depth study and mastery of brain cognitive science, electroencephalogram detection has become an important means of modern polygraph technology due to its subjective uncontrollability. Therefore, electroencephalogram detection has become an important means of modern polygraph technology due to its subjective uncontrollability.

[0003] The electroencephalogram signal processing process generally includes preprocessing, feature extraction and classification [1] . Preprocessing includes removing EEG artifacts such as noise, ECG artifacts and EOG artifacts. The first method to remove eye artifacts is regression [2] , followed by independent component analysis (ICA) [3] and wavelet transform [4] , etc. These methods adaptively adjust the window according to the frequency and have multi-resolution. Wavelet transform has multi-resolution function and can adaptively adjust the window according to the frequency. Wavelet denoising can eliminate sharp noise interference, while independent component denoising can not only eliminate noise, but also retain the small details of the electroencephalogram signal.

[0004] In the feature extraction process, most studies only focus on time domain features [5] or frequency domain features [6] . However, electroencephalogram signals are non-stationary random signals, so in some cases, only using features from a single domain may not be able to fully capture the signal differences. Common classifiers include support vector machines (SVM), k-nearest neighbors, random forests and linear discriminant analysis (LDA). However, due to the large amount of electroencephalogram data, a large number of features are left after feature extraction, and traditional classifiers may have difficulty obtaining satisfactory classification results in the face of these features.

[0005] Deep learning can improve feature representation through multiple linear combinations and improve classification accuracy. Therefore, more and more studies use deep learning models to classify electroencephalogram signals. At present, various feature extraction and recognition methods based on deep learning have been proposed [7][8] . Although these methods show good performance, they often lack interpretability. In addition, although deep learning models directly extract features from raw electroencephalogram data, they do not always guarantee accurate positioning of valuable features in electroencephalogram signals. Therefore, techniques that combine classical feature extraction methods with deep learning techniques are particularly important. SUMMARY

[0006] The purpose of the present application is to overcome the deficiencies in the prior art, and provide a lie detection method based on a multi-feature fusion network of electroencephalogram signals.

[0007] The purpose of the present application is realized through the following technical solutions.

[0008] The lie detection method based on the multi-feature fusion network of electroencephalogram signals comprises the following steps:

[0009] S1: Electroencephalogram signal acquisition

[0010] An improved double-probe complex test is used to obtain original electroencephalogram signal data of multiple subjects in different states, to construct an electroencephalogram lie detection data set, and to select original electroencephalogram signal data of some subjects as a training set;

[0011] S2: Electroencephalogram signal data preprocessing

[0012] ICA algorithm is used to separate and remove artifacts of the original electroencephalogram signal in the training set, and wavelet transform is used to decompose and reconstruct the artifact-removed electroencephalogram signal, to complete filtering and denoising of the electroencephalogram signal;

[0013] S3: Multi-domain feature extraction

[0014] Power spectrum estimation, wavelet transform and FBCSP algorithm are used to extract frequency domain features, time-frequency domain features and spatial features from the preprocessed electroencephalogram signal, respectively;

[0015] S4: Fusion feature classification training

[0016] The frequency domain features, time-frequency domain features and spatial features extracted in step S3 are used to train the MFNet network, to obtain a fusion feature classification model;

[0017] S5: Extract the original electroencephalogram signal data of the subject to be tested, and perform data preprocessing and multi-domain feature extraction in sequence according to steps S2 and S3, to obtain frequency domain features, time-frequency domain features and spatial features corresponding to the electroencephalogram signal of the subject to be tested, and input them into the fusion feature classification model trained in step S4, to obtain the classification label of the subject to be tested, i.e., lying or honesty.

[0018] Further, the specific process of the improved double-probe complex test in step S1: 24 subjects are divided into a lying group and an honest group, the lying group participates in the electroencephalogram test after the same scenario simulation, the honest group is divided into two cases, in the first case, the subject listens to the same content of the simulation case narration as the lying group and then participates in the electroencephalogram test, in the second case, the subject directly participates in the electroencephalogram test; then each subject is placed in the same soundproof electromagnetic shield room, and the same text test stimulation sequence is used to obtain the raw electroencephalogram signal data of each subject.

[0019] Further, the specific process of the multi-domain feature extraction in step S3:

[0020] S3.1 Extraction of frequency domain features

[0021] The Welch power spectrum estimation is used for frequency domain analysis of the electroencephalogram signal data after preprocessing, the electroencephalogram signal of any channel of the electroencephalogram signal data after preprocessing is segmented and windowed, the power spectrum of each segmented windowed signal is calculated, then the average power spectrum is obtained, the power values at t specific frequencies are selected to form the feature vector of the channel, the feature vectors of each channel are obtained by the same method, and the frequency domain feature matrix is spliced and formed;

[0022] S3.2 Extraction of time-frequency domain features

[0023] The wavelet transform is used for time-frequency domain analysis of the electroencephalogram signal data after preprocessing, the electroencephalogram signal of any channel of the electroencephalogram signal data after preprocessing is layered, the approximate coefficients corresponding to the approximate components of the corresponding frequency band are obtained through each layer decomposition, the feature vector of the channel is obtained, the time-frequency domain feature vectors of each channel are obtained by the above method, and the time-frequency domain feature matrix is spliced and formed;

[0024] S3.3 Extraction of spatial features

[0025] The FBCSP algorithm is used for spatial domain analysis of the electroencephalogram signal data after preprocessing, and the spatial features are extracted; first, the filter bank is used to filter the preprocessed electroencephalogram signal The electroencephalogram signal of each channel is filtered into multiple frequency bands; then the CSP algorithm is used to filter each frequency band of each channel, and the feature vector of each frequency band after CSP filtering is extracted, the feature vectors of all frequency bands of the same channel are normalized by variance to form the feature vector of the channel; finally, the feature vectors of all channels are spliced to obtain the final spatial feature matrix.

[0026] Further, the MFNet network in step S4 includes block 1, block 2, block 3, a connection layer, a flat layer, a dense layer, and a softmax classification layer, the block 1 includes a two-dimensional convolution layer, a batch normalization layer 1, an activation layer 1, and an average pooling layer 1 arranged in sequence along a data transmission direction, the block 2 includes a separable convolution layer, a batch normalization layer 2, an activation layer 2, and an average pooling layer 2 arranged in sequence along the data transmission direction, and the block 3 includes a depth convolution layer, a batch normalization layer 3, an activation layer 3, and an average pooling layer 3 arranged in sequence along the data transmission direction; the frequency domain feature, the time-frequency domain feature, and the spatial feature are input into the block 1, the block 2, and the block 3 respectively, the output features of the three blocks are connected through a feature connection layer, the multi-dimensional feature map after connection is flattened through the flat layer, the multi-dimensional feature map is converted into a one-dimensional vector, then the feature fusion is performed through the dense layer, and finally the class label of the electroencephalogram signal data, i.e., a lie or honesty, is obtained through the softmax classification layer

[0027] Compared with the prior art, the technical scheme of the present application has the following beneficial effects:

[0028] (1) The lie detection method based on the multi-feature fusion network of the electroencephalogram signal is proposed. Firstly, a double-probe composite test scheme is adopted in the lie detection experiment, and the experimental scheme is adjusted to more accurately reflect the real scene and more accurately simulate the real lie detection scene. By adjusting the experimental process, the possible interference factors are reduced, the collection of the electroencephalogram signal is closer to the actual application scene, and the credibility of the experimental results is further improved. Secondly, in the feature extraction of the electroencephalogram signal, the present application breaks through the traditional single feature extraction method and comprehensively captures the electroencephalogram information from the frequency, time-frequency, and space dimensions. The fusion of multi-dimensional features makes the electroencephalogram signal more accurate and more individualized. Especially in various situations, the method can capture the electroencephalogram activity characteristics related to lies in a short time, and has wide application potential.

[0029] (2) The present application proves the effectiveness of the method through a large number of experiments. Compared with the traditional electroencephalogram lie detection method, the lie detection method based on electroencephalogram signal shows higher accuracy. Especially in various situations, the method can capture the electroencephalogram activity characteristics related to lies in a short time, and has wide application potential. The traditional signal processing method is used to extract features, and then a neural network is used to enhance it to improve the classification accuracy and feature representativeness. This method solves the limitations of previous studies, that is, single-domain feature extraction cannot completely capture electroencephalogram features, and only using a neural network lacks interpretability. This lie detection method based on a multi-feature fusion network of electroencephalogram signals can effectively analyze the emotions of the subjects by using electroencephalogram to achieve the analysis of whether the subjects are lying. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The present application is a lie detection method based on a multi-feature fusion network of electroencephalogram signals.

[0031] Figure 2 It is a stimulation sequence example diagram.

[0032] Figure 3 It is a data preprocessing schematic diagram.

[0033] Figure 4 It is a multi-feature extraction schematic diagram.

[0034] Figure 5 It is a fusion feature classification model training schematic diagram. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the application more clear, the embodiments of the application will be further described in detail below with reference to the drawings.

[0036] Hidden information test is commonly used for lie detection, including 3 stimulation schemes [9] , complex test scheme

[10] and double probe complex test scheme

[11] .

[0037] The double probe complex test scheme is a major expansion and improvement of the P300 complex test scheme. One advantage of the complex test scheme is to delay the presentation of the target or non-target during the test, which separates the implicit detection and independent recognition task from the explicit target and non-target recognition task in time. This separation aims to reduce the competition for cognitive resources. The double probe complex test scheme doubles the probe content in each test block while retaining the countermeasure resistance and sensitivity of the simple complex test scheme, especially in terms of unintentionally acquired information during lie detection. In addition, subsequent experiments have improved the double probe complex test scheme to improve its effectiveness.

[0038] P300 is an important indicator in psychological tests for determining the meaning of stimuli to the subject. Initially, the amplitude and latency of P300 were used as identification indicators, but it was found that their accuracy was low. Subsequent studies adopted the guided amplitude difference method and the guided correlation difference method, which are closely related to the amplitude. However, relying solely on signal peaks cannot fully represent the various characteristics of ERP responses in the process of lie detection, and is easily affected by artifacts.

[0039] Traditional preprocessing methods include regression methods, which can eliminate artifacts and related electroencephalogram signals. On the other hand, subsequent methods such as ICA and wavelet transform have been shown to be more effective. Wavelet transform has a multi-resolution function and can adaptively adjust the window according to the frequency. Wavelet denoising can eliminate sharp noise interference, while independent component denoising can not only eliminate noise but also preserve the small details of the electroencephalogram signal.

[0040] For feature extraction, existing studies mostly proceed from the perspective of time and frequency domains. For example, Gao used the F-score algorithm to rank time domain features and found that amplitude and peak classification worked best. Tjandrasa

[12] It is proposed to use static wavelet transform (SWT) and discrete wavelet transform (DWT) for feature extraction, respectively. In addition, Arasteh

[13] A new EMD-based P300 deception detection method is proposed. These methods are mainly based on single-domain feature extraction. Although electroencephalogram has resolution in each domain, the features obtained through single-domain feature extraction may not be as sufficient as those obtained through multi-domain feature extraction.

[0041] Deep learning methods make up for the shortcomings of statistical methods. For example, wavelet packet transform (WPT) is used to extract features, and deep neural networks (DNN) are used for classification. After extracting multiple features of the electroencephalogram signal, the signal will be classified through a set framework that integrates multiple classifiers. Saini obtains intrinsic mode functions (IMF) after performing empirical mode decomposition (EMD) on the data. The first three IMFs are selected to extract statistical features in the time and frequency domains, and SVM is used for classification. In addition, a time-spectrum-based squeezing and excitation feature fusion network (TS-SEFFNet) is proposed for MI-EEG decoding. A practical hybrid unsupervised deep convolutional recurrent generative adversarial network model

[14] is proposed for electroencephalogram feature description and fusion (EEGFuseNet). A multi-branch deep learning fusion model (MBFM) is used to identify epileptogenic signals in the epileptogenic zone. Although deep learning models directly extract features from raw electroencephalogram data, they do not always guarantee accurate positioning of valuable features in electroencephalogram signals. Therefore, the present invention combines classical feature extraction methods with deep learning techniques.

[0042] The present application is directed to the adaptability of electroencephalogram in lie detection, and proposes a lie detection method based on a multi-feature fusion network of electroencephalogram, to solve the complexity problem of electroencephalogram in recognition. Figure 1 As shown, the following processes are included:

[0043] S1: Electroencephalogram acquisition

[0044] An improved double-probe complex test is used to obtain the original electroencephalogram data of multiple subjects in different states, to construct an electroencephalogram lie detection data set, and to select part of the original electroencephalogram data of the subjects as a training set.

[0045] The specific process of the improved double-probe complex test is as follows: 24 subjects are divided into a lying group and an honest group. The test is conducted in a quiet environment, all tests have been approved by the Institutional Review Board of the Institute of Psychology, Chinese Academy of Sciences, and are conducted in accordance with the principles set forth in the Helsinki Declaration.

[0046] All subjects in the lying group participate in electroencephalogram testing after the same scenario simulation. For example, each subject is required to enter a pre-arranged room and steal the most valuable item in the room. However, there is actually nothing valuable in the room except a check in the drawer. After claiming to have taken something, the subjects in the lying group are required to undergo electroencephalogram testing.

[0047] The subjects in the honest group are divided into two cases: in the first case, the subjects are required to listen to a narrative about a simulated case (the same as the scenario simulation of the lying group described above) read by the chief tester, which may include four test questions: 1. What was the stolen item? 2. What was in the stolen item? 3. How much was the stolen item worth? 4. Where was the stolen item taken from? Subsequently, the tester will ask the subjects to ensure that each subject has received the relevant information, and will ask each subject to provide correct answers to the four questions. Finally, each subject will participate in electroencephalogram testing. In the second case, the subjects directly participate in electroencephalogram testing.

[0048] At the beginning of the electroencephalogram test, each subject is placed in a soundproof electromagnetic shielded room in turn, the examiner reads the standardized experimental guide book, and the same text test stimulus sequence is used to obtain the original electroencephalogram data of each subject. Each subject can undergo multiple rounds of electroencephalogram testing, using a 64-channel ERP recording system from Neuroscan Company. The subject wears 64 electrodes on the head, and the electrode distribution position refers to the international 10-20 system. Preferably, the test stimulus sequence is displayed in the center of the screen, with black letters on a white background, in a bold font, and a size of 80. During the entire electroencephalogram test, the subject is required to focus their attention on the central region of the screen, with a distance of about 80 centimeters from the entire screen.

[0049] The test stimulus sequence is referenced to the DPCTP paradigm, and in order to keep consistent with the actual situation, the visual stimulus of stimulus 1 in the original paradigm is replaced by a word. For example, the test stimulus sequence of the whole electroencephalogram test is as shown in the following table: Figure 2 As shown in the table, the stimulus in the test experiment includes a check, an envelope, a check amount, and a drawer. In order to facilitate the subject to distinguish each test, a fixed point will be introduced before the test. In a single test, a fixed point in the form of a "+" symbol first appears and remains visible for 500 milliseconds. Subsequently, the word 1 appears for a duration of 300 milliseconds. After the subject perceives the word 1, the subject is required to press the left mouse button immediately. Subsequently, the subject has a reaction window of 1100 milliseconds to 1550 milliseconds, and the reaction time is randomly determined. After the reaction time ends, the word 2 will be displayed for 300 milliseconds. According to the instructions issued by the examiner before the test, the subject needs to press the left button when the specified irrelevant word (target stimulus) appears, and press the right button in other cases. The reaction window duration of this task is 2000 milliseconds, and the total test time is about 50 minutes. Before starting the formal test experiment, a series of exercises are performed.

[0050] According to the above test method, the original electroencephalogram signal data of 24 subjects can be obtained, and an electroencephalogram lie detection data set is constructed. Then, the original electroencephalogram signal data of 9 subjects can be randomly selected, of which the original electroencephalogram signal data of 8 subjects (preferably including subjects in both the lying group and the honest group) are used as a training set. The labeled data of the 8 subjects are used to train the MFNet network to obtain trained model parameters. The original electroencephalogram signal data of 1 subject is used as a test set to evaluate the performance of the model. The trained MFNet model is used to predict the classification label corresponding to the input data.

[0051] S2: Electroencephalogram signal data preprocessing

[0052] ICA algorithm (i.e., independent component analysis algorithm) is used to separate and remove artifacts from the original electroencephalogram signal in the training set. Wavelet transform is used to decompose and reconstruct the de-artifacted electroencephalogram signal to complete filtering and denoising of the electroencephalogram signal, as shown in the following table: Figure 3

[0053] In order to minimize redundant data, the electroencephalogram signal data collected by some electrodes can be selected, such as electrodes FP1, FP2, FZ, FCZ, CZ, C4, P3, PZ, P4, P6, PO3, and OZ. According to the MDPCTP mode, the first step of preprocessing uses the ICA algorithm to remove false differences, and the second step uses wavelet transform to filter and denoise the electroencephalogram signal through decomposition and reconstruction.

[0054] ​The first step of the preprocessing algorithm is to separate and remove the artifact components in the electroencephalogram signal from the mixed signal according to the ICA algorithm, so as to achieve the purpose of removing artifacts

[15] First, a non-mixing matrix B of separated independent signals is obtained, and then artifacts are found in the signal. The rows representing artifact components in the output signal Y are set to zero, and Y is corrected to Observed signal After correction, a relatively pure electroencephalogram signal is obtained.

[0055] After removing artifacts, the electroencephalogram signal still contains noise. The second step of the preprocessing algorithm is to decompose and reconstruct the artifact-removed electroencephalogram signal through wavelet transform, to complete the filtering and denoising of the electroencephalogram signal. Wavelet transform is based on the principle of signal decomposition, which decomposes the signal into a set of added wavelet functions. Wavelet analysis can simultaneously complete the time-frequency analysis of the electroencephalogram signal, and can also complete the "automatic scaling" in the frequency aspect, which is suitable for non-stationary signal processing of the electroencephalogram signal.

[0056] Assuming that the relatively pure electroencephalogram signal obtained after the above artifact removal processing is The electroencephalogram signal of one channel is The electroencephalogram signal is decomposed and reconstructed as follows, and φ(·) is the generator of multi-scale analysis. From the bi-scale equation, the following results can be obtained:

[0057]

[0058] where j is the decomposition layer number of the electroencephalogram signal, k is the time shift coefficient, t is the discrete time sequence number, φ j,k (t) is the multi-scale analysis generator of the electroencephalogram signal at the jth layer with a time shift coefficient k, Z represents the set of all integer values, φ j-1,r (t) is the multi-scale analysis generator of the electroencephalogram signal at the j-1th layer with a time shift coefficient r, h(·) is the dilation coefficient of the high-pass filter,

[0059] h((r-2k)=<φ j,r (t),φ j+1,r (t)> (2)

[0060] where φ j,r (t) is the multi-scale analysis generator of the electroencephalogram signal at the jth layer with a time shift coefficient r, and φ j+1,r (t) is the multi-scale analysis generator of the electroencephalogram signal at the j+1th layer with a time shift coefficient r.

[0061] c j,k represents the scale coefficient of the electroencephalogram signal at the jth layer with a time shift coefficient k, where is the input signal EEG signal of one channel, EEG signal The scale coefficient c of the EEG signal at layer j when the time shift coefficient is k is j,k Defined as:

[0062] c j,k =∫ R f(t)φ j,k (t)dt=∑ r∈Z h(r-2k)c j-1,r (3)

[0063] Among them, f(t) is the signal to be processed, that is, φ j,k The inner product of (t). The scale coefficient c of the current layer j j,k The expansion coefficient h(r-2k) of the high-pass filter and the scale coefficient c when the time shift coefficient of the previous layer j-1 is r j-1,r Similarly, the detail coefficient d of the EEG signal at layer j when the time shift coefficient is k j,k It can be introduced like this:

[0064] d j,k =∑ r∈Z g(r-2k)c j-1,r (4)

[0065] Where g(·) is the expansion coefficient of the low-pass filter. After decomposing into the corresponding frequency band, the coefficient c in the high-frequency noise frequency range is j,k and d j,k Zero it and then reconstruct it.

[0066] according to Through multi-level signal reconstruction, we can finally get a channel EEG signal The corresponding reconstructed signal The wavelet transform reconstruction formula is:

[0067] c j-1,r =∑ r∈Z h(k-2r)c j,r +∑ r∈Z g(k-2r)d j,r (5)

[0068] EEG signals All channel signals of the EEG signal are preprocessed through the above two steps. Can be reconstructed into a relatively pure signal

[0069] S3: Multi-domain feature extraction

[0070] The power spectrum estimation, wavelet transform and FBCSP algorithm are respectively used to extract the frequency domain features, time-frequency domain features and spatial features of the preprocessed electroencephalogram signal data, as shown in the following formula (1) to formula (8). Figure 4

[0071] S3.1 Extraction of frequency domain features

[0072] The power spectrum estimation is a method for estimating the relationship between the power of the received signal and the frequency by analyzing the correlation of the signal. The Welch algorithm based on the periodogram method is a simple algorithm which segments and windows the data to minimize the variance of the spectrum estimation while maintaining the resolution.

[0073] For the electroencephalogram signal, the Welch power spectrum estimation is used for frequency domain analysis. The general procedure includes segmenting and windowing the signal (adding a window function), calculating the power spectrum of each segment, and then obtaining the average power spectrum.

[0074] Suppose the preprocessed electroencephalogram signal data is The electroencephalogram signal of one channel is Divide into μ segments, each with a length of The data of segment l is represented as:

[0075]

[0076] Then, a window function w(i) is added to each data segment to obtain the periodogram of the data segment. The power spectrum of each windowed data segment is calculated as:

[0077]

[0078] where E is the normalization coefficient of the window function, The periodogram of each segment is approximately considered to be uncorrelated, and the power spectrum of all data segments is averaged to obtain the average power spectrum, which is the final power spectrum density estimate of the channel:

[0079]

[0080] Formula (9) represents the power distribution of the signal at different frequencies. Selecting the power values corresponding to t specific frequencies within a certain frequency range constitutes the feature vector of the channel (p ω1 ,p ω2 …p ωt ), where p ωt represents the power value corresponding to the tth selected frequency ω. The feature vector of each channel is obtained by the above method, and is spliced to form the frequency domain feature matrix F1. The dimension of F1 is σ × t, where σ represents the number of channels of the electroencephalogram signal, and each row corresponds to the feature vector of a channel. ​​

[0081] S3.2 Extraction of time-frequency domain features

[0082] Since the electroencephalogram has good time-frequency domain features, the features cannot be fully reflected from the time domain or the frequency domain alone, and therefore, the present application considers analyzing the signal from the time-frequency domain. Wavelet transform is generally used for time-frequency analysis. Assuming that the electroencephalogram signal after preprocessing of the electroencephalogram signal data is The electroencephalogram signal of one channel is The definition of the jth layer decomposition is as follows:

[0083]

[0084] Among them, c j is the approximate component corresponding to the signal in the jth layer decomposition, representing the low-frequency part, d j is the detail component of the signal in the jth layer decomposition, representing the high-frequency part. The approximate coefficients (i.e., scale coefficients) {c j,k} kΓZ and the detail coefficients {d j,k} k∈Z are defined by the above formulas (3) and (4), respectively. Using the wavelet coefficients of each layer (including the approximate coefficients and the detail coefficients), the reconstruction of the electroencephalogram signal of the channel is completed.

[0085] The detail coefficients capture high-frequency information, and the approximate coefficients capture low-frequency information, which can be used as time-frequency domain features. The wavelet transform can obtain the approximate coefficients corresponding to the approximate components of the corresponding frequency band through each layer decomposition, and the feature vector of the channel is {c j,k} k∈Z . The time-frequency domain feature vectors of each channel are obtained by the above method, and are spliced to form a time-frequency domain feature matrix F2. The number of rows of F2 is σ, and σ represents the number of channels of the electroencephalogram signal , and each row corresponds to a feature vector of a channel.

[0086] S3.3 Extraction of spatial features

[0087] The FBCSP algorithm is used to analyze the spatial domain of the electroencephalogram signal after preprocessing of the electroencephalogram signal data, and to extract spatial features. The FBCSP algorithm generally includes three stages: first, the preprocessed electroencephalogram signal is filtered into multiple frequency bands by using a filter bank; then, the feature vector of each frequency band of each channel is extracted by using the common spatial pattern (CSP) algorithm, and the feature vectors of all frequency bands of the same channel are normalized by variance to form the feature vector of the channel; finally, the feature vectors of all channels are spliced to obtain the final spatial feature matrix.

[0088] CSP is a spatial filtering method of binary classification, which can reflect various spatial distributions through multi-channel electroencephalogram signals

[16] The basic principle of the algorithm is to obtain a set of optimal filters through matrix diagonalization projection to maximize the variance between two signals, and obtain the feature vector with the highest discriminant.

[0089] First, the electroencephalogram signal data after preprocessing The electroencephalogram signal of one channel is filtered through a q-order band-pass IIR filter to obtain the filtered electroencephalogram data of μ frequency bands of the channel, and the filtered electroencephalogram data of the i-th frequency band is represented by , where i = 1, 2, …, μ, and the two-class original electroencephalogram signals of the frequency band are represented by two matrices: and They represent the electroencephalogram signals of the liars and the honest people in the frequency band, respectively.

[0090] Step 1, the mixed spatial covariance matrix of the two types of data is needed, and the covariance matrices Q1 and Q2 are obtained after normalizing the matrices and .

[0091]

[0092] In the above two formulas, is the transpose of matrix . trace represents the sum of all elements on the diagonal. Then the covariance matrix Q of the frequency band, i.e. the average covariance matrix of the two types of data, is calculated as: and represent the average of the normalized covariance matrices of the two classes (liars and honest people) signals, respectively.

[0093] Step 2, the feature matrix is calculated using principal component analysis. First, the covariance matrix Q of the frequency band is decomposed according to the following formula:

[0094] Q = VλV T (11)

[0095] where V is the eigenvector matrix of matrix Q, and λ is the diagonal matrix composed of eigenvalues. The whitening matrix can be obtained by descending the eigenvalues, as follows:

[0096]

[0097] where Q1 and Q2 can be transformed into G1 = JQ1J T and G2 = JQ2J T . G1 and G2 have a common eigenvector D, and G1 = Dλ1D TG2=Dλ2D T where λ1+λ2=I, I is an identity matrix.

[0098] Obviously, if the sum of the eigenvalues of the two types of matrices is always one, the largest eigenvalue in G1 can be minimized to G2. The eigenvalues in λ1 and λ2 are arranged in descending order and ascending order, respectively. Thus, the optimal variance of separating the two brain electrical signals is obtained. The projection matrix η is η=D T J.

[0099] Step 3 is feature extraction. Assuming that the test data is The feature Z can be obtained by the filter η i,j .

[0100]

[0101] The eigenvector of F is as follows:

[0102]

[0103] where VAR(·) is a variance operator, F cspi is the eigenvector of the i-th frequency band of the channel. According to the same method as described above, the eigenvector of each frequency band (a total of μ frequency bands) of the channel can be obtained, and then the eigenvectors of all frequency bands are normalized by variance to form the eigenvector of the channel {F cspi} i＝1…μ Finally, according to the number of channels σ of the brain electrical signal , the eigenvectors of all channels are spliced to obtain the final spatial feature matrix F3, the number of rows of the matrix F3 is σ, and each row corresponds to an eigenvector of a channel.

[0104] S4: fusion feature classification training

[0105] Through the above three feature extraction methods, the frequency domain, time-frequency domain and spatial domain brain electrical features F1, F2, F3 can be obtained. Next, the present application uses the MFNet network to further improve the feature representation capability, and uses several common convolution methods: two-dimensional convolution, deep convolution and separable convolution.

[0106] Based on the three convolutions, the MFNet network is divided into three branches, corresponding to block 1, block 2 and block 3, respectively. As Figure 5As shown, block 1 includes a two-dimensional convolution layer, a batch normalization layer 1, an activation layer 1, and an average pooling layer 1 arranged in sequence along the data transmission direction, block 1 is based on a two-dimensional convolution with a size of (1, 6) to further integrate the frequency domain features F1. Block 2 includes a separable convolution layer, a batch normalization layer 2, an activation layer 2, and an average pooling layer 2 arranged in sequence along the data transmission direction, block 2 uses separable convolution to explicitly decouple the relationship between feature maps and within feature maps to optimize the extracted time-frequency domain features F2, the separable convolution first applies convolution (depth convolution) to each input channel, and then integrates information through a point-by-point convolution (1x1 convolution). Block 3 includes a depth convolution layer, a batch normalization layer 3, an activation layer 3, and an average pooling layer 3 arranged in sequence along the data transmission direction, block 3 uses depth convolution to learn spatial features to further process spatial features F3, depth convolution can focus on spatial information of each input channel, helping the model to extract features related to spatial layout and further helping the model to understand the activity patterns of different brain regions. The output features of the three blocks are then connected through a feature connection layer, which means that the output feature maps of the three blocks are spliced in the feature dimension. The multi-dimensional feature map after connection usually needs to be flattened through a flattening layer to convert the multi-dimensional feature map into a one-dimensional vector, and then the three types of features are fused through a dense layer to form the final high-dimensional feature representation. Finally, the class label of the electroencephalogram signal data, i.e., lying or honesty, is obtained through a softmax classification layer.

[0107] The frequency domain features, time-frequency domain features, and spatial features extracted in step S3 are used to train the above MFNet network, and a deep learning-based fusion feature classification model is obtained after training. The overall training process is as follows: data input→F1 enters block 1 (two-dimensional convolution) to further integrate and extract frequency domain features; F2 enters block 2 (separable convolution) to further optimize the extraction of time-frequency domain features; F3 enters block 3 (depth convolution) to further process and extract spatial features→feature connection→flattening→dense layer fusion→softmax classification layer→output class label.

[0108] S5: Extract the original electroencephalogram signal data of the test liar, and perform data preprocessing and multi-domain feature extraction in steps S2 and S3 to obtain the frequency domain features, time-frequency domain features, and spatial features corresponding to the electroencephalogram signal of the test liar, which are input into the MFNet network (i.e., the fusion feature classification model) trained in step S4 to obtain the classification label of the test liar, i.e., lying or honesty. Here, the original electroencephalogram signal data of the test liar can also be extracted using the 64-channel ERP recording system of Neuroscan Company, and the liar wears 64 electrodes on the head, and the electrode distribution position refers to the international 10-20 system.

[0109] According to the MDPCTP paradigm, we first used ICA to remove artifacts, and then utilized wavelet transform to denoise the signal by decomposition and reconstruction. The sampling rate was reduced to 500 Hz. According to the sampling theorem, the highest frequency of the electroencephalogram signal is 250 Hz, while the typical frequency range of P300 is 1-10 Hz. In order to obtain the electroencephalogram data in this frequency band, the original data was decomposed into four layers. According to the approximation coefficients of the fourth layer, the signal was reconstructed in the time domain. Previous studies have shown that specific electrodes can detect prominent P300 components. In order to minimize redundant data, the following electrodes were particularly selected: FP1, FP2, FZ, FCZ, CZ, C4, P3, PZ, P4, P6, PO3, and OZ.

[0110] Welch power spectral estimation was performed on the electroencephalogram signals of the above channels. In the low frequency range, the power spectral estimation value of the deceiver was slightly higher than that under the honest condition. Therefore, the power spectral density value in the low frequency range was extracted as the frequency domain feature. The typical frequency range of P300 is 1-10 Hz, which corresponds to the approximation coefficient band of the fourth layer of wavelet decomposition. In the preprocessing stage, the electroencephalogram signal has undergone four-layer wavelet transform. Therefore, the fourth layer approximation coefficient can be directly extracted, and the average value is taken at each time point. Subsequently, spatial features are extracted using FBCSP.

[0111] The present invention designs a comparative experiment to compare five classical methods and four deep learning models. The accuracy (T), accuracy (L) and accuracy (H) represent the accuracy of all subjects, liars and honest people, respectively. The results are shown in Table 1. Traditional methods of processing electroencephalogram data usually involve single-domain feature extraction and classification using machine learning classifiers. However, due to the correlation between multi-domain electroencephalogram signals, the analysis of electroencephalogram signals by single-domain methods is limited. On the other hand, multi-domain feature extraction encounters challenges in feature dimensionality, making it difficult for traditional classifiers to achieve high-precision classification. In contrast, this paper proposes a model that combines traditional multi-domain feature extraction techniques with deep learning to achieve enhanced feature integration. This method not only retains the interpretability of traditional methods, but also improves the classification accuracy by utilizing deep learning.

[0112] Table 1

[0113]

[0114] The classification accuracy of the proposed multi-feature algorithm is compared with that of the single-domain feature extraction algorithm, and the results are shown in Table 2. This table clearly shows that the classification accuracy of any single-domain feature extraction method is lower than that of multi-domain feature extraction, thus highlighting the superiority of the proposed model.

[0115] Table 2

[0116] Domain Accuracy (T) Accuracy (L) Accuracy (H) Frequency domain 78.75 82.50 75.00 Time domain 85.42 87.50 83.33 Spatial domain 87.50 91.67 83.33 Frequency domain + spatial domain 86.25 85.83 86.67 Time domain + spatial domain 90.00 90.83 89.17 Time domain + frequency domain 85.83 83.33 88.33

[0117] To further demonstrate the advantages of the model proposed in the present application in the classification stage, the MFNet proposed in the present application is compared with traditional classifiers and EEGNet. The comparison results are shown in Table 3, which shows that the MFNet enhances the feature representation and achieves higher classification accuracy. In particular, the results show that the MFNet is superior to the random forest (RF) and adaboost (AB) classifiers.

[0118] Table 3

[0119] Method Accuracy Method Accuracy SVM 79.17 AB 87.23 KNN 83.78 LDA 86.07 RF 68.92 EEGNet 90.89 ELM 69.47 MFNet 92.04

[0120] The present application proposes a P300-based EEG spoofing detection method, which uses an improved dual-probe complex test scheme. In preprocessing, the present application uses a series of methods to denoise and improve the signal-to-noise ratio. We also propose a multi-feature extraction method. In the frequency domain, power spectrum estimation is used as the frequency domain feature; in the time-frequency domain, the approximate coefficients obtained by wavelet decomposition are used as the time-frequency domain feature; and in the spatial domain, FBCSP is used. This method can fully extract the features of P300 and has certain interpretability, while reducing the data volume. Experimental results show that compared with other competitive models, our model achieves better or close performance, which proves the effectiveness of the proposed model.

[0121] The multi-feature spoofing detection model proposed in the present application uses traditional signal processing methods to extract features, and then uses neural networks to enhance them to improve classification accuracy and feature representation. This method solves the limitations of previous studies, i.e., single-domain feature extraction cannot fully capture EEG features, and using neural networks alone lacks interpretability. Although the classification accuracy is high, the multi-domain model developed in this study still has certain limitations. First, in future research, the variation of P300 intensity at the same scalp position of different subjects should be considered. In addition, due to the limitation of experimental conditions, the available data is limited, and it is necessary to expand the data collection range in future research. Those skilled in the art, under the inspiration of the present application, without departing from the scope of the present application and the scope protected by the claims, can make many forms, which are all within the protection of the present application

[0122] Although the functions and working processes of the present application have been described above in combination with the drawings, the present application is not limited to the specific functions and working processes described above, and the above specific embodiments are only illustrative and not limiting. Those skilled in the art, under the inspiration of the present application, without departing from the scope of the present application and the scope protected by the claims, can make many forms, which are all within the protection of the present application.

Claims

1. A lie detection method based on a multi-feature fusion network of electroencephalogram signals, characterized in that, Comprising the following steps: S1: Electroencephalogram signal acquisition An improved double-probe complex test is used to obtain raw electroencephalogram signal data of multiple subjects in different states, to construct an electroencephalogram lie detection data set, and to select raw electroencephalogram signal data of some subjects as a training set; S2: Electroencephalogram signal data preprocessing ICA algorithm is used to separate and remove artifacts from the raw electroencephalogram signal data in the training set, and wavelet transform is used to decompose and reconstruct the artifact-removed electroencephalogram signal to complete the filtering and denoising of the electroencephalogram signal; S3: Multi-domain feature extraction Power spectrum estimation, wavelet transform, and FBCSP algorithm are used to extract frequency domain features, time-frequency domain features, and spatial features from the preprocessed electroencephalogram signal, respectively. The specific process is as follows: S3.1 Extraction of frequency domain features Welch power spectrum estimation is used for frequency domain analysis of the preprocessed electroencephalogram signal, the electroencephalogram signal of any channel of the preprocessed electroencephalogram signal is segmented and windowed, the power spectrum of each segment after windowing is calculated, then the average power spectrum is obtained, the power values at t specific frequencies are selected to form the feature vector of the channel, the same method is used to obtain the feature vector of each channel, and the frequency domain feature matrix is spliced; S3.2 Extraction of time-frequency domain features Wavelet transform is used for time-frequency domain analysis of the preprocessed electroencephalogram signal, the electroencephalogram signal of any channel of the preprocessed electroencephalogram signal is layered, the approximate coefficients corresponding to the approximate components of the corresponding frequency band are obtained through each layer decomposition, the feature vector of the channel is obtained, the same method is used to obtain the time-frequency domain feature vector of each channel, and the time-frequency domain feature matrix is spliced; S3.3 Extraction of spatial features The FBCSP algorithm is used for spatial domain analysis on the preprocessed electroencephalogram signal, and spatial features are extracted; first, the preprocessed electroencephalogram signal is filtered by using a filter bank The electroencephalogram signal of each channel is filtered into multiple frequency bands; then, the CSP algorithm is used for filtering of each frequency band of each channel, and a feature vector of each frequency band after CSP filtering is extracted; the feature vectors of all frequency bands of the same channel are normalized by variance to form a feature vector of the channel; finally, the feature vectors of all channels are spliced to obtain a final spatial feature matrix; S4: Fusion feature classification training The frequency domain features, time-frequency domain features, and spatial features extracted in step S3 are used to train the MFNet network to obtain a fusion feature classification model; S5: Extract the raw electroencephalogram signal data of the subject to be tested, and perform electroencephalogram signal data preprocessing and multi-domain feature extraction according to steps S2 and S3 to obtain the frequency domain features, time-frequency domain features, and spatial features corresponding to the electroencephalogram signal of the subject to be tested, and input them into the fusion feature classification model trained in step S4 to obtain the classification label of the subject to be tested, i.e., lying or honesty.

2. The lie detection method based on the electroencephalogram signal multi-feature fusion network according to claim 1, characterized in that, The specific process of the improved double-probe complex test in step S1 is as follows: 24 subjects are divided into a lying group and an honest group, the subjects in the lying group participate in electroencephalogram test after the same scenario simulation, the subjects in the honest group are divided into two cases, in the first case, the subjects listen to the same case description as the lying group and then participate in the electroencephalogram test, in the second case, the subjects directly participate in the electroencephalogram test; then each subject is placed in the same soundproof electromagnetic shielding room, and the same text test stimulus sequence is used to obtain the raw electroencephalogram signal data of each subject.

3. The lie detection method based on the electroencephalogram signal multi-feature fusion network according to claim 1, characterized in that, The MFNet network in step S4 includes block 1, block 2, block 3, a connection layer, a flattening layer, a dense layer and a softmax classification layer, the block 1 includes a two-dimensional convolution layer, a batch normalization layer 1, an activation layer 1 and an average pooling layer 1 arranged in sequence along the transmission direction of the electroencephalogram signal data, the block 2 includes a separable convolution layer, a batch normalization layer 2, an activation layer 2 and an average pooling layer 2 arranged in sequence along the transmission direction of the electroencephalogram signal data, and the block 3 includes a depth convolution layer, a batch normalization layer 3, an activation layer 3 and an average pooling layer 3 arranged in sequence along the transmission direction of the electroencephalogram signal data; The frequency domain features, the time-frequency domain features and the spatial features are respectively input into the block 1, the block 2 and the block 3, the output features of the three blocks are connected through the connection layer, the multi-dimensional feature maps after connection are flattened through the flattening layer, the multi-dimensional feature maps are converted into one-dimensional vectors, the feature fusion is performed through the dense layer, and finally the category label of the liar, that is, the lie or the honesty, is obtained through the softmax classification layer.

Images