A lie speech recognition method based on a hybrid neural network

Abstract

This invention discloses a lie detection method based on hybrid neural networks, comprising: extracting frequency domain information using a two-dimensional convolutional neural network and extracting temporal information using a recurrent neural network, and designing and constructing a CNN-LSTM lie detection model. Alternatively, a CNN-TCN lie detection model is designed and constructed based on extracting frequency domain information using a two-dimensional convolutional neural network and extracting temporal information using a temporal convolutional network. A comparative analysis of the proposed information fusion model and a single information model is conducted to address specific differences in training effectiveness and feature applicability. The results show that both information fusion methods effectively improve the test accuracy of lie detection. To reduce the adverse interference of redundant components on the information fusion process, an attention mechanism is introduced into the CNN-LSTM model. Analysis results show that the CNN-LSTM-ATT model can optimize the time step weights, further improving the lie detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of acoustic feature recognition technology, specifically a method for detecting lies in speech based on hybrid neural networks. Background Technology

[0002] Lie detection technology arose from and has been widely applied based on practical needs in fields such as case investigation and psychological analysis. Traditional lie detection methods rely on the tester's summary of existing lie detection experience and manual analysis of the subject's various physiological indicators to judge their psychological state and the veracity of a certain statement.

[0003] Lie detection technology has been extensively studied in recent decades. Manual lie detection based on physiological signals is a traditional technique, with multichannel recorders playing a crucial role in real-time monitoring of these signals. With the development of big data and artificial intelligence, intelligent lie detection algorithms based on physiological signals have advanced lie detection from manual analysis to intelligent identification. This technique effectively avoids human interference. However, like manual lie detection, it relies on contact-based physiological signal collection, which can cause adverse physiological and psychological interference to the subject in the lie detection environment. To reduce this interference, automatic lie detection technology based on speech signals and machine learning has emerged as a new method. This approach uses lie speech recognition algorithms to automatically identify lies in speech signals, ensuring the objectivity of the results while effectively reducing environmental impact on the subject. However, lie speech is complex and variable, and general automatic recognition algorithms struggle to be specifically designed for complex and subtle lie characteristics, leading to a decrease in lie detection accuracy.

[0004] Frequency domain and temporal information exhibit varying degrees of inter-class differences in lie speech detection tasks. While model construction based on either frequency domain or temporal information alone can objectively improve lie speech detection, such models inevitably lack effective representation of acoustic features, leading to a decline in both detection effectiveness and model performance. Integrating frequency domain and temporal information to design and construct a lie speech detection model can take into account the effective components of both types of information during lie speech detection, thereby improving the network model's ability to distinguish and fit both lie and real speech signals and their acoustic features. Summary of the Invention

[0005] To improve the accuracy of lie speech detection models, this invention proposes a lie speech identification method based on hybrid neural networks.

[0006] The technical solution adopted to achieve the purpose of this invention is as follows: A method for detecting lies in speech based on hybrid neural networks includes the following steps: 1) Convert the original speech samples into spectrograms by frame-segmentation, windowing, and fast Fourier transform; 2) The labeled speech dataset is preprocessed using speech spectrogram, Mel-frequency cepstral coefficient (MFCC), and Gamma-ton frequency cepstral coefficient (GFCC) to construct speech spectrogram feature dataset, Mel-frequency cepstral coefficient feature dataset, and Gamma-ton frequency cepstral coefficient feature dataset, respectively. 3) Construct a speech feature matrix using the three types of feature datasets mentioned above; the speech feature matrix and the speech samples in the corpus have a one-to-one correspondence. The overall feature dataset is a three-dimensional tensor, and the dimensions of the three-dimensional tensor include frequency domain dimension, temporal dimension, and sample order; among them, the frequency domain dimension of the speech spectrogram feature dataset is 512, the frequency domain dimension of the Mel frequency cepstral coefficient feature dataset is 20, and the frequency domain dimension of the gamma-ton frequency cepstral coefficient feature dataset is 64; the three types of feature datasets depend on the same preprocessing process, so the total temporal dimension is the same, which is 317. 4) For each category of samples, 80% of the samples are used as the training set and the remaining 20% ​​are used as the test set to ensure the balance between training and testing. 5) Use a two-dimensional convolutional neural network (CNN) to extract frequency domain information of acoustic features, and use a recurrent neural network (LSTM) and a temporal convolutional network (TCN) to extract temporal information of speech signals; 6) The two-dimensional convolutional neural network (CNN) is fused with the recurrent neural network (LSTM) and the temporal convolutional network (TCN) respectively to construct the CNN-LSTM hybrid neural network and the CNN-TCN hybrid neural network; 7) Using the speech spectrogram feature dataset, Mel frequency cepstral coefficient feature dataset, and Gamma-ton frequency cepstral coefficient feature dataset constructed in step 2), train and evaluate the CNN-LSTM hybrid neural network and the CNN-TCN hybrid neural network.

[0007] Further details show that the total number of samples in each category is 1135, including 600 honest samples and 535 lying samples. After selecting 80% of the training set and 20% of the test set, the training set contains 480 honest samples and 428 lying samples, and the test set contains 120 honest samples and 107 lying samples.

[0008] Further features, the methods for training CNN-LSTM hybrid neural networks and CNN-TCN hybrid neural networks include the following steps: a) Training the CNN-LSTM neural network model: Frequency domain information of the feature sequence of each frame is extracted based on a two-dimensional convolutional neural network, and temporal information of the overall speech signal is extracted based on a recurrent neural network; a fully connected layer is used as a classifier for lie speech detection. The fully connected layer takes the multi-channel output vector of the last time step of the recurrent layer as input data. The fully connected layer, together with each convolutional layer and recurrent layer, forms a lie speech recognition model that integrates temporal and frequency domain information. b) Training the CNN-TCN neural network model: Frequency domain information is extracted based on a two-dimensional convolutional neural network. In the temporal information extraction stage, a double-layer superimposed temporal convolutional network is used. The output of the first time step of the last layer of the temporal convolutional network is used as the input of the fully connected layer. The fully connected layer completes the final discrimination task of the model. c) Training the CNN-LSTM-ATT neural network model: The LSTM of the CNN-LSTM model is configured as a two-layer bidirectional recurrent layer structure, and the output of the last recurrent layer of the CNN-LSTM model is used as the input of the attention mechanism module. The attention mechanism module operates on the Long Short-Term Memory network and is embedded in the Encoder-Decoder model to output the predicted sequence. The attention mechanism is applied to automatic lie speech detection to generate an integrated sequence, and the integrated sequence is input into the fully connected layer for model training. The output of the attention mechanism module is adjusted according to the requirements of the receiver layer, and the optimization of the parameters within the attention mechanism module is carried out together with the optimization of the overall parameters of the Encoder-Decoder model.

[0009] Further features, in step 7), use the training and test sets of the constructed Spectrogram feature dataset, MFCC feature dataset, and GFCC feature dataset to complete the training and evaluation of the hybrid neural network.

[0010] A further feature is that the LSTM constructs recurrent layers along the forward and reverse directions of the time axis, and superimposes the hidden state vectors of the two layers; the output of the bidirectional recurrent neural network is represented as a single-channel state vector corresponding to each time step in sequence, while the state vector of the last time step of the unidirectional recurrent layer contains the historical information of all time steps.

[0011] A further feature is that the attention mechanism takes the state vector of the time step as input, assigns weight parameters to it through the inverse optimization of the overall model and performs temporal superposition, thereby obtaining the final integrated state vector. Based on the final integrated state vector, the LSTM represents the superimposed state vector in the bidirectional recurrent layer, using the state vectors from each time step. S = ( s 1; s 2; s 3; ···;s T ), as input s to the attention mechanism module t =(h t ,h t ), set the model parameter matrix W A and bias matrix B The softmax function is used to apply the softmax function to U=(u1;u2;u3;···;u T The weights are calculated at each time step. ;

[0012] right Perform dimensional transformation and multiply with the initial time step, then integrate the vectors to obtain the output sequence. C A :

[0013] The attention mechanism is modularly encapsulated, with the module taking the time step sequence as input and outputting an integrated state vector, thus embedding the attention mechanism module into the original model.

[0014] A further feature is that the attention mechanism is set only for recurrent neural networks, and the state vectors of each time step of the recurrent layer are used as the input feature vectors of the attention mechanism. The weights of the state vectors are optimized to enhance the LSTM recurrent feedback and form an LSTM-ATT network structure.

[0015] A further feature is that the TCN outputs a multi-channel feature matrix with the same shape as the original input, combining CNN with LSTM, LSTM-ATT and TCN respectively; The specific steps for combining CNN with LSTM and LSTM-ATT are as follows: The output of the CNN convolutional layer is used as the input of the LSTM, the final output of the LSTM recurrent layer is used as the input of the CNN fully connected layer, and the CNN fully connected layer is used as a classifier for lie speech detection, outputting the identification result. By concatenating CNN and TCN, and training the model using the three types of feature datasets that have already been constructed, the model is evaluated using a test set to verify that the frequency domain temporal fusion neural network can effectively improve the accuracy of lie detection.

[0016] The advantages of this invention are: This invention employs a two-dimensional convolutional neural network as the frequency domain information extraction algorithm for acoustic features, and recurrent neural networks and temporal convolutional networks as the temporal information extraction algorithms for speech signals, respectively. Lie detection models based on frequency domain and temporal information fusion are constructed for three types of speech acoustic features: speech spectrogram, Mel-frequency cepstral coefficients, and Gammatone-frequency cepstral coefficients. Comparative analysis of the proposed information fusion model and the single-information model shows that the fusion model effectively improves the accuracy of lie detection. To reduce the adverse interference of redundant components on the information fusion process, an attention mechanism is introduced into the CNN-LSTM model. Results show that the CNN-LSTM-ATT model can optimize the time step weights and improve lie detection accuracy. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the structure and partitioning of a dataset containing lie features; Figure 2 This is a schematic diagram of the CNN-LSTM model; Figure 3 It is the CNN-TCN neural network model structure; Figure 4 This is a schematic diagram of the attention mechanism module. Detailed Implementation

[0018] Since the methods described herein are implemented in a computer system, which can be located in the processor of a server or client, the methods described herein can be implemented as software executable as control logic, executed by the CPU in the server. The functions described herein can be implemented as a set of program instructions stored in a non-transitory tangible computer-readable medium. When implemented in this way, the computer program includes a set of instructions that, when executed by the computer, cause the computer to perform methods capable of implementing the described functions. The programmable logic can be temporarily or permanently mounted in a non-transitory tangible computer-readable medium, such as a read-only memory chip, computer memory, disk, or other storage medium. In addition to being implemented in software, the logic described herein can be embodied using discrete components, integrated circuits, programmable logic used in conjunction with programmable logic devices (such as field-programmable gate arrays (FPGAs) or microprocessors), or any other device including any combination thereof. All such implementations fall within the scope of this invention.

[0019] The following is in conjunction with the instruction manual appendix. Figure 1-4 To further describe the present invention in detail, a method for detecting lies in speech based on a hybrid neural network includes the following steps: 1) Convert the original speech samples into spectrograms by frame-segmentation, windowing, and fast Fourier transform; 2) The labeled speech dataset is preprocessed using speech spectrogram, Mel-frequency cepstral coefficient (MFCC), and Gamma-ton frequency cepstral coefficient (GFCC) to construct speech spectrogram feature dataset, Mel-frequency cepstral coefficient feature dataset, and Gamma-ton frequency cepstral coefficient feature dataset, respectively. 3) Construct a speech feature matrix using the three types of feature datasets mentioned above; the speech feature matrix and the speech samples in the corpus have a one-to-one correspondence. The overall feature dataset is a three-dimensional tensor, and the dimensions of the three-dimensional tensor include frequency domain dimension, temporal dimension, and sample order; among them, the frequency domain dimension of the speech spectrogram feature dataset is 512, the frequency domain dimension of the Mel frequency cepstral coefficient feature dataset is 20, and the frequency domain dimension of the gamma-ton frequency cepstral coefficient feature dataset is 64; the three types of feature datasets depend on the same preprocessing process, so the total temporal dimension is the same, which is 317. 4) For each category of samples, 80% of the samples are used as the training set and the remaining 20% ​​are used as the test set to ensure the balance between training and testing. 5) Use a two-dimensional convolutional neural network (CNN) to extract frequency domain information of acoustic features, and use a recurrent neural network (LSTM) and a temporal convolutional network (TCN) to extract temporal information of speech signals; 6) The two-dimensional convolutional neural network (CNN) is fused with the recurrent neural network (LSTM) and the temporal convolutional network (TCN) respectively to construct the CNN-LSTM hybrid neural network and the CNN-TCN hybrid neural network; 7) Using the speech spectrogram feature dataset, Mel frequency cepstral coefficient feature dataset, and Gamma-ton frequency cepstral coefficient feature dataset constructed in step 2), train and evaluate the CNN-LSTM hybrid neural network and the CNN-TCN hybrid neural network.

[0020] The original speech samples were converted into spectrograms by frame-segmented windowed Fast Fourier Transform. Then, the labeled speech dataset was preprocessed using the speech spectrogram, Mel frequency cepstral coefficients (MFCC), and Gammatone frequency cepstral coefficients (GFCC) to construct the Spectrogram feature dataset, MFCC feature dataset, and GFCC feature dataset, respectively.

[0021] Based on the aforementioned feature datasets, speech feature matrices are constructed. These feature matrices have a one-to-one correspondence with the speech samples in the corpus. The overall feature dataset can be viewed as a three-dimensional tensor (frequency domain dimension, temporal dimension, and sample order). The frequency domain dimension of the Spectrogram feature dataset is 512, the MFCC feature dataset is 20, and the GFCC feature dataset is 64. The total temporal dimension is determined by the speech preprocessing process. Since the three types of features share a common preprocessing step, their total temporal dimensions are the same, 317. For each category of samples, a fixed proportion of samples is used as the training set, and the remainder is used as the test set, thus maximizing the balance between training and testing. The original sample size is 1135, including 600 honest samples and 535 lying samples. With 80% of the samples used as the training set and 20% as the test set, the training set contains 480 honest samples and 428 lying samples, while the test set contains 120 honest samples and 107 lying samples.

[0022] Frequency domain information of acoustic features is extracted using two-dimensional convolutional neural networks (CNNs), and temporal information of speech signals is extracted using recurrent neural networks (LSTMs) and temporal convolutional networks (TCNs). CNNs are fused with LSTMs and TCNs respectively to construct CNN-LSTM and CNN-TCN neural networks. The constructed Spectrogram feature dataset, MFCC feature dataset, and GFCC feature dataset are used for training and evaluation testing.

[0023] The method for training a hybrid neural network of CNN-LSTM and CNN-TCN includes the following steps: The CNN-LSTM neural network model structure is as follows: Figure 2 As shown, the convolutional layer extracts the frequency domain information of the feature sequence of each frame, while the recurrent neural network extracts the temporal information of the overall speech signal. Finally, a fully connected layer is used as the segmenter for lie speech detection. The fully connected layer takes the multi-channel output vector of the last time step of the recurrent layer as input data, and together with each convolutional layer and recurrent layer, they form a lie speech recognition model that integrates temporal and frequency domain information.

[0024] The CNN-TCN neural network model structure is as follows: Figure 3 As shown, the CNN-TCN model still uses two-dimensional convolutional layers for frequency domain information extraction, but employs a double-layered temporal convolutional network for temporal information extraction, using the output of the last layer's initial time step as the input to the fully connected layer. The fully connected layer then completes the model's final discrimination task. The model structure is as follows: Figure 3 As shown.

[0025] The CNN-LSTM-ATT neural network model structure sets the LSTM of the CNN-LSTM model as a two-layer bidirectional recurrent layer structure, and uses the output of the last recurrent layer of the CNN-LSTM model as the input of the attention mechanism module. The attention mechanism is applied to the long short-term memory network, which is generally embedded in the Encoder-Decoder model to output the predicted sequence. The attention mechanism is applied to automatic lie speech detection. The integrated sequence is input into the fully connected layer for model training. The output of the attention mechanism module is adjusted according to the needs of the receiver layer. The optimization of the parameters within the module is carried out together with the optimization of the overall model parameters.

[0026] The models were trained and evaluated using the constructed Spectrogram feature dataset, MFCC feature dataset, and GFCC feature dataset, along with the test set. The test results and numerical differences between the two models are shown in the table below: Table 1 shows the average accuracy statistics of two lie detection models, CNN-LSTM and CNN-TCN.

[0027] Table 2 shows the test results statistics for the CNN-LSTM-ATT model. Example 1

[0028] This invention provides a method for detecting lies in speech based on hybrid neural networks, specifically:

[0029] Speech spectrograms, Mel-frequency cepstral coefficients (MFCCs), and Gammatone-frequency cepstral coefficients (GFCCs) are used to preprocess labeled speech datasets to construct feature vectors. The overall feature dataset can be viewed as a three-dimensional tensor (frequency domain dimension, temporal dimension, and sample order). The frequency domain dimension of the Spectrogram feature dataset is 512, that of the MFCC feature dataset is 20, and that of the GFCC feature dataset is 64. The total temporal dimension is determined by the speech preprocessing procedure. Since all three types of features depend on a single preprocessing step, the total temporal dimension is the same for all three: 317.

[0030] For CNN to learn the representation of speech feature frequency domain information, the convolution stride is 1, and the convolution kernel size is set differently for the three types of feature datasets; Table 3 shows the parameter settings for each layer of the convolution kernel.

[0031] LSTM constructs recurrent layers along both the forward and reverse time axes, and then superimposes their hidden state vectors. The output of the bidirectional recurrent neural network is represented as a single-channel state vector corresponding to each time step, while the state vector of the last time step of the unidirectional recurrent layer contains historical information from all time steps. Table 4 shows the recurrent layer parameter settings for the three types of feature datasets.

[0032] The ATT attention mechanism uses the state vectors at each time step as input to the attention mechanism. It assigns weight parameters to the overall model through inverse optimization and performs temporal superposition to obtain the final integrated state vector.

[0033] Based on the final integrated state vector, the LSTM represents the superimposed state vector in the bidirectional recurrent layer, using the state vectors from each time step. S = ( s 1; s 2; s 3; ···; s T ), as input s to the attention mechanism module t =(h t ,h t Set the model parameter matrix. W A and bias matrix B The softmax function is used to apply the softmax function to U=(u1;u2;u3;···;u T Calculate the weight vector at each time step. ;

[0034] right The dimensionality is transformed and multiplied with the initial time step, and then the vectors are integrated to obtain the output sequence (integrated state vector). C A :

[0035] The attention mechanism is modularly encapsulated. Each module takes a time-step sequence as input and outputs an integrated state vector, thus embedding the attention mechanism module into the original model. Figure 4 As shown.

[0036] ATT is configured only for recurrent neural networks, and the state vectors of each time step of the recurrent layer are used as the input feature vectors of the attention mechanism. The weights of the state vectors are optimized to enhance the recurrent feedback of LSTM, thus forming the LSTM-ATT network structure.

[0037] The TCN output is a multi-channel feature matrix with the same shape as the original input. The TCN layer parameter settings for the three types of feature datasets are shown in Table 5 below.

[0038] The specific steps for combining CNN with LSTM, LSTM-ATT, and TCN, and combining CNN with LSTM and LSTM-ATT, are as follows: The output of the CNN convolutional layer is used as the input of the LSTM, the final output of the LSTM recurrent layer is used as the input of the CNN fully connected layer, and the CNN fully connected layer is used as a classifier for lie speech detection, outputting the identification result.

[0039] The CNN-TCN concatenation model was trained using three types of feature datasets and then evaluated using a test set. The experiment verified that the frequency domain temporal fusion neural network can effectively improve the accuracy of lie detection.

[0040] Table 6: Average accuracy statistics of five lie detection models

[0041] Table 7 shows the statistical results of the test results for the CNN-LSTM-ATT model.

[0042]

[0043] The results show that the CNN-LSTM-ATT model can optimize the time step weights and improve the accuracy of lie detection.

Claims

1. A method for detecting lies in speech based on hybrid neural networks, characterized in that, Includes the following steps: 1) Convert the original speech samples into spectrograms by frame-segmentation, windowing, and fast Fourier transform; 2) The labeled speech dataset is preprocessed using speech spectrogram, Mel-frequency cepstral coefficient (MFCC), and Gamma-ton frequency cepstral coefficient (GFCC) to construct speech spectrogram feature dataset, Mel-frequency cepstral coefficient feature dataset, and Gamma-ton frequency cepstral coefficient feature dataset, respectively. 3) Construct a speech feature matrix using the three types of feature datasets mentioned above; the speech feature matrix and the speech samples in the corpus have a one-to-one correspondence. The overall feature dataset is a three-dimensional tensor, and the dimensions of the three-dimensional tensor include frequency domain dimension, temporal dimension, and sample order; among them, the frequency domain dimension of the speech spectrogram feature dataset is 512, the frequency domain dimension of the Mel frequency cepstral coefficient feature dataset is 20, and the frequency domain dimension of the gamma-ton frequency cepstral coefficient feature dataset is 64; the three types of feature datasets depend on the same preprocessing process, so the total temporal dimension is the same, which is 317. 4) For each category of samples, 80% of the samples are used as the training set and the remaining 20% ​​are used as the test set to ensure the balance between training and testing. 5) Use a two-dimensional convolutional neural network (CNN) to extract frequency domain information of acoustic features, and use a recurrent neural network (LSTM) and a temporal convolutional network (TCN) to extract temporal information of speech signals; 6) The two-dimensional convolutional neural network (CNN) is fused with the recurrent neural network (LSTM) and the temporal convolutional network (TCN) respectively to construct the CNN-LSTM hybrid neural network and the CNN-TCN hybrid neural network; 7) Using the speech spectrogram feature dataset, Mel frequency cepstral coefficient feature dataset, and Gamma-ton frequency cepstral coefficient feature dataset constructed in step 2), train and evaluate the CNN-LSTM hybrid neural network and the CNN-TCN hybrid neural network.

2. The lie speech detection method based on a hybrid neural network according to claim 1, characterized in that, The total number of samples in each category is 1135, including 600 honest samples and 535 lying samples. After selecting 80% of the training set and 20% of the test set, the training set contains 480 honest samples and 428 lying samples, and the test set contains 120 honest samples and 107 lying samples.

3. The lie speech detection method based on a hybrid neural network according to claim 1, characterized in that, The methods for training CNN-LSTM hybrid neural networks and CNN-TCN hybrid neural networks include the following steps: a) Training the CNN-LSTM neural network model: Frequency domain information of the feature sequence of each frame is extracted based on a two-dimensional convolutional neural network, and temporal information of the overall speech signal is extracted based on a recurrent neural network; a fully connected layer is used as a classifier for lie speech detection. The fully connected layer takes the multi-channel output vector of the last time step of the recurrent layer as input data. The fully connected layer, together with each convolutional layer and recurrent layer, forms a lie speech recognition model that integrates temporal and frequency domain information. b) Training the CNN-TCN neural network model: Frequency domain information is extracted based on a two-dimensional convolutional neural network. In the temporal information extraction stage, a double-layer superimposed temporal convolutional network is used. The output of the first time step of the last layer of the temporal convolutional network is used as the input of the fully connected layer. The fully connected layer completes the final discrimination task of the model. c) Training the CNN-LSTM-ATT neural network model: Set the LSTM of the CNN-LSTM model to a two-layer bidirectional recurrent layer structure, and use the output of the last recurrent layer of the CNN-LSTM model as the input of the attention mechanism module; The attention mechanism module operates on a long short-term memory network and is embedded in the Encoder-Decoder model to output a predicted sequence. The attention mechanism is applied to automatic lie speech detection to generate an integrated sequence, which is then input into a fully connected layer for model training. The output of the attention mechanism module is adjusted according to the requirements of the receiving layer, and the optimization of the parameters within the attention mechanism module is carried out in conjunction with the optimization of the overall parameters of the Encoder-Decoder model.

4. The lie speech detection method based on a hybrid neural network according to claim 1, characterized in that, In step 7), the training and evaluation of the hybrid neural network are completed using the training and testing sets of the constructed Spectrogram feature dataset, MFCC feature dataset, and GFCC feature dataset.

5. The lie speech detection method based on a hybrid neural network according to claim 1, characterized in that, The LSTM constructs recurrent layers along the forward and reverse time axes respectively, and superimposes the hidden state vectors of the two layers; the output of the bidirectional recurrent neural network is represented as a single-channel state vector corresponding to each time step in sequence, while the state vector of the last time step of the unidirectional recurrent layer contains the historical information of all time steps.

6. The lie speech detection method based on a hybrid neural network according to claim 3, characterized in that, The attention mechanism takes the state vector at each time step as input, assigns weight parameters to it through the inverse optimization of the overall model, and performs temporal superposition to obtain the final integrated state vector. Based on the final integrated state vector, the LSTM represents the superimposed state vector in the bidirectional recurrent layer, using the state vectors from each time step. S = ( s 1; s 2; s 3; ···; s T ), as input s to the attention mechanism module t =(h t ,h t ), set the model parameter matrix W A and bias matrix B The softmax function is used to apply the softmax function to U=(u1;u2;u3;···;u T Calculate the weight vector at each time step. ; ； right Perform dimensional transformation and multiply with the initial time step, then integrate the vectors to obtain the output sequence. C A : ； The attention mechanism is modularly encapsulated, with the module taking the time step sequence as input and outputting an integrated state vector, thus embedding the attention mechanism module into the original model.

7. The lie speech detection method based on a hybrid neural network according to claim 3, characterized in that, The attention mechanism is set only for recurrent neural networks, and the state vectors of each time step of the recurrent layer are used as the input feature vectors of the attention mechanism. The weights of the state vectors are optimized to enhance the LSTM recurrent feedback and form an LSTM-ATT network structure.

8. The lie speech detection method based on a hybrid neural network according to claim 1, characterized in that, The TCN outputs a multi-channel feature matrix with the same shape as the original input, and combines CNN with LSTM, LSTM-ATT and TCN respectively; The specific steps for combining CNN with LSTM and LSTM-ATT are as follows: The output of the CNN convolutional layer is used as the input of the LSTM, the final output of the LSTM recurrent layer is used as the input of the CNN fully connected layer, and the CNN fully connected layer is used as a classifier for lie speech detection, outputting the identification result. By cascading CNN and TCN, and training the model using the three types of feature datasets that have already been constructed, the model is evaluated using a test set to verify that the frequency domain temporal fusion neural network can effectively improve the accuracy of lie detection.

Images