Mental state detection method based on hierarchical multi-phonetic feature fusion model

Abstract

The present invention discloses a mental state detection method based on a hierarchical multi-phonetic feature fusion model, including preprocessing the original speech data and extracting phonetic features to obtain a phonetic feature set; inputting the established hierarchical fusion model according to the obtained phonetic feature set to perform feature level fusion and answer level fusion to obtain feature representations of all samples; and inputting the obtained sample feature representations into a classifier to determine the mental state of the sample. The present invention establishes a hierarchical fusion model and utilizes a designed convolutional neural network to mine deep features related to mental state from different phonetic features. In combination with channel perception and the classic attention mechanism, different weights are assigned to the local channels of the features and the global levels of different features, and the different importance of different features is learned, as well as different weights are learned for different answer fragments, thereby reducing the impact of data redundancy on the detection effect.

Description

Technical Field

[0001] The present invention relates to the technical field of mental state detection, and in particular to a mental state detection method based on a hierarchical multi-phonetic feature fusion model. Background Art

[0002] In recent years, with the need to improve the ability to manage social security, the public security system often needs to conduct preliminary mental assessments on suspects, so automatic mental state detection technology can provide assistance for mental assessments. Given that the public security system often records suspects in a question-and-answer format, using question-and-answer voice data to detect mental states has become the focus of this invention.

[0003] However, the existing mental state detection technology based on speech signal modeling is difficult to distinguish whether a certain mental state is positive or negative from the speech signals of different samples using traditional classifiers. With the development of deep learning technology, using deep neural networks to extract deep feature expressions related to mental states from phonetic features extracted from speech signals has become a mainstream method. However, on the one hand, different phonetic features reflect the characteristics of speech signals from different angles. Using only a small number of features may cause deviations for the detection of mental states and thus affect the detection effect. How to better utilize the complementarity between multiple different types of phonetic features and build a fusion model to improve the ability of mental state detection is an unresolved problem. On the other hand, since the question-and-answer speech data in mental state detection is often lengthy, the shortcomings of the existing technology are that directly modeling global information will lead to excessively large model parameters and it is difficult to mine feature expressions related to mental states. Although simply dividing the speech signal into equal lengths expands the data volume to a certain extent, each segmented data cannot represent each sample, and different questions and answers may not be highly correlated with the detection of mental states, resulting in a certain redundancy in the data. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art. To achieve the above purpose, a mental state detection method based on a hierarchical multi-phonetic feature fusion model is adopted to solve the problems raised in the above background technology.

[0005] A method for detecting mental state based on a hierarchical multi-phonetic feature fusion model, comprising:

[0006] Preprocessing the original speech data and extracting phonetic features to obtain a phonetic feature set;

[0007] The hierarchical fusion model established based on the obtained phonetic feature set is used to perform feature level fusion and answer level fusion to obtain feature representations of all samples;

[0008] The obtained feature representation of the sample is input into the classifier to determine the mental state of the sample.

[0009] As a further solution of the present invention: the specific steps of data preprocessing and phonetic feature extraction to obtain a speech feature set include:

[0010] Establish a mental state detection task and obtain a mapping function f, the prediction result y of the mapping function f i for:

[0011] y i =f(x i ),y i ∈{0,1};

[0012] Among them, y i =1 means the mental state is positive, y i =0 means the mental state is negative;

[0013] The original voice data is divided into a number of voice data segments according to the answer part of the question being asked;

[0014] The sample information is defined based on the obtained several voice data segments:

[0015] x i ={Q1,A1,…,Q j ,A j ,…,Q n ,A n};

[0016] Where n represents the number of questions, Q j (j∈[1,n]) represents each sample x i The jth problem in A j (j∈[1,n]) represents each sample x i Corresponding to Q j The jth answer of

[0017] According to several questions asked Q j Remove the question fragment and obtain a sample x′ that only retains the answer fragment i is: x′ i ={A1,…,A j ,…,A n};

[0018] According to the answer fragment A j Extract the phonetic feature set H j , expressed as:

[0019] Where k represents the number of phonetic feature types, j is the number of iThe index of the speech segments divided in ;

[0020] According to the obtained phonetic feature set H j Perform normalization and standardization.

[0021] As a further solution of the present invention: the phonetic feature set H obtained according to j The specific steps for normalization and standardization include:

[0022] Get the extracted phonetic feature set and perform m∈[1,k] is normalized, and the normalization formula is:

[0023]

[0024] Then, the standardization is performed according to the above results, and the standardization formula is:

[0025]

[0026] As a further solution of the present invention: the specific steps of the feature level fusion include:

[0027] Get the phonetic feature set H for preprocessing j Input level fusion model;

[0028] A convolutional neural network is established for each feature, and the deep features of each feature are extracted. The deep features are expressed as:

[0029]

[0030] in, l∈(1,k) represents the feature vector generated by the lth CNN. The convolutional neural network includes a one-dimensional convolutional neural network and a two-dimensional convolutional neural network. The output of the two-dimensional convolutional neural network is expressed as H, W, and C represent the height, width, and number of channels of the network output features, respectively;

[0031] At the same time, a channel-aware attention mechanism is used to learn the importance of different channels for the network output, and combined with global average pooling to encode it into a global feature vector

[0032] Then use the sigmoid gating mechanism to get the weight of each channel:

[0033] in, r is a hyperparameter;

[0034] Multiply each channel weight by the features output by the network:

[0035] Learn different weights for each feature:

[0036] Among them, W T , P are learnable parameters, Represents the weight of the lth feature;

[0037] The fusion representation of each answer segment obtained by weighted fusion is:

[0038] As a further solution of the present invention: the specific steps of answer level fusion include:

[0039] Attention mechanisms are integrated at the answer level, and different weights are learned for the feature representations of different answer fragments to represent the importance of different parts of the answer:

[0040]

[0041] Among them, W T , P are learnable parameters, Represents the weight of the j-th answer part;

[0042] The feature representation of each sample calculated by weighted fusion is:

[0043] The feature representation of each sample is input into the classifier to determine whether the mental state of the sample is positive. The classifier is: i =classifier(d i ).

[0044] Compared with the prior art, the present invention has the following technical effects:

[0045] By adopting the above technical solution, different phonetic features extracted from the original voice data are used, and multiple phonetic features are fused at the feature level and voice segments are fused at the answer level to obtain the mental state feature representation of each sample data, thereby effectively improving the model's effect on mental state detection. The answer content of each question is segmented, and then several types of phonetic features are extracted from each segmented voice segment. The convolutional neural network is combined with the channel-aware attention mechanism to obtain the representation of each feature, and then the attention mechanism is used to perform weighted fusion of the representations of different features to obtain the vector representation of each answer segment, and then the representations of different answer segments are fused and sent to the classifier to predict the mental state. Through the above design, the size of the deep neural network that needs to be constructed is reduced while improving the model's ability to detect mental states. It also reduces the impact of data redundancy on the detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings:

[0047] Figure 1 A schematic diagram of the steps of the mental state detection method of some embodiments disclosed in this application;

[0048] Figure 2 A mental state detection framework diagram of some embodiments disclosed in this application;

[0049] Figure 3 This is a data preprocessing flowchart for some embodiments disclosed in this application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Mental state detection can be considered as a binary classification problem. However, due to the characteristics of the data in the form of questions and answers, some of the questions asked are not related to a certain mental state, resulting in lengthy data. In addition, the features related to the mental state are hidden in the speech signal. It is difficult for traditional classifiers to effectively distinguish positive samples from negative samples directly from the speech signals of different samples. In addition, the differences in length and dimension of different initial phonetic features make it difficult to directly and effectively merge them. Features with weak correlation with a certain mental state may also become noise to a certain extent. In order to solve the above problems, most existing methods use a small number of phonetic features to directly divide the speech data into equal lengths to solve the problem of large model parameters caused by too long data, thereby performing supervised learning on different deep neural networks. However, on the one hand, this does not provide a good model for each sample for the mental state detection problem itself, and on the other hand, using a small number of features may only partially reflect the connection between data and mental state, thereby affecting the generalization ability and robustness of the model itself.

[0052] Please refer to Figure 1 and Figure 2 In an embodiment of the present invention, a method for detecting a mental state based on a hierarchical multi-phonetic feature fusion model includes:

[0053] S1. Preprocessing the original speech data and extracting phonetic features to obtain a phonetic feature set. The specific steps include:

[0054] S11. Establish the mental state detection task, obtain the mapping function f, and transform the data x of each sample i Input to the mapping function f to get the prediction result y i , the prediction result y of the mapping function f i for:

[0055] y i =f(x i ),y i ∈{0,1};

[0056] Among them, y i =1 means the mental state is positive, y i =0 means the mental state is negative;

[0057] S12, dividing the original voice data into a plurality of voice data segments according to the answer part of the question being asked;

[0058] Specifically, Figure 3 As shown in the figure, the interview data of a sample consists of several questions and answers. Therefore, directly modeling the global data will result in too many model parameters and it is difficult to mine the feature representation related to the mental state from the lengthy data. The data needs to be preprocessed.

[0059] S13, according to the obtained several voice data segments, a sample x i It can be seen as a series of questions and answers, and the sample information definition is expressed as:

[0060] x i ={Q1,A1,…,Q j ,A j ,…,Q n ,A n};

[0061] Where n represents the number of questions, Q j (j∈[1,n]) represents each sample x i The jth problem in A j (j∈[1,n]) represents each sample x i Corresponding to Q j The jth answer of

[0062] S14. According to the start and end time of each answer, several answer segments can be divided from the original speech signal, and then the question segments are removed according to several questions asked to obtain the answer segment x′ i is: x′ i ={A1,…,A j ,…,A n};

[0063] After getting the voice data x i After each answer segment in A, it is necessary to extract several phonetic features for each segment. j Extract the phonetic feature set H j , expressed as:

[0064] Where k represents the number of phonetic feature types, j is the number of i The index of the speech segments divided in ;

[0065] In the specific implementation steps, four types of manual features are extracted in the present invention, namely low-level descriptors, high-level statistical features, Bag-of-AudioWords (BoAW) and spectral features. The low-level descriptors use Mel frequency cepstral coefficients (MFCC) and feature sets eGeMAPs, where MFCC is calculated from 26 Mel spectrum bands calculated by FFT power spectrum with 25 milliseconds as an audio frame, using Hamming window framing, and the sampling rate is set to 10ms. It describes the energy of the cepstral on the Mel scale, and eGeMAPs contains frequency-related features (pitch, jitter, formant), energy-related features (flicker, loudness, harmonic to noise ratio), spectral parameters (alpha ratio, Hammarburg ratio, spectral slope -500hz and 500-1500Hz, formant 1,2,3 relative energy), harmonic differences between H1-H2 and H1-A3 and their statistical features, and six time features related to speech rate. Advanced statistical features are the arithmetic mean of variance and correlation coefficients calculated by mfcc and eGeMAPs. BoAW is generated using the openXBOW toolkit, which summarizes the audio into information consisting of different pseudo-audio words. Spectrogram is a visual representation of the audio signal, representing the amplitude of frequency components changing over time, generated by short-time Fourier transform (STFT).

[0066] S15, based on the obtained phonetic feature set H j Perform normalization and standardization. The specific steps include:

[0067] Get the extracted phonetic feature set H j , and for each feature m∈[1,k] is normalized and the value is constrained to the range of [0, 1]. The normalization formula is:

[0068]

[0069] Then, the standardization is performed according to the above results, and the standardization formula is:

[0070]

[0071] S2. Based on the obtained phonetic feature set, the hierarchical fusion model is input to perform feature level fusion and answer level fusion to obtain feature representations of all samples;

[0072] S21. The specific steps of feature level fusion include:

[0073] Get the phonetic feature set H for preprocessing j Input level fusion model;

[0074] A convolutional neural network is established for each feature, and the deep features of each feature are extracted. The deep features are expressed as:

[0075]

[0076] in, l∈(1,k) represents the feature vector generated by the lth CNN, and the convolutional neural network includes a one-dimensional convolutional neural network and a two-dimensional convolutional neural network;

[0077] In the specific implementation steps, since the low-level descriptor has the characteristics of a temporal one-dimensional signal, the present invention designs a one-dimensional convolutional neural network for it. The one-dimensional convolutional neural network used is composed of 9 convolutional blocks, each of which contains two identical structures, namely, a one-dimensional convolutional layer, a batch normalization, a ReLU activation function, and a dropout layer. The convolution kernel length is 5, and jump connections are used between layers to reduce the number of layers required by the model and accelerate convergence. The output dimensions of different layers are [64, 64, 64, 64, 128, 128, 128, 256, 256].

[0078] For other types of features:

[0079] The present invention is designed to adopt a 10-layer two-dimensional convolutional neural network, which is similar to the above-mentioned one-dimensional convolutional neural network structure. Each convolution block contains two identical structures consisting of a one-dimensional convolutional layer, batch normalization, a ReLU activation function and a dropout layer. The convolution kernel length is 5x5, and the skip connection structure is also adopted. The dimensions of the outputs of different layers are: [64, 64, 64, 64, 128, 128, 128, 256, 256, 256].

[0080] Among them, the output of the two-dimensional convolutional neural network is expressed as H, W, and C represent the height, width, and number of channels of the network output features, respectively;

[0081] At the same time, a channel-aware attention mechanism is used to learn the importance of different channels for the network output, and combined with global average pooling to encode it into a global feature vector

[0082] Then use the sigmoid gating mechanism to get the weight of each channel:

[0083] in, A structure with two fully connected layers is adopted, and r is a hyperparameter used to adjust the dimensionality reduction degree of the fully connected layer to reduce the complexity of the model and improve the generalization ability;

[0084] Multiply each channel weight by the features of the network output, so that different channels have different importance and improve the channel discrimination ability:

[0085] After performing channel-aware attention on each feature itself, different features are fused at the feature level, and different weights are learned for each feature:

[0086] Among them, W T , P are learnable parameters, Represents the weight of the lth feature;

[0087] Then, the fusion representation of each answer segment is obtained through weighted fusion:

[0088] S22, the specific steps of answer level fusion include:

[0089] In the specific implementation steps, feature level fusion only represents the representation of the complementary information fusion of different phonetic features of a speech segment, but it cannot represent the characteristics of a sample, because there are question and answer parts that are not related to the mental state in the question and answer data, and different questions and answers have different importance. Based on this, the second layer fusion of the present invention is at a higher semantic modeling level.

[0090] Attention mechanisms are integrated at the answer level, and different weights are learned for the feature representations of different answer fragments to represent the importance of different parts of the answer:

[0091]

[0092] Among them, W T , P are learnable parameters, Represents the weight of the j-th answer part;

[0093] The feature representation of each sample calculated by weighted fusion is:

[0094] S3. Input the obtained feature representation of the sample into the classifier to determine the mental state of the sample.

[0095] The feature representation of each sample is input into the classifier to determine whether the mental state of the sample is positive. The classifier is: i =classifier(d i ).

[0096] Specifically, a two-layer fully connected structure is used as a classifier to predict the result of each sample.

[0097] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included in the scope of protection of the present invention.

Claims

1. A method for detecting mental state based on a hierarchical multi-phonetic feature fusion model, characterized in that: include: Preprocessing the original speech data and extracting phonetic features to obtain a phonetic feature set; The hierarchical fusion model established based on the obtained phonetic feature set is used to perform feature level fusion and answer level fusion to obtain feature representations of all samples; The obtained feature representation of the sample is input into the classifier to determine the mental state of the sample; The specific steps of data preprocessing and phonetic feature extraction to obtain a speech feature set include: Establish a mental state detection task and obtain a mapping function f, the prediction result y of the mapping function f i for: y i =f(x i ),y i ∈{0,1}; Among them, y i =1 means the mental state is positive, y i =0 means the mental state is negative; The original voice data is divided into several voice data segments according to the answer part of the question being asked; sample information is defined according to the obtained several voice data segments: x i ={Q1,A1,…,Q j ,A j ,…,Q n ,A n }; Where n represents the number of questions, Q j (j∈[1,n]) represents each sample x i The jth problem in A j (j∈[1,n]) represents each sample x i Corresponding to Q j The jth answer of j Remove the question fragment and obtain a sample x that only retains the answer fragment i ′ is: x i ′={A1,…,A j ,…,A n }; According to the answer fragment A j Extract the phonetic feature set H j , expressed as: Among them, k represents the number of types of phonetic features, j is in x i The index of the speech segment divided in ′; According to the obtained phonetic feature set H j Perform normalization and standardization; The phonetic feature set H obtained according to j The specific steps for normalization and standardization include: Get the extracted phonetic feature set and perform Normalization is performed, and the normalization formula is: Then, the standardization is performed according to the above results, and the standardization formula is: The specific steps of the feature level fusion include: Get the phonetic feature set H for preprocessing j Input level fusion model; A convolutional neural network is established for each feature, and the deep features of each feature are extracted. The deep features are expressed as: in, represents the feature vector generated by the lth CNN, the convolutional neural network includes a one-dimensional convolutional neural network and a two-dimensional convolutional neural network, and the output of the two-dimensional convolutional neural network is expressed as H, W, and C represent the height, width, and number of channels of the network output features, respectively; At the same time, a channel-aware attention mechanism is used to learn the importance of different channels for the network output, and combined with global average pooling to encode it into a global feature vector Then use the sigmoid gating mechanism to get the weight of each channel: in, r is a hyperparameter; Multiply each channel weight by the features output by the network: Learn different weights for each feature: Among them, W T , P are learnable parameters, Represents the weight of the lth feature; The fusion representation of each answer segment obtained by weighted fusion is: The specific steps of answer level fusion include: Attention mechanisms are integrated at the answer level, and different weights are learned for the feature representations of different answer fragments to represent the importance of different parts of the answer: Among them, W T , P are learnable parameters, Represents the weight of the j-th answer part; The feature representation of each sample calculated by weighted fusion is: The feature representation of each sample is input into the classifier to determine whether the mental state of the sample is positive. The classifier is: i =classifier(d i ).

Images