An intelligent voice question-answering method and system based on deep learning and emotion recognition

This intelligent voice question answering method, which combines deep learning and emotion recognition, utilizes convolutional neural networks and bidirectional long short-term memory networks to construct semantic representation and matching models. This solves the problems of low accuracy and data redundancy in existing intelligent voice question answering systems, and achieves more efficient question answering system matching.

CN114203177BActive Publication Date: 2024-11-22SHENZHEN ZHENGTONG ELECTRONIC CO LTD +1

Patent Information

Application Number
CN202111475872.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-06
Publication Date
2024-11-22
Estimated Expiration
2041-12-06

AI Technical Summary

Technical Problem

Existing intelligent voice question answering systems suffer from low accuracy due to the ambiguity and complexity of natural language and the insufficient application of deep learning algorithms. Furthermore, they suffer from data redundancy and a single dimension when capturing Chinese sentence information, which can easily lead to system matching errors.

Method used

We employ an intelligent voice question answering method based on deep learning and emotion recognition. By acquiring user voice, we extract feature parameters and perform sentiment analysis. We then combine convolutional neural networks and bidirectional long short-term memory networks to construct a semantic representation and matching model for question answering database matching.

Benefits of technology

It improved the accuracy of the question-answering system, solved the problems of data redundancy and single dimension, and enhanced the input dimensions and matching accuracy of the question-answering system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114203177B_ABST
    Figure CN114203177B_ABST
Patent Text Reader

Abstract

The present invention discloses an intelligent voice question-answering method based on deep learning and emotion recognition, comprising: obtaining a voice from a user, and processing the voice to extract the corresponding text information, extracting feature parameters of the user's voice, and performing emotion analysis on the voice according to the extracted feature parameters to form an emotion label; inputting the generated text information into a trained semantic representation and matching model, combining the obtained emotion label, matching with a question-answering library, obtaining an answer to the question and outputting it. The present invention can solve the technical problem that the existing intelligent question-answering system still has a low accuracy rate due to the ambiguity and complexity of natural language and the deep learning algorithm is not widely used, and when capturing Chinese sentence information, there is still data redundancy and a single dimension, which easily causes a technical problem of system matching deviation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and more specifically, relates to an intelligent voice question-answering method and system based on deep learning and emotion recognition. Background Art

[0002] With the advent of the Internet era, data has grown exponentially, and people have begun to use search engines to find the information they need. As one of the forms of intelligent question-answering systems, search engines provide users with answers to their questions in the form of lists. As people have higher requirements for the efficiency and quality of information retrieval, the research on intelligent voice question-answering systems has also begun to develop further.

[0003] Existing intelligent voice question-and-answer systems usually require two steps: first, voice recognition, and then answering questions based on the recognized voice. Voice recognition technology is usually divided into three steps: feature extraction, deep neural network training and decoding. Question-and-answer systems are mainly divided into three categories: extractive intelligent question-and-answer, generative intelligent question-and-answer, and intelligent question-and-answer systems based on question-and-answer pairs. They all need to use natural language processing technology to convert sentences into structured data that can be recognized by computers, analyze questions, capture keywords, and finally match them with the answer library.

[0004] However, the above-mentioned intelligent question-answering system still has certain defects: first, the natural language processing technology it uses, due to the ambiguity and complexity of natural language, the deep learning algorithm is not widely used, resulting in the low accuracy of the question-answering system; second, when it captures Chinese sentence information, there is still data redundancy and a single dimension, which can easily cause system matching deviations. Summary of the invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides an intelligent voice question and answer system based on deep learning and emotion recognition, which aims to solve the technical problems that the accuracy of the existing intelligent question and answer system is relatively low due to the ambiguity and complexity of natural language and the limited application of deep learning algorithms, as well as the technical problem that when capturing Chinese sentence information, there is still data redundancy and a single dimension, which easily causes system matching deviation.

[0006] To achieve the above object, according to one aspect of the present invention, an intelligent voice question-answering method based on deep learning and emotion recognition is provided, comprising the following steps:

[0007] (1) Acquire the voice from the user and process the voice to extract the corresponding text information.

[0008] (2) Extract feature parameters of the user's speech and perform sentiment analysis on the speech based on the extracted feature parameters to form an emotion label.

[0009] (3) The text information generated in step (1) is input into the trained semantic representation and matching model, combined with the emotion label obtained in step (2), matched with the question-answer database, and the answer to the question is obtained and output.

[0010] Preferably, step (1) specifically includes the following sub-steps:

[0011] (1-1) Acquire speech from a user and perform digital encoding processing on the speech to convert it into an audio graph;

[0012] (1-2) using a Hamming window to perform frame division and windowing processing on the audio image obtained in step (1-1), and performing Fourier transform on the obtained processing result to obtain a spectrogram;

[0013] (1-3) Input the spectrogram obtained in step (1-2) into the trained convolutional neural network CNN to obtain the corresponding text information.

[0014] Preferably, step (2) specifically includes the following sub-steps:

[0015] (2-1) Use a Hamming window to frame the audio image obtained in step (1-1) to obtain preprocessed audio data.

[0016] (2-2) For each frame of audio data in the audio data preprocessed in step (2-1), the time domain features of the frame of audio data are obtained, and the time domain features corresponding to all frames are merged to obtain num initial feature parameters, whose dimension is 4.

[0017] (2-3) Using the principal component analysis method, PCA, the num initial feature parameters obtained in step (2-2) are successively subjected to dimensionality reduction and feature selection processing to obtain a mapping matrix A consisting of a subset of feature parameters.

[0018] (2-4) Based on the mapping matrix A obtained in step (2-3), calculate the comprehensive probability P corresponding to each principal component of the audio data preprocessed in step (2-1), and select the emotion type corresponding to the maximum comprehensive probability as the emotion label of the speech.

[0019] Preferably, step (2-2) comprises the following sub-steps:

[0020] (2-2-1) For each frame of audio data in the audio data preprocessed in step (2-1), obtain the short-time energy E of the frame of audio data n ;

[0021] (2-2-2) For each frame of audio data in the audio data preprocessed in step (2-1), obtain the short-time average zero crossing rate ZCR of the frame of audio data n :

[0022]

[0023] (2-2-3) For each frame of audio data in the audio data preprocessed in step (2-1), obtain the resonance peak characteristic parameter HZ of the frame of audio data n ;

[0024] (2-2-4) For each frame of audio data in the audio data preprocessed in step (2-1), the fundamental frequency period is extracted in the time domain using the short-time autocorrelation function, that is, d (the value range of d is 1 to 5, preferably 3) groups of signals are selected around the nth frame of audio data using the window function for autocorrelation calculation to obtain the fundamental frequency R of the frame of audio data n :

[0025]

[0026] Preferably, the short-time energy E of the n-th frame of audio data in step (2-2-1) n for:

[0027]

[0028] where x n (m) represents the audio graph obtained in step (1-1), which is the mth sample data in the nth (n∈[1,num])th frame of audio data after frame processing, and N represents the number of samples contained in one frame, which depends on the specific audio format.

[0029] Preferably, step (2-3) comprises the following sub-steps:

[0030] (2-3-1) For the num 4-dimensional initial feature parameters obtained in step (2-2), construct a feature vector A with 4 rows and num columns 4×num =[a 1 ,a 2 ,...,a num ]Enter the collected feature data;

[0031] (2-3-2) The feature vector A obtained in step (2-3-1) 4×num =[a 1 ,a 2 ,...,a num ], calculate its mean μ and covariance matrix COV 4×num ;

[0032] (2-3-3) For the covariance matrix COV in step (2-3-2) 4×num , calculate its eigenvalue and eigenvector, and the eigenvector and eigenvalue of each element form a coordinate form according to a one-to-one correspondence (λ i ,e i ) vector combination, sorted by eigenvalue to obtain the eigenvector sequence (λ 1 ,e 1 ),(λ 2 ,e 2 ),...,(λ num ,e num );

[0033] (2-3-4) For the feature vector sequence (λ 1 ,e 1 ),(λ 2 ,e 2 ),...,(λ num ,e num ), select the first k eigenvectors as the directions of the principal components (i.e., the basis vectors of the space after dimensionality reduction), and construct a 4-row k-column mapping matrix A, where the i-th column in the mapping matrix A represents the i-th eigenvector, where k∈[1,num].

[0034] Preferably, step (2-4) comprises the following sub-steps:

[0035] (2-4-1) For different emotion types, use the original data in the corpus to calculate the mean μ of the kth principal component of the mapping matrix A in the jth emotion type jk With variance σ jk , and perform the following maximum separability processing to obtain the kth principal component's ability to discriminate emotion types H k .

[0036]

[0037]

[0038]

[0039] Where J is the number of sentiment categories in the corpus, L k represents the separation of the kth principal component in the sentiment category, M k represents the concentration of the kth principal component in the sentiment category, C is the permutation and combination function; H k It is used to reflect the discriminative ability of the principal component in the emotional category. k When it is larger, the extracted principal component has a stronger ability to discriminate the emotion type;

[0040] (2-4-2) The ability of the principal component obtained in step (2-4-1) to distinguish the emotion type H k , after sorting the k principal components of the mapping matrix A, select H k The first p principal components with the largest size are used as the principal components for emotion recognition. The first p principal components are normalized to obtain the standardized principal components for emotion recognition. The projection matrix A of all principal components is summed to obtain the score value S of each principal component for emotion recognition. k .

[0041] (2-4-3) The score value S of each principal component of emotion recognition obtained in step (2-4-2) k , calculate the comprehensive probability P corresponding to each principal component in the user's voice, and select the emotion type corresponding to the maximum comprehensive probability as the emotion label of the user's voice; the comprehensive probability calculation formula is as follows:

[0042]

[0043] Preferably, step (3) specifically includes the following sub-steps:

[0044] (3-1) Performing word segmentation preprocessing on the Chinese information in the corpus to obtain a corpus after word segmentation;

[0045] (3-2) The word2vec model is used to encode and extract features from the word-segmented corpus obtained in step (3-1) to obtain a word matrix W of order c×v, where c is the number of all words in the sentence, and v is the vector dimension corresponding to each word. The dimension of the model is set to v=300. The word matrix W is used as the input for training the semantic representation and matching model;

[0046] (3-3) Input the word matrix W obtained in step (3-2) into the trained semantic representation and matching model to obtain the trained semantic representation and matching model.

[0047] (3-4) After word segmentation preprocessing of the text information generated in step (1), the word2vec model is used for encoding and feature extraction, and the result is input into the semantic representation and matching model trained in step (3-3) to obtain the feature vector of speech analysis.

[0048] (3-5) Calculate the Euclidean distance between the feature vector of the speech analysis obtained in step (3-4) and the existing question vector as vector similarity, and combine it with the emotion label obtained in step (2) to match the question answer in the question and answer library corresponding to the emotion label to obtain the answer to the question and output it.

[0049] Preferably, the semantic representation and matching model in step (3-3) are trained by the following steps:

[0050] (3-3-1) Use the c×v word matrix W obtained in step (3-2) as the input of the CNN network;

[0051] (3-3-2) Use the back propagation algorithm to update and optimize the weight parameters and bias parameters of each layer in the CNN convolutional neural network to obtain an updated CNN convolutional neural network;

[0052] (3-3-3) Iteratively train the CNN convolutional neural network updated in step (3-3-2) until the loss function of the CNN convolutional neural network reaches a minimum, thereby obtaining a preliminarily trained CNN convolutional neural network;

[0053] The loss function L of the CNN convolutional neural network is:

[0054]

[0055] Sample represents the total number of samples in the training set, Z represents the number of categories in the training set, and t i,z It represents the prediction result after the i-th training sample of the z-th class is input into the CNN convolutional neural network, y i,z represents the true result corresponding to the i-th training sample of the z-th class, i∈[1,Sample], λ represents the degree of regularization, and its value is 0.007, w i,z It represents the weight parameter when the i-th training sample of the z-th category is input into the CNN convolutional neural network, which changes automatically with the training of the CNN convolutional neural network.

[0056] (3-3-4) For the CNN convolutional neural network preliminarily trained in step (3-3-3), the 1-max pooling strategy is used to extract the maximum eigenvalue matrix C after the convolution of the convolution kernel vector, which is calculated as follows:

[0057] C = max[c 1 ,c 2 ,...,c n-h+1 ]

[0058] where c i Represents the features calculated by the convolution kernel for the i-th word in the sentence, which is produced during the training process of the CNN convolutional neural network.

[0059] (3-3-5) The maximum eigenvalue matrix C obtained in step (3-3-4) is used as the training input of the BiLSTM network, and features are extracted from the maximum eigenvalue matrix C. A feature vector for speech analysis is constructed as the output of the semantic representation and matching model to obtain a trained semantic representation and matching model.

[0060] According to another aspect of the present invention, an intelligent voice question-answering system based on deep learning and emotion recognition is provided, comprising:

[0061] The first module is used to obtain the voice from the user and process the voice to extract the corresponding text information.

[0062] The second module is used to extract feature parameters of the user's voice and perform sentiment analysis on the voice based on the extracted feature parameters to form an emotion label.

[0063] The third module is used to input the text information generated by the first module into the trained semantic representation and matching model, combine it with the emotion label obtained by the second module, match it with the question and answer library, obtain the answer to the question and output it.

[0064] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0065] (1) The present invention adopts step (3), integrates the convolutional neural network and the BiLSTM network, constructs a semantic representation and matching model to obtain a feature vector for speech analysis, and applies a more complete deep learning algorithm compared to the existing question-answering system.

[0066] (2) Since the present invention adopts step (2) to perform sentiment analysis on the user's voice and form sentiment labels, it enriches the input dimension of the question-answering system and can thus solve the problem of data redundancy and single dimension in the existing question-answering system. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a flow chart of the intelligent voice question-answering system based on deep learning and emotion recognition of the present invention;

[0068] Figure 2 It is a detailed flow chart of the intelligent voice question-answering system based on deep learning and emotion recognition of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0070] like Figure 1 and Figure 2As shown, the present invention provides an intelligent voice question-answering method based on deep learning and emotion recognition, comprising the following steps:

[0071] (1) Acquire the voice from the user and process the voice to extract the corresponding text information.

[0072] Furthermore, step (1) specifically includes the following sub-steps:

[0073] (1-1) Acquire speech from a user and perform digital encoding processing on the speech to convert it into an audio graph;

[0074] (1-2) using a Hamming window to perform frame division and windowing processing on the audio image obtained in step (1-1), and performing Fourier transform on the obtained processing result to obtain a spectrogram;

[0075] (1-3) Input the spectrogram obtained in step (1-2) into the trained convolutional neural network (CNN) to obtain the corresponding text information.

[0076] (2) Extract feature parameters of the user's speech and perform sentiment analysis on the speech based on the extracted feature parameters to form an emotion label.

[0077] Furthermore, step (2) specifically includes the following sub-steps:

[0078] (2-1) Using a Hamming window, the audio image obtained in step (1-1) is framed (after framing, each recording file generates a corresponding one-dimensional frame array), thereby obtaining preprocessed audio data.

[0079] (2-2) For each frame of audio data in the audio data preprocessed in step (2-1), the time domain features of the frame of audio data (including average pronunciation rate, fundamental frequency, short-time energy change, short-time average zero-crossing rate and resonance peak frequency) are obtained, and the time domain features corresponding to all frames are merged to obtain num initial feature parameters (whose dimension is 4).

[0080] Specifically, the processing of audio data in this step includes the following sub-steps:

[0081] (2-2-1) For each frame of audio data in the audio data preprocessed in step (2-1), obtain the short-time energy E of the frame of audio data n ;

[0082] Specifically, the audio graph obtained in step (1-1) has the mth sample data in the nth (n∈[1,num])th frame of audio data after frame processing as x n(m), then the short-time energy E of the nth frame of audio data is n for:

[0083]

[0084] Wherein N represents the number of samples contained in one frame, which depends on the specific audio format (eg, AAC has a fixed number of 1024 samples per frame, while the number of samples in MP3 format is 1152).

[0085] (2-2-2) For each frame of audio data in the audio data preprocessed in step (2-1), obtain the short-time average zero crossing rate ZCR of the frame of audio data n :

[0086]

[0087] (2-2-3) For each frame of audio data in the audio data preprocessed in step (2-1), obtain the resonance peak characteristic parameter HZ of the frame of audio data n ;

[0088] (2-2-4) For each frame of audio data in the audio data preprocessed in step (2-1), the fundamental frequency period is extracted in the time domain using the short-time autocorrelation function, that is, d (the value range of d is 1 to 5, preferably 3) groups of signals are selected around the nth frame of audio data using the window function for autocorrelation calculation to obtain the fundamental frequency R of the frame of audio data n :

[0089]

[0090] (2-3) Principal components analysis (PCA) is used to perform dimensionality reduction and feature selection on the num initial feature parameters obtained in step (2-2) to obtain a mapping matrix A consisting of a subset of feature parameters.

[0091] Specifically, this step includes the following sub-steps:

[0092] (2-3-1) For the num 4-dimensional initial feature parameters obtained in step (2-2), construct a feature vector A with 4 rows and num columns 4×num =[a 1 ,a 2 ,...,a num ]Enter the collected feature data;

[0093] (2-3-2) The feature vector A obtained in step (2-3-1) 4×num =[a 1 ,a2 ,...,a num ], calculate its mean μ and covariance matrix COV 4×num ;

[0094] (2-3-3) For the covariance matrix COV in step (2-3-2) 4×num , calculate its eigenvalue and eigenvector, and the eigenvector and eigenvalue of each element form a coordinate form according to a one-to-one correspondence (λ i ,e i ) vector combination, sorted by eigenvalue to obtain the eigenvector sequence (λ 1 ,e 1 ),(λ 2 ,e 2 ),...,(λ num ,e num );

[0095] (2-3-4) For the feature vector sequence (λ 1 ,e 1 ),(λ 2 ,e 2 ),...,(λ num ,e num ), select the first k (k∈[1,num]) eigenvectors as the directions of the principal components (i.e., the basis vectors of the space after dimensionality reduction), and construct a 4-row k-column mapping matrix A, where the i-th column in the mapping matrix A represents the i-th eigenvector.

[0096] (2-4) Based on the mapping matrix A obtained in step (2-3), calculate the comprehensive probability P corresponding to each principal component of the audio data preprocessed in step (2-1), and select the emotion type corresponding to the maximum comprehensive probability as the emotion label of the speech.

[0097] This step includes the following sub-steps:

[0098] (2-4-1) For different emotion types, use the original data in the corpus to calculate the mean μ of the kth (k∈[1,num])th principal component of the mapping matrix A in the jth emotion type jk With variance σ jk , and perform the following maximum separability processing to obtain the kth principal component's ability to discriminate emotion types H k .

[0099]

[0100]

[0101]

[0102] Where J is the number of sentiment categories in the corpus, L k represents the separation of the kth principal component in the sentiment category, M k represents the concentration of the kth principal component in the sentiment category, C is the permutation and combination function; H k It is used to reflect the discriminative ability of the principal component in the emotional category. k The larger it is, the stronger the ability of the extracted principal component to distinguish the emotion type.

[0103] (2-4-2) The ability of the principal component obtained in step (2-4-1) to distinguish the emotion type H k , after sorting the k principal components of the mapping matrix A, select H k The first p principal components with the largest size are used as the principal components for emotion recognition. The first p principal components are normalized to obtain the standardized principal components for emotion recognition. The projection matrix A of all principal components is summed to obtain the score value S of each principal component for emotion recognition. k .

[0104] (2-4-3) The score value S of each principal component of emotion recognition obtained in step (2-4-2) k , calculate the comprehensive probability P corresponding to each principal component in the user's voice, and select the emotion type corresponding to the maximum comprehensive probability as the emotion label of the user's voice.

[0105] The comprehensive probability calculation formula is as follows:

[0106]

[0107] (3) The text information generated in step (1) is input into the trained semantic representation and matching model, combined with the emotion label obtained in step (2), matched with the question-answer database, and the answer to the question is obtained and output.

[0108] Furthermore, step (3) specifically includes the following sub-steps:

[0109] (3-1) Perform word segmentation preprocessing on the Chinese information in the corpus to obtain a corpus after word segmentation.

[0110] (3-2) The word2vec model is used to encode and extract features from the word-segmented corpus obtained in step (3-1) to obtain a word matrix W of order c×v, where c is the number of all words in the sentence, and v is the vector dimension corresponding to each word. The dimension of the model is set to v=300. The word matrix W is used as the input for training the semantic representation and matching model.

[0111] (3-3) Input the word matrix W obtained in step (3-2) into the trained semantic representation and matching model (the model can analyze the semantics and structure of the question pair and construct a feature vector for speech analysis) to obtain a trained semantic representation and matching model.

[0112] Furthermore, the semantic representation and matching model in step (3-3) is trained by the following steps:

[0113] (3-3-1) Use the c×v word matrix W obtained in step (3-2) as the input of the CNN network;

[0114] (3-3-2) Use the back propagation algorithm to update and optimize the weight parameters and bias parameters of each layer in the CNN convolutional neural network to obtain an updated CNN convolutional neural network;

[0115] Specifically, the initial value of the weight parameter is a random value output by a truncated normal distribution with a standard deviation of 0.1, and the initial value of the bias parameter is set to 0;

[0116] (3-3-3) Iteratively train the CNN convolutional neural network updated in step (3-3-2) until the loss function of the CNN convolutional neural network reaches a minimum, thereby obtaining a preliminarily trained CNN convolutional neural network;

[0117] The loss function L of the CNN convolutional neural network is:

[0118]

[0119] Sample represents the total number of samples in the training set, Z represents the number of categories in the training set, and t i,z It represents the prediction result after the i-th training sample of the z-th class is input into the CNN convolutional neural network, y i,z represents the true result corresponding to the i-th training sample of the z-th class, i∈[1,Sample], λ represents the degree of regularization, and its value is 0.007, w i,z It represents the weight parameter when the i-th training sample of the z-th category is input into the CNN convolutional neural network, which changes automatically with the training of the CNN convolutional neural network.

[0120] (3-3-4) For the CNN convolutional neural network preliminarily trained in step (3-3-3), the 1-max pooling strategy is used to extract the maximum eigenvalue matrix C after the convolution of the convolution kernel vector, which is calculated as follows:

[0121] C = max[c 1 ,c 2 ,...,c n-h+1 ]

[0122] where ci Represents the features calculated by the convolution kernel for the i-th word in the sentence, which is produced during the training process of the CNN convolutional neural network.

[0123] (3-3-5) The maximum eigenvalue matrix C obtained in step (3-3-4) is used as the training input of the BiLSTM network, and features are extracted from the maximum eigenvalue matrix C. A feature vector for speech analysis is constructed as the output of the semantic representation and matching model to obtain a trained semantic representation and matching model.

[0124] (3-4) After word segmentation preprocessing of the text information generated in step (1), the word2vec model is used for encoding and feature extraction, and the result is input into the semantic representation and matching model trained in step (3-3) to obtain the feature vector of speech analysis.

[0125] (3-5) Calculate the Euclidean distance between the feature vector of the speech analysis obtained in step (3-4) and the existing question vector as vector similarity, and combine it with the emotion label obtained in step (2) to match the question answer in the question and answer library corresponding to the emotion label to obtain the answer to the question and output it.

[0126] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent voice question-answering method based on deep learning and emotion recognition, It is characterized in that The following steps are involved: (1) Acquire a voice from a user and process the voice to extract the corresponding text information; step (1) specifically includes the following sub-steps: (1-1) Acquire speech from a user and perform digital encoding processing on the speech to convert it into an audio graph; (1-2) using a Hamming window to perform frame division and windowing processing on the audio image obtained in step (1-1), and performing Fourier transform on the obtained processing result to obtain a spectrogram; (1-3) Inputting the spectrogram obtained in step (1-2) into the trained convolutional neural network CNN to obtain the corresponding text information; (2) extracting feature parameters of the user's speech, and performing sentiment analysis on the speech according to the extracted feature parameters to form an emotion label; step (2) specifically includes the following sub-steps: (2-1) using a Hamming window to frame the audio image obtained in step (1-1), thereby obtaining preprocessed audio data; (2-2) for each frame of audio data in the audio data preprocessed in step (2-1), obtaining the time domain features of the frame of audio data, and merging the time domain features corresponding to all frames to obtain num initial feature parameters, whose dimension is 4; (2-3) using the principal component analysis method PCA to successively perform dimension reduction and feature selection processing on the num initial feature parameters obtained in step (2-2) to obtain a mapping matrix A consisting of a subset of feature parameters; (2-4) calculating the comprehensive probability P corresponding to each principal component of the audio data preprocessed in step (2-1) according to the mapping matrix A obtained in step (2-3), and selecting the emotion type corresponding to the maximum comprehensive probability as the emotion label of the speech; (3) The text information generated in step (1) is input into the trained semantic representation and matching model, combined with the emotion label obtained in step (2), matched with the question-answer database, and the answer to the question is obtained and output.

2. According to claim 1, the intelligent voice question-answering method based on deep learning and emotion recognition, It is characterized in that Step (2-2) includes the following sub-steps: (2-2-1) For each frame of audio data in the audio data preprocessed in step (2-1), obtain the short-time energy E of the frame of audio data n ; (2-2-2) For each frame of audio data in the audio data preprocessed in step (2-1), obtain the short-time average zero crossing rate ZCR of the frame of audio data n : (2-2-3) For each frame of audio data in the audio data preprocessed in step (2-1), obtain the resonance peak characteristic parameter HZ of the frame of audio data n ; (2-2-4) For each frame of audio data preprocessed in step (2-1), the fundamental frequency period is extracted in the time domain using the short-time autocorrelation function, that is, d groups of signals are selected around the nth frame of audio data using the window function for autocorrelation calculation to obtain the fundamental frequency R of the frame of audio data n : where x n (m) represents the audio graph obtained in step (1-1), which is the mth sample data in the nth frame of audio data after frame processing. N represents the number of samples contained in a frame, which depends on the specific audio format. The value range of d is 1 to 5, n∈[1,num].

3. The intelligent voice question-answering method based on deep learning and emotion recognition according to claim 2, It is characterized in that The short-time energy E of the nth frame of audio data in step (2-2-1) n for:

4. The intelligent voice question-answering method based on deep learning and emotion recognition according to claim 3, It is characterized in that Step (2-3) includes the following sub-steps: (2-3-1) For the num 4-dimensional initial feature parameters obtained in step (2-2), construct a feature vector A with 4 rows and num columns 4×num =[a 1 ,a 2 ,...,a num ]Enter the collected feature data; (2-3-2) The feature vector A obtained in step (2-3-1) 4×num =[a 1 ,a 2 ,...,a num ], calculate its mean μ and covariance matrix COV 4×num ; (2-3-3) For the covariance matrix COV in step (2-3-2) 4×num , calculate its eigenvalue and eigenvector, and the eigenvector and eigenvalue of each element form a coordinate form according to a one-to-one correspondence (λ i ,e i ) vector combination, sorted by eigenvalue to obtain the eigenvector sequence (λ 1 ,e 1 ),(λ 2 ,e 2 ),...,(λ num ,e num ); (2-3-4) For the feature vector sequence (λ 1 ,e 1 ),(λ 2 ,e 2 ),...,(λ num ,e num ), select the first k eigenvectors as the directions of the principal components, and construct a 4-row, k-column mapping matrix A, where the w-th column in the mapping matrix A represents the w-th eigenvector, where k∈[1,num].

5. The intelligent voice question-answering method based on deep learning and emotion recognition according to claim 4, It is characterized in that Step (2-4) includes the following sub-steps: (2-4-1) For different emotion types, use the original data in the corpus to calculate the mean μ of the kth principal component of the mapping matrix A in the jth emotion type jk With variance σ jk , and perform the following maximum separability processing to obtain the kth principal component's ability to discriminate emotion types H k ; Where J is the number of sentiment categories in the corpus, L k represents the separation of the kth principal component in the sentiment category, M k represents the concentration of the kth principal component in the sentiment category, C is the permutation and combination function; H k It is used to reflect the discriminative ability of the principal component in the emotional category. k When it is larger, the extracted principal component has a stronger ability to discriminate the emotion type; (2-4-2) The ability of the principal component obtained in step (2-4-1) to distinguish the emotion type H k , after sorting the k principal components of the mapping matrix A, select H k The first p principal components with the largest size are used as the principal components for emotion recognition. The first p principal components are normalized to obtain the standardized principal components for emotion recognition. The projection matrix A of all principal components is summed to obtain the score value S of each principal component for emotion recognition. k ; (2-4-3) The score value S of each principal component of emotion recognition obtained in step (2-4-2) k , calculate the comprehensive probability P corresponding to each principal component in the user's voice, and select the emotion type corresponding to the maximum comprehensive probability as the emotion label of the user's voice; The comprehensive probability calculation formula is as follows:

6. The intelligent voice question-answering method based on deep learning and emotion recognition according to claim 5, It is characterized in that Step (3) specifically includes the following sub-steps: (3-1) Performing word segmentation preprocessing on the Chinese information in the corpus to obtain a corpus after word segmentation; (3-2) The word2vec model is used to encode and extract features from the word-segmented corpus obtained in step (3-1) to obtain a word matrix W of order c×v, where c is the number of all words in the sentence, and v is the vector dimension corresponding to each word. The dimension of the model is set to v=300; the word matrix W is used as the input for training the semantic representation and matching model; (3-3) Inputting the word matrix W obtained in step (3-2) into the trained semantic representation and matching model to obtain the trained semantic representation and matching model; (3-4) After pre-processing the text information generated in step (1) by word segmentation, the word2vec model is used for encoding and feature extraction, and the result is input into the semantic representation and matching model trained in step (3-3) to obtain the feature vector of speech analysis; (3-5) Calculate the Euclidean distance between the feature vector of the speech analysis obtained in step (3-4) and the existing question vector as vector similarity, and combine it with the emotion label obtained in step (2) to match the question answer in the question and answer library corresponding to the emotion label to obtain the answer to the question and output it.

7. The intelligent voice question-answering method based on deep learning and emotion recognition according to claim 6, It is characterized in that The semantic representation and matching model in step (3-3) is trained by the following steps: (3-3-1) Use the c×v word matrix W obtained in step (3-2) as the input of the CNN network; (3-3-2) Use the back propagation algorithm to update and optimize the weight parameters and bias parameters of each layer in the CNN convolutional neural network to obtain an updated CNN convolutional neural network; (3-3-3) Iteratively train the CNN convolutional neural network updated in step (3-3-2) until the loss function of the CNN convolutional neural network reaches a minimum, thereby obtaining a preliminarily trained CNN convolutional neural network; The loss function L of the CNN convolutional neural network is: Sample represents the total number of samples in the training set, Z represents the number of categories in the training set, and t i,z It represents the prediction result after the i-th training sample of the z-th class is input into the CNN convolutional neural network, y i,z represents the true result corresponding to the i-th training sample of the z-th class, i∈[1,Sample], λ represents the degree of regularization, and its value is 0.007, w i,z It represents the weight parameter when the i-th training sample of the z-th category is input into the CNN convolutional neural network, which changes automatically with the training of the CNN convolutional neural network; (3-3-4) For the CNN convolutional neural network preliminarily trained in step (3-3-3), the 1-max pooling strategy is used to extract the maximum eigenvalue matrix C after the convolution of the convolution kernel vector, which is calculated as follows: C=max[c 1 ,c 2 ,…,c n-h+1 ] where c i Represents the features calculated by the convolution kernel for the i-th word in the sentence, which is produced during the training process of the CNN convolutional neural network; (3-3-5) The maximum eigenvalue matrix C obtained in step (3-3-4) is used as the training input of the BiLSTM network, and features are extracted from the maximum eigenvalue matrix C. A feature vector for speech analysis is constructed as the output of the semantic representation and matching model to obtain a trained semantic representation and matching model.

8. An intelligent voice question-answering system based on deep learning and emotion recognition. It is characterized in that include: The first module is used to obtain the voice from the user and process the voice to extract the corresponding text information; the first module specifically includes the following submodules: The first submodule is used to obtain the voice from the user and perform digital encoding processing on the voice to convert it into an audio graph; The second submodule is used to use a Hamming window to perform frame division and windowing processing on the audio image obtained by the first submodule, and perform Fourier transform on the obtained processing result to obtain a spectrogram; The third submodule is used to input the spectrogram obtained by the second submodule into the trained convolutional neural network CNN to obtain the corresponding text information; The second module is used to extract feature parameters of the user's voice and perform sentiment analysis on the voice according to the extracted feature parameters to form an emotion label; the second module specifically includes the following submodules: The fourth submodule is used to frame the audio image obtained by the first submodule using a Hamming window, so as to obtain preprocessed audio data; A fifth submodule is used to obtain the time domain features of each frame of audio data in the audio data preprocessed by the fourth submodule, and merge the time domain features corresponding to all frames to obtain num initial feature parameters, whose dimension is 4; The sixth submodule is used to perform dimension reduction and feature selection processing on the num initial feature parameters obtained by the fifth submodule by using the principal component analysis method PCA, so as to obtain a mapping matrix A composed of feature parameter subsets; The seventh submodule is used to calculate the comprehensive probability P corresponding to each principal component of the audio data preprocessed by the fourth submodule according to the mapping matrix A obtained by the sixth submodule, and select the emotion type corresponding to the maximum comprehensive probability as the emotion label of the speech; The third module is used to input the text information generated by the first module into the trained semantic representation and matching model, combine it with the emotion label obtained by the second module, match it with the question and answer library, obtain the answer to the question and output it.

Citation Information

Patent Citations

  • Question-answering system combining emotion recognition and output

    CN103593054A

  • Intelligent robot, and emotional interaction method and system thereof

    CN107301168A

  • Method for performing emotion recognition on nature language

    CN107944008A

  • Intelligent dialogue generation method and device, computer equipment and computer storage medium

    CN110990543A

  • Oral communication device and computing architecture for processing data and outputting user feedback, and related methods

    CN111201566A

Cited By

  • Usage of emotion recognition in trade monitoring

    US20240127335A1