Voice theme classification model based on pre-training and graph network
The model integrates semantic pre-training and graph networks to address the limitations of existing models by capturing local and global speech context, improving classification accuracy and reducing data requirements.
Patent Information
- Application Number
- CN202310752002.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-21
- Publication Date
- 2024-12-24
AI Technical Summary
Existing speech topic classification models primarily rely on acoustic features, which capture only local time-frequency information, leading to information loss in long speech segments and neglecting semantic content, while deep learning models like CNN and RNN prioritize locality and sequence, lacking global speech context understanding, resulting in low accuracy and high demand for labeled training data.
A model combining pre-training with wavlm to extract semantic features, using BiLSTM and CNN for local context, and a graph network to capture global discourse context, integrating local and global features through attention mechanisms for improved classification accuracy and interpretability.
Enhances classification accuracy by comprehensively understanding speech content with combined local and global features, reducing the need for large amounts of labeled data through pre-training and graph network generalization.
Smart Images
Figure FT_1 
Figure BDA0004301539000000041 
Figure BDA0004301539000000042
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of speech recognition, speech understanding and machine learning, and in particular to a model for constructing speech topic recognition by using pre-training and graph networks. Background Art
[0002] With the development of the times and the continuous advancement of network technology, personalized recommendations of short videos, recordings of meetings and other related audio information are playing an increasingly important role in our lives. It is particularly important to accurately classify and locate voice information in a large amount of voice information, thereby reducing the time it takes to obtain information.
[0003] Speech topic classification aims to automatically classify a large number of speech documents according to the topic of the content. It is often cascaded by Automatic Speech Recognition (ASR) and natural language understanding. In view of the rapid development of the two fields in recent years, the performance of the cascade system has been improved and gradually put into practical use. However, this type of design method has shortcomings such as error propagation, large training data requirements, and complex calculations. Originating from the end-to-end idea, direct semantic understanding based on speech waveforms has gradually become a mainstream research idea. Representative works include: Liu Chunxi proposed a scalable unsupervised unit, using UTD and AUD to process acoustic features, and automatically discovering a vocabulary of similar words or phoneme units to obtain speech tokenization; Dmitriy Serdyuk proposed an encoder-decoder structure that can directly map the acoustic features of speech to intent, and achieved good results in spoken language understanding tasks; Dong Zexian also proposed a similar encoder-decoder for end-to-end topic modeling. The proposed model self-learns some statistical features for topic identification based on MFCC acoustic features without relying on other information; Liu Tan et al. proposed a method of fusion of deep acoustic features and language features for speech classification. First, a traditional CTC-based acoustic model with phonemes as output units is trained, and the output of the previous layer of the linear phoneme classifier is used as the deep acoustic features of speech information. These deep acoustic features are input into a phoneme to word module to obtain deep language features, and finally a multi-head attention module is used to fuse these two deep features for topic classification. Most of the previous studies by scholars are based on the acoustic features of speech. However, acoustic features can only capture local time-frequency information in a very short time, which is defective for research such as speech topic classification that requires understanding of long-term speech information.
[0004] At the same time, deep learning models have been widely used in previous studies to learn feature representations, including CNN and RNN, such as LSTM. Because CNN and RNN give priority to locality and sequentiality, these deep learning models can better capture the sequence information of local continuous speech. However, when understanding the entire voice document, from a temporal point of view, the target speech is not only related to the local continuous speech sequence, but also to all (temporally continuous and non-continuous) speech sequences in the document. This relationship structure is called speech context here. Therefore, there is a lack of exploration of the global speech context in current research, which leads to the low accuracy of classifying audio with a long input time. Summary of the invention
[0005] The present invention aims at the deficiencies existing in the traditional classification methods and provides a model for speech topic classification based on pre-training and graph network. Most of the previous models are based on the acoustic features of speech. However, the acoustic features can only capture the local time-frequency information in a very short time. Extracting only the acoustic features will cause the information loss of long speech segments and the neglect of the semantic content. At the same time, because CNN and RNN give priority to locality and sequence, they can better capture the sequence information of local continuous speech. However, there is a lack of understanding of global information. Therefore, in view of the above problems, the scheme of the present invention first uses a pre-training model to better focus on the semantic information of speech, and at the same time combines the local sequential context features extracted by CNN with the global features constructed by the graph network, which not only greatly improves the accuracy of classification, but also improves the interpretability of the overall network structure. At the same time, the ability of large-scale pre-training to "learn from unlabeled speech" is combined with the ability of the graph network to "generalize from labeled data to unlabeled data", which alleviates the problem of large demand for training labeled data. Its specific technical scheme is as follows:
[0006] (1) Pre-training semantic feature extraction stage
[0007] 1.1) According to the actual usage environment, collect voice clips in advance, perform text topic annotation, and build a voice database.
[0008] 1.2) Perform silence removal and enhancement processing on speech to reduce the impact of noise and other interference factors.
[0009] 1.3) Input M voice data into the wavlm pre-training model to extract semantic features. The specific steps are: first, convert the voice features into the input form of the wavlm model. Generally speaking, the input of the wavlm model is a string of tokenized text data. Therefore, it is necessary to convert the voice features into text data for input into the wavlm model. This process can use a speech recognition engine to convert audio into text data. Then input the text data into the wavlm model to obtain the semantic vector of each word. Wavlm can automatically learn the semantic features of a language. By inputting the text data into the model, the semantic representation of each word in the model can be obtained. This semantic representation is usually a vector that can represent the position of the word in the semantic space. Finally, the semantic vectors of all words are weighted averaged to obtain the semantic feature vector of the entire voice segment.
[0010] 1.4) Use BiLSTM to extract local context features. The specific steps are: divide the input M voice data into multiple time series, and the input of each time series is a vector; input each time series into an LSTM layer, which can automatically learn the dependencies in the sequence data; use the output of the LSTM layer as the input of another LSTM layer, and use the opposite time order to obtain the output of the reverse LSTM layer; splice the outputs of the forward LSTM layer and the reverse LSTM layer to obtain the final effective features of the sequence data. 1.5) Use CNN to enhance local context features. CNN extracts N-gram information by stacking different convolution kernel sizes, and then highlights the most critical features extracted by each convolution operation through the maximum pooling operation, obtaining a more complex and abstract feature representation to enhance the local continuous speech semantic information.
[0011] (2) Graph Network Discourse Context Extraction Stage
[0012] 2.1) The semantic feature vector s of the entire speech segment obtained in (1) is i According to the number of utterances, it is transformed into a feature vector h of the utterance form. i .
[0013] s i =h1+h2+…+h n
[0014] where s i It represents the feature vector of the i-th speech segment.
[0015] 2.2) Construct the nodes of the graph network. The nodes in the directed graph are composed of N utterances in the speech document, and the feature vector h i initialization.
[0016] 2.3) Construct the edges of the graph network. Because the central node and adjacent nodes in the directed graph are connected, the present invention uses the events before and after the central node as context and constructs edges within a context window of size W. Therefore, each node in the directed graph is connected to at most 2W (including itself) adjacent nodes.
[0017] 2.4) Calculate the attention coefficient between nodes. The attention mechanism is used to automatically learn and optimize the connection relationship between nodes.
[0018] e ij =a([Wh i ||Wh j ])
[0019] The linear mapping of the shared parameter W increases the dimension of the node features, [·||·] concatenates the transformed features of nodes i and j, and finally a(·) maps the concatenated high-dimensional features to a real number to learn the correlation between nodes i and j. After obtaining the correlation coefficient, softmax is used to calculate the similarity weight between nodes i and j.
[0020]
[0021] 2.5) Top-k mask strategy. Here, α ij Contains the edge weights between node i and all nodes, including node i itself. We then adopt a top-k masking strategy for node selection to prioritize the important relationships between audio feature nodes and select the k most relevant nodes while mitigating interference from noise nodes.
[0022]
[0023] 2.6) Extraction of global features: A multi-head Attention mechanism is used to concatenate the features obtained, so that each output node i integrates the features of more related adjacent nodes to extract global information.
[0024]
[0025] K represents the independent calculation of K attentions, which can better extract information from the coarse and fine granularity. It can obtain a more comprehensive feature representation and is more conducive to learning the global information of the discourse context in the voice document.
[0026] (3) Feature fusion stage
[0027] The semantic feature vector s of the local context extracted by the pre-trained model iand the global feature vector h of the discourse context learned through the graph network i Concatenate and apply similarity-based attention mechanism to obtain the final feature representation:
[0028]
[0029]
[0030]
[0031] where h i The semantic feature vector s of the local context i and the global feature vector of the discourse context learned through the graph network The concatenated vector. i It is based on the weight value after attention. Finally, different weight values are added to the corresponding feature vectors so that the model can obtain feature representation based on similarity. Finally, through the ReLU activation function and combined with the fully connected network, and then through the softmax function for classification, the classification result is finally output:
[0032]
[0033] P i =softmax(W smax l i +b smax )
[0034]
[0035] is the topic category predicted by the model. Compare it with the topic category annotated in the text to determine the correctness of the classification.
[0036] The characteristics and beneficial effects of the present invention are: based on acoustic features, only local time-frequency information in a very short time can be captured. The present invention uses semantic features extracted by the wavlm pre-training model to replace the original acoustic features and combines them with the features of the speech context constructed by the graph network to greatly improve the accuracy of classification. The combination of local features and global features enables the model to understand the speech content more comprehensively and improves the overall interpretability. At the same time, the ability of large-scale pre-training to "learn from unlabeled speech" is combined with the ability of the graph network to "generalize from labeled data to unlabeled data", which alleviates the problem of large demand for training labeled data. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 This is a modeling process of a speech topic classification model based on pre-training and graph network proposed by the present invention. DETAILED DESCRIPTION
[0038] The present invention proposes a speech topic classification model based on pre-training and graph network. In order to make the above model of the present invention more obvious and easy to understand, it is further described in detail below in conjunction with the accompanying drawings and specific embodiments:
[0039] (1) Pre-training semantic feature extraction stage
[0040] 1.1) According to the actual usage environment, collect voice clips in advance, perform text topic annotation, and build a voice database.
[0041] 1.2) Perform silence removal and enhancement processing on speech to reduce the impact of noise and other interference factors.
[0042] 1.3) Input M pieces of speech data into the wavlm pre-training model to extract semantic features. The specific steps are: the present invention uses a pre-training model based on base, and uses a fourth-order language model trained with a 100-hour Librispeech label data set. And use the output of the last coding layer as a feature to perform downstream tasks. First, convert the speech features into the input form of the wavlm model. Generally speaking, the input of the wavlm model is a string of tokenized text data. Therefore, it is necessary to convert the speech features into text data for easy input into the wavlm model. This process can use a speech recognition engine to convert audio into text data. Then input the text data into the wavlm model to obtain the semantic vector of each word. Wavlm can automatically learn the semantic features of a language. By inputting text data into the model, the semantic representation of each word in the model can be obtained. This semantic representation is usually a vector that can represent the position of the word in the semantic space. The semantic vectors of all words are weighted averaged to obtain the semantic feature vector s of the entire speech segment. i .
[0043] 1.4) Create a dictionary and convert the semantic features s of M speech segments into i Splice by category so that each category topic corresponds to the voice data.
[0044] 1.5) Use BiLSTM to extract local context features. The specific steps are: divide the input M voice data into multiple time series, and the input of each time series is a vector; input each time series into an LSTM layer, and the LSTM layer can automatically learn the dependencies in the sequence data; use the output of the LSTM layer as the input of another LSTM layer, and use the opposite time order to obtain the output of the reverse LSTM layer; splice the outputs of the forward LSTM layer and the reverse LSTM layer to obtain the final effective features of the sequence data. The present invention uses a layer of BiLSTM with a hidden layer dimension of 32. One layer of Attention, the hidden layer dimension is 64.
[0045] 1.6) Use CNN to enhance local context features. CNN stacks three different convolution kernels with sizes of 2, 3, and 4, and all convolution layers use ReLU as the activation function. The outputs of the three parallel convolution layers are merged and connected into a 1200-dimensional feature vector to extract N-gram information, and then a one-dimensional maximum pooling operation is performed to highlight the most critical features extracted by each convolution operation, resulting in a more complex and abstract feature representation to enhance the local continuous speech semantic information.
[0046] (2) Graph Network Discourse Context Extraction Stage
[0047] 2.1) The semantic feature vector s of the entire speech segment obtained in (1) is i According to the number of utterances, it is transformed into a feature vector h of the utterance form. i .
[0048] s i =h1+h2+…+h n
[0049] where s i It represents the feature vector of the ith speech segment. In the present invention, the entire speech semantic feature is divided into 30 discourse features.
[0050] 2.2) Construct the nodes of the graph network. The nodes in the directed graph are composed of N = 30 utterances in the speech document, and the nodes are composed of the feature vectors h in the form of utterances. i initialization.
[0051] 2.3) Construct the edges of the graph network. Because the central node and adjacent nodes in a directed graph are connected, we use the events before and after the central node as context and construct edges within a context window of size W. Therefore, each node in the directed graph is connected to at most 2W = 30 (including itself) adjacent nodes.
[0052] 2.4) Calculate the attention coefficient between nodes. The attention mechanism is used to automatically learn and optimize the connection relationship between nodes.
[0053] e ij =a([Wh i ||Wh j ])
[0054] The linear mapping of the shared parameter W increases the dimension of the node features, [·||·] concatenates the transformed features of nodes i and j, and finally a(·) maps the concatenated high-dimensional features to a real number to learn the correlation between nodes i and j. After obtaining the correlation coefficient, softmax is used to calculate the similarity weight between nodes i and j.
[0055]
[0056] 2.5) Top-k mask strategy. Here, α ij Contains the edge weights between node i and all nodes, including node i itself. Then, we adopt the top-k mask strategy for node selection, prioritize the important relationships between audio feature nodes, and select the k most relevant nodes while mitigating interference from noise nodes. The present invention selects k=15.
[0057]
[0058] 2.6) Extraction of global features: A multi-head Attention mechanism is used to concatenate the features obtained, so that each output node i integrates the features of more related adjacent nodes to extract global information.
[0059]
[0060] K represents K independent calculations of attention. The present invention uses 8 attentions in total, which can better extract information from the coarse and fine granularity, obtain a more comprehensive feature representation, and is more conducive to learning the global information of the discourse context in the voice document.
[0061] (3) Feature fusion stage
[0062] The semantic feature vector s of the local context extracted by the pre-trained model i and the global feature vector of the discourse context learned through the graph network Concatenate and apply similarity-based attention mechanism to obtain the final feature representation:
[0063]
[0064]
[0065]
[0066] where h i The semantic feature vector s of the local context i and the global feature vector of the discourse context learned through the graph network The concatenated vector. i It is based on the weight value after attention. Finally, different weight values are added to the corresponding feature vectors so that the model can obtain feature representation based on similarity. Finally, through the ReLU activation function and combined with the fully connected network, and then through the softmax function for classification, the classification result is finally output:
[0067]
[0068] P i =softmax(W smax l i +b smax )
[0069]
[0070] is the topic category predicted by the model. Compare it with the topic category annotated in the text to determine the correctness of the classification.
Claims
1. The present invention proposes a model for speech topic classification based on pre-training and graph network, which is characterized by combining the semantic features extracted by pre-training with the features of the speech context constructed by the graph network to achieve better classification effect. It includes the following steps: (1) Pre-training semantic feature extraction stage 1.1) According to the actual usage environment, collect voice clips in advance, perform text topic annotation, and build a voice database. 1.2) Perform silence removal and enhancement processing on speech to reduce the impact of noise and other interference factors. 1.3) Input M speech data into the wavlm pre-trained model to extract semantic features. wavlm aggregates the semantic features of each speech data and uses methods such as average, maximum value, and weighted average to merge the features of multiple time steps into a fixed-length vector representation. 1.4) Extract local context features using BiLSTM. Build a BiLSTM model and use the serialized features as input. Adjust the model's hyperparameters such as the number of LSTM layers and hidden units as needed, and finally output the corresponding local feature representation. 1.5) Enhance local context features using CNN. Build a CNN model and take the local context feature sequence as input. You can adjust the model's hyperparameters, such as convolution kernel size, sliding window size, number of channels, etc., according to the task requirements, and finally output the corresponding enhanced feature representation. (2) Graph Network Discourse Context Extraction Stage 2.1) The semantic feature vector s of the entire speech segment obtained in (1) is i According to the number of utterances, it is transformed into a feature vector h of the utterance form. i . s i =h1+h2+...+h n where s i It represents the feature vector of the i-th speech segment. 2.2) Construct the nodes of the graph network. The nodes in the directed graph are composed of N utterances in the speech document, and the nodes are composed of the feature vector h in the form of utterances. i initialization. 2.3) Construct the edges of the graph network. The central node and adjacent nodes in a directed graph are connected, so the events before and after the central node are used as context, and edges are constructed within a context window of size W. Therefore, each node in a directed graph is connected to at most 2W (including itself) adjacent nodes. 2.4) Calculate the attention coefficient between nodes. The attention mechanism is used to automatically learn and optimize the connection relationship between nodes. e ij =a([Wh i ||Wh j ]) The linear mapping of the shared parameter W increases the dimension of the node features, [·||·] concatenates the transformed features of nodes i and j, and finally a(·) maps the concatenated high-dimensional features to a real number to learn the correlation between nodes i and j. After obtaining the correlation coefficient, softmax is used to calculate the similarity weight between nodes i and j. 2.5) Top-k mask strategy. Here, α ij Contains the edge weights between node i and all nodes, including node i itself. We then adopt a top-k masking strategy for node selection to prioritize the important relationships between audio feature nodes and select the k most relevant nodes while mitigating interference from noise nodes. 2.6) Extraction of global features. A multi-head mechanism of Attention is used to concatenate the features obtained, so that each output node i integrates the features of more related adjacent nodes to obtain a more comprehensive feature representation. This is more conducive to the model learning the global information of the discourse context in the voice document. (3) Feature fusion stage The semantic feature vector s of the local context extracted by the wavlm pre-training model i and the global feature vector h of the discourse context learned through the graph network i Concatenate and apply a similarity-based attention mechanism to obtain the final feature representation: Finally, a fully connected network is used to classify the speech features: P i =softmax(W smax L i +b smax ) is the topic category predicted by the model. Compare it with the topic category annotated in the text to determine the correctness of the classification.