A context cue based speech event extraction method
By constructing a context-cue-based speech event extraction model and utilizing deep neural networks and attention mechanisms, the problem of event extraction difficulties caused by the differences between speech and text is solved, achieving efficient and accurate speech event extraction, especially with significant performance improvements on Chinese and English datasets.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHEAST UNIV
- Filing Date
- 2023-09-26
- Publication Date
- 2026-06-02
Smart Images
Figure CN117373448B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer natural language processing, specifically relating to a method for extracting speech events based on contextual clues. Background Technology
[0002] Speech is the primary mode of communication in everyday conversations, widely used in meetings and interviews. Extracting event information directly from speech is an important but poorly researched problem. A common approach is to perform this task using a cascaded automatic speech recognition (ASR) system and a text-based event extraction system. In this process, the ASR system transcribes speech into text, and then the event extraction system extracts event information from the transcribed text. However, this pipelined approach suffers from a problem: errors in the ASR system can propagate to downstream event extraction systems, affecting overall performance—a phenomenon known as error propagation.
[0003] Unlike text-based event extraction, extracting event information directly from speech involves processing continuous speech signals as input. This presents additional challenges for the following reasons: First, speech is a continuous signal without well-defined word boundaries, making event element identification more complex. Second, the inherent differences between speech and text make mapping speech to text more difficult. Furthermore, semantic events are generally more complex than auditory events. Auditory events are directly identifiable, such as a dog barking or a door slamming, while semantic events involve further interpretation of these auditory events. For example, a car horn might indicate "traffic jam," and understanding such semantic events often requires considering additional contextual information, such as time, space, and the people involved. These factors make extracting events directly from speech significantly more challenging. Summary of the Invention
[0004] Objective: To overcome the shortcomings of existing technologies, this invention provides a method for efficient and accurate speech-based event extraction. By transcribing speech signals into text as contextual clues, the information between speech and text is coordinated to improve the quality and overall performance of event extraction.
[0005] Technical Solution: To achieve the above objectives, this invention provides a method for extracting speech events based on contextual clues, comprising the following steps:
[0006] S1: Constructing a context-cue-based speech event extraction model. This model aims to utilize the contextual information in the speech signal to make event extraction more accurate and refined. First, for the speech event extraction task, a deep neural network model based on contextual cues is constructed. The model is designed to fully utilize the contextual information in the speech signal to achieve accurate event extraction. The model includes two main components: a speech encoder and a text decoder, used to map the speech signal to event labels.
[0007] S2: Implementing the speech encoder. The speech encoder uses a deep neural network to convert the input speech signal into a high-dimensional feature representation. By capturing the time-frequency characteristics of the speech signal, the speech encoder generates speech features suitable for subsequent event extraction. The speech encoder uses a deep neural network, whose structure includes a multi-layer convolutional neural network and a position encoder. The convolutional neural network is used to extract high-dimensional feature representations from the input speech signal, and these features will play a key role in the subsequent event extraction process. The position encoder is used to enhance the positional information of the features to help the model understand the temporal characteristics of the speech signal.
[0008] S3: Implementing the text decoder. The text decoder employs an attention mechanism to map the feature representations generated by the speech encoder to a sequence of text events. Guided by contextual information, the text decoder decodes speech features frame by frame into corresponding event labels, thereby extracting speech events. The text decoder uses an attention mechanism to map the high-dimensional feature representations generated by the speech encoder to a sequence of text events. During decoding, the model can decode speech features frame by frame into corresponding event labels based on contextual information. The attention mechanism allows the model to assign different attention weights to different parts of the input, thereby better capturing the correlation between speech and text.
[0009] S4: Training the Speech Event Extraction Model. During the training phase, labeled speech data is used to optimize the parameters of the speech event extraction model by minimizing the negative log-likelihood loss function. The training process enables the model to accurately predict the event label corresponding to each frame of speech. The training data includes speech signals and their corresponding event labels, as well as relevant parameters used for model optimization.
[0010] S5: Test the speech event extraction model. During the inference phase, the speech signal of the event to be extracted is input into the pre-trained model. After feature extraction and decoding, the event label corresponding to each frame of speech is predicted, thereby realizing the extraction of the input speech event.
[0011] As an improvement to the present invention, the speech-oriented event extraction task in step S1 specifically includes:
[0012] Considering a dataset consisting of three subsets These subsets include the training set. Validation set and test set In this dataset, each instance is represented as a tuple. , of which Each instance consists of data from speech. Input of a digital signal sequence It consists of, and includes, the output tag sequence. .
[0013] in, This represents a special marker used to indicate event elements, such as "event type" or "role," while This represents the various content extracted from speech, including trigger words and arguments.
[0014] As an improvement to the present invention, the context-cue-based speech event extraction model in step S1 specifically consists of a speech encoder and a text decoder, with the transcribed text used as context cues to guide event generation. Under the stimulus of conditional generation, the output of Automatic Speech Recognition (ASR) is used as context cues for subsequent text decoding.
[0015] As an improvement to the present invention, the speech encoder in step S2 is specifically as follows:
[0016] First, for the speech input x, the feature extraction module performs an initial downsampling operation, where x is represented as an 80-channel log-amplitude Mel-spectrum. These representations are processed by two convolutional layers, each using a filter of width 3 and applying the GELU activation function for non-linear mapping. Simultaneously, to enhance the representation of positional information, sinusoidal position embeddings are incorporated into the generated feature representations, forming a representation vector. .
[0017] Next, the context representation vector is computed through a multi-layer Transformer encoder. This encoder consists of multiple Transformer blocks, each containing two main components: a self-attention layer and a feedforward neural network. These components work together to help capture contextual information and feature representations of the input data to generate a contextual representation.
[0018] As an improvement to the present invention, the text decoder in step S3 specifically includes:
[0019] The decoder generates words from the target text one by one using an autoregressive approach. In this generation process, the generation of each word is influenced by previously generated words and the encoded representation of the input speech. Specifically, the generation process can be represented as follows: .
[0020] in, Each layer contains a self-attention mechanism to handle the current state of the decoder. It also includes a cross-attention mechanism to take encoder state into account. These attention mechanisms help the decoder select the next word to generate based on previously generated content and speech encoding information.
[0021] As an improvement to the present invention, minimizing the negative log-likelihood loss function in step S4 is specifically as follows:
[0022]
[0023] in It is the training set. Indicates the optimal parameters. It's inputting voice. It is a sequence of generated transcribed text and predicted event structure.
[0024] As an improvement to the present invention, the likelihood function in step S4 is specifically as follows:
[0025]
[0026] in Defined as The accumulation, It is the output sequence The first in A tag.
[0027] As an improvement of the present invention, the training process in step S4 includes the following steps:
[0028] 4-1: Data preprocessing, which includes feature extraction of speech signals, conversion of event labels, etc., to provide suitable input for model training;
[0029] 4-2: Model initialization. The model initialization stage initializes the speech encoder and text decoder to prepare for subsequent training.
[0030] 4-3: Iterative training. The iterative training stage involves multiple rounds of iteration, using training data to train the model, continuously optimizing the model parameters, and improving the accuracy and robustness of speech event extraction.
[0031] As an improvement of the present invention, the reasoning process in step S5 includes the following steps:
[0032] 5-1: Voice input, wherein the voice input is the voice signal of the event to be extracted.
[0033] 5-2: Feature extraction. The feature extraction stage preprocesses the speech input and extracts speech feature representations.
[0034] 5-3: Event Extraction. The event extraction stage utilizes a trained speech event extraction model to decode the feature representation and predict the event label corresponding to the speech, thereby realizing the extraction of input speech events.
[0035] As an improvement of the present invention, the model employs an attention mechanism for information transfer between the speech encoder and the text decoder to improve the effectiveness and accuracy of speech event extraction.
[0036] Beneficial effects: Compared with existing technologies, this invention introduces speech recognition-transcribed text as contextual cues into the speech event extraction model, effectively narrowing the gap between text and speech modalities. This method brings significant improvements on Chinese and English datasets, achieving a maximum F1 gain of 10.7%. Attached Figure Description
[0037] Figure 1 This is a specific example diagram of the speech event extraction task introduced in the background technology;
[0038] Figure 2 This is a structural diagram of the context-cue-based speech event extraction model in an embodiment of the present invention. Detailed Implementation
[0039] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.
[0040] Example 1: See Figure 2 A method for extracting speech events based on contextual clues includes the following steps:
[0041] S1: Construct a speech event extraction model based on contextual cues. This model learns contextual information from the speech signal to achieve accurate event extraction.
[0042] S2: Implement a speech encoder that uses a deep neural network to convert the input speech signal into a high-dimensional feature representation. By capturing the time-frequency characteristics of the speech signal, it generates speech features for subsequent event extraction.
[0043] S3: Implement a text decoder. The text decoder employs an attention mechanism to map the feature representations generated by the speech encoder to a sequence of text events. It then uses contextual information to decode the speech features into corresponding event labels, thereby achieving speech event extraction.
[0044] S4: During the training phase, labeled speech data is used to optimize the parameters of the speech event extraction model by minimizing the negative log-likelihood loss function, enabling it to accurately predict the event labels corresponding to the speech.
[0045] S5: In the inference stage, the speech signal of the event to be extracted is input into the trained model. After feature extraction and decoding, the event label corresponding to the speech is predicted, thereby realizing the extraction of the input speech event.
[0046] Specifically, the speech-oriented event extraction task in step S1 is as follows:
[0047] Considering a dataset consisting of three subsets These subsets include the training set. Validation set and test set In this dataset, each instance is represented as a tuple. , of which Each instance consists of data from speech. Input of a digital signal sequence It consists of, and includes, the output tag sequence. .
[0048] in, This represents a special marker used to indicate event elements, such as "event type" or "role," while This represents the various content extracted from speech, including trigger words and arguments.
[0049] The context-cue-based speech event extraction model in step S1 specifically consists of a speech encoder and a text decoder. The transcribed text is used as context cues to guide event generation. Under the stimulus of conditional generation, the output of Automatic Speech Recognition (ASR) is used as context cues for subsequent text decoding.
[0050] The speech encoder in step S2 is specifically:
[0051] First, for the speech input x, the feature extraction module performs an initial downsampling operation, where x is represented as an 80-channel log-amplitude Mel-spectrum. These representations are processed by two convolutional layers, each using a filter of width 3 and applying the GELU activation function for non-linear mapping. Simultaneously, to enhance the representation of positional information, sinusoidal position embeddings are incorporated into the generated feature representations, forming a representation vector. .
[0052] Next, the context representation vector is computed through a multi-layer Transformer encoder. This encoder consists of multiple Transformer blocks, each containing two main components: a self-attention layer and a feedforward neural network. These components work together to help capture contextual information and feature representations of the input data to generate a contextual representation.
[0053] The text decoder in step S3 is specifically as follows:
[0054] The decoder generates words from the target text one by one using an autoregressive approach. In this generation process, the generation of each word is influenced by previously generated words and the encoded representation of the input speech. Specifically, the generation process can be represented as follows: .
[0055] in, Each layer contains a self-attention mechanism to handle the current state of the decoder. It also includes a cross-attention mechanism to take encoder state into account. These attention mechanisms help the decoder select the next word to generate based on previously generated content and speech encoding information.
[0056] The specific steps for minimizing the negative log-likelihood loss function in step S4 are as follows:
[0057]
[0058] in It is the training set. Indicates the optimal parameters. It's inputting voice. It is a sequence of generated transcribed text and predicted event structure.
[0059] As an improvement to the present invention, the likelihood function in step S4 is specifically as follows:
[0060]
[0061] in Defined as The accumulation, It is the output sequence The first in A tag.
[0062] The training process in step S4 includes the following steps:
[0063] 4-1: Data preprocessing, which includes feature extraction of speech signals, conversion of event labels, etc., to provide suitable input for model training;
[0064] 4-2: Model initialization. The model initialization stage initializes the speech encoder and text decoder to prepare for subsequent training.
[0065] 4-3: Iterative training. The iterative training stage involves multiple rounds of iteration, using training data to train the model, continuously optimizing the model parameters, and improving the accuracy and robustness of speech event extraction.
[0066] The reasoning process in step S5 includes the following steps:
[0067] 5-1: Voice input, wherein the voice input is the voice signal of the event to be extracted.
[0068] 5-2: Feature extraction. The feature extraction stage preprocesses the speech input and extracts speech feature representations.
[0069] 5-3: Event Extraction. The event extraction stage utilizes a trained speech event extraction model to decode the feature representation and predict the event label corresponding to the speech, thereby realizing the extraction of input speech events.
[0070] The model employs an attention mechanism between the speech encoder and the text decoder to improve the effectiveness and accuracy of speech event extraction.
[0071] Through the above steps, the context-cue-based speech event extraction method described in this embodiment can effectively convert speech signals into corresponding event tags, thus realizing the speech event extraction process. The advantage of this method is that it can fully utilize contextual cues to improve the accuracy and robustness of event extraction.
[0072] Example 2: This example applies a speech event extraction method based on contextual clues provided by the present invention, which specifically includes the following steps:
[0073] Step 1): Data preparation and model building
[0074] First, a large amount of voice data and corresponding event tags were collected from real-world scenarios. This voice data covers different types of voice events, such as "attack." Next, based on the technical solution provided by this invention, a context-cue-based voice event extraction model was constructed. This model includes a voice encoder and a text decoder to achieve the mapping from voice to event tags.
[0075] Step 2): Applying voice data
[0076] The collected actual speech data is input into the trained model for extraction. The model first uses a speech encoder to convert the input speech signal into a high-dimensional feature representation, and then uses a text decoder to map these features to a text event sequence. Through the model's decoding process, the event label corresponding to each frame of speech can be obtained.
[0077] Step 3): Results Analysis and Optimization
[0078] After obtaining the extracted event labels, the results are analyzed to evaluate the model's performance on different speech event types. Based on the analysis results, the model is optimized in a targeted manner, such as adjusting the model's hyperparameters and increasing training data, to improve the accuracy and robustness of event extraction.
[0079] To verify the effectiveness of the method of the present invention, this embodiment compares the performance of the method of the present invention with that of a baseline method. The comparison results are shown in Tables 1 and 2.
[0080] Table 1: Performance comparison results of the proposed method and the baseline method on Chinese and English datasets.
[0081]
[0082] As can be seen, the method of this invention outperforms the baseline method in F1-score on the speech event extraction datasets Speech-ACE05 (English), Speech-MAVEN (English), and Speech-DuEE (Chinese). This demonstrates that the method proposed in this invention can effectively improve the performance of speech event extraction.
[0083] In summary, the present invention provides a speech event extraction method based on contextual clues. By making full use of the contextual information in the speech signal and combining deep neural networks and attention mechanisms, it achieves accurate extraction and prediction of speech events. It is applicable to a variety of speech event extraction tasks and has broad application prospects and practical value.
[0084] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention. Equivalent transformations or substitutions made based on the above technical solutions all fall within the scope of protection of the claims of the present invention.
Claims
1. A method for extracting speech events based on contextual cues, characterized in that, Includes the following steps: S1: Construct a speech event extraction model based on contextual cues. This model learns contextual information from the speech signal to achieve accurate event extraction. S2: Implement a speech encoder that uses a deep neural network to convert the input speech signal into a high-dimensional feature representation. By capturing the time-frequency characteristics of the speech signal, it generates speech features for subsequent event extraction. S3: Implement a text decoder. The text decoder employs an attention mechanism to map the feature representations generated by the speech encoder to a sequence of text events. It then uses contextual information to decode the speech features into corresponding event labels, thereby achieving speech event extraction. S4: During the training phase, labeled speech data is used to optimize the parameters of the speech event extraction model by minimizing the negative log-likelihood loss function, enabling it to accurately predict the event labels corresponding to the speech. S5: In the inference stage, the speech signal of the event to be extracted is input into the trained model. After feature extraction and decoding, the event label corresponding to the speech is predicted, thereby realizing the extraction of the input speech event. Specifically, the speech-oriented event extraction task in step S1 is as follows: The dataset consists of three subsets. These subsets include the training set. Validation set and test set In this dataset, each instance is represented as a tuple. , of which Each instance consists of data from speech. Input of a digital signal sequence It consists of, and includes, the output tag sequence. , in, This represents a special marker used to indicate an event element. This represents the various content extracted from speech, including trigger words and arguments.
2. The method for extracting speech events based on contextual cues according to claim 1, characterized in that, The context-cue-based speech event extraction model in step S1 is specifically as follows: It consists of a speech encoder and a text decoder. The transcribed text is used as a contextual cue to guide event generation. Under the stimulus of conditional generation, the output of Automatic Speech Recognition (ASR) is used as a contextual cue for subsequent text decoding.
3. The method for extracting speech events based on contextual cues according to claim 1, characterized in that, The speech encoder in step S2 is specifically: First, for the speech input x, the feature extraction module performs an initial downsampling operation, where x is represented as an 80-channel log-amplitude Mel-spectrum. These representations are processed by two convolutional layers, each using a filter of width 3 and applying the GELU activation function for non-linear mapping. Simultaneously, to enhance the representation of positional information, sinusoidal position embeddings are incorporated into the generated feature representations, forming a representation vector. , Next, the context representation vector is computed through a multi-layer Transformer encoder. This encoder consists of multiple Transformer blocks, each containing two main components: a self-attention layer and a feedforward neural network. These components work together to help capture contextual information and feature representations of the input data to generate contextual representations.
4. The method for extracting speech events based on contextual cues according to claim 1, characterized in that, The text decoder in step S3 is specifically as follows: The decoder uses an autoregressive approach to generate words from the target text one by one. In this generation process, the generation of each word is influenced by previously generated words and the encoded representation of the input speech, as detailed below. The generation process is represented as follows: , in, Each layer contains a self-attention mechanism to handle the current state of the decoder. It also includes a cross-attention mechanism to take encoder state into account. These attention mechanisms help the decoder select the next word to generate based on previously generated content and speech encoding information.
5. The method for extracting speech events based on contextual cues according to claim 1, characterized in that, The specific steps for minimizing the negative log-likelihood loss function in step S4 are as follows: in It is the training set. Indicates the optimal parameters. It's inputting voice. It is a sequence of generated transcribed text and predicted event structure.
6. The method for extracting speech events based on contextual cues according to claim 1, characterized in that, The likelihood function in step S4 is specifically: in Defined as The accumulation, It is the output sequence The first in A tag.
7. The speech event extraction method based on contextual clues according to claim 1, characterized in that, The training process in step S4 includes the following steps: 4-1: Data preprocessing, which includes feature extraction of speech signals and conversion of event labels, to provide suitable input for model training; 4-2: Model Initialization. The model initialization phase involves initializing the speech encoder and text decoder to prepare for subsequent training. 4-3: Iterative training. The iterative training stage involves multiple rounds of iteration, using training data to train the model, continuously optimizing the model parameters, and improving the accuracy and robustness of speech event extraction.
8. The speech event extraction method based on contextual cues according to claim 1, characterized in that, The reasoning process in step S5 includes the following steps: 5-1: Voice input, wherein the voice input is the voice signal of the event to be extracted. 5-2: Feature extraction. The feature extraction stage preprocesses the speech input and extracts speech feature representations. 5-3: Event Extraction. The event extraction stage utilizes a trained speech event extraction model to decode the feature representation and predict the event label corresponding to the speech, thereby realizing the extraction of input speech events.
9. The speech event extraction method based on contextual clues according to claim 1, characterized in that, The speech event extraction model employs an attention mechanism between the speech encoder and the text decoder to improve the effectiveness and accuracy of speech event extraction.