Mandarin stutter detection and identification method based on data enhancement

Abstract

The invention discloses a mandarin stutter detection and recognition method based on data enhancement. The method comprises the following steps: establishing a stutter event detection model and an automatic speech recognition module; training the built stutter event model and the automatic speech recognition module, wherein a data-enhanced stutter speech data set is adopted for training; and adopting the trained stutter event detection model and the automatic speech recognition module to detect and recognize the speech to obtain a stutter type and a recognized speech text. Through multiple data enhancement technologies, high-quality stutter voice data is systematically generated, a training data set is enriched, and the enhanced training set is adopted to train two models, namely a stutter event category model and a stutter text recognition model respectively, so that the performance of the stutter event detection and ASR system is improved.

Description

Technical Field

[0001] The present invention relates to the field of speech detection and recognition, and is applicable to scenarios of automobile loan telephone credit review, and in particular to a Mandarin stuttering detection and recognition method based on data enhancement for automobile finance telephone credit review in the automobile finance industry. Background Art

[0002] In the financial sector, telephone credit review is a key business process, mainly used to verify applicants' personal information, credit status, etc., in order to assess their credit risk. Credit review uses a speech recognition model to identify the user's voice information, but when the voice is stuttering, the traditional recognition effect is greatly reduced or even unusable.

[0003] Stuttering is a complex speech disorder characterized by disrupted speech fluency, including repetition, prolongation, and blocking, affecting approximately 1% of the world's population. Stuttering not only affects social functions, but also brings psychological stress, shame, and inferiority, and even causes patients to be afraid to speak and self-isolate. Stuttering event detection and recognition technology developed using deep learning technology is essential for the development of effective treatment tools and support systems. However, the training of deep learning models often requires the support of a large amount of speech data, but there is relatively little research and data in this area, resulting in existing speech event detection and recognition systems generally showing high error rates and low reliability when processing stuttered speech.

[0004] The prior art has the following defects when recognizing stuttered speech:

[0005] a. Existing automatic speech recognition (ASR) systems are trained to recognize normal speech. Therefore, when processing stuttering speech, the recognition accuracy is low and the error rate is high.

[0006] b. The stuttering event detection model lacks abundant training data, which leads to poor performance of the model in practical applications.

[0007] c. The lack of simulation methods for Mandarin stuttering speech has hindered the research and development of effective treatment tools and support systems, making it impossible to obtain speech recognition solutions for stuttering situations. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for detecting and identifying stuttering in Mandarin based on data enhancement, which is suitable for speech recognition in telephone credit review and other fields in the financial field, and is designed for stuttering speech to make its recognition accuracy higher.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is: a method for detecting and identifying stuttering in Mandarin based on data enhancement, the method comprising:

[0010] Build a stuttering event detection model and automatic speech recognition module;

[0011] Train the constructed stuttering event model and automatic speech recognition module using a data-enhanced stuttering speech dataset;

[0012] The trained stuttering event detection model and automatic speech recognition module are used to detect and recognize speech to obtain the stuttering type and the recognized speech text.

[0013] The stuttering event model built is used to detect stuttering events, which includes an Input layer, a Zipformer layer, and an Output layer. The speech data is input into the input layer and passes through the Zipformer layer and the Output layer in turn, and then the stuttering event category corresponding to the speech is output.

[0014] The Input layer in the stuttering event detection model consists of a filter bank fbank of features extracted from the speech signal; the Zipformer layer combines convolution operations and the Transformer self-attention mechanism to capture the time and context information in the speech signal; the Zipformer layer also has a U-Net-like encoder structure, which reduces the sequence to different low frame rates by downsampling.

[0015] The Output layer in the stuttering event detection model includes a softmax function layer, which performs probability prediction on the output of the Zipformer layer and divides the input speech into corresponding stuttering categories. The softmax function layer predicts the probabilities of five stuttering categories based on the features obtained by the Zipformer layer of the input speech, and outputs the stuttering category with the highest probability as the predicted stuttering event type; the five stuttering event categories are: extension, obstruction, sound repetition, word repetition and insertion.

[0016] The automatic speech recognition model uses an end-to-end model to transcribe stuttered speech into text. The automatic speech recognition model uses a transducer approach.

[0017] The automatic speech recognition model includes an Input layer, a Zipformer Encoder, a Transducer Decoder, and an Output layer. After the speech signal is input into the Input layer, it passes through the Zipformer Encoder and the Transducer Decoder in sequence, and then the Output layer outputs the corresponding recognized speech text.

[0018] Automatic speech recognition model

[0019] The input layer includes a filter bank fbank that extracts features from speech signals. The Zipformer Encoder uses a self-attention mechanism to encode the input features into a high-dimensional representation, capturing long-range dependencies and contextual information in speech signals through the self-attention mechanism.

[0020] Transducer Decoder uses a joint network to combine the encoder output with the prediction network to generate an output sequence; Output layer: finally predicts the probability distribution of a set of 4418 Chinese characters and selects the Chinese character with the highest probability distribution as the recognized text output.

[0021] The establishment of a data-enhanced stuttering speech database includes: selecting a Mandarin speech dataset as a benchmark corpus, using the Montreal forced alignment tool MFA to perform phoneme-level alignment on the speech data, and then performing data enhancement through various types of enhancement methods to obtain a speech dataset that simulates various stuttering events, namely the stuttering speech dataset.

[0022] Use the MFA tool to perform phoneme-level alignment on the Mandarin speech dataset to obtain accurate phoneme boundary information; divide the aligned data into smaller segments according to the phoneme boundaries; apply the corresponding data enhancement processing method to each segment to generate simulated stuttering speech data; and build a stuttering speech dataset based on the generated simulated stuttering speech data.

[0023] Data enhancement processing methods include:

[0024] Blocking: inserting random silence segments of set duration between words;

[0025] Sound repetition: a syllable is randomly selected and repeated for half its duration;

[0026] Word Repeat: Apply tempo perturbation to the selected segment, causing it to repeat multiple times;

[0027] Combined enhancement: applying blocking and sound repetition or blocking and word repetition enhancement to the same segment;

[0028] Normal: Randomly select segments from the original dataset without any modification.

[0029] The advantages of the present invention are: through a variety of data enhancement technologies, high-quality stuttering speech data is systematically generated to enrich the training data set, thereby improving the performance of stuttering event detection and ASR system. It is suitable for speech recognition in the financial field for telephone letter review and other aspects. It is designed for stuttering speech to make its recognition accuracy higher. The enhancement of stuttering data greatly increases the stuttering data set, improves the basic data for training, and increases the accuracy and reliability of the trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The following is a brief description of the contents expressed in the drawings of the present invention and the symbols in the drawings:

[0031] Figure 1 The process of constructing the data set of the present invention;

[0032] Figure 2 is a flow chart of stuttering category identification of the present invention;

[0033] Figure 3 The figure is a flow chart of outputting text for stuttering speech recognition according to the present invention. DETAILED DESCRIPTION

[0034] The specific implementation of the present invention will be further explained in detail below by describing the optimal embodiment with reference to the accompanying drawings.

[0035] This application is mainly aimed at improving the following defects in the existing technology for stuttering speech detection and recognition, so as to achieve accurate recognition of stuttering speech. The main defects of the existing technology include:

[0036] a. Lack of data: Existing stuttering event detection and ASR systems lack sufficient stuttering speech data for training, resulting in poor performance of the model in practical applications. Most existing speech recognition systems rely mainly on fluent speech data for training and lack specific training data for stuttering speech, resulting in low recognition accuracy and high error rate when processing stuttering speech.

[0037] b. Low recognition accuracy: The existing technology has low recognition accuracy and high error rate when processing stuttering speech. This is mainly because the existing event detection and speech recognition models lack sufficient stuttering speech data for training and cannot effectively recognize and process complex phenomena in stuttering speech, such as repetition, extension and blocking.

[0038] c. Weak data augmentation methods: Existing speech data augmentation methods lack systematicity and specificity, and cannot effectively simulate complex stuttering events, resulting in low quality of generated data that cannot meet training requirements. For example, inserting silent segments or repeating syllables usually cannot effectively simulate complex phenomena in stuttering speech.

[0039] In view of the above shortcomings, the present invention proposes a Mandarin stuttering speech simulation method based on data enhancement. Through a variety of data enhancement techniques, high-quality stuttering speech data is systematically generated to enrich the training data set, thereby improving the performance of stuttering event detection and ASR systems.

[0040] For defects a and c, we designed data enhancement technology based on normal Mandarin speech, which increased the richness of the stuttering speech dataset and improved the model training effect. For defect c, we designed a stuttering event detection model and an automatic speech recognition module to perform stuttering category recognition and stuttering speech conversion to text respectively. The following is a detailed introduction:

[0041] A data-enhanced Mandarin stuttering detection and recognition method in this solution is suitable for scenarios such as financial credit review where stuttering speech recognition is required. It includes the following steps:

[0042] S1. Build a stuttering event detection model and automatic speech recognition module;

[0043] S2, training the constructed stuttering event model and automatic speech recognition module, using a data-enhanced stuttering speech dataset;

[0044] S3. Use the trained stuttering event detection model and automatic speech recognition module to detect and recognize speech to obtain the stuttering type and the recognized speech text.

[0045] The stuttering event model built is used to detect stuttering events. It includes an input layer, a Zipformer layer, and an output layer. The speech data is input into the input layer and passes through the Zipformer layer and the Output layer in sequence to output the stuttering event category corresponding to the speech. The input layer in the stuttering event detection model consists of a filter group fbank of features extracted from the speech signal. The Zipformer layer combines convolution operations and the Transformer self-attention mechanism to capture the time and context information in the speech signal. The Zipformer layer also has a U-Net-like encoder structure, which reduces the sequence to different low frame rates by downsampling. The Output layer in the stuttering event detection model includes a softmax function layer, which predicts the probability of the output of the Zipformer layer and divides the input speech into corresponding stuttering categories. The softmax function layer predicts the probability of five stuttering categories for the features obtained by the Zipformer layer of the input speech, and outputs the stuttering category with the highest probability as the predicted stuttering event type. The five stuttering event categories are: extension, obstruction, sound repetition, word repetition, and insertion.

[0046] The automatic speech recognition model uses an end-to-end model to transcribe stuttering speech into text. The automatic speech recognition model uses the transducer method. The automatic speech recognition model includes the Input layer, Zipformer Encoder, Transducer Decoder, and Output layer. After the speech signal is input to the Input layer, it passes through the Zipformer Encoder and Transducer Decoder in sequence, and then the Output layer outputs the corresponding recognized speech text.

[0047] The Input layer in the automatic speech recognition model includes a filter group fbank that extracts features from speech signals; the Zipformer Encoder uses a self-attention mechanism to encode input features into a high-dimensional representation, capturing long-distance dependencies and contextual information in speech signals through the self-attention mechanism; the Transducer Decoder uses a joint network to combine the encoder output with the prediction network to generate an output sequence; the Output layer: finally predicts the probability distribution of a set of 4418 Chinese characters, and selects the Chinese character with the highest probability distribution as the recognized text output.

[0048] The establishment of a data-enhanced stuttering speech database includes: selecting a Mandarin speech dataset as a benchmark corpus, using the Montreal forced alignment tool MFA to perform phoneme-level alignment on the speech data, and then performing data enhancement through various types of enhancement methods to obtain a speech dataset that simulates various stuttering events, namely the stuttering speech dataset.

[0049] Use the MFA tool to perform phoneme-level alignment on the Mandarin speech dataset to obtain accurate phoneme boundary information; divide the aligned data into smaller segments according to the phoneme boundaries; apply the corresponding data enhancement processing method to each segment to generate simulated stuttering speech data; and build a stuttering speech dataset based on the generated simulated stuttering speech data.

[0050] There are five data enhancement processing methods:

[0051] Blocking: inserting random silence segments of set duration between words;

[0052] Sound repetition: a syllable is randomly selected and repeated for half its duration;

[0053] Word Repeat: Apply tempo perturbation to the selected segment, causing it to repeat multiple times;

[0054] Combined enhancement: applying blocking and sound repetition or blocking and word repetition enhancement to the same segment;

[0055] Normal: Randomly select segments from the original dataset without any modification.

[0056] These five data enhancement processing methods are used to perform data enhancement processing on the normal Mandarin speech dataset to obtain the corresponding stuttering dataset, which is used to train or process the constructed model.

[0057] Each component is introduced as follows:

[0058] 1. Dataset Selection and Alignment

[0059] Firstly, a Mandarin speech dataset was selected as the benchmark corpus, and the Montreal Forced Alignment tool (MFA) was used to perform phoneme-level alignment on the speech data to ensure the accuracy and consistency of the data.

[0060] 2. Data enhancement technology: This solution develops five types of enhanced data to simulate different stuttering events:

[0061] Block: insert random silence segments of 200-600 milliseconds between words.

[0062] Sound Repetition: Randomly selects a syllable and repeats it for half its duration.

[0063] Word Repetition: Applies a tempo perturbation to the selected segment, causing it to repeat 2-5 times.

[0064] Combination enhancement: Apply blocking and sound repetition or blocking and word repetition techniques to the same segment.

[0065] Normal: Randomly select segments from the original dataset without any modification.

[0066] The process of data enhancement method includes the following categories and their specific steps:

[0067] Alignment: Use the MFA tool to perform phoneme-level alignment on the Mandarin speech dataset to obtain accurate phoneme boundary information.

[0068] Segmentation: Split the aligned data into smaller segments based on phoneme boundaries for applying enhancement techniques.

[0069] Enhancement: Apply the corresponding enhancement technology to each segment to generate simulated stuttering speech data. Each segment randomly selects the enhancement type, and the number of each type of enhanced segments is the same.

[0070] Combination: Combine augmented segments to form new training examples, ensuring data diversity and coverage.

[0071] After the above enhancement is completed, verification is required, which includes: verifying the quality of the enhanced data through manual and automatic methods to ensure that it accurately represents the stuttering event.

[0072] 3. System Architecture

[0073] The stuttering recognition and detection architecture of this scheme consists of two major components: the stuttering event detection (SED) model and the automatic speech recognition (ASR) model. Both are based on the Zipformer-L model, which combines convolution operations and the Transformer self-attention mechanism and is suitable for speech processing tasks.

[0074] 1. Stuttering Event Detection (SED) model: used to detect five types of stuttering events: prolongation, obstruction, sound repetition, word repetition and insertion.

[0075] The key components of the model include:

[0076] a.Input layer: The input of the model consists of filter bank (fbank) features extracted from the speech signal.

[0077] b. Zipformer layers: These layers combine convolution operations with the Transformer self-attention mechanism to effectively capture temporal and contextual information in speech signals. The Zipformer layer also has an encoder structure similar to U-Net, which downsamples the sequence to different low frame rates. This makes the model more efficient when processing long sequences.

[0078] c.Output layer: The output layer is a softmax layer that classifies the input into one of the five stuttering event categories and outputs the stuttering type with the maximum probability. Based on the probability of stuttering events of the species output by the softmax layer, the stuttering event type with the maximum probability is selected as the output.

[0079] 2. Automatic Speech Recognition (ASR) Model: An end-to-end ASR model for transcribing stuttered speech into text. This model uses a transducer approach and is suitable for processing variable-length input and output sequences.

[0080] The key components of the model include:

[0081] Input layer: The input consists of filter bank (fbank) features extracted from the speech signal.

[0082] Zipformer Encoder: The encoder uses a self-attention mechanism to encode input features into a high-dimensional representation. Through the self-attention mechanism, the encoder can effectively capture long-range dependencies and contextual information in speech signals.

[0083] Transducer Decoder: The decoder uses a joint network to combine the encoder output with the prediction network to generate an output sequence (Chinese characters). The transducer decoder is able to handle variable-length input and output sequences, making it suitable for processing complex stuttered speech.

[0084] Output layer: The final prediction is the probability distribution of a set of 4418 Chinese characters. 4418 refers to the character statistics of the data set + the number of commonly used Chinese characters. The output layer specifically consists of the softmax layer. The output layer outputs the probability distribution. Based on the probability distribution, the dictionary corresponding to the category and the highest probability character is converted to obtain the final semantic text.

[0085] By combining the Zipformer Encoder and the Transducer Decoder, the model performs well in speech recognition tasks and is able to accurately transcribe stuttering speech into text.

[0086] The innovation of this invention lies in the rich and systematic Mandarin stuttering data enhancement technology. Based on this technology, we have built a set of excellent Mandarin stuttering detection and recognition systems, which can be effectively applied to the recognition of stuttered speech in speech recognition in various fields.

[0087] Obviously, the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention, they are all within the protection scope of the present invention.

Claims

1. A method for detecting and identifying stuttering in Mandarin based on data enhancement, characterized in that: The method comprises: Build a stuttering event detection model and automatic speech recognition module; Train the constructed stuttering event model and automatic speech recognition module using a data-enhanced stuttering speech dataset; The trained stuttering event detection model and automatic speech recognition module are used to detect and recognize speech to obtain the stuttering type and the recognized speech text.

2. The method for detecting and identifying stuttering in Mandarin based on data enhancement as claimed in claim 1, characterized in that: The stuttering event model built is used to detect stuttering events, which includes an Input layer, a Zipformer layer, and an Output layer. The speech data is input into the input layer and passes through the Zipformer layer and the Output layer in turn, and then the stuttering event category corresponding to the speech is output.

3. The method for detecting and identifying stuttering in Mandarin based on data enhancement as claimed in claim 2, characterized in that: The Input layer in the stuttering event detection model consists of a filter bank fbank of features extracted from the speech signal; the Zipformer layer combines convolution operations and the Transformer self-attention mechanism to capture the time and context information in the speech signal; the Zipformer layer also has a U-Net-like encoder structure, which reduces the sequence to different low frame rates by downsampling.

4. The method for detecting and identifying stuttering in Mandarin based on data enhancement as claimed in claim 3, characterized in that: The Output layer in the stuttering event detection model includes a softmax function layer, which performs probability prediction on the output of the Zipformer layer and divides the input speech into corresponding stuttering categories. The softmax function layer predicts the probabilities of five stuttering categories based on the features obtained by the Zipformer layer of the input speech, and outputs the stuttering category with the highest probability as the predicted stuttering event type; the five stuttering event categories are: extension, obstruction, sound repetition, word repetition and insertion.

5. A method for detecting and identifying stuttering in Mandarin based on data enhancement according to any one of claims 1 to 4, characterized in that: The automatic speech recognition model uses an end-to-end model to transcribe stuttered speech into text. The automatic speech recognition model uses a transducer approach.

6. The method for detecting and identifying stuttering in Mandarin based on data enhancement as claimed in claim 5, characterized in that: The automatic speech recognition model includes an Input layer, a Zipformer Encoder, a Transducer Decoder, and an Output layer. After the speech signal is input into the Input layer, it passes through the Zipformer Encoder and the Transducer Decoder in sequence, and then the Output layer outputs the corresponding recognized speech text.

7. The method for detecting and identifying stuttering in Mandarin based on data enhancement as claimed in claim 6, characterized in that: Automatic speech recognition model The input layer includes a filter bank fbank that extracts features from speech signals. The Zipformer Encoder uses a self-attention mechanism to encode the input features into a high-dimensional representation, capturing long-range dependencies and contextual information in speech signals through the self-attention mechanism. Transducer Decoder uses a joint network to combine the encoder output with the prediction network to generate an output sequence; Output layer: finally predicts the probability distribution of a set of 4418 Chinese characters and selects the Chinese character with the highest probability distribution as the recognized text output.

8. A method for detecting and identifying stuttering in Mandarin based on data enhancement according to any one of claims 1 to 4, characterized in that: The establishment of a data-enhanced stuttering speech database includes: selecting a Mandarin speech dataset as a benchmark corpus, using the Montreal forced alignment tool MFA to perform phoneme-level alignment on the speech data, and then performing data enhancement through various types of enhancement methods to obtain a speech dataset that simulates various stuttering events, namely the stuttering speech dataset.

9. The method for detecting and identifying stuttering in Mandarin based on data enhancement as claimed in claim 8, characterized in that: Use the MFA tool to perform phoneme-level alignment on the Mandarin speech dataset to obtain accurate phoneme boundary information; divide the aligned data into smaller segments according to the phoneme boundaries; apply the corresponding data enhancement processing method to each segment to generate simulated stuttering speech data; and build a stuttering speech dataset based on the generated simulated stuttering speech data.

10. The method for detecting and identifying stuttering in Mandarin based on data enhancement as claimed in claim 9, characterized in that: Data enhancement processing methods include: Blocking: inserting random silence segments of set duration between words; Sound repetition: a syllable is randomly selected and repeated for half its duration; Word Repeat: Apply tempo perturbation to the selected segment, causing it to repeat multiple times; Combined enhancement: applying blocking and sound repetition or blocking and word repetition enhancement to the same segment; Normal: Randomly select segments from the original dataset without any modification.

Images