Multi-dialect voice text conversion system and processing method

Abstract

The invention relates to the technical field of language recognition, in particular to a multi-dialect voice text conversion system and processing method, and the system comprises an audio collection module which is responsible for collecting voice input of a user and supports PCM, MP3 and WAV formats and sampling rates of 16kHz and 44.1 kHz; the dialect detection module is used for recognizing a dialect type of voice input by utilizing a pre-training language classifier; the voice feature extraction module is used for converting the voice signal into feature representation, such as Mel spectrum and MFCC (Mel Frequency Cepstrum Coefficient); the multi-dialect speech recognition module is used for mapping the features into a text sequence based on a speech recognition model of multi-task learning; and the text post-processing module is used for outputting a final standardized text according to a language rule and a context optimization recognition result. The multi-dialect support covers main dialect languages and can be quickly expanded to new dialects through transfer learning.

Description

Technical Field

[0001] The present invention relates to the technical field of language recognition, and in particular to a multi-dialect speech-to-text conversion system and a processing method. Background Art

[0002] In order to meet the needs of practical applications, it is of great significance to develop a speech recognition system that can accurately recognize multiple dialects, which can improve people's quality of life and work efficiency. Through this system, not only can the misunderstandings and communication barriers when dialect users interact with machines be effectively reduced, but also more personalized and customized services can be provided. In human-computer interaction, dialect voice interaction with robots can provide a more natural and friendly way of communication. This way of interaction makes users feel more comfortable and at ease, and more easily express their intentions and needs, increasing the interactive experience between users and robots. In addition, the use of dialects can promote cultural exchanges and protect regional culture, making the communication between people and machines more rich and diverse;

[0003] Existing speech conversion systems have limited coverage of dialects and cannot support dialects from multiple regions or complex language environments. In addition, due to the complex pronunciation characteristics of dialects, existing systems are prone to misjudgment during speech recognition due to accent differences.

[0004] Therefore, a multi-dialect speech-to-text conversion system and processing method are proposed. Summary of the invention

[0005] The object of the present invention is to provide a multi-dialect speech-to-text conversion system and processing method to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a multi-dialect speech-to-text conversion system and processing method, comprising:

[0007] Audio acquisition module: responsible for collecting user voice input, supporting PCM, MP3, WAV formats and 16kHz, 44.1kHz sampling rates;

[0008] Dialect Detection Module: Uses pre-trained language classifiers to identify the dialect type of speech input;

[0009] Speech feature extraction module: converts speech signals into feature representations, such as Mel spectrum and MFCC;

[0010] Multi-dialect speech recognition module: a speech recognition model based on multi-task learning that maps features into text sequences; text post-processing module: optimizes recognition results according to language rules and context, and outputs the final standardized text; model training module: responsible for system training and optimization, supporting transfer learning to expand new dialects.

[0011] A processing method for a multi-dialect speech-to-text conversion system comprises the following steps:

[0012] Step 1: Audio acquisition and preprocessing:

[0013] a. Audio acquisition: Real-time recording through microphone, sampling rate is set to 16kHz, bit depth is 16 bits;

[0014] b. Preprocessing:

[0015] Denoising: remove environmental noise through low-pass filter;

[0016] Normalization: Normalize the audio amplitude to [-1,1];

[0017] Framing and windowing: The audio signal is framed, each frame is 25ms long, the frame shift is 10ms, and a Hamming window is added; Step 2: Dialect detection:

[0018] Dialect Classification Model:

[0019] Train classification models based on convolutional neural networks and long short-term memory networks;

[0020] Input: Mel spectrogram or MFCC features;

[0021] Output: Dialect category, such as Cantonese, Sichuanese, and Mandarin;

[0022] Formula: P(y|x)=softmax(W·f(x)+b)

[0023] Where: P(y|x) is the dialect classification probability, W is the classifier weight, and f(x) is the hidden layer representation after feature extraction; Step 3: Speech feature extraction:

[0024] Extract audio features such as Mel-spectrogram or MFCC: Where: X[k] is the energy of the kth frequency band, n is the MFCC feature dimension;

[0025] Step 4: Multi-dialect speech recognition

[0026] Model structure:

[0027] Based on end-to-end deep learning models, such as Transformer and CTC+Attention combined architecture;

[0028] Input: feature sequence;

[0029] Output: text sequence;

[0030] formula:

[0031] Where: Y is the output text sequence, X is the input feature sequence, (y t |y 1:t-1 ,X) is the output probability at time step t;

[0032] Step 5: Text Post-Processing

[0033] Using language models to optimize the coherence of recognized text:

[0034] The language model uses a bidirectional LSTM or GPT pre-trained model;

[0035] Contextual correction: correct ambiguous words based on the context;

[0036] Example: Correct "I stir it well" to "I feel good" (Cantonese dialect recognition).

[0037] Step 6: Model training and optimization

[0038] Multi-task learning:

[0039] Dialect recognition and speech recognition are jointly trained to improve the generalization ability of the model by sharing underlying features;

[0040] Transfer Learning:

[0041] Use the existing dialect model to initialize the weight of the new dialect model to reduce training costs;

[0042] Optimization goal:

[0043] Using the cross entropy loss function:

[0044] Preferably, the audio acquisition expands the training corpus and enhances the model's recognition ability for low-resource dialects by collecting dialect speech data in real scenarios. In this step, environmental noise, long periods of silence, and invalid speech data are removed from the recording, erroneous labels are corrected through manual or semi-automatic annotation, and audio enhancement technology is used to generate more samples.

[0045] Preferably, the data enhancement includes time stretching and compression, adding background noise, such as white noise and street noise, and changing the speech frequency. The data enhancement formula is:

[0046] x′=αx+(1-α)n

[0047] Where x is the original audio signal, n is the noise signal, and α is the enhancement weight parameter.

[0048] Preferably, the audio acquisition also includes real-time optimization, specifically model architecture optimization, model lightweighting and accelerated decoding algorithm.

[0049] Preferably, the model architecture is optimized, using a streaming model architecture of RNN-T or CTC+Attention, using multi-threaded parallel processing to perform data preprocessing, feature extraction, and recognition decoding in steps, and each step uses independent threads for parallelization to improve throughput;

[0050] Parallel efficiency formula:

[0051] Where: T parallel is the parallel processing time, T total : Total task time, N is the number of parallel threads;

[0052] The model is lightweight, quantizing model parameters from 32-bit floating point to INT8 while maintaining recognition accuracy, removing unnecessary model parameters such as low-weight connections, and guiding student model training through the teacher model, thus retaining performance while significantly reducing model size;

[0053] Accelerated decoding algorithm

[0054] Introducing multi-layer BeamSearch to optimize decoding efficiency, reduce the number of candidate sequences, and increase output speed;

[0055] Decoding optimization formula:

[0056] Where: S: optimal decoding sequence, logP(y|x): speech model probability, logP LM (y): language model probability, λ: language model weight.

[0057] Preferably, the step 2: dialect detection includes domain adaptive optimization, adopts an adaptive learning mechanism, counts the frequency of occurrence of dialect words in real time, dynamically updates the priority of the language model, and the word frequency adjustment formula is:

[0058]

[0059] Where: P′(w): adjusted word probability, P(w): original word probability, count(w): frequency of occurrence of the word.

[0060] Preferably, the step 2: dialect detection includes multimodal integrated optimization when the scene requires, such as video conferencing or education scenes, combined with lip reading recognition or scene text information;

[0061] Multimodal feature fusion formula: Z = αX audio +βX visual

[0062] Among them: Z: fused feature vector, X audio : Voice features, X visual: visual features, α, β: fusion weights.

[0063] Preferably, the step 2: dialect detection includes a scenario adaptation model, and for specific scenarios, such as multi-person conversations, a role recognition function is introduced, the speaker identity is identified through a speech separation model, and different transcription outputs are allocated.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. Multi-dialect support: covers major dialects and can be quickly expanded to new dialects through transfer learning.

[0066] 2. Improve recognition accuracy: Improve the accuracy of speech-to-text conversion through deep learning and feature enhancement algorithms.

[0067] 3. Strong real-time performance: Optimize the model structure and calculation process to significantly reduce voice processing delay.

[0068] 4. Strong adaptability: Supports dialect detection and adaptive adjustment, allowing the system to adjust recognition strategies based on the characteristics of different dialects. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0070] Figure 1 A flow chart of a processing method of a multi-dialect speech-to-text conversion system of the present invention;

[0071] Figure 2 It is a comparison table of multiple dialect samples of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] See also Figure 1 to Figure 2 , the present invention provides a technical solution:

[0074] A multi-dialect speech-to-text conversion system and processing method are provided to solve the problems raised in the above-mentioned background technology.

[0075] To achieve the above object, the present invention provides the following technical solution: a multi-dialect speech-to-text conversion system and processing method, comprising:

[0076] Audio acquisition module: responsible for collecting user voice input, supporting PCM, MP3, WAV formats and 16kHz, 44.1kHz sampling rates;

[0077] Dialect Detection Module: Uses pre-trained language classifiers to identify the dialect type of speech input;

[0078] Speech feature extraction module: converts speech signals into feature representations, such as Mel spectrum and MFCC;

[0079] Multi-dialect speech recognition module: A speech recognition model based on multi-task learning that maps features into text sequences;

[0080] Text post-processing module: optimizes recognition results according to language rules and context, and outputs the final standardized text;

[0081] Model training module: responsible for system training and optimization, supporting transfer learning to expand new dialects.

[0082] A processing method for a multi-dialect speech-to-text conversion system comprises the following steps:

[0083] Step 1: Audio acquisition and preprocessing:

[0084] a. Audio acquisition: Real-time recording through microphone, sampling rate is set to 16kHz, bit depth is 16 bits;

[0085] b. Preprocessing:

[0086] Denoising: remove environmental noise through low-pass filter;

[0087] Normalization: Normalize the audio amplitude to [-1,1];

[0088] Framing and windowing: The audio signal is framed, each frame is 25ms long, the frame shift is 10ms, and a Hamming window is added;

[0089] Step 2: Dialect Detection:

[0090] Dialect Classification Model:

[0091] Train classification models based on convolutional neural networks and long short-term memory networks;

[0092] Input: Mel spectrogram or MFCC features;

[0093] Output: Dialect category, such as Cantonese, Sichuanese, and Mandarin;

[0094] Formula: P(y|x)=softmax(W·f(x)+b)

[0095] Where: P(y|x) is the dialect classification probability, W is the classifier weight, and f(x) is the hidden layer representation after feature extraction; Step 3: Speech feature extraction:

[0096] Extract audio features such as Mel-spectrogram or MFCC:

[0097] Where: X[k] is the energy of the kth frequency band, n is the MFCC feature dimension;

[0098] Step 4: Multi-dialect speech recognition

[0099] Model structure:

[0100] Based on end-to-end deep learning models, such as Transformer and CTC+Attention combined architecture;

[0101] Input: feature sequence;

[0102] Output: text sequence;

[0103] formula:

[0104] Where: Y is the output text sequence, X is the input feature sequence, (y t |y 1:t-1 ,X) is the output probability of time step t; Step 5: Text post-processing

[0105] Using language models to optimize the coherence of recognized text:

[0106] The language model uses a bidirectional LSTM or GPT pre-trained model;

[0107] Contextual correction: correct ambiguous words based on the context;

[0108] Example: Correct "I stir it well" to "I feel good" (Cantonese dialect recognition).

[0109] Step 6: Model training and optimization

[0110] Multi-task learning:

[0111] Dialect recognition and speech recognition are jointly trained to improve the generalization ability of the model by sharing underlying features;

[0112] Transfer Learning:

[0113] Use the existing dialect model to initialize the weight of the new dialect model to reduce training costs;

[0114] Optimization goal:

[0115] Using the cross entropy loss function:

[0116] Specifically, the audio acquisition collects dialect speech data in real scenarios, expands the training corpus, and enhances the model's recognition ability for low-resource dialects. In this step, environmental noise, long periods of silence, and unclear speech data in the recording are removed, incorrect labels are corrected through manual or semi-automatic annotation, and audio enhancement technology is used to generate more samples.

[0117] Specifically, the data enhancement includes time stretching and compression, adding background noise such as white noise and street noise, and changing the speech frequency. The data enhancement formula is:

[0118] x′=αx+(1-α)n

[0119] Where x is the original audio signal, n is the noise signal, and α is the enhancement weight parameter.

[0120] Specifically, the audio acquisition also includes real-time optimization, specifically model architecture optimization, model lightweighting and accelerated decoding algorithm.

[0121] Specifically, the model architecture is optimized, using the streaming model architecture of RNN-T or CTC+Attention, using multi-threaded parallel processing for data preprocessing, feature extraction, and recognition decoding in steps, using independent threads for parallelization of each step to improve throughput;

[0122] Parallel efficiency formula:

[0123] Where: T parallel is the parallel processing time, T total : Total task time, N is the number of parallel threads;

[0124] The model is lightweight, quantizing model parameters from 32-bit floating point to INT8 while maintaining recognition accuracy, removing unnecessary model parameters such as low-weight connections, and guiding student model training through the teacher model, thus retaining performance while significantly reducing model size;

[0125] Accelerated decoding algorithm

[0126] Introducing multi-layer BeamSearch to optimize decoding efficiency, reduce the number of candidate sequences, and increase output speed;

[0127] Decoding optimization formula:

[0128] Where: S: optimal decoding sequence, logP(y|x): speech model probability, logP LM(y): language model probability, λ: language model weight.

[0129] Specifically, the step 2: dialect detection includes domain adaptive optimization, adopts an adaptive learning mechanism, counts the frequency of occurrence of dialect words in real time, dynamically updates the priority of the language model, and the word frequency adjustment formula is:

[0130]

[0131] Where: P′(w): adjusted word probability, P(w): original word probability, count(w): frequency of occurrence of the word.

[0132] Specifically, the step 2: dialect detection includes multimodal integrated optimization when the scenario requires, such as video conferencing or education scenarios, combined with lip reading recognition or scene text information;

[0133] Multimodal feature fusion formula: Z = αX audio +βX visual

[0134] Among them: Z: fused feature vector, X audio : Voice features, X visual : visual features, α, β: fusion weights.

[0135] Specifically, the step 2: dialect detection includes a scenario adaptation model, which introduces a role recognition function for specific scenarios, such as multi-person conversations, identifies the speaker's identity through a speech separation model, and allocates different transcription outputs.

[0136] Lightweight deployment optimization

[0137] 1. Cloud and edge collaboration

[0138] The cloud handles complex computing tasks, and edge devices handle simple local tasks:

[0139] The deep speech model runs in the cloud and is responsible for accurate recognition;

[0140] Edge devices process front-end data in real time and make preliminary inferences.

[0141] Edge collaboration formula: Ttotal = Tedge + TcloudT total =T edge +T cloud

[0142] 2. Distributed computing

[0143] Use distributed computing frameworks (such as TensorFlow Serving) to process multi-dialect speech data in parallel on multiple devices to improve processing efficiency

[0144] Data expansion and optimization: dialect coverage increased to over 90%; data noise interference rate decreased by 15%.

[0145] Real-time optimization: The delay is reduced to less than 200ms to meet real-time processing requirements; the size of the model is reduced by 50% after compression, and device adaptability is enhanced.

[0146] Domain adaptive optimization: speech recognition accuracy in different industries increased by 10%-20%.

[0147] Multimodal optimization: The recognition accuracy of complex scenes is improved by 15% after multimodal fusion.

[0148] Lightweight deployment optimization: Resource usage is reduced by 30%-40% when running on edge devices while maintaining high recognition accuracy.

[0149] Through the above detailed optimization, the system has achieved significant improvements in the performance, applicability and resource efficiency of multi-dialect speech-to-text conversion, and is suitable for speech recognition needs in complex multi-language and multi-scenario environments.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-dialect speech-to-text conversion system and processing method, characterized in that: include: Audio acquisition module: responsible for collecting user voice input, supporting PCM, MP3, WAV formats and 16kHz, 44.1kHz sampling rates; Dialect Detection Module: Uses pre-trained language classifiers to identify the dialect type of speech input; Speech feature extraction module: converts speech signals into feature representations, such as Mel spectrum and MFCC; Multi-dialect speech recognition module: A speech recognition model based on multi-task learning that maps features into text sequences; Text post-processing module: optimizes recognition results according to language rules and context, and outputs the final standardized text; Model training module: responsible for system training and optimization, supporting transfer learning to expand new dialects.

2. A processing method for a multi-dialect speech-to-text conversion system according to claim 1, characterized in that: The steps include: Step 1: Audio acquisition and preprocessing: a. Audio acquisition: Real-time recording through microphone, sampling rate is set to 16kHz, bit depth is 16 bits; b. Preprocessing: Denoising: remove environmental noise through low-pass filter; Normalization: Normalize the audio amplitude to [-1,1]; Framing and windowing: The audio signal is framed, each frame is 25ms long, the frame shift is 10ms, and a Hamming window is added; Step 2: Dialect Detection: Dialect Classification Model: Train classification models based on convolutional neural networks and long short-term memory networks; Input: Mel spectrogram or MFCC features; Output: Dialect category, such as Cantonese, Sichuanese, and Mandarin; Formula: P(y|x)=softmax(W·f(x)+b) Where: P(y|x) is the dialect classification probability, W is the classifier weight, and f(x) is the hidden layer representation after feature extraction; Step 3: Speech feature extraction: Extract audio features such as Mel-spectrogram or MFCC: Where: X[k] is the energy of the kth frequency band, n is the MFCC feature dimension; Step 4: Multi-dialect speech recognition Model structure: Based on end-to-end deep learning models, such as Transformer and CTC+Attention combined architecture; Input: feature sequence; Output: text sequence; official: Where: Y is the output text sequence, X is the input feature sequence, (y t |y 1:t-1 ,X) is the output probability at time step t; Step 5: Text Post-Processing Using language models to optimize the coherence of recognized text: The language model uses a bidirectional LSTM or GPT pre-trained model; Contextual correction: correct ambiguous words based on the context; Step 6: Model training and optimization Multi-task learning: Dialect recognition and speech recognition are jointly trained to improve the generalization ability of the model by sharing underlying features; Transfer Learning: Use the existing dialect model to initialize the weight of the new dialect model to reduce training costs; Optimization goal: Using the cross entropy loss function:

3. The processing method of a multi-dialect speech-to-text conversion system according to claim 2, characterized in that: The audio acquisition collects dialect speech data in real scenarios, expands the training corpus, and enhances the model's recognition ability for low-resource dialects. In this step, environmental noise, long periods of silence, and invalid speech data are removed from the recording, incorrect labels are corrected through manual or semi-automatic annotation, and audio enhancement technology is used to generate more samples.

4. The processing method of a multi-dialect speech-to-text conversion system according to claim 3, characterized in that: The data enhancement includes time stretching and compression, adding background noise such as white noise and street noise, and changing the speech frequency. The data enhancement formula is: x′=αx+(1-α)n Where x is the original audio signal, n is the noise signal, and α is the enhancement weight parameter.

5. The processing method of a multi-dialect speech-to-text conversion system according to claim 4, characterized in that: The audio acquisition also includes real-time optimization, specifically model architecture optimization, model lightweighting and accelerated decoding algorithm.

6. The processing method of a multi-dialect speech-to-text conversion system according to claim 5, characterized in that: The model architecture is optimized, using RNN-T or CTC+Attention streaming model architecture, using multi-threaded parallel processing for data preprocessing, feature extraction, and recognition decoding in steps, using independent threads for parallelization of each step to improve throughput; Parallel efficiency formula: Where: T parallel is the parallel processing time, T total : Total task time, N is the number of parallel threads; The model is lightweight, quantizing model parameters from 32-bit floating point to INT8 while maintaining recognition accuracy, removing unnecessary model parameters such as low-weight connections, and guiding student model training through the teacher model, thus retaining performance while significantly reducing model size; Accelerated decoding algorithm Introducing multi-layer BeamSearch to optimize decoding efficiency, reduce the number of candidate sequences, and increase output speed; Decoding optimization formula: Where: S: optimal decoding sequence, logP(y|x): speech model probability, logP LM (y): language model probability, λ: language model weight.

7. The processing method of a multi-dialect speech-to-text conversion system according to claim 6, characterized in that: The step 2: dialect detection includes domain adaptive optimization, adopts an adaptive learning mechanism, counts the frequency of occurrence of dialect words in real time, dynamically updates the priority of the language model, and adjusts the word frequency formula: Where: P′(w): adjusted word probability, P(w): original word probability, count(w): frequency of occurrence of the word.

8. The processing method of a multi-dialect speech-to-text conversion system according to claim 7, characterized in that: The step 2: dialect detection includes multimodal integrated optimization when the scene requires, such as video conferencing or education scenes, combined with lip reading recognition or scene text information; Multimodal feature fusion formula: Z = αX audio +βX visual Among them: Z: fused feature vector, X audio : Voice features, X visual : visual features, α, β: fusion weights.

9. The processing method of a multi-dialect speech-to-text conversion system according to claim 8, characterized in that: The step 2: dialect detection includes a scenario adaptation model. For specific scenarios, such as multi-person conversations, a role recognition function is introduced, the speaker identity is identified through a speech separation model, and different transcription outputs are allocated.

Images