A multi-dialect speech-to-text conversion system and processing method

Abstract

The present application relates to the technical field of language recognition, in particular to a multi-dialect speech text conversion system and processing method, comprising: an audio acquisition module: responsible for collecting the speech input of a user, supporting PCM, MP3, WAV format and 16kHz, 44.1kHz sampling rate; a dialect detection module: using a pre-trained language classifier to identify the dialect type of the speech input; a speech feature extraction module: converting the speech signal into feature representation, such as mel spectrum, MFCC; a multi-dialect speech recognition module: based on a multi-task learning speech recognition model, mapping the features into a text sequence; a text post-processing module: optimizing the recognition result according to language rules and context, and outputting the final standardized text. Multi-dialect support: covering main dialect languages, and can be quickly expanded to new dialects through transfer learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of language recognition, in particular to a multi-dialect speech text conversion system and processing method. BACKGROUND

[0002] In order to meet the actual application requirements, it is of great significance to develop a speech recognition system that can accurately recognize multiple dialects, which can improve people's quality of life and work efficiency. Through the system, not only can misunderstandings and communication barriers be effectively reduced when dialect users interact with machines, but also more personalized and customized services can be provided. In human-computer interaction, speech interaction with robots in dialects can provide a more natural and friendly way of communication. This interaction makes users feel more comfortable and at ease, and more easily express their intentions and needs, increasing the interactive experience between users and robots. In addition, the use of dialects can promote cultural exchange and protect regional culture, making communication between people and machines more diverse;

[0003] The existing speech conversion system has limited coverage of dialects, cannot support multiple regional dialects or complex language environments, and due to the complexity of dialect pronunciation characteristics, the existing system is prone to misjudgment due to differences in accent when recognizing speech.

[0004] Therefore, a multi-dialect speech text conversion system and processing method are proposed. SUMMARY

[0005] The purpose of the present application is to provide a multi-dialect speech text conversion system and processing method to solve the problems raised in the background.

[0006] To achieve the above purpose, the present application provides the following technical solution: a multi-dialect speech text conversion system and processing method, comprising:

[0007] An audio acquisition module: responsible for collecting user speech input, supporting PCM, MP3, WAV format and 16kHz, 44.1kHz sampling rate;

[0008] A dialect detection module: uses a pre-trained language classifier to identify the dialect type of the speech input;

[0009] A speech feature extraction module: converts speech signals into feature representations, such as Mel frequency spectrum, MFCC;

[0010] A multi-dialect speech recognition module: a speech recognition model based on multi-task learning, maps features to text sequences; a text post-processing module: optimizes the recognition result according to language rules and context, and outputs the final standardized text; a model training module: responsible for system training and optimization, supporting transfer learning to expand new dialects.

[0011] A processing method of a multi-dialect speech-to-text system, comprising the following steps:

[0012] Step 1: Audio acquisition and preprocessing:

[0013] a. Audio acquisition: Real-time recording through microphone, sampling rate set to 16 kHz, bit depth to 16 bits;

[0014] b. Preprocessing:

[0015] De-noising: Remove environmental noise through a low-pass filter;

[0016] Normalization: Standardize the audio amplitude to [-1, 1];

[0017] Frame division and windowing: Divide the audio signal into frames, each frame length is 25 ms, frame shift is 10 ms, and add a Hamming window; Step 2: Dialect detection:

[0018] Dialect classification model:

[0019] Train a classification model based on convolutional neural network and long short-term memory network;

[0020] Input: Mel spectrogram or MFCC feature;

[0021] Output: Dialect category, such as Cantonese, Sichuan dialect, and Mandarin;

[0022] Formula: P(y|x) = softmax(W·f(x) + b)

[0023] Where: P(y|x) is the dialect classification probability, W is the classifier weight, and f(x) is the hidden layer representation after feature extraction; Step 3: Speech feature extraction:

[0024] Extract audio features such as Mel spectrogram or MFCC: Where: X[k] is the energy of the kth frequency band, and n is the MFCC feature dimension;

[0025] Step 4: Multi-dialect speech recognition

[0026] Model structure:

[0027] End-to-end deep learning model based on architecture such as Transformer, CTC + Attention combination;

[0028] Input: Feature sequence;

[0029] Output: Text sequence;

[0030] Formula:

[0031] Where: Y is the output text sequence, X is the input feature sequence, (y t |y 1:t-1 ,X) is the output probability at time step t.

[0032] Step 5: Text Post-processing

[0033] Optimize the coherence of recognized text using language models:

[0034] Language models use bidirectional LSTM or GPT pre-training models.

[0035] Context correction: correct ambiguous words based on previous and subsequent context.

[0036] Example: Correct "I stir well" to "I feel well" (Cantonese dialect recognition).

[0037] Step 6: Model Training and Optimization

[0038] Multi-task learning:

[0039] Joint training of dialect recognition and speech recognition, improving model generalization ability by sharing underlying features.

[0040] Transfer learning:

[0041] Use existing dialect models to initialize new dialect model weights, reducing training costs.

[0042] Optimization goals:

[0043] Use cross-entropy loss function:

[0044] Preferably, the audio collection collects dialect speech data in real scenarios, expands the training corpus, and enhances the model's recognition ability for low-resource dialects. In this step, remove environmental noise, long silence, and speech ambiguity from the recording, correct incorrect labels through manual or semi-automatic labeling, and generate more samples using audio enhancement techniques.

[0045] Preferably, the data augmentation includes time stretching and compression, adding background noise such as white noise and street noise, and changing speech frequency. Data augmentation formula:

[0046] x' = αx + (1-α)n

[0047] Where x is the original audio signal, n is the noise signal, and α is the enhancement weight parameter.

[0048] Preferably, the audio collection also includes real-time optimization, specifically model architecture optimization, model lightweight, and accelerated decoding algorithm.

[0049] Preferably, the model architecture is optimized, using RNN-T or CTC+Attention streaming model architecture, using multi-thread parallel processing data preprocessing, feature extraction and recognition decoding step-by-step execution, each step using independent threads parallelization, improving throughput;

[0050] Parallel efficiency formula:

[0051] Where: T parallel is the parallel processing time, T total : total task time, N is the number of parallel threads;

[0052] Model lightening, model parameters from 32-bit floating point quantization to INT8, while maintaining the recognition accuracy, removing unnecessary model parameters such as low weight connection, through the teacher model to guide the student model training, while preserving the performance, greatly reducing the model size;

[0053] Accelerated decoding algorithm

[0054] Introducing multi-layer BeamSearch optimization decoding efficiency, reducing the number of candidate sequences, improving the output speed;

[0055] Decoding optimization formula:

[0056] Where: S: the best decoding sequence, logP(y|x): speech model probability, logP LM (y): language model probability, λ: language model weight.

[0057] Preferably, the step 2: dialect detection includes domain adaptation optimization, using adaptive learning mechanism, real-time statistics of dialect vocabulary frequency, dynamic updating of language model priority, word frequency adjustment formula:

[0058]

[0059] Where: P'(w): adjusted word probability, P(w): original word probability, count(w): word frequency.

[0060] Preferably, the step 2: dialect detection includes multi-modal integration optimization, when needed in the scene, such as video conference or education scene, combined with lip reading or scene text information;

[0061] Multi-modal feature fusion formula: Z=αX audio +βX visual

[0062] Where: Z: the fused feature vector, X audio : speech features, X visualVisual features, a, b: fusion weights.

[0063] Preferably, the step 2: dialect detection includes a scene adaptation model, which introduces a role recognition function for specific scenes such as multi-person conversation, identifies the speaker's identity through a speech separation model, and assigns different transcription outputs.

[0064] Compared with the prior art, the beneficial effects of the present application are:

[0065] 1. Multi-dialect support: covering major dialects and quickly expanding to new dialects through transfer learning.

[0066] 2. Improve recognition accuracy: improve speech-to-text conversion accuracy through deep learning and feature enhancement algorithms.

[0067] 3. Strong real-time performance: optimize model structure and computing process, significantly reduce speech processing delay.

[0068] 4. Strong adaptability: support dialect detection and adaptive adjustment, so that the system can adjust the recognition strategy according to different dialect characteristics. BRIEF DESCRIPTION OF DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the drawings needed in the specific embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0070] Fig. 1 The flow chart of the processing method of the multi-dialect speech text conversion system of the present application;

[0071] Fig. 2 The multi-dialect sample control table of the present application. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0073] Please refer to Figs. 1-2 The present application provides a technical solution:

[0074] A multi-dialect speech text conversion system and processing method are provided to solve the problems in the background art.

[0075] To achieve the above object, the present application provides the following technical solutions: a multi-dialect speech text conversion system and processing method, comprising:

[0076] An audio acquisition module: responsible for collecting user's voice input, supporting PCM, MP3, WAV format and 16kHz, 44.1kHz sampling rate;

[0077] A dialect detection module: using a pre-trained language classifier to identify the dialect type of the voice input;

[0078] A speech feature extraction module: converting the speech signal into feature representation, such as mel spectrum, MFCC;

[0079] A multi-dialect speech recognition module: a speech recognition model based on multi-task learning, mapping the features to a text sequence;

[0080] A text post-processing module: optimizing the recognition result according to language rules and context, and outputting the final standardized text;

[0081] A model training module: responsible for system training and optimization, supporting transfer learning to expand new dialects.

[0082] A processing method of a multi-dialect speech text conversion system, comprising the following steps:

[0083] Step 1: Audio acquisition and preprocessing:

[0084] a. Audio acquisition: real-time recording through a microphone, sampling rate set to 16kHz, bit depth to 16 bits;

[0085] b. Preprocessing:

[0086] De-noising: removing environmental noise through a low-pass filter;

[0087] Normalization: standardizing the audio amplitude to [-1, 1];

[0088] Frame division and windowing: dividing the audio signal into frames, each frame length is 25ms, frame shift is 10ms, and Hamming window is added;

[0089] Step 2: Dialect detection:

[0090] Dialect classification model:

[0091] Training a classification model based on convolutional neural network and long short-term memory network;

[0092] Input: mel spectrum or MFCC feature;

[0093] Output: dialect category, such as Cantonese, Sichuan dialect, Mandarin;

[0094] Formula: P(y|x) = softmax(W·f(x) + b)

[0095] Where: P(y|x) is the dialect classification probability, W is the classifier weight, f(x) is the hidden layer representation after feature extraction; Step 3: Speech Feature Extraction:

[0096] Extract audio features such as Mel-spectrogram or MFCC:

[0097] Where: X[k] is the energy of the kth frequency band, n is the MFCC feature dimension;

[0098] Step 4: Multi-dialect Speech Recognition

[0099] Model Structure:

[0100] End-to-end deep learning model based on architecture such as Transformer, CTC + Attention combination;

[0101] Input: Feature sequence;

[0102] Output: Text sequence;

[0103] Formula:

[0104] Where: Y is the output text sequence, X is the input feature sequence, (y t |y 1:t-1 , X) is the output probability at time step t; Step 5: Text Post-processing

[0105] Use language model to optimize the coherence of recognized text:

[0106] Language model uses bidirectional LSTM or GPT pre-training model;

[0107] Context correction: correct ambiguous words according to the context before and after;

[0108] Example: correct "I stir good" to "I feel good" (Cantonese dialect recognition).

[0109] Step 6: Model Training and Optimization

[0110] Multi-task learning:

[0111] Joint training of dialect recognition and speech recognition, improve model generalization ability by sharing underlying features;

[0112] Transfer learning:

[0113] Use existing dialect model to initialize new dialect model weight, reduce training cost;

[0114] Optimization goal:

[0115] Using cross-entropy loss function:

[0116] Specifically, the audio collection expands the training corpus by collecting dialect speech data in real scenes, enhances the model's recognition ability for low-resource dialects, removes environmental noise, long silence, and invalid speech data in the recording, corrects incorrect labels through manual or semi-automatic labeling, and generates more samples using audio enhancement techniques.

[0117] Specifically, the data enhancement includes time stretching and compression, adding background noise such as white noise and street sound, and changing speech frequency, with the data enhancement formula:

[0118] x' = alpha x + (1-alpha) n

[0119] Where x is the original audio signal, n is the noise signal, and alpha is the enhancement weight parameter.

[0120] Specifically, the audio collection also includes real-time optimization, specifically model architecture optimization, model lightweight, and accelerated decoding algorithm.

[0121] Specifically, the model architecture optimization uses RNN-T or CTC+Attention streaming model architecture, uses multi-thread parallel processing data preprocessing, feature extraction, and recognition decoding, and uses independent threads for each step to improve throughput;

[0122] Parallel efficiency formula:

[0123] Where: T parallel is the parallel processing time, T total : total task time, N is the number of parallel threads.

[0124] Model lightweight, quantize model parameters from 32-bit floating point to INT8 while maintaining recognition accuracy, remove unnecessary model parameters such as low weight connections, use teacher model to guide student model training, retain performance while significantly reducing model size.

[0125] Accelerated decoding algorithm

[0126] Introduce multi-layer BeamSearch to optimize decoding efficiency, reduce the number of candidate sequences, and improve output speed.

[0127] Decoding optimization formula:

[0128] Where: S: best decoding sequence, logP(y|x): speech model probability, logP LM(y): Language model probability, λ: Language model weight.

[0129] Specifically, the step 2: dialect detection includes field adaptation optimization, using adaptive learning mechanism, real-time statistics of dialect vocabulary frequency, dynamic updating of language model priority, word frequency adjustment formula:

[0130]

[0131] Where: P'(w): adjusted word probability, P(w): original word probability, count(w): word frequency.

[0132] Specifically, the step 2: dialect detection includes multi-modal integration optimization, when needed in the scene, such as video conference or education scene, combined with lip reading or scene text information;

[0133] Multi-modal feature fusion formula: Z = aX audio + βX visual

[0134] Where: Z: fused feature vector, X audio : speech features, X visual : visual features, a, β: fusion weight.

[0135] Specifically, the step 2: dialect detection includes scenario adaptation model, for specific scenarios such as multi-person dialogue, introducing role recognition function, identifying speaker identity through speech separation model, and assigning different transcription output.

[0136] Lightweight deployment optimization

[0137] 1. Cloud and edge collaboration

[0138] Cloud handles complex computing tasks, edge devices handle local simple tasks:

[0139] Cloud runs deep speech model, responsible for accurate recognition;

[0140] Edge devices process front-end data in real time and make preliminary inferences.

[0141] Edge collaboration formula: Ttotal = Tedge + TcloudT total = T edge + T cloud

[0142] 2. Distributed computing

[0143] Use distributed computing framework (such as TensorFlow Serving), parallel processing of multi-dialect speech data on multiple devices to improve processing efficiency

[0144] Data expansion optimization: dialect coverage rate increased to over 90%; data noise interference rate decreased by 15%.

[0145] Real-time optimization: delay reduced to within 200ms, meeting real-time processing requirements; model compression size reduced by 50%, enhancing device adaptability.

[0146] Domain adaptation optimization: different industry speech recognition accuracy improved by 10%-20%.

[0147] Multi-modal optimization: multi-modal fusion improved complex scene recognition accuracy by 15%.

[0148] Lightweight deployment optimization: reduced resource usage by 30%-40% in edge device runtime, while maintaining high recognition accuracy.

[0149] Through the above detailed optimization, the system has achieved significant improvement in performance, applicability and resource efficiency in multi-dialect speech text conversion, and is suitable for speech recognition requirements in complex multi-language and multi-scenario environments.

[0150] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A processing method of a multi-dialect speech-to-text system, characterized by: Comprising the following steps: Step 1: Audio acquisition and preprocessing: a. Audio acquisition: Real-time recording through microphone, sampling rate set to 16kHz, bit depth set to 16 bits; b. Preprocessing: Noise removal: Remove environmental noise through a low-pass filter; Normalization: Standardize audio amplitude to [-1, 1]; Frame and windowing: Divide audio signal into frames, each frame length is 25ms, frame shift is 10ms, and add Hamming window; Step 2: Dialect detection: Dialect classification model: Train a classification model based on convolutional neural network and long short-term memory network; Input: Mel spectrogram or MFCC feature; Output: Dialect category, including Cantonese, Sichuan dialect and Mandarin; Formula: P(y|x) = softmax(W·f(x) + b) Where: P(y|x) is the dialect classification probability, W is the classifier weight, f(x) is the hidden layer representation after feature extraction; Dialect detection includes domain adaptation optimization, using adaptive learning mechanism, real-time statistics of dialect vocabulary frequency, dynamic updating of language model priority, word frequency adjustment formula: Where: P'(w): adjusted word probability, P(w): original word probability, count(w): word frequency; Step 3: Speech feature extraction: extracting audio features, including mel-spectrograms or MFCCs: Where: X[k] is the energy of the kth frequency band, n is the MFCC feature dimension; Step 4: Multi-dialect speech recognition Model structure: End-to-end deep learning model based on Transformer and CTC+Attention combined architecture; Input: Feature sequence; Output: Text sequence; Equation: where: Y is the output text sequence, X is the input feature sequence, P(y t |y 1:t-1 ,X) is the output probability at time step t; Step 5: Text post-processing Use language model to optimize the coherence of recognized text: Language model uses bidirectional LSTM or GPT pre-training model; Context correction: Correct ambiguous words according to the context before and after; Step 6: Model training and optimization Multi-task learning: Dialect recognition and speech recognition are trained together to improve the generalization ability of the model by sharing the underlying features; Transfer learning: Use existing dialect model to initialize new dialect model weight to reduce training cost; Optimization goal: Using a cross-entropy loss function:

2. The processing method of a multi-dialect speech-to-text system according to claim 1, wherein: The audio acquisition collects dialect speech data in real scenarios, expands the training corpus, and enhances the model's recognition ability for low-resource dialects. In this step, environmental noise, long silent, and ambiguous speech data are removed, and incorrect labels are corrected through manual or semi-automatic labeling. Audio enhancement techniques are used to generate more samples.

3. The processing method of a multi-dialect speech-to-text system according to claim 2, wherein: The audio enhancement includes time stretching and compression, adding background noise, and changing speech frequency. The data augmentation formula is: x' = αx + (1-α)n Where x is the original audio signal, n is the noise signal, and α is the enhancement weight parameter.

4. The processing method of a multi-dialect speech-to-text system according to claim 3, wherein: The audio acquisition also includes real-time optimization, specifically model architecture optimization and model lightweight, and accelerated decoding algorithm.

5. The processing method of a multi-dialect speech-to-text system according to claim 4, wherein: The model architecture optimization uses RNN-T or CTC+Attention streaming model architecture, uses multi-thread parallel processing for data preprocessing, feature extraction and recognition decoding, and uses independent threads for each step to improve throughput; Parallel efficiency formula: where: T parallel is the parallel processing time, T total : total task time, N is the number of parallel threads; Model lightweight, the model parameters from 32-bit floating-point quantization to INT8, while maintaining the recognition accuracy, remove low weight connection, through the teacher model guide student model training, while retaining performance while significantly reducing the model size; Accelerate decoding algorithm Introducing multi-layer BeamSearch to optimize decoding efficiency, reduce the number of candidate sequences, and improve output speed; Decoding optimization formula: where: S: best decoding sequence, log P(y|x): speech model probability, log P LM (y): language model probability, λ: language model weight.

6. The processing method of a multi-dialect speech-to-text system according to claim 5, wherein: The step 2: dialect detection includes multi-modal integration optimization, when needed in specific video conference or education scene, combined with lip reading or scene text information; Multi-modal feature fusion formula: Z = aX + βX audio visual ​ Wherein: Z: the feature vector after fusion, X audio : speech features, X visual : visual features, a, b: fusion weights.

7. The processing method of a multi-dialect speech-to-text system according to claim 6, wherein: The step 2: dialect detection includes scene adaptation model, for multi-person dialogue scene, introduce role recognition function, through speech separation model to identify the speaker identity, assign different transcription output.

8. The processing method of a multi-dialect speech-to-text conversion system of claim 1, wherein The method is applied to a multi-dialect speech text conversion system, comprising: Audio acquisition module: responsible for collecting user's voice input, supporting PCM, MP3, WAV format and 16kHz, 44.1kHz sampling rate; Dialect detection module: use pre-trained language classifier to identify the dialect type of voice input; Speech feature extraction module: convert speech signal to feature representation, including mel spectrum and MFCC; Multi-dialect speech recognition module: based on multi-task learning speech recognition model, map features to text sequence; Text post-processing module: according to language rules and context to optimize the recognition result, output the final standardized text; Model training module: responsible for system training and optimization, support transfer learning to extend new dialect.

Images