Multi-accent adaptive english speech recognition method, system, device, and medium

Abstract

The application relates to a multi-accent adaptive English speech recognition method, system, device and medium, which mixes open source data, multi-accent TTS synthesis data and noise data of English audio into a training set in proportion; mel filter bank features corresponding to each audio data in the training set are extracted, and noise data is superimposed on each mel filter bank feature based on a preset probability; a Zipformer model is improved to construct an improved Zipformer model; the improved Zipformer model is trained based on mel filter bank features with and without superimposed noise data; the mel filter bank features of English audio to be recognized are input into the trained improved Zipformer model, and an acoustic coding vector is output; the acoustic coding vector is post-processed to obtain a recognized text, and the English recognition accuracy in a non-standard accent scene is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, system, device and medium for multi-accent adaptive English speech recognition. Background Technology

[0002] Currently, mainstream Automatic Speech Recognition (ASR) technology has evolved from traditional HMM-GMM hybrid systems to end-to-end deep neural network models. Among them, the convolutional-enhanced Transformer structure, represented by Conformer, has achieved leading recognition accuracy in various ASR benchmark tests. Existing research has proposed the Zipformer structure, which, based on Conformer, achieves a better balance between recognition accuracy and inference efficiency through improvements such as encoder layer design, normalization methods, and optimization strategies. It has become one of the important solutions for lightweight edge ASR.

[0003] However, the fixed downsampling strategies in existing open-source English ASR systems based on Zipformer (including ecosystems such as Icefall, K2, and Sherpa-ONNX) cannot adapt to varying prosodic rhythms, lack robustness to environmental noise and accent superposition, and the training data is still mainly based on standard American English corpora such as LibriSpeech, resulting in poor system generalization ability and ultimately insufficient coverage of multi-accent scenarios such as Indian English, Middle Eastern English, and Southeast Asian English. In devices targeting the global consumer electronics market, such as smart set-top boxes, smart remote controls, and IoT voice assistants, their cross-accent generalization ability still has significant shortcomings. Summary of the Invention

[0004] The purpose of this application is to provide a multi-accent adaptive English speech recognition method, system, device and medium, which aims to solve the defects of existing open-source English ASR systems based on Zipformer, such as the inability of the fixed downsampling strategy to adapt to the changing rhythms, insufficient robustness to environmental noise and accent superposition, and poor system generalization ability caused by the long tail distribution of data.

[0005] This application provides a multi-accent adaptive English speech recognition method, including: The training set is formed by mixing open-source English audio data, multi-accent TTS synthesized data, and noise data in a certain proportion. Extract the Mel filter bank features corresponding to each audio data in the training set, and superimpose the noise data onto each Mel filter bank feature based on a preset probability; Based on the Zipformer model, an accent embedding auxiliary branch is added to calculate the accent embedding vector based on the features output from the first layer of the Zipformer model's Stack. Each downsample module in the Zipformer model is replaced with a prosody-aware adaptive downsampling module based on adaptive weights. Each deep convolution module in each Zipformer block of the second to sixth layers of the Stack is replaced with a multi-scale parallel convolution module based on the fusion of accent embedding vectors and parallel deep convolution results. Each bypass module in each Zipformer block is replaced with an accent-aware bypass gating module based on the fusion of accent embedding vectors and the input and processed features of the Zipformer block. This constructs an improved Zipformer model. The improved Zipformer model is trained based on the features of the Mel filter bank of superimposed and unsuperimposed noise data; The Mel filter bank features of the English audio to be identified are trained on an improved Zipformer model, which outputs an acoustic coding vector. The acoustic coding vector is then post-processed to obtain the identified text.

[0006] Preferably, the process of acquiring multi-accent TTS synthesis data includes: Select the target scene English text, and use an open-source TTS acoustic model to convert the target scene English text. During the conversion process, apply phoneme perturbation, fundamental frequency perturbation, speech rate scaling, and use different speaker embeddings to generate the multi-accent TTS synthesis data.

[0007] Preferably, while superimposing noise data onto the features of each Mel filter bank based on a preset probability, the method further includes: The noise data is enhanced by performing time masking and frequency masking.

[0008] Preferably, the forward computation process of the accent embedding auxiliary branch includes: The low-level acoustic features output from the first-layer Stack are subjected to global average pooling in the time dimension to obtain a compressed vector; The dimension of the compressed vector is mapped to the target dimension using a two-layer multilayer perceptron to obtain the accent embedding vector.

[0009] Preferably, the forward computation process of any prosody-aware adaptive downsampling module includes: Based on any pair of adjacent frame features in the feature sequence output by the previous Stack layer, the three-dimensional auxiliary statistical vector of the corresponding frame is calculated using the following formula: ; in, This represents the three-dimensional auxiliary statistical vector of frame t. This represents the features of the t-th frame in the feature sequence. This represents the features of the (t+1)th frame in the feature sequence. Represents the L2 norm. Indicates cosine similarity; The adjacent frame features and the corresponding frame's three-dimensional auxiliary statistical vector are concatenated to form the corresponding frame's local statistical features. A two-layer multilayer perceptron is used to map the dimensions of the local statistical features to the hidden dimensions to obtain the corresponding frame's gating factor. The gate factor is normalized using the Softmax function to obtain adaptive weights; Based on the adaptive weighted fusion of the adjacent frame features, the downsampled features of the corresponding frame are obtained; By traversing the feature sequence of each frame output by the previous Stack, the downsampled features of each frame are obtained. The downsampled features of each frame are the downsampled feature sequence output by the prosody-aware adaptive downsampling module.

[0010] Preferably, the forward computation process of any multi-scale parallel convolution module includes: The features output by the previous SA module in the corresponding Zipformer block are obtained by pointwise convolution and gated linear unit activation. The gated features are input into three parallel depthwise convolution branches with different kernel sizes, and the first convolution result, the second convolution result, and the third convolution result are output respectively. Based on the features output by the SA module and the accent embedding vector, the fusion weights of the three deep convolutional branches are predicted and calculated as follows: ; in, This represents the fusion weights of the i-th depthwise convolutional branch. This represents the Softmax function. This represents a multilayer perceptron. This represents global average pooling over the time dimension. This indicates the characteristics output by the SA module. Represents the accent embedding vector. Indicates feature splicing; The first, second, and third convolutional results are weighted and fused based on the fusion weights of each deep convolutional branch to obtain the fused convolutional features; The fused convolutional features are sequentially processed through BiasNorm, SwooshR activation, and pointwise convolution to obtain the multi-scale parallel convolutional features output by the multi-scale parallel convolution module.

[0011] Preferably, the forward calculation process of any accent-sensing bypass gating module includes: The accent embedding vector is mapped to a channel gate using the Sigmoid activation function; Based on the input features of the corresponding Zipformer block obtained by the channel gate weighted fusion and the features of the input features in the corresponding Zipformer block after being processed by each module before the accent perception bypass gating module, the intermediate output features or the final output features of the corresponding Zipformer block are obtained.

[0012] This application also provides a multi-accent adaptive English speech recognition system, including: The training set construction module is used to mix open-source English audio data, multi-accent TTS synthesized data, and noise data in a proportional manner to form a training set; The preprocessing module is used to extract the Mel filter bank features corresponding to each audio data in the training set, and to superimpose the noise data onto each Mel filter bank feature based on a preset probability. The model building module is used to build an improved Zipformer model by adding an accent embedding auxiliary branch that calculates accent embedding vectors based on the features output from the first layer of the Stack in the Zipformer model. It replaces all the downsample modules in the Zipformer model with prosodic-aware adaptive downsampling modules that perform downsampling based on adaptive weights. It replaces all the deep convolution modules in each Zipformer block of the Stack layers 2-6 with multi-scale parallel convolution modules that fuse the results of parallel deep convolution based on accent embedding vectors. It also replaces all the bypass modules in each Zipformer block with accent-aware bypass gating modules that fuse the input and processed features of the Zipformer block based on accent embedding vectors. The model training module is used to train the improved Zipformer model based on the features of the Mel filter bank of superimposed and unsuperimposed noise data. The speech recognition module is used to train an improved Zipformer model on the Mel filter bank features of the English audio to be recognized, output an acoustic coding vector, and post-process the acoustic coding vector to obtain the recognized text.

[0013] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0014] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0015] The beneficial effects of this application are as follows: This method mixes open-source English audio data, multi-accent TTS synthesis data, and noise data in a proportional manner to form a training set; extracts the Mel filter bank features corresponding to each audio data in the training set, and superimposes the noise data onto each Mel filter bank feature based on a preset probability; improves the Zipformer model to construct an improved Zipformer model; trains the improved Zipformer model based on the Mel filter bank features of superimposed and unsuperimposed noise data; outputs an acoustic coding vector by passing the Mel filter bank features of the English audio to be recognized through the trained improved Zipformer model; and performs post-processing on the acoustic coding vector to obtain the recognized text. This method overcomes the defects of existing open-source English ASR systems based on Zipformer and improves the English recognition accuracy in non-standard accent scenarios. Attached Figure Description

[0016] Figure 1 This is an optional flowchart of the multi-accent adaptive English speech recognition method provided in the embodiments of this application.

[0017] Figure 2 This is an optional structural diagram of the improved Zipformer model provided in the embodiments of this application.

[0018] Figure 3 This is a schematic diagram of an optional structure of the accent embedding auxiliary branch provided in the embodiments of this application.

[0019] Figure 4 This is an optional structural diagram of the prosody-aware adaptive downsampling module provided in the embodiments of this application.

[0020] Figure 5 This is an optional structural diagram of the multi-scale parallel convolution module provided in the embodiments of this application.

[0021] Figure 6 This is an optional structural diagram of the accent-sensing bypass gating module provided in an embodiment of this application.

[0022] Figure 7 This is an optional structural diagram of the end-side deployment architecture provided in the embodiments of this application.

[0023] Figure 8 This is an optional structural diagram of the multi-accent adaptive English speech recognition system provided in the embodiments of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0027] Based on this, embodiments of this application provide a multi-accent adaptive English speech recognition method, system, device, and medium, aiming to solve the defects of existing open-source English ASR systems based on Zipformer, such as the inability of the fixed downsampling strategy to adapt to varying prosody and rhythm, insufficient robustness to environmental noise and accent superposition, and poor system generalization ability caused by long-tailed data distribution.

[0028] The multi-accent adaptive English speech recognition method provided in this application relates to the field of network link fault diagnosis technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; the software can be an application implementing text classification methods, but is not limited to the above forms.

[0029] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers (PCs), minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0030] Please see Figure 1 , Figure 1 This is an optional flowchart of the multi-accent adaptive English speech recognition method provided in the embodiments of this application. In some embodiments of this application, Figure 1 The method described below may include, but is not limited to, steps S101 to S105. Figure 1 These five steps will be explained in detail.

[0031] S101: The training set is formed by proportionally mixing open-source English audio data, multi-accent TTS synthesized data, and noise data.

[0032] In some optional implementations, the open-source data uses the LibriSpeech public dataset (approximately 1000 hours of American English reading) as the foundation for the model, ensuring basic language comprehension and acoustic modeling capabilities.

[0033] In some optional embodiments, the process of acquiring multi-accent TTS synthesis data includes: Select target scenario English text (including IoT control commands, daily English conversations, etc.), and use an open-source TTS acoustic model to convert the target scenario English text. During the conversion process, apply phoneme perturbation (including replacing or deleting specific IPA phonemes according to preset rules, such as simulating Indian English pronouncing / v / as / w / , simulating Chinese English ignoring final consonant plosives, etc.), fundamental frequency perturbation (such as ±15%), speech rate scaling (such as randomly scaling the speech rate of the synthesized speech to 0.8~1.5 times speed), and use different speaker embeddings (such as using TTS speaker embeddings from different native language backgrounds to generate a mixture). The resulting large-scale pseudo data with accent features is the multi-accent TTS synthesized data.

[0034] In some optional embodiments, the noise data uses the MUSAN dataset, which includes three main categories: background music, loud human voices, and natural ambient noise, for online dynamic mixing.

[0035] In some alternative embodiments, the mixing ratio of the three types of data is open source data: multi-accent TTS synthesized data: noise data = 1:1:0.5, in order to balance the model's capabilities.

[0036] In a preferred embodiment, the method further includes: merging open-source data and multi-accent TTS synthesis data into a unified corpus, training a joint word segmenter using the BPE algorithm, and preferably using a vocabulary size of 500-2000 (for English) or 3000-5000 (for multilingual mixing) to ensure that the vocabulary simultaneously covers general English vocabulary (general words covered by LibriSpeech) and domain keywords (such as IPC control commands, XHome smart home terms, etc.) to avoid the situation where rare words cannot be recognized.

[0037] S102: Extract the Mel filter bank features corresponding to each audio data in the training set, and superimpose the noise data onto each Mel filter bank feature based on a preset probability.

[0038] In some optional embodiments, the extraction process of Mel filter bank features includes: resampling all audio data at a uniform sampling rate of 16kHz, and extracting 80-dimensional Mel filter bank features according to a frame length of 25ms and a frame shift of 10ms. Mel filter bank features simulate the critical bandwidth characteristics of the human ear and retain more spectral details compared to Mel frequency cepstral coefficients, making them suitable for deep neural network modeling.

[0039] In some optional embodiments, superimposing noise data onto the features of each Mel filter bank based on a preset probability includes: During training, noise data is superimposed onto the features of each Mel filter bank with a probability of p=0.5, and the signal-to-noise ratio (SNR) is uniformly sampled within the range of [0dB, 20dB].

[0040] In a preferred embodiment, while superimposing noise data onto the features of each Mel filter bank based on a preset probability, the method further includes: The noise data is enhanced by performing time masking and frequency masking.

[0041] By dynamically mixing and masking noisy data online, the model becomes accustomed to the combined distribution of noisy environment and accent during training, eliminating the need for denoising preprocessing at the inference end.

[0042] S103: Based on the Zipformer model, an accent embedding auxiliary branch is added to calculate the accent embedding vector based on the features output from the first layer of the Zipformer model's Stack. Each downsample module in the Zipformer model is replaced with a prosody-aware adaptive downsampling module based on adaptive weights. Each deep convolution module in each Zipformer block of the second to sixth layers of the Stack is replaced with a multi-scale parallel convolution module based on the result of fusing parallel deep convolutions with accent embedding vectors. Each bypass module in each Zipformer block is replaced with an accent-aware bypass gating module based on the fusion of the Zipformer block's input and processed features with accent embedding vectors, thus constructing an improved Zipformer model.

[0043] It is worth noting that the structure and process of the improved Zipformer model constructed in this embodiment are as follows: Figure 2 As shown, the core improvements include an Accent Embedding Auxiliary Branch (AEAB), a Rhythm-Aware Adaptive Downsampling (RAAD) module, a Multi-Scale Parallel Convolution (MSPC) module, and an Accent-Aware Bypass Gating (AABG) module. The remaining modules retain the original Zipformer U-Net-style downsampling encoder structure (i.e., 6 stacks with frame rates sequentially from 50Hz to 25Hz to 12.5Hz to 6.25Hz to 12.5Hz to 25Hz). The specific modifications are as follows: 1. Replace all three downsample modules (50→25Hz, 25→12.5Hz, 12.5→6.25Hz) between each stack with RAAD modules; 2. Within each Zipformer block from Stack-2 to Stack-6, replace the original depthwise convolutional module with the MSPC module; 3. Within each Zipformer block from Stack-2 to Stack-6, replace the original Bypass module with the AABG module; 4. Add a new AEAB branch, branching from the Stack-1 output to generate a 32-dimensional accent embedding vector. It broadcasts and injects into MSPC and AABG in a FiLM-style manner.

[0044] Both the MSPC and AABG modules rely on a 32-dimensional accent embedding vector. As a conditional signal, how to efficiently extract the embedding from the original audio without significantly increasing inference overhead is a key sub-problem that this embodiment needs to solve. To this end, this embodiment designs an accent embedding auxiliary branch AEAB, such as... Figure 3 As shown, the forward computation process of the accent embedding auxiliary branch includes: Branch entry: A low-level acoustic feature branch is generated from the output of Stack-1 (50Hz, not yet deeply downsampled). The reason for choosing Stack-1 instead of a higher Stack is that lower-level acoustic features are more sensitive to accents (mainly manifested in timbre, rhythm, and other low-level acoustic features), while higher-level representations have been abstracted into semantically related content.

[0045] Temporal Global Average Pooling: The low-level acoustic features output from the first-layer Stack are subjected to global average pooling in the temporal dimension, compressing the entire audio segment into a single-dimensional vector, resulting in a compressed vector. .

[0046] A two-layer multilayer perceptron is used to map the dimension of the compressed vector to the target dimension to obtain the accent embedding vector, calculated as follows: ; in, Represents the accent embedding vector. This represents the Swish activation function. This represents the first weight of the first layer of the multilayer perceptron. , This represents the first bias of the first layer of the multilayer perceptron. This represents the first weight of the second layer of the multilayer perceptron. , This represents the first bias of the second-layer multilayer perceptron.

[0047] Since the branch entry point has reused the forward computation results from Stack-1, the additional computations added by the AEAB module during inference are only: one time-dimensional average pooling + one two-layer MLP forward propagation (approximately D×128 + 128×32 = 32,768 + 4,096 ≈ 37K multiply-accumulate operations). Compared to the GFLOPs-level computation of the entire encoder, the inference overhead of the AEAB module is negligible (measured RTF increase <1%).

[0048] It is worth noting that during training, the mapping weights W are used. cls After mapping the accent embedding vector, the cross-entropy loss of the classification head is calculated using the Softmax function to supervise accent classification.

[0049] The original Downsample module within the Zipformer block uses the same weighting method regardless of whether the input frame is a voiceless consonant, a voiced consonant, a plosive, or a transitional frame. For multi-accented speech with significant prosodic differences, this fixed-weight downsampling method leads to information loss, ultimately resulting in a significant decrease in recognition rate for non-standard accents. To overcome this deficiency, this embodiment designs a prosodic-aware adaptive downsampling module. A lightweight gating network dynamically infers the relative importance of features in each pair of adjacent frames, generating adaptive weights for the corresponding frames. This weighted fusion of downsampled features for the corresponding frames enhances the recognition of accent-discriminating features such as plosive initiation and fricative bursts, thereby improving the recognition rate for non-standard accents.

[0050] Figure 4 This diagram illustrates the structure of any prosody-aware adaptive downsampling module with a downsampling rate r=2. Its forward computation process includes: Based on any pair of adjacent frame features in the feature sequence output from the previous layer Stack , D represents the feature dimension. The three-dimensional auxiliary statistical vector of the corresponding frame is calculated using the following formula: ; in, This represents the three-dimensional auxiliary statistical vector of frame t. This represents the features of the t-th frame in the feature sequence. This represents the features of the (t+1)th frame in the feature sequence. The L2 norm is used to characterize frame energy. Cosine similarity is used to characterize spectral differences, and the sensitivity to short plosives is improved by combining L2 norm with cosine similarity.

[0051] The adjacent frame features and the corresponding frame's three-dimensional auxiliary statistical vector are concatenated to form the corresponding frame's local statistical features. A two-layer multilayer perceptron is used to map the dimensions of local statistical features to the hidden dimensions, obtaining the gating factor for the corresponding frame. The calculation formula is as follows: ; in, Denotes the gating factor of frame t. This represents the Swish activation function. This represents the second weight of the first layer of the multilayer perceptron. , This represents the second bias of the first layer of the multilayer perceptron. This represents the second weight of the second layer of the multilayer perceptron. , represents the second bias of the second layer multilayer perceptron, and h represents the hidden dimension, which is preferably D / 4, so as to control the increase of the number of parameters to within 5% of the original Downsample module.

[0052] The gate factor is normalized using the Softmax function to obtain adaptive weights. Preferably, a learnable temperature can be added to the Softmax function. The adaptive weight calculation formula is as follows: ; ; in, This represents the adaptive weights for frame t. This represents the adaptive weights for frame t+1. This represents the Softmax function. This represents the learnable temperature, initialized to 1.0. These are learnable parameters.

[0053] Based on the adaptive weighted fusion of the features of adjacent frames, the downsampled features of the corresponding frame are obtained, and the calculation formula is as follows: ; in, This represents the downsampling features of frame t. This represents element-wise multiplication. Preferably, when downsampling is accompanied by changes in embedding dimension, the downsampled features can be adapted to the dimension through a linear layer.

[0054] By traversing the feature sequence of each frame output by the previous Stack, the downsampled features of each frame are obtained. The downsampled features of each frame are the downsampled feature sequence output by the prosody-aware adaptive downsampling module.

[0055] The RAAD module is a strict superset of the original Downsample module in terms of design: when the parameters of the gated network... , When initialized to zero and with bias initialization so that the Softmax output is [0.5, 0.5], the RAAD module degenerates into the behavior of the original Downsample module. After training begins, the gating network gradually learns to deviate from uniform weights on frame pairs with significant accents / prosody. This ensures a smooth transition during training, allowing for direct fine-tuning of the RAAD module on a pre-trained Zipformer model. Furthermore, under the same parameter scale, compared to the original Downsample module, the RAAD module reduces the error rate by 5%~12% (relatively) on non-standard accent test sets such as Hindi English and Chinglish, while maintaining a similar WER on the standard LibriSpeech test-clean (fluctuation <0.1% absolute).

[0056] Because the original deep convolutional module within the Zipformer block uses a single kernel size (e.g., 6 stacks configured with kernels of 31, 31, 15, 15, 15, and 31 respectively), these kernel sizes can adequately cover the duration range of English syllables (approximately 50-200ms) for standard English. However, the duration of phonemes varies greatly depending on the accent. For example, American English requires a larger kernel size to capture stress rhythms, Hindi / Chinglish requires a medium-sized kernel, and fast-paced accents or heavily accented plosive consonants require a smaller kernel size. A single kernel size cannot simultaneously adapt to these three time scales. To address this deficiency, this embodiment designs a multi-scale parallel convolutional module. A router with an accent condition dynamically allocates the fusion weights of the three parallel deep convolutional branches, adapting the three time scales through each fusion weight.

[0057] like Figure 5 As shown, the forward computation process of any multi-scale parallel convolution module includes: The features output by the previous SA module in the corresponding Zipformer block are processed by 1×1 pointwise convolution and gated linear unit activation to obtain gated features.

[0058] The gated features are input into three parallel depthwise convolution branches with different kernel sizes, and the first convolution result, the second convolution result, and the third convolution result are output respectively. Specifically: In the first depthwise convolution branch, the kernel size is set to 7 to capture short-term dependencies (plosive / acrid consonants, fast speech scenarios). In the second deep convolution branch, the kernel size is set to 15 to capture time-dependent (syllable / vowel segments). In the third deep convolution branch, the kernel size is set to 31 to capture long-term dependencies (prosody / accent patterns).

[0059] The router is configured to predict the fusion weights of the three deep convolutional branches based on the features output by the SA module and the accent embedding vector, calculated as follows: ; in, This represents the fusion weights of the i-th depthwise convolutional branch. This represents the Softmax function. This represents a multilayer perceptron. This represents global average pooling over the time dimension. This indicates the characteristics output by the SA module. This represents a 32-dimensional accent embedding vector. This indicates feature splicing.

[0060] The fused convolutional features are obtained by weighting and fusing the first, second, and third convolutional results based on the fusion weights of each deep convolutional branch. The calculation formula is as follows: ; in, This indicates the fusion of convolutional features. , , These represent the fusion weights of the first, second, and third depthwise convolutional branches, respectively. , , These represent the first depthwise convolution branch, the second depthwise convolution branch, and the third depthwise convolution branch, respectively.

[0061] The fused convolutional features are sequentially processed through BiasNorm, SwooshR activation, and pointwise convolution to obtain the multi-scale parallel convolutional features output by the multi-scale parallel convolution module.

[0062] Compared to the original depthwise convolutional module with a single kernel of 15, the MSPC module introduces two additional branches: kernel=7 and kernel=31. Since the number of parameters in the original depthwise convolutional module is only proportional to the number of channels × kernel size, the total number of parameters with the three branches is approximately (7+15+31) / 15 ≈ 3.5 times the original, but this only accounts for a small portion (<15%) of the total number of parameters in the Zipformer block. Overall, the MSPC module increases the number of parameters in Zipformer-M from 65.6M to approximately 72M, an increase of <10%, and an increase of <8% in inference computation. Furthermore, after training, the output... , , The results of cross-accent statistical analysis are as follows: American English sample: The mean value for (kernel=31) is approximately 0.45~0.55, reflecting that the model uses a long kernel to model the rereading rhythm.

[0063] Hindi / Chinglish: , The proportions are relatively balanced (approximately 0.3 to 0.4 for each syllable), reflecting the uniformity of the syllables.

[0064] Spanish-accented English (shorter plosive consonants): (kernel=7) The mean is significantly higher than other accents.

[0065] The results show a clear accent-branch preference correlation, which provides important support for the interpretability of the recognition method provided in this embodiment.

[0066] The Zipformer block has a Bypass module at the middle and end, which performs channel-level weighted fusion of input and output features. Once trained, the channel-level scalars are globally shared across all inputs (i.e., regardless of whether the input is American, Indian, or Chinglish, the bypass opening degree of each channel is the same). To address this limitation, this example designs an accent-aware bypass gating module. The globally shared channel-level scalars in the original Bypass module are replaced with channel gates dynamically generated from accent embedding vectors. These gates dynamically change with the accent embedding vectors, ensuring that different accents receive different bypass opening degrees in each Zipformer block.

[0067] like Figure 6 As shown, the forward computation process of any accent-aware bypass gating module includes: The accent embedding vector is mapped to a channel gate using the Sigmoid activation function, calculated as follows: ; in, Indicates a passageway door. This represents the Sigmoid activation function. Indicates learnable weights, , This represents the dimension of the accent embedding vector. , This indicates a learnable bias.

[0068] Based on the input features of the corresponding Zipformer block obtained by the channel gate weighted fusion and the features of the input features in the corresponding Zipformer block after being processed by the modules before the accent perception bypass gating module, the intermediate output features or final output features of the corresponding Zipformer block are obtained. The calculation formula is as follows: ; in, Indicates intermediate or final output features, and represents element-wise multiplication. This represents the input features corresponding to the Zipformer block. Representing input features The features are processed sequentially by the modules preceding the accent perception bypass gating module. The gating network in this module is very lightweight, introducing only... The number of parameters is approximately 512×32+512=16896 in Zipformer-M, which is negligible compared to the 65M parameters of the entire model.

[0069] because As the accent embedding vector dynamically changes, different accents will receive different degrees of bypass opening in each Zipformer block: For American English (the mainstream training distribution): the model has already learned a good representation on the backbone. It tends to be larger, meaning it goes through more modules.

[0070] For marginal accents such as Indian / Chinese: the core representation is imperfect. The bias tends to be smaller, meaning that more of the original input is passed through to avoid amplifying the erroneous bias of the backbone.

[0071] The AABG module essentially learns a unique residual mixing strategy for each accent, making it a very lightweight accent-conditional mixture of experts system.

[0072] S104: Train the improved Zipformer model based on the features of the Mel filter bank of superimposed and unsuperimposed noise data.

[0073] Specifically, the weighted sum of the multi-task losses is used as the total loss function, expressed as: ; in, Represents the total loss function. This represents the pruning Transducer loss for the recognition task. Indicates the auxiliary loss weight, This represents the cross-entropy loss of the classification head. Regarding the training strategy... It is preferable to set the value to 0.3 during the long training period (the first 10 epochs) to guide the accent embedding vector to converge quickly to a discriminative space, and then linearly decay it to 0.05 to avoid over-reliance on accent labels.

[0074] Backpropagation is performed based on the total loss function, and the model parameters of the improved Zipformer model are updated through the optimizer until the total loss function converges or the maximum number of iterations is reached, thus completing the training of the improved Zipformer model.

[0075] In some optional embodiments, the optimizer used during training is the original Zipformer's ScaledAdam, and the learning rate scheduling strategy is Eden-schedule (initial learning rate set to 0.045, warm-up time set to 500) to accelerate convergence and improve training stability.

[0076] It's worth noting that the Transducer loss used in the recognition task, compared to the standard RNN-T loss, retains only the alignment paths with higher posterior probabilities during forward computation and backward propagation, pruning the remaining paths. This significantly reduces memory usage (by more than 70%) while maintaining approximately equivalent gradient directions. This characteristic allows the pruning mechanism to naturally ignore unreasonable alignment paths, improving training stability.

[0077] As a preferred embodiment, after training is completed, the three functional modules of the Transducer framework are exported to ONNX respectively: encoder.onnx: Contains Conv-Embed, 6 stacks (containing all RAAD / MSPC / AABG modules), and the AEAB branch (only retaining the generated branch). The part (the classification head has been pruned) is responsible for converting Fbank features into acoustic coding vectors.

[0078] decoder.onnx: Stateless Decoder (context-based embedding layer) is responsible for predicting the embedding of the next token based on the decoded tokens.

[0079] joiner.onnx: Combines the outputs of the Encoder and Decoder, outputting the probability distribution over the vocabulary.

[0080] The export process supports variable-length inputs (i.e., dynamic shapes) to accommodate the dynamic accumulation of blocks in streaming recognition.

[0081] Applying ONNX-Runtime's INT8 dynamic quantization to encoder.onnx and joiner.onnx, quantizing only the weights (activations preserved in FP32), can compress the model size by approximately 75% (e.g., from 250MB to approximately 60MB), with an accuracy loss typically <0.1% WER absolute. The decoder, due to its extremely small number of parameters, is preserved in FP32 and not quantized.

[0082] like Figure 7 As shown, the three quantized ONNX files mentioned above are loaded through the Sherpa-onnx inference framework on embedded devices such as set-top boxes (e.g., models based on RK3568) and smart remote controls (ARM Cortex-A series). Sherpa-onnx is implemented in C++ and onnxruntime, and has been specifically optimized for Zipformer's chunk-wise streaming inference, supporting simultaneous speech and word output (a forward pass is performed every 320ms), fully meeting the real-time requirements of consumer electronics devices.

[0083] S105: The Mel filter bank features of the English audio to be recognized are processed by the improved Zipformer model after training, and the acoustic coding vector is output. The acoustic coding vector is then post-processed to obtain the recognized text.

[0084] In some optional embodiments, post-processing of the acoustic coding vector includes: The acoustic coding vector is decoded by a decoder to obtain a token sequence; By fusing the acoustic coding vector with the token sequence, a probability distribution of the vocabulary size is output; Greedy decoding or cluster search is performed on the probability distribution to generate recognition text containing the text sequence.

[0085] The multi-accent adaptive English speech recognition method provided in this embodiment has the following beneficial effects: 1. The designed RAAD module enables the model to adaptively allocate downsampling weights based on accent prosodic features (plosives, syllable boundaries, etc.), preserving key acoustic information in multi-accent scenarios.

[0086] 2. The designed MSPC module enables the model to simultaneously process different phoneme duration distributions (short plosives, medium syllables, and long prosody) corresponding to different accents.

[0087] 3. The designed AABG module enables different accent inputs to learn exclusive residual mixing strategies, achieving accent specialization without changing the main architecture.

[0088] 4. The accent embedding vector output by the designed AEAB branch is used as the conditional input of both the MSPC module and the AABG module, realizing accent conditionalization of the two modules, and the inference overhead increases by <1%.

[0089] 5. A complete training data pipeline was formed through online dynamic noise mixing, while the use of the BPE joint word segmenter ensured consistent coverage of general vocabulary and domain vocabulary.

[0090] Please see Figure 8 This application also provides a multi-accent adaptive English speech recognition system, which can implement the above-mentioned multi-accent adaptive English speech recognition method. The system includes: Training set construction module 801 is used to mix open source data of English audio, multi-accent TTS synthesis data, and noise data in a proportional manner to form a training set; The preprocessing module 802 is used to extract the Mel filter bank features corresponding to each audio data in the training set, and to superimpose the noise data onto each Mel filter bank feature based on a preset probability. Model building module 803 is used to build an improved Zipformer model by adding an accent embedding auxiliary branch that calculates accent embedding vectors based on the features output from the first layer of the Stack in the Zipformer model. It replaces each downsample module in the Zipformer model with a prosody-aware adaptive downsampling module that performs downsampling based on adaptive weights. It replaces each deep convolution module in each Zipformer block of the second to sixth layers of the Stack in the Zipformer model with a multi-scale parallel convolution module that fuses the results of parallel deep convolution based on accent embedding vectors. It also replaces each bypass module in each Zipformer block with an accent-aware bypass gating module that fuses the input and processed features of the Zipformer block based on accent embedding vectors. Model training module 804 is used to train the improved Zipformer model based on the features of the Mel filter bank of superimposed and unsuperimposed noise data. The speech recognition module 805 is used to train an improved Zipformer model on the Mel filter bank features of the English audio to be recognized, output an acoustic coding vector, and perform post-processing on the acoustic coding vector to obtain the recognized text.

[0091] The specific implementation of this multi-accent adaptive English speech recognition system is basically the same as the specific implementation of the multi-accent adaptive English speech recognition method described above, and will not be repeated here.

[0092] The electronic device shown in this embodiment is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments disclosed herein.

[0093] In this embodiment, the electronic device is presented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit, at least one storage unit, a bus connecting different system components (including the storage unit and the processing unit), a display unit, etc.

[0094] The storage unit stores program code, which can be executed by the processing unit to perform the steps described in the method section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit can perform actions such as... Figure 1 The steps are shown in the figure.

[0095] The storage unit may include readable media in the form of volatile storage units, such as random access memory (RAM) and / or cache storage units, and may further include read-only memory (ROM).

[0096] The storage unit may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0097] A bus can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus that uses any of the various bus structures.

[0098] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter can communicate with other modules of the electronic device via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0099] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0100] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the methods described above according to the embodiments of this disclosure.

[0101] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0102] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0103] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0104] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0105] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0106] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A multi-accent adaptive English speech recognition method, characterized in that, include: The training set is formed by mixing open-source English audio data, multi-accent TTS synthesized data, and noise data in a certain proportion. Extract the Mel filter bank features corresponding to each audio data in the training set, and superimpose the noise data onto each Mel filter bank feature based on a preset probability; Based on the Zipformer model, an accent embedding auxiliary branch is added to calculate the accent embedding vector based on the features output from the first layer of the Zipformer model's Stack. Each downsample module in the Zipformer model is replaced with a prosody-aware adaptive downsampling module based on adaptive weights. Each deep convolution module in each Zipformer block of the second to sixth layers of the Stack is replaced with a multi-scale parallel convolution module based on the fusion of accent embedding vectors and parallel deep convolution results. Each bypass module in each Zipformer block is replaced with an accent-aware bypass gating module based on the fusion of accent embedding vectors and the input and processed features of the Zipformer block. This constructs an improved Zipformer model. The improved Zipformer model is trained based on the features of the Mel filter bank of superimposed and unsuperimposed noise data; The Mel filter bank features of the English audio to be identified are trained on an improved Zipformer model, which outputs an acoustic coding vector. The acoustic coding vector is then post-processed to obtain the identified text.

2. The method according to claim 1, characterized in that, The process of acquiring multi-accent TTS synthesis data includes: Select the target scene English text, and use an open-source TTS acoustic model to convert the target scene English text. During the conversion process, apply phoneme perturbation, fundamental frequency perturbation, speech rate scaling, and use different speaker embeddings to generate the multi-accent TTS synthesis data.

3. The method according to claim 1, characterized in that, In addition to superimposing noise data onto the features of each Mel filter bank based on preset probabilities, the process also includes: The noise data is enhanced by performing time masking and frequency masking.

4. The method according to claim 1, characterized in that, The forward computation process of the accent embedding auxiliary branch includes: The low-level acoustic features output from the first-layer Stack are subjected to global average pooling in the time dimension to obtain a compressed vector; The dimension of the compressed vector is mapped to the target dimension using a two-layer multilayer perceptron to obtain the accent embedding vector.

5. The method according to claim 1, characterized in that, The forward computation process of any prosody-aware adaptive downsampling module includes: Based on any pair of adjacent frame features in the feature sequence output by the previous Stack layer, the three-dimensional auxiliary statistical vector of the corresponding frame is calculated using the following formula: ； in, This represents the three-dimensional auxiliary statistical vector of frame t. This represents the features of the t-th frame in the feature sequence. This represents the features of the (t+1)th frame in the feature sequence. Represents the L2 norm. Indicates cosine similarity; The adjacent frame features and the corresponding frame's three-dimensional auxiliary statistical vector are concatenated to form the corresponding frame's local statistical features. A two-layer multilayer perceptron is used to map the dimensions of the local statistical features to the hidden dimensions to obtain the corresponding frame's gating factor. The gate factor is normalized using the Softmax function to obtain adaptive weights; Based on the adaptive weighted fusion of the adjacent frame features, the downsampled features of the corresponding frame are obtained; By traversing the feature sequence of each frame output by the previous Stack, the downsampled features of each frame are obtained. The downsampled features of each frame are the downsampled feature sequence output by the prosody-aware adaptive downsampling module.

6. The method according to claim 1, characterized in that, The forward computation process of any multi-scale parallel convolution module includes: The features output by the previous SA module in the corresponding Zipformer block are obtained by pointwise convolution and gated linear unit activation. The gated features are input into three parallel depthwise convolution branches with different kernel sizes, and the first convolution result, the second convolution result, and the third convolution result are output respectively. Based on the features output by the SA module and the accent embedding vector, the fusion weights of the three deep convolutional branches are predicted and calculated as follows: ； in, This represents the fusion weights of the i-th depthwise convolutional branch. This represents the Softmax function. This represents a multilayer perceptron. This represents global average pooling over the time dimension. This indicates the characteristics output by the SA module. Represents the accent embedding vector. Indicates feature splicing; The first, second, and third convolutional results are weighted and fused based on the fusion weights of each deep convolutional branch to obtain the fused convolutional features; The fused convolutional features are sequentially processed through BiasNorm, SwooshR activation, and pointwise convolution to obtain the multi-scale parallel convolutional features output by the multi-scale parallel convolution module.

7. The method according to claim 1, characterized in that, The forward computation process of any accent-aware bypass gating module includes: The accent embedding vector is mapped to a channel gate using the Sigmoid activation function; Based on the input features of the corresponding Zipformer block obtained by the channel gate weighted fusion and the features of the input features in the corresponding Zipformer block after being processed by each module before the accent perception bypass gating module, the intermediate output features or the final output features of the corresponding Zipformer block are obtained.

8. A multi-accent adaptive English speech recognition system, characterized in that, include: The training set construction module is used to mix open-source English audio data, multi-accent TTS synthesized data, and noise data in a proportional manner to form a training set; The preprocessing module is used to extract the Mel filter bank features corresponding to each audio data in the training set, and to superimpose the noise data onto each Mel filter bank feature based on a preset probability. The model building module is used to build an improved Zipformer model by adding an accent embedding auxiliary branch that calculates accent embedding vectors based on the features output from the first layer of the Stack in the Zipformer model. It replaces all the downsample modules in the Zipformer model with prosodic-aware adaptive downsampling modules that perform downsampling based on adaptive weights. It replaces all the deep convolution modules in each Zipformer block of the Stack layers 2-6 with multi-scale parallel convolution modules that fuse the results of parallel deep convolution based on accent embedding vectors. It also replaces all the bypass modules in each Zipformer block with accent-aware bypass gating modules that fuse the input and processed features of the Zipformer block based on accent embedding vectors. The model training module is used to train the improved Zipformer model based on the features of the Mel filter bank of superimposed and unsuperimposed noise data. The speech recognition module is used to train an improved Zipformer model on the Mel filter bank features of the English audio to be recognized, output an acoustic coding vector, and post-process the acoustic coding vector to obtain the recognized text.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images