Speech recognition model training method and device, equipment and medium

Abstract

The invention relates to a speech recognition technology, discloses a speech recognition model training method and device, equipment and a medium, and aims to improve the speech recognition rate in a noise environment. The method comprises the following steps: firstly, training an automatic speech recognition network to obtain a pre-trained network; a noise reduction module is introduced in front of an output classification layer, the noise reduction module is trained by taking embedded features output by the pre-trained automatic speech recognition network as a reference, and parameters of the pre-trained automatic speech recognition network are fixed during training to form an initial speech recognition model; and finally, noise reduction module parameters are fixed, the pre-trained automatic speech recognition network is retrained, and a final model is obtained. Through staged training and a parameter fixing strategy, training target conflicts among modules are avoided, and the training stability and the convergence speed are improved; the noise reduction module focuses on feature denoising required by recognition, is high in adaptability, has the characteristics of light weight and low delay, and can be widely applied to low-resource real-time scenes such as embedded equipment and edge computing.

Description

Technical Field

[0001] This invention relates to speech recognition technology, and in particular to a method, apparatus, device, and medium for training a speech recognition model. Background Technology

[0002] In the field of speech recognition technology, automatic speech recognition networks can achieve high recognition accuracy in clean speech environments, but their recognition performance will drop significantly in complex acoustic environments with noise or reverberation, making it difficult to meet the needs of practical application scenarios (such as embedded devices, edge computing, and online voice interaction).

[0003] To improve speech recognition performance in noisy environments, existing technologies typically employ a "front-end noise reduction module + back-end recognition model" architecture. This involves adding a noise reduction module to the input of the automatic speech recognition network to preprocess and denoise the noisy speech before feeding the processed speech signal into the recognition model for identification. However, this type of existing solution suffers from three core problems: First, unreasonable training logic leads to convergence difficulties. Existing technologies often adopt the method of "joint training of noise reduction module and automatic speech recognition network". The training objectives of the two are conflicting - the noise reduction module focuses on "the noise removal effect of speech signal", while the automatic speech recognition network focuses on "accurate recognition of speech content". When training together, parameters are prone to mutual interference, resulting in slow overall convergence speed and poor stability of the model, making it difficult to simultaneously achieve both noise reduction effect and recognition performance.

[0004] Secondly, the training of the noise reduction module lacks an effective benchmark. Existing technologies for training noise reduction modules mostly rely on the traditional mean squared error (MSE) loss function, optimizing module parameters only by calculating the amplitude difference between noisy and clean speech. This fails to establish an effective training benchmark by combining the feature extraction capabilities of automatic speech recognition networks. As a result, although the noise-reduced speech is close to the clean speech in amplitude, the matching degree is low in the feature dimensions required for speech recognition (such as embedded features), which cannot effectively improve the subsequent recognition accuracy.

[0005] Third, the lack of parameter fixing strategies affects performance optimization. Existing technologies do not design phased parameter fixing and training strategies for the "automatic speech recognition network" and the "noise reduction module". Either the parameters of the two are updated simultaneously throughout the process (causing target conflicts), or the noise reduction module is trained separately and then directly connected to the recognition network (without allowing the recognition network to adapt to the noise-reduced features). Ultimately, this results in limited improvement in recognition performance in noisy environments, making it difficult to meet the recognition accuracy requirements of practical applications.

[0006] Therefore, there is an urgent need for an automatic speech recognition model training method that can achieve stable improvement in speech recognition performance under noisy environments by using a reasonable phased training logic, a clear parameter fixing strategy, and an effective training benchmark. Summary of the Invention

[0007] This invention provides a training method, apparatus, device, and medium for a speech recognition model, aiming to solve the technical problem of unstable performance of speech recognition models in noisy environments in the prior art.

[0008] To achieve the aforementioned objectives, the first aspect of this invention proposes a method for training a speech recognition model, the method comprising: The automatic speech recognition network is trained to obtain a pre-trained automatic speech recognition network; A noise reduction module is introduced before the output classification layer of the pre-trained automatic speech recognition network, and the noise reduction module is trained based on the embedding features output by the pre-trained automatic speech recognition network to obtain an initial speech recognition model that includes the pre-trained automatic speech recognition network and the noise reduction module; wherein, when training the noise reduction module, the parameters of the pre-trained automatic speech recognition network are fixed. By fixing the parameters of the noise reduction module in the initial speech recognition model, the pre-trained automatic speech recognition network is retrained to obtain the final speech recognition model.

[0009] Further, training the automatic speech recognition network to obtain a pre-trained automatic speech recognition network includes: The automatic speech recognition network is trained using a corpus set that can support the automatic speech recognition network in learning speech feature rules and achieving generalization ability. Data augmentation technology is used during the training process, and the corpus set includes clean speech and noisy speech. When the word error rate of the automatic speech recognition network on the validation set no longer decreases, or when the loss function of the automatic speech recognition network converges, training is stopped, and the pre-trained automatic speech recognition network is obtained.

[0010] Furthermore, the noise reduction module includes: An encoder used for dimensionality transformation and preliminary feature extraction of input features, the encoder being composed of several convolutional layers; A dual-path recurrent neural network layer used to model bidirectional dependency in the frequency dimension and causal relationship in the time dimension of encoder output features, respectively. A decoder is used to restore the dimension of the output features of a dual-path recurrent neural network layer and fuse multi-scale features. The decoder is structurally symmetrical with the encoder, and the decoder fuses the multi-scale features output by the encoder through skip connections.

[0011] Furthermore, in the several convolutional layers of the encoder, the kernel size of the convolutional layer in the time dimension is fixed at 1, while the kernel size of the convolutional layer in the frequency dimension is greater than 1; and except for the first convolutional layer of the encoder, the remaining convolutional layers all use grouped convolution to extract features, so as to reduce computational complexity.

[0012] Furthermore, the dual-path recurrent neural network layer includes a bidirectional recurrent neural network sublayer and a unidirectional recurrent neural network sublayer; wherein, the bidirectional recurrent neural network sublayer is used to perform bidirectional dependency modeling of the encoder output features in the frequency dimension, and the unidirectional recurrent neural network sublayer is used to perform causal relationship modeling of the features output by the bidirectional recurrent neural network sublayer in the time dimension, so as to ensure low latency and real-time performance of speech recognition.

[0013] Furthermore, training the noise reduction module includes: The clean speech is input into the pre-trained automatic speech recognition network to obtain the clean embedding features of the clean speech output by the network. The noisy speech is input into the pre-trained automatic speech recognition network, and then the noisy features output by the network are input into the noise reduction module to obtain the denoised embedding features output by the noise reduction module. The denoising module is trained using a combined loss function, which includes embedding space similarity loss and mean squared error loss. The embedding space similarity loss is obtained by calculating the similarity between the clean embedding feature and the denoised embedding feature, and the mean squared error loss is obtained by calculating the magnitude difference between the clean embedding feature and the denoised embedding feature. Training stops when the combined loss function converges, and the parameters of the pre-trained automatic speech recognition network are fixed throughout the training process.

[0014] Furthermore, when retraining the pre-trained automatic speech recognition network, the process includes: The noisy speech is input into the initial speech recognition model, and then processed sequentially by the pre-trained automatic speech recognition network and the noise reduction module with fixed parameters to output speech features. Based on the aforementioned speech features, the loss function corresponding to the speech recognition task is used for training; Training stops when the loss function converges, or when the word error rate of the initial speech recognition model on the noisy speech validation set no longer decreases, and the final speech recognition model is obtained.

[0015] A second aspect of the present invention provides a training apparatus for a speech recognition model, the apparatus comprising: The first training module is used to train the automatic speech recognition network to obtain a pre-trained automatic speech recognition network. The second training module is used to introduce a noise reduction module before the output classification layer of the pre-trained automatic speech recognition network, and to train the noise reduction module based on the embedded features output by the pre-trained automatic speech recognition network to obtain an initial speech recognition model that includes the pre-trained automatic speech recognition network and the noise reduction module; wherein, when training the noise reduction module, the parameters of the pre-trained automatic speech recognition network are fixed. The third training module is used to fix the parameters of the noise reduction module in the initial speech recognition model and retrain the pre-trained automatic speech recognition network to obtain the final speech recognition model.

[0016] A third aspect of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the training method for the speech recognition model described in any of the preceding claims.

[0017] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the training method for the speech recognition model as described in any of the preceding claims.

[0018] Beneficial effects: The training method, apparatus, device, and medium of the speech recognition model of this invention significantly improve training stability. Through a phased training strategy of "pre-training the automatic speech recognition network → training the denoising module → fine-tuning the automatic speech recognition network," coupled with fixed parameter design at each stage, the target conflict between the denoising module and the recognition network in traditional joint training is avoided, making the model training process more stable and convergence faster. Performance optimization in noisy environments is achieved by training the denoising module based on the embedded features output by the pre-trained automatic speech recognition network, ensuring that the denoising module focuses on optimizing the core features required for speech recognition. This makes the denoising effect highly compatible with recognition needs, significantly improving the speech recognition rate in noisy environments. It exhibits strong adaptability and versatility; the method is not limited to specific architectures of automatic speech recognition networks or denoising modules, does not rely on complex hardware resources, and can be flexibly applied to various speech recognition scenarios, especially adaptable to deployment requirements in low-resource scenarios such as embedded devices and edge computing. Excellent model synergy is achieved through phased fine-tuning to achieve deep adaptation between the automatic speech recognition network and the denoising module, avoiding feature incompatibility problems caused by traditional independent module combinations, and improving the overall inference efficiency and recognition reliability of the model. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a training method for a speech recognition model according to an embodiment of this application. Figure 2This is a schematic block diagram of the structure of a training device for a speech recognition model according to an embodiment of this application; Figure 3 This is a schematic block diagram of the structure of a computer device according to an embodiment of this application.

[0020] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any modules and all combinations of one or more associated listed items.

[0023] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0024] This application takes Project A, "Development of Speech Recognition Model under Noise in Embedded Devices," as an example for further explanation.

[0025] Reference Figure 1 This invention provides a method for training a speech recognition model, the method comprising: S1: Train the automatic speech recognition network to obtain a pre-trained automatic speech recognition network.

[0026] The aforementioned automatic speech recognition network refers to a deep learning network capable of converting speech signals into text information. It typically includes a feature extraction layer, an intermediate encoding layer, and an output classification layer, enabling the mapping from speech to text. In this embodiment, a lightweight and deployable automatic speech recognition network is generally selected.

[0027] Pre-trained automatic speech recognition network: refers to an automatic speech recognition network that has undergone initial training, has stable performance (such as word error rate) on the validation set or whose loss function has converged, and has basic speech recognition capabilities.

[0028] In Project A, an automatic speech recognition network based on the Transformer architecture was selected (the input was a speech signal with a sampling rate of 16kHz, and the feature extraction layer used Mel-frequency cepstral coefficients (MFCC) to extract speech features). A corpus containing 8000 hours of clean speech (covering speakers of different genders, ages, and accents) and 4000 hours of noisy speech (including common scene noises such as street noise and office noise) was used for training. During training, data augmentation techniques such as "speech speed perturbation (adjusting the speech speed to 0.8-1.2 times)" and "volume perturbation (adjusting the speech volume to 0.7-1.3 times)" were used. A stochastic gradient descent (SGD) optimizer was employed, with the cross-entropy loss function as the optimization objective. Iterative training continued until the word error rate (WER) on the validation set stabilized below 5% and no longer decreased for five consecutive epochs. At this point, training was stopped, resulting in a pre-trained automatic speech recognition network.

[0029] By training with a large-scale mixed corpus (including clean and noisy speech), the automatic speech recognition network can learn the speech feature patterns under different acoustic environments in advance, avoiding the problem of "poor generalization ability in noisy environments" caused by training with only clean speech. Data augmentation technology further expands the diversity of the corpus, improves the robustness of the pre-trained automatic speech recognition network, and lays a good foundation for subsequent noise reduction module training and network fine-tuning.

[0030] S2: A noise reduction module is introduced before the output classification layer of the pre-trained automatic speech recognition network, and the noise reduction module is trained based on the embedded features output by the pre-trained automatic speech recognition network to obtain an initial speech recognition model that includes the pre-trained automatic speech recognition network and the noise reduction module; wherein, when training the noise reduction module, the parameters of the pre-trained automatic speech recognition network are fixed.

[0031] The output classification layer refers to the final layer in an automatic speech recognition network that maps intermediate features to text categories (such as phonemes and lexical units). Its input is the network's intermediate features (i.e., embedded features), and its output is the probability distribution of each category.

[0032] The noise reduction module refers to the module used to remove noise components from the speech signal and retain effective speech features. In this embodiment, a lightweight convolutional recurrent network (CRN) structure is adopted.

[0033] Embedded features refer to the intermediate features that precede the classification layer output by the automatic speech recognition network. They can represent the core semantic and acoustic information of speech and are a key bridge connecting "speech feature extraction" and "text classification".

[0034] The initial speech recognition model refers to a model consisting of a pre-trained automatic speech recognition network and a trained noise reduction module, which has basic noise removal and speech recognition capabilities.

[0035] In Project A, a lightweight CRN denoising module (with only 1 / 5 the number of parameters of a traditional front-end denoising module) is introduced before the output classification layer (i.e., the word classification layer) of the pre-trained Transformer automatic speech recognition network. During training, all parameters of the pre-trained automatic speech recognition network are frozen in the code (ensuring stable feature extraction capabilities). The embedding features output by the pre-trained Transformer automatic speech recognition network are used as a benchmark: clean speech is input into the pre-trained Transformer automatic speech recognition network to obtain its clean embedding features; noisy speech is input into the pre-trained Transformer automatic speech recognition network, and then the noisy embedding features output by the network are input into the denoising module to obtain denoised embedding features. The denoising module is trained using a combined loss function of "embedding space similarity loss + mean squared error (MSE) loss," iterating until the combined loss function converges (loss value fluctuation is less than 0.001 for 5 consecutive epochs). At this point, an initial speech recognition model containing the pre-trained Transformer automatic speech recognition network and the trained lightweight CRN denoising module is obtained.

[0036] Fixed pre-trained automatic speech recognition network parameters can prevent interference with its feature extraction capabilities and ensure the stability of the training benchmark (embedded features) for the noise reduction module. Training with embedded features as the benchmark allows the noise reduction module to focus on "noise reduction of the core features required for speech recognition" rather than simply pursuing amplitude matching of the speech signal, thus solving the problem of "improved speech quality after noise reduction but no improvement in recognition rate" in traditional noise reduction modules. The lightweight noise reduction module design adapts to the "embedded device deployment" requirements of Project A, reducing hardware resource consumption.

[0037] S3: Fix the parameters of the noise reduction module in the initial speech recognition model, and retrain the pre-trained automatic speech recognition network to obtain the final speech recognition model.

[0038] The final speech recognition model refers to the model that, after being fine-tuned by the pre-trained automatic speech recognition network, has the best fit between the noise reduction module and the automatic speech recognition network, and has a high recognition accuracy in noisy environments.

[0039] In Project A, all parameters of the CRN noise reduction module in the initial speech recognition model were frozen in the code (to ensure its noise reduction capability was stable). The pre-trained Transformer automatic speech recognition network was retrained using the WER (Warranty Error Rate) of a noisy speech validation set (containing 200 hours of noisy speech from different scenarios) as the core metric. The SGD optimizer was still used during training, with the cross-entropy loss function as the optimization objective. Iteration continued until the WER of the validation set stabilized (no longer decreasing after 5 consecutive epochs), at which point the final speech recognition model was obtained.

[0040] Fixing the parameters of the noise reduction module can prevent its noise reduction capability from being interfered with. By fine-tuning the pre-trained automatic speech recognition network, the network can be adapted to the "features processed by the noise reduction module", further optimizing the recognition performance in noisy environments. Compared with the traditional "joint training", this step can avoid mutual interference between model parameters, improve training stability, and ensure that the WER of the final model of Project A is significantly reduced compared with the initial model in noisy environments on embedded devices.

[0041] The core of the speech recognition model training method in this embodiment is a dual strategy of "phased training + parameter fixing": The first stage trains the automatic speech recognition network (APR) with a large-scale mixed corpus, giving it basic recognition capabilities. The second stage fixes the parameters of the pre-trained APR and trains the denoising module based on the embedded features output by the network, ensuring that the denoising module is compatible with the feature extraction logic of the pre-trained APR. The third stage fixes the parameters of the denoising module and fine-tunes the pre-trained APR, achieving a deep integration of "denoising capability" and "recognition capability." In this embodiment, training stability is significantly improved. The phased training and parameter fixing strategy avoids the target conflict between the denoising module and the recognition network in traditional "joint training," making the model training process more stable in terms of loss convergence and reducing the number of iterations compared to traditional methods. Deployment adaptability is better; the lightweight denoising module and phased training logic are adapted to low-resource scenarios such as embedded devices and edge computing. The recognition rate in noisy environments is significantly improved. Through "denoising module training based on embedded features + network fine-tuning," the final model's WER in noisy environments such as streets is significantly lower than traditional methods, fully meeting the needs of practical applications. With strong versatility, this method is not limited to automatic speech recognition networks with specific architectures (such as Transformer, CNN-RNN) or specific types of noise. It can be widely applied to the development of speech recognition models in various noisy environments, such as smart speakers, in-vehicle voice interaction, and voice control of wearable devices.

[0042] In one embodiment, step S1, which trains the automatic speech recognition network to obtain a pre-trained automatic speech recognition network, includes: S11: The automatic speech recognition network is trained using a corpus set that can support the automatic speech recognition network in learning speech feature rules and achieving generalization ability. Data augmentation technology is used during the training process, and the corpus set includes clean speech and noisy speech.

[0043] The aforementioned corpus set capable of supporting automatic speech recognition networks in learning speech feature patterns and achieving generalization ability refers to a corpus set whose size and diversity (including different scenarios, speakers, and noise types) are sufficient to enable the automatic speech recognition network to learn general speech feature patterns and possess stable recognition ability on unknown speech data. Data augmentation techniques refer to techniques that expand the diversity of the corpus and improve the generalization ability of the model by reasonably transforming the original speech data (without changing the semantics). Common methods include speed perturbation, volume perturbation, and noise superposition.

[0044] In Project A, the corpus is specifically a “12,000-hour mixed corpus”: including 8,000 hours of clean speech (covering the speech of speakers of different accents of Mandarin Chinese (such as Northeast accent, Sichuan accent), different genders (male and female), and different ages (children, youth, and the elderly), with text content covering daily conversations, news broadcasts, command control, etc.) and 4,000 hours of noisy speech (noise types include street noise (car horns, crowd noise), office noise (keyboard typing, people talking), and household noise (television sound, kitchen sound), with a signal-to-noise ratio range of 0-20dB). Three data augmentation techniques were employed during training: 1. Speed ​​perturbation: The speech speed was randomly adjusted to 0.8x, 0.9x, 1.1x, and 1.2x (without altering the semantic meaning); 2. Volume perturbation: The decibel value of the speech was randomly adjusted to 0.7x, 0.8x, 1.2x, and 1.3x of the original value; 3. Noise superposition: Noise segments from the above noisy speech were randomly superimposed onto clean speech (signal-to-noise ratio randomly ranging from 5-15dB). The corpus was divided into training, validation, and test sets in an 8:1:1 ratio for training the Transformer automatic speech recognition network.

[0045] The scale and diversity of the mixed corpus are sufficient to support the network in learning speech feature patterns and avoid the problem of "overfitting caused by training with small corpus"; the introduction of noisy speech allows the network to be exposed to noise features in advance, improving its generalization ability in noisy environments; data augmentation technology further expands the diversity of the corpus. After adopting three augmentation techniques, the equivalent size of the training set is expanded, and the recognition stability of the network in unknown noise scenarios is greatly improved.

[0046] S12: When the word error rate of the automatic speech recognition network on the validation set no longer decreases, or when the loss function of the automatic speech recognition network converges, training is stopped, and the pre-trained automatic speech recognition network is obtained.

[0047] The word error rate (WER) mentioned above is a core metric for measuring the accuracy of speech recognition results. The calculation formula is WER = {insertion errors (I) + deletion errors (D) + substitution errors (S)} / {total number of words in the reference text (N)} × 100%, where "insertion errors" refer to extra words in the recognition result, "deletion errors" refer to omitted words in the recognition result, and "substitution errors" refer to words in the recognition result that do not match the reference text. The lower the WER value, the higher the recognition accuracy. Loss function convergence means that the value of the loss function no longer decreases significantly with the number of iterations during training. It is usually defined as "the loss value fluctuates less than a set threshold (e.g., 0.001) for several consecutive epochs," indicating that the model's learning process has stabilized and further training cannot significantly improve performance.

[0048] In Project A, when training the Transformer automatic speech recognition network, the WER and cross-entropy loss function values ​​of the validation set are calculated every epoch (one iteration of the training set). Training stops when one of the following two conditions is met: 1. The validation set WER no longer decreases for five consecutive epochs (e.g., WER is 5.2% in epoch 10, and WERs for epochs 11-15 are 5.1%, 5.2%, 5.1%, 5.2%, and 5.1%, respectively); 2. The cross-entropy loss function value fluctuates less than 0.001 for five consecutive epochs (e.g., loss value is 0.123 in epoch 12, and loss values ​​for epochs 13-17 are 0.122, 0.123, 0.122, 0.123, and 0.122, respectively). In Project A, when training reached epoch 15, the validation set WER stabilized below 5% for five consecutive epochs, at which point training stopped, resulting in the pre-trained automatic speech recognition network.

[0049] Using "WER no longer decreases or loss converges" as the stopping condition can avoid "overfitting caused by overtraining" and "underfitting caused by undertraining", ensuring that the pre-trained automatic speech recognition network achieves the optimal balance between "fit degree" and "generalization ability". The WER of the validation set of the pre-trained automatic speech recognition network is stable below 5%, providing a high-quality feature extraction foundation for the subsequent training of the noise reduction module.

[0050] In this embodiment, the core is to clarify the "technical requirements for the corpus" and the "criterion for stopping training": by limiting the corpus to "support the network's learning of feature patterns and achieve generalization ability" and "containing both clean and noisy speech," the pre-trained automatic speech recognition network is ensured to have basic robustness; by limiting the stopping condition to "WER no longer decreases or loss converges," the performance stability of the pre-trained automatic speech recognition network is ensured. Specifically, the robustness of the pre-trained automatic speech recognition network is improved. The combination of corpus and data augmentation techniques allows the pre-trained automatic speech recognition network to adapt to noisy environments in advance, and its recognition stability in unknown noise scenarios is significantly improved compared to "networks trained only with clean corpus"; training efficiency is optimized. The dual stopping condition of "WER + loss" avoids invalid training iterations, and the number of training iterations is significantly reduced compared to the traditional method of "fixed iterations"; deployment adaptability is enhanced. The pre-trained automatic speech recognition network achieves stable performance with limited corpus and number of iterations, reducing the requirements for training hardware and lowering development costs.

[0051] In one embodiment, the noise reduction module includes: An encoder used to perform dimensionality transformation and preliminary feature extraction on input features, the encoder consisting of several convolutional layers.

[0052] The encoder is the core component in the noise reduction module used to perform "dimensionality compression" and "key feature extraction" on the embedded features of the input. It achieves dimensionality reduction and abstraction of features through convolution operations.

[0053] A convolutional layer is a neural network layer that uses convolution operations (by sliding the convolution kernel to extract local features) to process input features. It is a key unit for realizing feature extraction and dimension transformation.

[0054] Dimension transformation refers to adjusting the dimensions of input features (such as the number of channels or spatial dimension) to a dimension that is more suitable for subsequent modeling. In this embodiment, it mainly refers to "channel compression" (reducing the feature dimension to reduce the amount of computation).

[0055] Preliminary feature extraction refers to extracting key noise and speech features from the input features through convolutional layers, laying the foundation for subsequent "noise separation".

[0056] In the lightweight CRN noise reduction module of Project A, the encoder consists of 3 convolutional layers (the number of parameters is only 1 / 4 of that of a traditional encoder): The first convolutional layer (512 input channels, 256 output channels, kernel size (3,1) - 3 for frequency dimension and 1 for time dimension, stride (1,1), activation function is ReLU) achieves preliminary feature extraction and channel number compression (from 512 to 256); the second and third convolutional layers (both 256 input channels, 128 output channels, kernel size (3,1), stride (1,1), activation function is ReLU, and grouped convolution is used - 8 groups) further compress the number of channels (from 256 to 128) and extract more abstract features. The encoder takes as input the embedded features (with dimensions [batch_size, time_step, 512], where batch_size is the batch size, time_step is the time step, and 512 is the number of channels) from the pre-trained automatic speech recognition network, and outputs as the preliminary features after dimensionality compression (with dimensions [batch_size, time_step, 128]).

[0057] The multi-convolutional layer design achieves "gradual dimensionality reduction and feature abstraction," avoiding the loss of feature information caused by a single dimensionality reduction. The use of grouped convolutions (layers 2 and 3) reduces the encoder's computational load by 7 / 8 compared to traditional convolutions, meeting the "low computational load" requirement of Project A for embedded devices. The temporal dimension convolution kernel size is 1 (only frequency dimension convolution), ensuring the temporal continuity of speech features and avoiding the "temporal feature destruction" problem of traditional convolutions.

[0058] A dual-path recurrent neural network layer used to model bidirectional dependency in the frequency dimension and causal relationship in the time dimension of encoder output features.

[0059] The dual-path recurrent neural network (RNN) layer consists of a "bidirectional RNN sub-layer in the frequency dimension" and a "unidirectional RNN sub-layer in the time dimension," which respectively model the dependence of features in the frequency and time dimensions. It is the core of achieving "accurate noise separation" and "low latency."

[0060] Frequency-dimensional bidirectional dependency modeling refers to learning the sequential dependencies of features in the frequency dimension (such as the correlation between high-frequency noise and low-frequency speech) through bidirectional RNNs (such as bidirectional gated recurrent units (Bi-GRU)) to achieve accurate differentiation between noise and speech features.

[0061] Time-dimensional causal relationship modeling refers to using a unidirectional RNN (such as a unidirectional gated recurrent unit (Uni-GRU)) to model only the features of the "current moment and previous moments", ensuring that the output depends only on historical information and meeting the "causality" requirement of real-time speech recognition (avoiding the delay problem of traditional bidirectional RNNs that "have to wait for the complete speech sequence before outputting").

[0062] Gated Recurrent Unit (GRU) is an improved RNN that solves the problem of "vanishing gradients in long sequences" in traditional RNNs by controlling the transmission and forgetting of information through updating and resetting gates, thus possessing better long sequence modeling capabilities.

[0063] In the lightweight CRN noise reduction module of Project A, the dual-path RNN layer consists of one Bi-GRU sublayer and one Uni-GRU sublayer (the number of parameters is only 1 / 3 of that of a traditional dual-path RNN): 1. Bi-GRU sublayer (hidden layer dimension is 128, bidirectional computation—forward GRU from time step 1 to time step T, reverse GRU from time step T to time step 1), input is the features output by the encoder (dimension [batch_size, time_step, 128]), output is the features modeled in the frequency dimension (dimension [batch_size, time_step, 128]). 1. **[time_step,256] (bidirectional output concatenation)**, used to learn the dependence of noise and speech in the frequency dimension; 2. **Uni-GRU sublayer (hidden layer dimension is 128, only forward computation—from the 1st time step to the Tth time step), input is the output of the Bi-GRU sublayer (dimension [batch_size,time_step,256]), the number of channels is compressed to 128 through linear projection before input, and the output is the features modeled in the time dimension (dimension [batch_size,time_step,128]), used to ensure temporal causality.

[0064] The bidirectional modeling capability of the Bi-GRU sublayer enables the module to accurately distinguish noise and speech features in the frequency dimension (e.g., street noise is mostly concentrated in high frequencies, while speech is mostly concentrated in low frequencies). In Project A, this sublayer improves the noise separation accuracy. The unidirectional modeling of the Uni-GRU sublayer ensures that the module output relies only on historical information, and the latency is controlled within a short time, such as less than 10ms, which fully meets the real-time requirements of "online speech recognition for embedded devices" in Project A. The lightweight design (reasonable control of hidden layer dimensions) reduces the computational load of the dual-path RNN layer compared to traditional designs, making it suitable for low-hardware resources.

[0065] A decoder is used to restore the dimension of the output features of a dual-path recurrent neural network layer and fuse multi-scale features. The decoder is structurally symmetrical with the encoder, and the decoder fuses the multi-scale features output by the encoder through skip connections.

[0066] The decoder, in the noise reduction module, is the component used to restore the low-dimensional features output by the dual-path RNN layer to the same dimension as the input embedded features and to fuse multi-scale features. It is the key to achieving the adaptation of the noise-reduced features to the pre-trained automatic speech recognition network.

[0067] Dimension restoration refers to restoring the number of channels and spatial dimension of low-dimensional features to the dimension of the input embedded features through deconvolution or upsampling operations, ensuring that the output of the noise reduction module can be directly used for subsequent speech recognition.

[0068] Multi-scale feature fusion refers to fusing the features output by different convolutional layers of the encoder (features with different levels of abstraction, such as shallow features containing detailed information and deep features containing semantic information) to improve the completeness of noise separation.

[0069] Skip connection refers to directly connecting the output of a layer of the encoder to the input of the corresponding layer of the decoder, thereby achieving direct fusion of multi-scale features and avoiding the loss of feature information in deep networks.

[0070] Structural symmetry refers to the fact that the number of layers, kernel size, stride, and other parameters of the decoder correspond to those of the encoder, ensuring the symmetry of dimensionality transformation and restoration (e.g., the encoder reduces dimensionality through 3 convolutional layers, and the decoder increases dimensionality through 3 deconvolutional layers).

[0071] In the lightweight CRN noise reduction module of Project A, the decoder consists of three deconvolutional layers (symmetrical to the three convolutional layers of the encoder): The first deconvolutional layer (128 input channels, 256 output channels, kernel size (3,1), stride (1,1), activation function ReLU) takes as input features (dimensionality [batch_size, time_step, 128]) from the output of the dual-path RNN layer and fuses the output (dimensionality [batch_size, time_step, 128]) of the encoder's third convolutional layer through skip connections to achieve feature fusion between shallow and deep layers; the second deconvolutional layer (256 input channels, 256 output channels, kernel size (3,1), stride (1,1), activation function ReLU)... The first convolutional layer (with a kernel size of (3,1), a stride of (1,1), an activation function of ReLU, and 8 groups) is connected to the output of the second convolutional layer of the fusion encoder via a skip connection (dimension [batch_size, time_step, 256]). The third deconvolutional layer (with 256 input channels, 512 output channels, a kernel size of (3,1), a stride of (1,1), and an activation function of ReLU) is connected to the output of the first convolutional layer of the fusion encoder via a skip connection (dimension [batch_size, time_step, 256]). The final output is a denoised embedding feature with a dimension of [batch_size, time_step, 512] (consistent with the dimension of the input embedding feature).

[0072] The symmetrical structural design ensures the accuracy of dimensionality recovery and avoids the problem of "feature distortion after dimensionality recovery" in traditional decoders. In Project A, this design achieves 100% dimensionality matching between the denoised embedded features and the input embedded features. Skip connections enable multi-scale feature fusion. The fused features contain both shallow details from the encoder (such as pitch features of speech) and deep semantic information (such as lexical features of speech), making noise separation more thorough and improving the recognition adaptability of the denoised features. The use of grouped convolution (the second deconvolution layer) significantly reduces the computational cost of the decoder compared to traditional designs, making it suitable for the low computational requirements of embedded devices.

[0073] In this embodiment, the core is to define the three-component architecture of "encoder-dual-path RNN layer-decoder": the encoder realizes feature dimensionality reduction and preliminary extraction, the dual-path RNN layer realizes frequency and time dimension dependency modeling, and the decoder realizes dimension restoration and multi-scale fusion. The three components work together to achieve the noise reduction function of "lightweight, low latency and high noise reduction effect". Specifically, the noise reduction effect is excellent. The three components work together to achieve a complete noise reduction process of "feature dimensionality reduction - dependency modeling - dimensionality restoration - multi-scale fusion". This module improves the signal-to-noise ratio of noisy speech and enhances the recognition adaptability of the denoised speech. It balances lightweight design and low latency. Through grouped convolution, reasonable hidden layer dimensions, and unidirectional RNN design, the number of module parameters is reduced compared to traditional front-end noise reduction modules, reducing computational load and keeping latency within 10ms, making it fully adaptable to low-resource real-time scenarios such as embedded devices and edge computing. It has strong compatibility with automatic speech recognition networks. The decoder dimensionality restoration and structural symmetry design ensure that the output features of the noise reduction module perfectly match the input feature dimensions of the pre-trained automatic speech recognition network, avoiding the problem of "features needing additional transformation after denoising" in traditional modules, thus improving the overall efficiency of the model.

[0074] In one embodiment, in the several convolutional layers of the encoder, the kernel size of the convolutional layer in the time dimension is fixed at 1, and the kernel size of the convolutional layer in the frequency dimension is greater than 1; and except for the first convolutional layer of the encoder, the remaining convolutional layers all use grouped convolution to extract features, so as to reduce computational complexity.

[0075] The kernel size refers to the size of the "sliding window" used to extract local features in the convolution operation. It is usually expressed as (frequency dimension size, time dimension size). In this embodiment, the kernel size is (3,1) (frequency dimension 3, time dimension 1).

[0076] The convolution kernel size in the time dimension is fixed at 1, which means that the window size of the convolution kernel in the time dimension is 1. This means that the convolution operation is only performed on the features of the current time step, without involving the features of the previous and next time steps, thus ensuring the temporal continuity of speech features.

[0077] A frequency dimension convolution kernel size greater than 1 means that the window size of the convolution kernel in the frequency dimension is greater than 1 (3 in this embodiment). That is, convolution operation is performed on the features of the current frequency point and the frequency points before and after it to extract the dependencies in the frequency dimension (such as the frequency distribution difference between noise and speech).

[0078] In the encoder of Project A (3 convolutional layers), the kernel size of all convolutional layers is set to (3,1): 1. Time dimension size is 1: the convolution operation only operates on the features of the current time step (e.g., the features of the t-th time step are only convolved with themselves), avoiding the disruption of temporal features caused by the convolution kernel covering the time steps before and after (e.g., t-1, t+1), and ensuring the temporal continuity of speech (speech is a temporal signal, and the correlation between time steps before and after is extremely strong); 2. Frequency dimension size is 3: the convolution operation operates on the current frequency point (e.g., the f-th frequency point) and one frequency point before and after it (f-1, f+1), extracting local dependencies in the frequency dimension (e.g., street noise is mostly concentrated in f=1000-2000Hz, and speech is mostly concentrated in f=200-800Hz, and the two can be accurately distinguished by convolution of 3 frequency points). The convolution kernel size of 1 in the time dimension ensures that the temporal features of speech are not destroyed. In Project A, this design improves the retention rate of speech temporal information and avoids the "speech semantic ambiguity" problem caused by traditional convolution (time dimension size greater than 1). The convolution kernel size of 3 in the frequency dimension enables the module to accurately extract the frequency dimension dependency relationship, improves the accuracy of distinguishing noise and speech features, and lays a good foundation for the subsequent modeling of dual-path RNN layers. The fixed convolution kernel size (unified as (3,1)) simplifies the model design and reduces the difficulty of development and training.

[0079] Group convolution refers to dividing the input and output channels of a convolutional layer into several groups. Each input channel within a group is convolved only with the output channel of its corresponding group, rather than the traditional convolution method where all input channels are fully connected to all output channels. The calculation formula is as follows: ,in For computational load, Input the number of channels. , (where G is the height and width of the convolution kernel, and B is the number of groups and B is the batch size). It is evident that the larger the number of groups G, the higher the computational cost. The smaller.

[0080] Computational complexity refers to the number of computations (such as the number of multiplications and additions) required during model training and inference. The lower the computational complexity, the less demand the model has on hardware resources (such as CPU and GPU), and the more suitable it is for deployment on embedded devices.

[0081] In the encoder of Project A, except for the first convolutional layer (input channels 512 → output channels 256), the second and third convolutional layers both adopt a grouped convolution design: 1. The number of groups is set to 8 (appropriately selected based on the hardware computing power of the embedded device (e.g., a 4-core CPU) to avoid feature information loss due to excessive grouping); 2. The second convolutional layer (input channels 256 → output channels 128): the input channels 256 are divided into 8 groups (32 channels per group), and the output channels 128 are divided into 8 groups (16 channels per group). Within each group, the 32 input channels are convolved only with the 16 output channels; 3. The third convolutional layer (input channels 256 → output channels 128): the grouping method is the same as the second layer. Calculations using the formula show that the computational cost of traditional convolution (without grouping, G=1) is... (Simplified calculation, ignoring batch size B), the computational cost of grouped convolution (G=8) is... The computational load is reduced by 87.5%. Grouped convolution significantly reduces computational complexity. In Project A, the computational load of the 2nd and 3rd convolutional layers is reduced by 87.5% compared to traditional convolution, resulting in a 70% reduction in the overall computational load of the encoder. This fully adapts to the low-computing power requirements of embedded devices (such as a 4-core CPU that can run in real time). Choosing 8 groups (instead of a larger number such as 16) reduces computational load while avoiding the loss of feature information. In Project A, this number of groups achieves a feature information retention rate of 95%, ensuring the subsequent noise reduction effect. Grouped convolution is only used in the 2nd and 3rd layers (not in the first layer) to avoid insufficient feature extraction in the first layer (the first layer needs to extract basic features, and fully connected convolution is better), thus balancing "reduced computational load" and "feature extraction quality".

[0082] In this embodiment, the core is to clarify the "convolution kernel size" and the "group convolution application rules": by limiting the convolution kernel size of the time dimension to 1, the continuity of speech temporal sequence is ensured; by limiting the convolution kernel size of the frequency dimension to be greater than 1, the frequency feature discrimination is improved; by limiting the use of group convolution except for the first layer, the computational complexity is reduced. The three work together to achieve "both encoder lightweighting and feature extraction quality". Specifically, the integrity of speech features is improved. The convolution kernel size of 1 in the time dimension ensures that temporal features are not destroyed, while the convolution kernel size of 3 in the frequency dimension improves the discriminative power of frequency features. The combination of these two features improves the integrity of the features extracted by the encoder compared to traditional designs, providing a high-quality feature foundation for noise separation in subsequent noise reduction modules. The computational complexity is significantly reduced. The grouped convolution design greatly reduces the computational load of the encoder compared to traditional designs. The inference time of the encoder on embedded devices (4-core CPU) is shortened from the traditional 50ms to 15ms, meeting real-time requirements. Hardware adaptability is enhanced. The lightweight design greatly reduces the encoder's hardware resource requirements, allowing it to be deployed on embedded devices with less than 1GB of memory and less than 8 CPU cores, expanding the application scenarios of the model (such as smartwatches and automotive microcontrollers).

[0083] In one embodiment, the dual-path recurrent neural network layer includes a bidirectional recurrent neural network sublayer and a unidirectional recurrent neural network sublayer; wherein, the bidirectional recurrent neural network sublayer is used to perform bidirectional dependency modeling of the encoder output features in the frequency dimension, and the unidirectional recurrent neural network sublayer is used to perform causal relationship modeling of the features output by the bidirectional recurrent neural network sublayer in the time dimension, so as to ensure low latency and real-time performance of speech recognition.

[0084] Bidirectional recurrent neural network (Bi-RNN) sublayers refer to sublayers composed of forward RNNs and backward RNNs, which model feature dependencies from two directions, "from the start point to the end point" and "from the end point to the start point", respectively. In this embodiment, a bidirectional gated recurrent unit (Bi-GRU) is used, which has better long sequence modeling capabilities.

[0085] Frequency-dimensional bidirectional dependency refers to the relationship between the current frequency point and the previous and next frequency points in the frequency dimension (such as the mutual influence between high-frequency noise and mid-frequency speech, and the relationship between low-frequency speech and second-low-frequency noise). Bidirectional modeling can learn both "forward dependency" (current frequency point and previous frequency point) and "backward dependency" (current frequency point and next frequency point) at the same time.

[0086] In the dual-path RNN layer of Project A, the parameters of the Bi-GRU sublayer are designed as follows: 1. The hidden layer dimension is 128 (reasonably set according to the memory size of the embedded device (e.g., 1GB) to avoid memory overflow due to excessive dimension); 2. The input is the feature output by the encoder (dimension [batch_size, time_step, 128]), which is converted into [batch_size, frequency_step, time_step × 128] (where frequency_step is the frequency step size, which is 64 in this embodiment) through "frequency dimension expansion", so that Bi-GRU can directly model the frequency dimension; 3. The forward GRU models the forward dependency from frequency_step=1 to frequency_step=64, and the backward GRU models the backward dependency from frequency_step=64 to frequency_step=1. The outputs of the two are concatenated to obtain the feature with dimension [batch_size, frequency_step, 256], which is then converted into the feature with dimension [batch_size, time_step, 256] through "frequency dimension compression" and input into the subsequent unidirectional RNN sublayer. The bidirectional modeling capability of the Bi-GRU sublayer enables the module to fully learn the frequency dimension dependencies. In Project A, this sublayer improves the accuracy of distinguishing noise and speech features (such as accurately identifying the mixed feature of "high-frequency noise accompanied by mid-frequency speech"). The 128-dimensional hidden layer design balances "modeling capability" and "memory usage", making the memory usage of the sublayer on embedded devices only 1 / 2 of that of the traditional Bi-GRU (256-dimensional hidden layer). Frequency dimension expansion and compression operations ensure that Bi-GRU can directly model the frequency dimension, avoiding the limitation of traditional RNNs that "can only model the time dimension", thus expanding the modeling dimension.

[0087] A Unidirectional Recurrent Neural Network (Uni-RNN) sublayer refers to a sublayer composed only of forward RNNs that models feature dependencies in a single direction from the start point to the end point of the feature sequence. In this embodiment, a Unidirectional Gated Recurrent Unit (Uni-GRU) is used.

[0088] The temporal causal relationship refers to the relationship in the time dimension where "the output at the current moment depends only on the input at the current moment and the previous moments", that is, y_t=f(x_1,x_2,...,x_t) (where y_t is the output at time t, and x_1-x_t is the input at time 1-t), avoiding dependence on input at future moments (x_{t+1},x_{t+2},...) and ensuring real-time performance.

[0089] Low latency and real-time performance refer to the time interval (latency) between the module receiving input features and outputting processed features being less than the maximum allowable latency of the application scenario (such as online speech recognition, which typically requires a latency of less than 20ms), ensuring smooth user interaction.

[0090] In the dual-path RNN layer of Project A, the parameters of the Uni-GRU sublayer are designed as follows: 1. The hidden layer dimension is 128 (consistent with the Bi-GRU sublayer, to avoid feature information loss due to dimensional abrupt changes); 2. The input is the [batch_size, time_step, 256] features output by the Bi-GRU sublayer, which is compressed to 128 channels through linear projection (weight matrix dimension [256, 128]) to obtain the [batch_size, time_step, 128] input features; 3. Only the forward GRU is used to model the temporal causal relationship from time_step=1 to time_step=T, and the output is the [batch_size, time_step, 128] features (i.e., the final output of the dual-path RNN layer). The latency is calculated through code: the Uni-GRU sublayer processes a single frame of speech features (time step 1) in 2ms, plus the Bi-GRU sublayer's processing time of 8ms, for a total latency of 10ms for the dual-path RNN layer, which is less than the "20ms real-time threshold" of Project A. Uni-GRU sublayer unidirectional modeling ensures causal relationships in the time dimension, with latency controlled within 10ms, fully meeting the real-time requirements of Project A's "online speech recognition for embedded devices," avoiding the latency of over 200ms caused by traditional Bi-RNN's requirement to "wait for a complete speech sequence (e.g., 1 second) before outputting." Linear projection operation matches the input feature dimension with the Uni-GRU hidden layer dimension, avoiding computational errors caused by dimension mismatch. The 128-dimensional hidden layer design results in a sublayer memory footprint of only 50MB, adapting to the low memory requirements of embedded devices.

[0091] In this embodiment, the core is to define the dual structure of "Bi-RNN sub-layer + Uni-RNN sub-layer": the Bi-RNN sub-layer focuses on "bidirectional dependency modeling in the frequency dimension" to improve the accuracy of noise discrimination; the Uni-RNN sub-layer focuses on "causal relationship modeling in the time dimension" to ensure low latency and real-time performance. The two work together to achieve a balance between "high noise reduction effect" and "low latency". Specifically, noise separation accuracy is improved. The bidirectional frequency modeling of the Bi-RNN sublayer significantly improves the accuracy of distinguishing noise and speech features compared to the traditional Uni-RNN. For example, in Project A, this sublayer improved the signal-to-noise ratio of the denoised speech by 15dB, laying the foundation for subsequent recognition rate improvement. Real-time performance is significantly enhanced. The unidirectional time modeling of the Uni-RNN sublayer controls the total latency of the dual-path RNN layer to within 10ms, which is 95% lower than the traditional Bi-RNN (latency of more than 200ms), fully meeting the real-time requirements of online speech recognition. Hardware adaptability is excellent. The hidden layer dimension of both Bi-GRU and Uni-GRU is set to 128, so the memory usage of the dual-path RNN layer is only 100MB, and the computational load is only 1 / 3 of that of the traditional dual-path RNN, which can be deployed on low-configuration embedded devices (such as 4-core CPU, 1GB memory).

[0092] In one embodiment, training the noise reduction module includes: S21: Input the clean speech into the pre-trained automatic speech recognition network and obtain the clean embedding features of the clean speech output by the network.

[0093] Clean speech refers to pure speech signals without any noise components. In this embodiment, it refers to clean speech from the A project corpus (such as noise-free Mandarin command speech "turn on the lights" and "turn off the air conditioner").

[0094] Clean embedding features refer to the intermediate features output before the output classification layer after the clean speech has been processed by the pre-trained automatic speech recognition network. They can represent the core semantic and acoustic information of clean speech and are the "baseline features" for training the noise reduction module.

[0095] In Project A, 1000 clean speech samples (each 2-3 seconds long, 16kHz sampling rate) were selected from the corpus. Each clean speech sample was input into a pre-trained Transformer automatic speech recognition network (with fixed parameters). The intermediate features (dimensions [1, 60-90, 512], where 1 represents batch size=1, 60-90 represent time steps, and 512 represent the number of channels) before the network output classification layer (lexical classification layer) were extracted using code. These features are the clean embedding features. All clean embedding features were stored as binary files as baseline data for subsequent noise reduction module training.

[0096] Clean embedding features serve as "benchmark features," providing a "target direction" for training the denoising module (even if the denoised embedding features are as close as possible to the clean embedding features), avoiding the problem of "denoising direction deviation caused by the lack of a clear benchmark" in traditional denoising modules; selecting 1000 samples ensures the diversity of benchmark data, covering the features of different command speech. For example, in Project A, this sample size improves the representativeness of the benchmark data and avoids "benchmark deviation caused by a single sample."

[0097] S22: Input the noisy speech into the pre-trained automatic speech recognition network, and then input the noisy features output by the network into the noise reduction module to obtain the denoised embedding features output by the noise reduction module.

[0098] Noisy speech refers to speech signals containing noise components. In this embodiment, it is noisy speech from the A project corpus (street noise and office noise are superimposed on clean speech, with a signal-to-noise ratio of 5-15dB).

[0099] Noisy embedding features refer to the intermediate features output before the output classification layer after the pre-trained automatic speech recognition network processes noisy speech. These features contain mixed information of noise and speech.

[0100] Denoising embedding features refer to the features output after noisy embedding features are processed by a denoising module. The goal is to be as close as possible to clean embedding features, that is, to remove noise information from noisy embedding features.

[0101] In Project A, 1000 noisy speech samples (each noisy speech sample has the same content as a clean speech sample, only with different noise superimposed) were selected as noisy speech samples. Each noisy speech sample was input into a pre-trained Transformer automatic speech recognition network (with fixed parameters), and the noisy embedding features (dimensions [1, 60-90, 512]) output by the network were extracted. These noisy embedding features were then input into a lightweight CRN denoising module to be trained (with initial parameters randomly initialized), and the features output by the module became the denoised embedding features (dimensions [1, 60-90, 512]). During training, for each batch (containing 32 noisy speech samples), the corresponding noisy embedding features and denoised embedding features were obtained according to the above process.

[0102] The content of the noisy speech and the clean speech are consistent (only the noise is different), ensuring that the subsequent loss calculation only reflects the "noise removal effect" and avoids the problem of "loss bias caused by content difference"; the parameters of the pre-trained automatic speech recognition network are fixed to ensure that the "feature extraction logic" of the noisy embedding features is consistent with that of the clean embedding features, so that the noise reduction module focuses on "noise removal" rather than "feature extraction logic adaptation"; the noisy and denoised embedding features are acquired in real time to ensure the continuity of the training process. For example, in Project A, this process improved the training iteration efficiency by 20%.

[0103] S23: The denoising module is trained using a combined loss function, which includes embedding space similarity loss and mean squared error loss. The embedding space similarity loss is obtained by calculating the similarity between the clean embedding feature and the denoised embedding feature, and the mean squared error loss is obtained by calculating the magnitude difference between the clean embedding feature and the denoised embedding feature.

[0104] Combined loss function: refers to a loss function composed of a weighted sum of two or more loss functions. In this embodiment, it is... ,in For combined loss, For embedding space similarity loss, For mean square error loss, , As the weighting coefficient, in this embodiment =0.6, =0.4, determined through validation set optimization.

[0105] Embedding space similarity loss This refers to measuring the directional consistency between clean and denoised embedded features by calculating the similarity (cosine similarity in this example) in the embedding space. The calculation formula is as follows: Where N is the number of feature samples, Let i be the clean embedding feature vector. Let i be the denoised embedded feature vector. For vector dot product, (where L2 norm is a vector) The smaller the value, the higher the consistency in direction between the two.

[0106] Mean Squared Error (MSE) Loss This refers to measuring the numerical similarity between clean and denoised embedded features by calculating the magnitude difference between them. The calculation formula is as follows: Where N is the number of feature samples and D is the feature dimension. Let j be the value of the j-th dimension of the i-th clean embedded feature. (where j is the value of the j-th dimension of the i-th denoised embedded feature). The smaller the value, the smaller the difference in amplitude between the two.

[0107] The above weighting coefficients ( , (), refers to the parameter used to balance the importance of different loss functions. The larger the value, the greater the impact of the embedding space similarity loss; The larger the value, the greater the impact of MSE loss.

[0108] In Project A, the above combined loss function is used when training the noise reduction module: 1. Calculate For each batch of clean embedding features (X) and denoised embedding features (Y), calculate the similarity of each sample using the cosine similarity formula, then average the results and subtract 1 to obtain the result. (For example, if the average cosine similarity of a batch is 0.9, then...) =1-0.9=0.1); 2. Calculate For each batch of X and Y, calculate the amplitude difference of each sample using the MSE formula, and then average them to obtain the result. (For example, if the average sum of squares of amplitude differences in a certain batch is 0.05, then...) =0.05); 3. Calculate :according to =0.6、 =0.4 weighted sum, i.e. =0.6×0.1+0.4×0.05=0.06+0.02=0.08. Minimize using the Adam optimizer (learning rate 0.001). The parameters of the noise reduction module are trained iteratively.

[0109] The combined loss function balances "directional consistency" and "amplitude similarity": To address the problem of traditional MSE loss "focusing only on amplitude and ignoring direction" (e.g., in speech features, direction represents semantics, and amplitude represents intensity; directional deviation can lead to recognition errors), as in project A... This improves the directional consistency between denoised embedded features and clean embedded features; Ensure that the amplitudes of the two are close to avoid distortion of feature intensity after denoising; =0.6、 The weight coefficient of 0.4 balances the importance of both factors, improving the convergence stability of the combined loss function compared to the traditional single MSE loss, and avoiding excessive loss fluctuations during training.

[0110] S24: When the combined loss function converges, training is stopped, and the parameters of the pre-trained automatic speech recognition network are fixed throughout the training process.

[0111] The convergence of the combined loss function means that... The value no longer decreases significantly with increasing iteration count, and is usually defined as "five consecutive epochs". "The fluctuation is less than 0.001", which means that the noise reduction capability of the noise reduction module has become stable and further training will not significantly improve the performance.

[0112] The parameters of the pre-trained automatic speech recognition network are fixed throughout the entire training process. This means that all parameters (weights, biases, etc.) of the pre-trained automatic speech recognition network are frozen in the code throughout the training process of the noise reduction module. This ensures that the feature extraction logic remains unchanged and avoids changes in the baseline (clean embedded features) caused by interference with the network parameters.

[0113] In Project A, after each epoch, the validation set (containing 200 noisy speech samples) is calculated. Value. When "5 consecutive epochs" occur. When the fluctuation is less than 0.001, the combined loss function is considered to have converged, and the training of the denoising module is stopped. Throughout the training process, all parameters of the pre-trained Transformer network are frozen using the "model.eval()" function of the PyTorch framework to ensure that it only serves as a feature extractor and does not participate in parameter updates.

[0114] The convergence stopping condition of the combined loss avoids overtraining and undertraining, ensuring that the noise reduction module achieves an optimal balance between "noise reduction effect" and "generalization ability." For example, in project A, this condition ensures that the validation set of the noise reduction module... Stable below 0.05, with excellent noise reduction effect; the pre-trained automatic speech recognition network parameters are fixed throughout the process to ensure that the extraction logic of clean embedding features and noisy embedding features is consistent, avoiding the deviation of the training direction of the noise reduction module caused by the change of the benchmark, solving the problem of mutual interference of network parameters in traditional "joint training", and improving the stability of the training process.

[0115] In this embodiment, the core is to clarify the three-step process of "feature acquisition - combined loss training - convergence and stopping": features are acquired through a dual path of "clean speech → pre-trained automatic speech recognition network → clean embedded features" and "noisy speech → pre-trained automatic speech recognition network → noise reduction module → noise-reduced embedded features"; the module parameters are optimized through a combined loss function of "embedding space similarity loss + MSE loss"; and training stability is ensured through "combined loss convergence + fixed pre-trained automatic speech recognition network parameters". The three work together to achieve "precise training of the noise reduction module". Specifically, the training accuracy of the denoising module is improved. The combined loss function takes into account both direction and magnitude, increasing the similarity between the denoised embedded features and the clean embedded features compared to the traditional MSE loss, thus improving the noise separation accuracy of the denoising module. The training process is more stable. By fixing the parameters of the pre-trained automatic speech recognition network and the convergence stopping condition of the combined loss, the loss fluctuation during training is reduced compared to traditional joint training, and the number of iterations is reduced, significantly improving training efficiency. It has excellent compatibility with the pre-trained automatic speech recognition network. The dual-path feature acquisition process ensures that the denoised embedded features output by the denoising module are completely matched with the input feature dimensions of the pre-trained automatic speech recognition network, avoiding the problem of "features needing additional conversion after denoising" in traditional modules, and improving the overall inference efficiency of the model.

[0116] In one embodiment, retraining the pre-trained automatic speech recognition network includes: S31: Input the noisy speech into the initial speech recognition model, and process it sequentially through the pre-trained automatic speech recognition network and the noise reduction module with fixed parameters to output speech features.

[0117] The initial speech recognition model refers to a model consisting of a pre-trained automatic speech recognition network and a trained noise reduction module, which already has basic noise removal and speech recognition capabilities. In this embodiment, it is the initial model obtained in step 2 of project A.

[0118] Speech features refer to the features output by the noisy speech after processing by the initial model. Specifically, they are the denoised embedding features output by the denoising module (with dimensions consistent with the input features of the pre-trained automatic speech recognition network), which are the core input features for fine-tuning the pre-trained automatic speech recognition network.

[0119] A denoising module with fixed parameters refers to a denoising module whose parameters are frozen in the code to ensure stable denoising capabilities. It only participates in the feature processing of the initial model as a "feature preprocessing component".

[0120] In Project A, 5000 noisy speech samples (covering scenarios such as streets, offices, and homes, with a signal-to-noise ratio of 0-20dB) were selected from the corpus as the fine-tuning training set. Each noisy speech sample was input into the initial speech recognition model and processed according to the following steps: 1. The noisy speech sample was first input into the pre-trained Transformer automatic speech recognition network, which outputs noisy embedding features; 2. The noisy embedding features were input into a CRN denoising module with fixed parameters (all parameters of the module were frozen through code), which outputs denoised embedding features (i.e., speech features, dimensions [1, 60-90, 512]). During the fine-tuning training process, for each iteration of one batch (containing 32 noisy speech samples), the speech features were acquired in real time according to the above process and used as input for fine-tuning the pre-trained automatic speech recognition network.

[0121] A noise reduction module with fixed parameters ensures stable noise reduction capabilities and avoids fluctuations in speech feature quality caused by changes in module parameters during fine-tuning. The noisy speech covers multiple scenarios and a wide signal-to-noise ratio range, enabling the pre-trained automatic speech recognition network to learn feature patterns under different noise environments. For example, in Project A, this training set improves the network's generalization ability in unknown noise scenarios. Real-time acquisition of speech features ensures the continuity of fine-tuning training and avoids the problem of "excessive memory usage caused by offline feature storage" (5000 features require 10GB for offline storage, while real-time acquisition only requires 1GB of memory).

[0122] S32: Based on the aforementioned speech features, train the system using the loss function corresponding to the speech recognition task.

[0123] The loss function for a speech recognition task refers to the function used to measure the difference between the text output of a pre-trained automatic speech recognition network and the actual text output. In this embodiment, the cross-entropy loss function (suitable for word classification tasks) is used, and the calculation formula is as follows: Where N is the number of samples and C is the number of word categories. Let be the true label (0 or 1) of the c-th class for the i-th sample. (where c is the predicted probability of the i-th sample in the c-th category). The smaller the value, the closer the text prediction result of the network is to the actual result.

[0124] Fine-tuning of the pre-trained automatic speech recognition network refers to the process of retraining the network parameters with a small amount of data while keeping the overall structure of the pre-trained automatic speech recognition network unchanged, so that the network can adapt to "new scene features" (in this embodiment, "speech features output by the noise reduction module"). In Project A, when fine-tuning the pre-trained Transformer automatic speech recognition network: 1. Input the obtained speech features (denoised embedding features) into the output classification layer (lexical classification layer) of the pre-trained automatic speech recognition network. The network outputs the predicted probabilities of lexical terms (dimensions [1, 60-90, C], where C=1000 is the number of lexical terms); 2. Calculate the loss value between the predicted probabilities and the true lexical terms using the cross-entropy loss function formula. (For example, if the true label of a sample is "turn on the light" and the predicted label is "turn on / off the light", then...) 3. Use the Adam optimizer (learning rate 0.0001, lower than the initial training rate of 0.001, to avoid large parameter fluctuations) to minimize Only the parameters of the pre-trained automatic speech recognition network are updated (the parameters of the noise reduction module are fixed).

[0125] The cross-entropy loss function directly measures the core metric of speech recognition tasks (text prediction accuracy), allowing the fine-tuning process to focus on "improving recognition rate" and avoiding the problem of "no improvement in recognition rate caused by fine-tuning with MSE loss" in traditional methods. The low learning rate (0.0001) ensures that the basic feature extraction capability of the pre-trained automatic speech recognition network is not destroyed, and it only adapts to "noise-reduced speech features". For example, in Project A, this learning rate keeps the fluctuation of network parameters within 5%, avoiding overfitting. Only the parameters of the pre-trained automatic speech recognition network are updated, which reduces the computational cost of fine-tuning by 50% compared to "full model fine-tuning", meeting the needs of "fine-tuning on low-computing-power servers" in Project A.

[0126] S33: When the loss function converges, or when the word error rate of the initial speech recognition model on the noisy speech validation set no longer decreases, training is stopped, and the final speech recognition model is obtained.

[0127] Loss function convergence refers to the convergence of the cross-entropy loss function. The value no longer decreases significantly with increasing iteration count, defined as "for 5 consecutive epochs". Fluctuation less than 0.001.

[0128] The noisy speech validation set refers to the set of noisy speech used to validate the performance of the fine-tuned network. In this embodiment, it is 200 hours of noisy speech from the A project corpus (different from the fine-tuning training set scenario).

[0129] The final speech recognition model refers to the pre-trained automatic speech recognition network and noise reduction module that have been fine-tuned and have the best fit, and which has a high recognition accuracy in noisy environments.

[0130] In Project A, the validation set is calculated every epoch. Values ​​and WER values. Fine-tuning stops when one of the following two conditions is met: 1. 1. The fluctuation is less than 0.001 for 5 consecutive epochs; 2. The WER of the noisy speech validation set no longer decreases for 5 consecutive epochs.

[0131] The dual stopping condition of "loss convergence + WER no longer decreasing" ensures that the fine-tuned network achieves an optimal balance between "loss value" and "actual recognition rate", avoiding the problem of "loss convergence but no improvement in recognition rate". The noisy validation set is different from the training set, ensuring the generalization ability of the final model. For example, in Project A, this validation set improves the stability of the model's recognition rate in unknown noise scenarios. The final model obtained after stopping fine-tuning has a lower WER in noisy environments than the initial model, which fully meets the requirement of "high recognition rate under embedded device noise" in Project A.

[0132] In this embodiment, the core is to clarify the three-step process of "feature acquisition - loss function fine-tuning - convergence and stopping": fine-tuning input is obtained through "noisy speech → initial model → speech features"; the parameters of the pre-trained automatic speech recognition network are optimized through the "cross-entropy loss function"; and the fine-tuning effect is ensured by "loss convergence or WER no longer decreasing." These three steps work together to achieve "adaptive optimization between the pre-trained automatic speech recognition network and the noise reduction module." Specifically, the recognition rate in noisy environments is significantly improved; the pre-trained automatic speech recognition network and the noise reduction module have optimal adaptability after fine-tuning; fine-tuning efficiency and stability are balanced, with a low learning rate and double stopping conditions reducing parameter fluctuations during fine-tuning and decreasing the number of iterations compared to the traditional "fixed iteration" method, thus significantly shortening the fine-tuning time; and excellent generalization ability is achieved, with multi-scene noisy training sets and independent validation sets ensuring improved recognition stability of the final model in unknown noise scenarios, avoiding the limitation of "excellent performance only in specific noise scenarios" and expanding the model's application scope.

[0133] Reference Figure 2 This invention also provides a training apparatus for a speech recognition model, used to implement the training method for the speech recognition model described in any of the above embodiments, the apparatus comprising: The first training module 10 is used to train the automatic speech recognition network to obtain a pre-trained automatic speech recognition network. The second training module 20 is used to introduce a noise reduction module before the output classification layer of the pre-trained automatic speech recognition network, and to train the noise reduction module based on the embedded features output by the pre-trained automatic speech recognition network to obtain an initial speech recognition model that includes the pre-trained automatic speech recognition network and the noise reduction module; wherein, when training the noise reduction module, the parameters of the pre-trained automatic speech recognition network are fixed. The third training module 30 is used to fix the parameters of the noise reduction module in the initial speech recognition model and retrain the pre-trained automatic speech recognition network to obtain the final speech recognition model.

[0134] Reference Figure 3 The present invention also provides a computer device, the internal structure of which can be as follows: Figure 3 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor in this computer device is designed to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores operating devices, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store speech training samples, etc. The network interface of the computer device is used for communication with external terminals via a network connection. Furthermore, the computer device may also be equipped with an input device and a display screen, etc. When the computer program is executed by the processor, it implements the training method of the speech recognition model of any of the above embodiments. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.

[0135] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the training method of the speech recognition model of any of the above embodiments. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0136] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in this application and in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0137] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0138] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for training a speech recognition model, characterized in that, The method includes: The automatic speech recognition network is trained to obtain a pre-trained automatic speech recognition network; A noise reduction module is introduced before the output classification layer of the pre-trained automatic speech recognition network, and the noise reduction module is trained based on the embedding features output by the pre-trained automatic speech recognition network to obtain an initial speech recognition model that includes the pre-trained automatic speech recognition network and the noise reduction module; wherein, when training the noise reduction module, the parameters of the pre-trained automatic speech recognition network are fixed. By fixing the parameters of the noise reduction module in the initial speech recognition model, the pre-trained automatic speech recognition network is retrained to obtain the final speech recognition model.

2. The training method for the speech recognition model according to claim 1, characterized in that, The process of training the automatic speech recognition network to obtain a pre-trained automatic speech recognition network includes: The automatic speech recognition network is trained using a corpus set that can support the automatic speech recognition network in learning speech feature rules and achieving generalization ability. Data augmentation technology is used during the training process, and the corpus set includes clean speech and noisy speech. When the word error rate of the automatic speech recognition network on the validation set no longer decreases, or when the loss function of the automatic speech recognition network converges, training is stopped, and the pre-trained automatic speech recognition network is obtained.

3. The training method for the speech recognition model according to claim 1, characterized in that, The noise reduction module includes: An encoder used for dimensionality transformation and preliminary feature extraction of input features, the encoder being composed of several convolutional layers; A dual-path recurrent neural network layer used to model bidirectional dependency in the frequency dimension and causal relationship in the time dimension of encoder output features, respectively. A decoder is used to restore the dimension of the output features of a dual-path recurrent neural network layer and fuse multi-scale features. The decoder is structurally symmetrical with the encoder, and the decoder fuses the multi-scale features output by the encoder through skip connections.

4. The training method for the speech recognition model according to claim 3, characterized in that, In the several convolutional layers of the encoder, the kernel size of the convolutional layer in the time dimension is fixed at 1, while the kernel size of the convolutional layer in the frequency dimension is greater than 1. In addition to the first convolutional layer of the encoder, the remaining convolutional layers all use grouped convolution to extract features, so as to reduce computational complexity.

5. The training method for the speech recognition model according to claim 3, characterized in that, The dual-path recurrent neural network layer includes a bidirectional recurrent neural network sublayer and a unidirectional recurrent neural network sublayer; wherein, the bidirectional recurrent neural network sublayer is used to perform bidirectional dependency modeling of the encoder output features in the frequency dimension, and the unidirectional recurrent neural network sublayer is used to perform causal relationship modeling of the features output by the bidirectional recurrent neural network sublayer in the time dimension, so as to ensure low latency and real-time performance of speech recognition.

6. The training method for the speech recognition model according to claim 3, characterized in that, Training the noise reduction module includes: The clean speech is input into the pre-trained automatic speech recognition network to obtain the clean embedding features of the clean speech output by the network. The noisy speech is input into the pre-trained automatic speech recognition network, and then the noisy features output by the network are input into the noise reduction module to obtain the denoised embedding features output by the noise reduction module. The denoising module is trained using a combined loss function, which includes embedding space similarity loss and mean squared error loss. The embedding space similarity loss is obtained by calculating the similarity between the clean embedding feature and the denoised embedding feature, and the mean squared error loss is obtained by calculating the magnitude difference between the clean embedding feature and the denoised embedding feature. Training stops when the combined loss function converges, and the parameters of the pre-trained automatic speech recognition network are fixed throughout the training process.

7. The training method for the speech recognition model according to claim 1, characterized in that, Retraining the pre-trained automatic speech recognition network includes: The noisy speech is input into the initial speech recognition model, and then processed sequentially by the pre-trained automatic speech recognition network and the noise reduction module with fixed parameters to output speech features. Based on the aforementioned speech features, the loss function corresponding to the speech recognition task is used for training; Training stops when the loss function converges, or when the word error rate of the initial speech recognition model on the noisy speech validation set no longer decreases, and the final speech recognition model is obtained.

8. A training device for a speech recognition model, characterized in that, The device includes: The first training module is used to train the automatic speech recognition network to obtain a pre-trained automatic speech recognition network. The second training module is used to introduce a noise reduction module before the output classification layer of the pre-trained automatic speech recognition network, and to train the noise reduction module based on the embedded features output by the pre-trained automatic speech recognition network to obtain an initial speech recognition model that includes the pre-trained automatic speech recognition network and the noise reduction module; wherein, when training the noise reduction module, the parameters of the pre-trained automatic speech recognition network are fixed. The third training module is used to fix the parameters of the noise reduction module in the initial speech recognition model and retrain the pre-trained automatic speech recognition network to obtain the final speech recognition model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the training method for the speech recognition model as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the training method for the speech recognition model as described in any one of claims 1 to 7.

Images