Speech recognition method and device, equipment and storage medium

Abstract

The invention relates to the technical field of speech recognition, and discloses a speech recognition method and device, equipment and a storage medium. Aiming at the defects of fixed filter coefficient, static context modeling and difficulty in deep training and deployment of the existing deep feed-forward sequence memory network (DFSMN), the method comprises the following steps of: acquiring a frame-level acoustic feature sequence of input voice, and inputting an acoustic model containing a neural network unit; in the unit, the filter weight of the memory module is generated in real time through a parameter generation network based on the intermediate feature of the current frame; performing weighted aggregation on the historical and future frame features by using the weight to obtain a context enhancement feature; and outputting features based on the feature generation unit, and decoding to obtain an identification result. According to the method, context dynamic self-adaptive modeling is realized, the recognition precision and robustness are improved, the efficient reasoning characteristic of a pure feed-forward network is kept, the calculation overhead is not remarkably increased, and the method is adaptive to a server and terminal equipment and is suitable for multi-scene speech recognition of keywords, command words and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, and particularly relates to a speech recognition method, device, equipment and storage medium. BACKGROUND

[0002] Speech recognition is one of the key technologies of human-computer interaction, and especially in the scenes of intelligent sound boxes, vehicle-mounted systems, wearable devices and the like, fast and accurate recognition of key words or command words has important application value. Acoustic modeling is a core link of a speech recognition system, and its performance directly determines the overall recognition accuracy.

[0003] In recent years, deep neural networks are widely applied to acoustic modeling. Early schemes mostly adopt recurrent neural networks (RNN) and their variants (such as long short-term memory networks LSTM), and this kind of model can effectively capture long-time dependence in speech signals. However, the RNN type model needs to rely on the back-propagation through time (BPTT) algorithm during training, which not only has high computational complexity, but also is easily affected by the gradient vanishing or explosion problem, and is difficult to be efficiently deployed on resource-limited terminal devices.

[0004] In order to overcome the above defects, researchers have proposed a number of sequence modeling methods with pure feedforward structure, among which the feedforward sequential memory network (FSMN) has attracted widespread attention due to its simple and efficient characteristics. FSMN introduces a “memory module” (Memory Block) in the neural network layer, which aggregates the history and future of the input feature sequence in the form of a finite impulse response (FIR) filter, thereby explicitly modeling the context information. On this basis, the deep feedforward sequential memory network (Deep FSMN, DFSMN) further introduces inter-layer skip connection, effectively alleviating the gradient degradation problem caused by deep stacking, and significantly improving the model performance and training stability.

[0005] While DFSMN achieves a good balance between inference efficiency and modeling capability, its memory module has a fundamental limitation: the filter coefficients are completely fixed after model training and cannot be dynamically adjusted according to the input speech content. Specifically, the memory module in DFSMN is typically implemented as a grouped convolutional layer unfolded along the time dimension, whose convolutional kernels (i.e., filters) learn a fixed set of weight parameters through backpropagation during the training phase. Once training is complete, regardless of whether the input is vowels, consonants, silence segments, or speech from different speakers at different speeds, the filter performs weighted fusion of context frames in the exact same way. This static context modeling mechanism ignores the inherent content diversity and time-varying characteristics of speech signals—for example, voiceless consonants (such as s and t) typically have short durations and rapid spectral changes, requiring narrow time windows; while vowels (such as a and o) have longer steady-state segments, suitable for wider context modeling. Using the same fixed set of filters to handle all cases inevitably leads to limited modeling accuracy.

[0006] Furthermore, since the filter structure is immutable, the model cannot dynamically "turn off" the contribution of certain time steps according to the importance of the current speech segment (i.e., it cannot achieve structural sparsity), which further limits its expressive flexibility.

[0007] Therefore, how to maintain the advantages of DFSMN's efficient feedforward structure while endowing it with content-aware, dynamically adaptive context modeling capabilities has become a key technical challenge for improving speech recognition performance, especially the robustness of small-vocabulary command word recognition. Summary of the Invention

[0008] This invention provides a speech recognition method, apparatus, device, and medium, aiming to solve the technical problem that using the same set of fixed filters to handle all situations results in limited modeling accuracy and poor expressive flexibility.

[0009] To achieve the above-mentioned objective, the first aspect of the present invention provides a speech recognition method, the method comprising: Acquire the input speech signal and extract frame-level acoustic feature sequences; The acoustic feature sequence is input into an acoustic model for processing, the acoustic model containing at least one neural network unit with a memory module; In the neural network unit, the filter weights of the memory module are generated in real time through a parameter generation network based on the intermediate features of the current frame. The filter weights are used to perform weighted aggregation of historical and future frames in the acoustic feature sequence to obtain context-enhanced features. Based on the context enhancement features, the output features of the neural network unit are generated; The recognition result of the speech signal is obtained by decoding based on the output features of the neural network unit.

[0010] Furthermore, the parameter generation network is a multilayer perceptron, which includes two parallel output heads.

[0011] Furthermore, the two parallel output heads include a first output head and a second output head, wherein the first output head outputs a filter coefficient vector and the second output head outputs a structure mask vector; The method further includes: multiplying the filter coefficient vector element-wise with the structure mask vector to obtain the filter weights.

[0012] Furthermore, the structure mask vector is obtained by applying the Sigmoid function to the original output of the second output head, and the values ​​of its elements are located in the interval [0, 1].

[0013] Furthermore, in the neural network unit, before performing backbone processing on the current input features, a global linear transformation is performed on the current input features, the global linear transformation employing a trainable scalar scaling parameter and a trainable scalar offset parameter.

[0014] Furthermore, the trainable scalar scaling parameters and trainable scalar offset parameters are fused into the weights and biases of the fully connected layers immediately following the neural network unit during the model inference phase.

[0015] Further, generating the output features of the neural network unit based on the context enhancement features includes: The residuals of the context-enhanced features and the current input features are added together to obtain the residual fusion features; A layer normalization operation is performed on the residual fusion features to obtain the unit output features.

[0016] A second aspect of the present invention provides a speech recognition apparatus for performing the speech recognition method described in any of the preceding claims, comprising: The acquisition unit is used to acquire the input speech signal and extract the frame-level acoustic feature sequence. An input unit is used to input the acoustic feature sequence into an acoustic model for processing, the acoustic model containing at least one neural network unit with a memory module; A computing unit is used in the neural network unit to generate the filter weights of the memory module in real time through a parameter generation network based on the intermediate features of the current frame. An aggregation unit is used to perform weighted aggregation of historical and future frames in the acoustic feature sequence using the filter weights to obtain context-enhanced features. An output unit is used to generate the output features of the neural network unit based on the context enhancement features; The recognition unit is used to decode the speech signal to obtain the recognition result based on the output features of the neural network unit.

[0017] A third aspect of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the speech recognition method described in any of the preceding claims.

[0018] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the speech recognition method described in any of the preceding claims.

[0019] Beneficial effects: The speech recognition method of this invention, through an innovative dynamic context modeling mechanism, achieves the following significant advantages compared to existing speech recognition technologies such as Deep Feedforward Sequence Memory Network (DFSMN): Overcoming the limitations of static modeling, this method achieves context-adaptive modeling: Existing DFSMN memory module filter coefficients remain fixed after training, failing to adapt to the time-varying characteristics of speech signals (such as differences in phonemes, speech rates, and accents). This method uses a parameter generation network to generate filter weights in real-time based on intermediate features of the current frame, enabling the context-dependent mode of the memory module to dynamically adjust with the input speech content. For example, for short-duration, rapidly changing voiceless consonants, short-effective-length filters can be generated; for long-duration, steady-state vowels, long-effective-length filters can be generated. This accurately captures the temporal dependencies of different types of speech segments, significantly improving the model's expressive power and speech recognition accuracy, and effectively reducing the recognition error rate.

[0020] Maintaining efficient inference while balancing performance and deployment requirements: This method adopts the core architecture of a pure feedforward network, with a lightweight parameter generation network (such as a simple multilayer perceptron). It avoids introducing recurrent structures or complex computational modules, thus preventing the high computational complexity caused by traditional recurrent neural networks (such as LSTM) relying on backpropagation over time (BPTT). While achieving dynamic modeling, it does not significantly increase the computational overhead of the inference phase, ensuring efficient model operation. This approach is suitable for large-scale speech recognition scenarios on servers and also meets the deployment requirements of low-latency, low-resource-consumption terminals such as smart speakers and in-vehicle devices.

[0021] Optimizing the training foundation of deep networks and improving model scalability: This method effectively alleviates the gradient vanishing problem in deep network training through a residual fusion mechanism (adding the residual of context-enhanced features to the current input features), providing structural support for building deeper acoustic models. Compared to the limitation of existing DFSMNs in deep stacking, this method can stably train more layers of neural network units, further improving the model's feature extraction ability and generalization performance in complex speech scenarios, enabling the model to maintain high recognition robustness in complex scenarios such as noisy environments and dialect accents.

[0022] The technology is highly versatile and adaptable to various speech recognition scenarios: The core innovation of this method lies in the generation of dynamic filter weights and the context aggregation mechanism. It does not rely on specific feature extraction algorithms or decoding strategies and is compatible with existing mainstream frame-level acoustic features (such as FBank and MFCC) and decoding algorithms (such as CTC and attention decoding). Whether in scenarios with small vocabulary such as keyword recognition and command word recognition, or in continuous speech recognition scenarios with large vocabulary, this method can be used to construct acoustic models, demonstrating broad application scenarios and practical value. Attached Figure Description

[0023] Figure 1 A flowchart illustrating a speech recognition method according to an embodiment of the invention; Figure 2 This is a schematic diagram of the structure of a voice recognition device according to an embodiment of the invention; Figure 3 This is a schematic diagram of a computer device according to an embodiment of the invention. The realization of the object of the invention, its functional features and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any modules and all combinations of one or more associated listed items.

[0026] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0027] Reference Figure 1 This invention provides a speech recognition method, including: S1: Acquire the input speech signal and extract the frame-level acoustic feature sequence.

[0028] Frame-level acoustic feature sequences refer to the set of feature vectors extracted from each frame of a continuous speech signal after dividing it into frames of fixed time length. These vectors characterize the acoustic properties of the speech signal and serve as the core intermediate carrier connecting the original speech with the model processing.

[0029] First, input speech signals from real-world scenarios (such as a user's "play music" command to a smart device) are collected. The signals are preprocessed to reduce noise interference, and then transformed into a frame-level feature sequence that the model can process using a feature extraction algorithm. For example, the speech signal is sampled at a 16kHz sampling rate, with frames of 20ms length and 10ms frame shift. A Hanning window is added to each frame to suppress spectral leakage, and then 40-dimensional filter bank (FBank) features are extracted. If the "play music" speech duration is 0.8 seconds, 80 frames are obtained after framing, forming a frame-level acoustic feature sequence {X1, X2, ..., X...} with dimensions 80×40. 80 Each frame-level acoustic feature is a 40-dimensional feature vector.

[0030] By standardizing preprocessing and feature extraction, redundant noise in the speech signal is removed, while core acoustic information is retained, providing high-quality input for subsequent acoustic modeling and laying the foundation for recognition accuracy.

[0031] S2: Input the acoustic feature sequence into an acoustic model for processing. The acoustic model contains at least one neural network unit with a memory module.

[0032] The acoustic model is the core module in the speech recognition system responsible for mapping acoustic features to semantically relevant representations; the neural network unit refers to the basic computing unit used to model context dependencies in this invention, and its core component is the memory module; the memory module refers to the functional module (essentially a two-dimensional grouped convolutional layer) used to weight and aggregate the features of the speech sequence's history and future frames in order to capture temporal dependencies.

[0033] The acoustic model is constructed by stacking several neural network units, and can be combined with auxiliary layers such as input mapping layers, bottleneck layers, and output mapping layers to form an end-to-end feature processing chain. The core design logic is to overcome the limitations of traditional static modeling by using the memory modules within the neural network units to achieve dynamic context capture. For example, in constructing an acoustic model containing 8 layers of neural network units, the input mapping layer maps 40-dimensional FBank features to 256-dimensional features, each layer of the neural network unit maintains a 256-dimensional output feature dimension, and the output mapping layer transforms the 256-dimensional features into 48-dimensional phoneme probability distribution features. The 80×40 feature sequence obtained in step S1 is input into this model and processed sequentially through each layer of the network.

[0034] The stacked structure inherits the efficient inference advantage of pure feedforward networks and avoids the high computational complexity of recurrent networks. The introduction of memory modules enables the model to explicitly model long-term contextual dependencies, providing structural support for subsequent dynamic weight adjustments and improving the model's adaptability to speech temporal characteristics.

[0035] S3: In the neural network unit, the filter weights of the memory module are generated in real time through the parameter generation network based on the intermediate features of the current frame.

[0036] The intermediate features of the current frame (denoted as p_t) refer to the low-dimensional condensed features obtained after the current frame features are compressed by the input mapping layer and the bottleneck layer, which can accurately represent the core information of the current frame; the parameter generation network refers to the lightweight network used to dynamically output the filter parameters required by the memory module based on the intermediate features; the filter weights refer to the core parameters in the memory module used to perform weighted calculations on the features of the context frames, which determine the degree of contribution of each frame to the features of the current frame.

[0037] Within each neural network unit, the current frame features are first transformed in dimension by an input mapping layer, then compressed into intermediate features p_t by a bottleneck layer (1×1 convolution). Subsequently, p_t is input to a parameter generation network, which outputs filter weights that match the current speech content in real time, replacing the fixed filter coefficients in traditional DFSMN. For example, the bottleneck layer compresses 256-dimensional features into 64-dimensional intermediate features p_t. The parameter generation network uses a lightweight structure with two fully connected layers: the first fully connected layer takes 64-dimensional p_t as input and outputs 32-dimensional features activated by ReLU; the second fully connected layer maps the 32-dimensional features to a 16-dimensional output (corresponding to a maximum filter length L_max=16), which is the dynamic filter weight W_final for the current frame (dimension [16,1,1,1], adapted to grouped convolution format). For the features of frame 40, the W_final_40 generated by the network processing of p_40 differs significantly from the W_final_20 generated by p_20 of frame 20, each adapting to the speech content of the corresponding frame.

[0038] By accurately capturing the current speech characteristics through intermediate features, and generating dynamic weights in real time through the parameter generation network, the context modeling method of the memory module is adaptively adjusted according to the speech content. This completely solves the problem of insufficient adaptability caused by the fixed coefficients of the traditional DFSMN filter and improves the model's ability to adapt to different phonemes and speech rates.

[0039] S4: Use the filter weights to perform weighted aggregation on several historical and future frames in the acoustic feature sequence to obtain context-enhanced features.

[0040] Weighted aggregation refers to the linear weighted summation of features from N1 frames (historical frames) before the current frame and N2 frames (future frames) after the current frame, using filter weights as coefficients; context-enhanced features refer to enhanced features that fuse current frame features with information from surrounding context frames, enabling a more comprehensive representation of the temporal correlation of speech sequences.

[0041] The memory module (two-dimensional grouped convolutional layer) uses a dynamically generated W_final as the convolution kernel to perform convolution operations on the acoustic feature sequence along the time dimension. Essentially, it performs a weighted fusion of features from the current frame's historical N1 frames and the future N2 frames, where N1 + N2 + 1 is the effective length of the filter. For example, if the current frame is frame 40, the effective length of W_final is 10 (N1 = 6, historical frames are frames 34-39; N2 = 3, future frames are frames 41-43), and the 10 coefficients of W_final are [0.15, 0.12, 0.18, 0.22, 0.16, 0.09, 0.07, 0.05, 0.04, 0.02]. These coefficients are then multiplied element-wise with the 256-dimensional feature vectors of frames 34-43 and summed to obtain the context enhancement feature F_40 (256-dimensional) for frame 40. This feature fuses key information from the current frame and the surrounding 10 frames.

[0042] Dynamic filter weights enable different types of speech segments to obtain the optimal context modeling range. Short-duration voiceless consonants can be adapted to a shorter effective length, while long-duration vowels can be adapted to a longer effective length, effectively capturing long-term dependencies in speech and improving the discriminative and expressive power of features.

[0043] S5: Based on the context enhancement features, generate the output features of the neural network unit.

[0044] The output feature (denoted as X^(l+1)_t) refers to the feature vector obtained after residual fusion of the context enhancement features. It is the final output of the current neural network unit and will be passed to the next layer or the output mapping layer.

[0045] To preserve the original input information and optimize gradient propagation, the context-enhanced features are added to the features processed by the input mapping layer in the current frame using an Add operation, directly obtaining the unit output features without additional normalization or transformation operations, thus ensuring computational efficiency. For example, the context-enhanced features F_40 (256 dimensions) of frame 40 are added to the features S_40 (256 dimensions) processed by the input mapping layer using an Add operation, i.e., X^(l+1)_40=F_40+S_40, resulting in the 256-dimensional unit output features. These features contain both context-enhanced information and retain the core characteristics of the original input.

[0046] The residual addition operation effectively alleviates the gradient vanishing problem in deep networks, enabling the model to train deeper network structures; at the same time, it preserves the original input information, avoids information loss during context aggregation, and improves the model training stability and the integrity of feature representation.

[0047] S6: Based on the output features of the neural network unit, the recognition result of the speech signal is obtained by decoding.

[0048] Decoding refers to the process of converting the phoneme probability distribution features output by the acoustic model into natural language text, which is a key link connecting acoustic modeling and the final recognition result.

[0049] The feature sequence output by the last layer of the acoustic model's neural network is transformed into a phoneme probability distribution for each frame via an output mapping layer. This distribution, combined with a language model and dictionary, is then used to search for the optimal text sequence using a decoding algorithm to obtain the speech recognition result. For example, the 80×256 feature sequence output by the 8-layer neural network is transformed into an 80×48 phoneme probability distribution (corresponding to 48 Chinese phonemes) via the output mapping layer. A Connectionist Temporal Classification (CTC) decoding algorithm, combined with a Chinese common word dictionary and an n-gram language model, decodes the probability distribution sequence, ultimately outputting the text "Play music," thus completing the speech recognition.

[0050] The decoding process integrates the precise feature output of the acoustic model with the semantic constraints of the language model, transforming the advantages of dynamic context modeling into improved recognition performance and ensuring the accuracy and fluency of the recognition results.

[0051] The core innovation of this embodiment lies in breaking through the limitations of traditional DFSMN static filters. Through the design of an "intermediate feature-driven parameter generation network," it achieves real-time dynamic adjustment of the filter weights in the memory module. Compared with existing DFSMN speech recognition methods, it has the following key advantages: 1. It achieves dynamic adaptation of context modeling, with filter weights adjusted in real time according to the speech content, significantly improving the model's adaptability to different phonemes, speech rates, and accents, and effectively reducing the recognition error rate; 2. It maintains a pure feedforward network structure, and the parameter generation network adopts a lightweight design, without significantly increasing the inference computation, balancing recognition performance and deployment efficiency; 3. The residual addition operation optimizes the gradient propagation path, supporting the construction of deeper acoustic models and further improving the model's expressive power; 4. The technical solution has strong versatility and can be widely applied to various speech recognition scenarios such as keyword recognition and command word recognition, demonstrating high practical value.

[0052] In one embodiment, the parameter generation network is a multilayer perceptron, which includes two parallel output heads.

[0053] A multilayer perceptron (MLP) is a feedforward neural network composed of multiple fully connected layers. It can learn the mapping relationship between input and output through nonlinear transformation. It has a simple structure and high computational efficiency. Parallel output head means that the last layer of the MLP has two independent fully connected branches, which can output two sets of vectors with different functions at the same time. The calculation process of the two sets of outputs is executed in parallel without adding extra latency.

[0054] This embodiment further clarifies the specific implementation of the parameter generation network—an MLP with two parallel output heads. The core purpose of this design is to provide a foundation for the subsequent dynamic adjustment of weights in a two-dimensional manner (coefficients + mask). The two output heads generate parameters with different functions, and parallel computation ensures that inference efficiency is not affected. For example, using the aforementioned acoustic model structure (8 layers of neural network units, 64-dimensional intermediate features), the optimized MLP structure of the parameter generation network is: input layer (64-dimensional) → first fully connected layer (32 neurons, ReLU activation) → two parallel second fully connected layers (i.e., output heads). Each output head contains 16 neurons (corresponding to a maximum filter length of 16), with no additional activation function, directly outputting a 16-dimensional vector. The two output heads compute in parallel, extracting the required information from the 32-dimensional features output by the first fully connected layer, and simultaneously generating two sets of 16-dimensional outputs. The total computational load increases by only 10% compared to a single-output-head MLP, while the inference latency remains essentially unchanged.

[0055] The nonlinear transformation capability of the multilayer perceptron ensures the accuracy of parameter generation and enables efficient learning of the mapping relationship between intermediate features and filter parameters. The parallel output head design can generate two sets of core parameters simultaneously, providing support for the subsequent dynamic adjustment of the filter's "values" and "structure". Parallel computing does not significantly increase inference overhead, balancing functional scalability and deployment efficiency. The explicit structural design reduces the difficulty of implementing the technical solution, and those skilled in the art can directly implement the parameter generation network based on this structure.

[0056] In one embodiment, the two parallel output heads include a first output head and a second output head, wherein the first output head outputs a filter coefficient vector and the second output head outputs a structure mask vector; the method further includes: multiplying the filter coefficient vector element-wise with the structure mask vector to obtain the filter weights.

[0057] The filter coefficient vector (denoted as C) is a basic vector generated by the first output head, used to characterize the weighting strength of the context frames. Its element values ​​are real numbers, determining the basic contribution of each context frame to the features of the current frame. The structure mask vector (denoted as M) is a vector generated by the second output head, used to dynamically adjust the effective structure of the filter. Its element values ​​are located in a specific range after processing, which can "turn off" or "reduce" the effect of some filter coefficients. Element-wise multiplication refers to the operation of multiplying corresponding elements of two vectors with the same dimension to obtain a new vector of the same dimension. It is the core operation for realizing the fusion of coefficients and masks.

[0058] The first output head provides the context-weighted "basic values," and the second output head provides the "dynamic control signals" for the filter structure. The "basic weights + dynamic structure" are fused through element-wise multiplication to obtain the final filter weights, enabling both numerical and structural adaptation. For example, using the aforementioned MLP structure, the maximum filter length L_max = 16. The filter coefficient vector C = [0.8, 0.6, -0.5, 0.4, 0.3, -0.2, 0.15, 0.1, -0.08, 0.06, 0.04, -0.03, 0.02, 0.01, -0.01, 0.005] output by the first output head has element values ​​distributed in the range of [-1, 1], representing the basic weighting strength of each context frame; the structure mask vector M = [0.93, 0.89, 0.16, 0.82, 0.76, 0.21, 0.68, 0.61, 0.24, 0.14, 0.11, 0.07, 0.04, 0.02, 0.01, 0.008] output by the second output head has element values ​​initially distributed around [0, 1], used to control the activation level of the corresponding coefficients.

[0059] The functional division of the two output heads makes the generation of dynamic weights more logical and targeted. The coefficient vector ensures the basic modeling effect, while the mask vector realizes dynamic structural adjustment. Together, they solve the dual limitations of traditional DFSMN weights, which are "fixed in value and fixed in structure". The clear functional division makes it easier for the model to learn the rules of coefficient generation and mask generation respectively, thereby improving training efficiency and parameter generation accuracy.

[0060] Weight fusion refers to the process of transforming the filter coefficient vector and the structure mask vector into usable filter weights through element-wise multiplication. It is the core link in realizing "numerical-structure" dual adaptation.

[0061] The weight fusion process is performed after the parameter generation network output and before the convolution operation of the memory module. It is part of the feedforward computation and requires no additional iterations or complex calculations. The fused filter weights W_final are directly used as the convolution kernel of the memory module for weighted aggregation of context frames. For example, multiplying C and M element-wise yields the final filter weights W_final=[0.8×0.93,0.6×0.89,-0.5×0.16,0.4×0.82,0.3×0.76,-0.2×0.21,0.15×0.68,0.1×0.61,-0.08×0.24,0.06×0.14,0.04×0.11,-0.03×0.07, 0.02×0.04,0.01×0.02,-0.01×0.01,0.005×0.008], i.e. [0.744,0.534,-0.08,0.328,0.228,-0.042,0.102,0.061,-0.019,0.008,0.004,-0.002,0.001,0.0002,-0.0001,0.00004]. It can be seen that the coefficients at positions 3, 6, and 9 in the original C are almost ineffective after masking, and the effective filter length is reduced from 16 to 10, adapting to the contextual requirements of the current speech segment.

[0062] Element-wise multiplication is simple and efficient, with a computational complexity of O(L_max) (where L_max is the maximum length of the filter), and it adds almost no inference latency. The fused weights have both numerical and structural adaptive capabilities, which can adjust the weighting intensity of each frame according to the speech content, and can also dynamically adjust the effective length and sparsity of the filter, significantly improving the accuracy and flexibility of context modeling. The fusion process is reversible and interpretable, which facilitates model training and debugging.

[0063] This embodiment further clarifies the functions of the two parallel output heads and the weight fusion method, representing a deeper refinement of the aforementioned "dynamically generated filter weights" technical solution. The core logic achieves dual "numerical-structural" adaptation of the filter through a combination of "coefficient vector (numerical basis) + mask vector (structural control) + element-by-element multiplication (fusion)". The coefficient vector matches the weighting requirements of the speech content, while the mask vector dynamically adjusts the effective structure of the filter, completely overcoming the limitations of traditional DFSMN static modeling and significantly improving the model's adaptability and recognition accuracy for different types of speech segments. The weight fusion method is simple and efficient; the element-by-element multiplication operation does not add extra computational burden, ensuring model inference efficiency and balancing performance improvement with deployment requirements. The modular design allows for independent optimization of the three stages: coefficient generation, mask generation, and weight fusion, facilitating subsequent adjustments based on actual scenarios (such as optimizing the activation function for mask generation) and enhancing the scalability of the technical solution.

[0064] In one embodiment, the structure mask vector is obtained by applying a Sigmoid function to the original output of the second output head, and the values ​​of its elements are in the range [0,1].

[0065] The original output of the second output head refers to the real number output vector of the MLP's second output head before it has been processed by the activation function. Its element value range is not explicitly constrained (usually [-∞, +∞]). The Sigmoid function is a non-linear activation function of the form σ(x) = 1 / (1 + e^(-x)). It can map any real number range of inputs to the interval [0, 1]. It is a commonly used normalized activation function in neural networks and has the characteristics of being smooth and differentiable.

[0066] Since the original output of the second output head has no range constraint, it cannot be directly used as a mask vector to achieve stable "enable / disable" control. The normalization effect of the Sigmoid function can fix the element values ​​of the mask vector in the interval [0,1], giving it a clear physical meaning: close to 1 indicates that the corresponding filter coefficient is fully enabled, and close to 0 indicates that the corresponding coefficient is disabled or significantly weakened. For example, using the above MLP structure, the original output vector of the second output head is [2.1,1.7,-1.6,2.0,1.5,-1.3,0.8,0.6,-1.1,0.2,0.05,-1.9,-2.1,-2.6,-3.1,-3.6], with element values ​​distributed in the interval [-3.6,2.1], without a clear range constraint; applying the Sigmoid function to each element, the structure mask vector M=[σ(2.1),σ(1.7), σ(-1.6),...,σ(-3.6)], i.e. [0.89,0.84,0.17,0.88,0.81,0.21,0.69,0.65,0.25,0.55,0.51,0.13,0.11,0.07,0.04,0.03], all elements are strictly located in the interval [0,1], of which the first 5 elements are close to 0.8-0.9 (corresponding coefficients are fully enabled), and the last 4 elements are close to 0.03-0.13 (corresponding coefficients are disabled).

[0067] The normalization effect of the Sigmoid function fixes the range of element values ​​in the structure mask vector, avoiding mask failure caused by fluctuations in the original output (such as excessive enhancement of coefficients due to extreme large values), and improving the stability of model training; the output in the [0,1] interval has a clear semantic interpretation, which makes it easier for the model to learn the mask pattern corresponding to different speech content (such as the mask element values ​​corresponding to vowels being generally large, and the element values ​​corresponding to voiceless consonants being relatively small), making the structure adaptation more targeted; the smooth and differentiable properties of the Sigmoid function ensure stable gradient propagation during training, avoiding gradient vanishing or exploding, and improving the efficiency of model training.

[0068] In one embodiment, in the neural network unit, a global linear transformation is performed on the current input features before backbone processing. The global linear transformation employs a trainable scalar scaling parameter and a trainable scalar offset parameter.

[0069] The backbone processing refers to the core processing flow in a neural network unit from input features to intermediate feature generation (including input mapping, bottleneck layer compression, etc., i.e., the relevant steps of "generating intermediate features" in the above embodiments); global linear transformation refers to performing a uniform linear scaling and translation transformation on all dimensions of the input feature vector, rather than setting parameters for each dimension individually; trainable scalar scaling parameter (denoted as w) refers to a globally shared real-number parameter that can be learned and adjusted through backpropagation, used to scale the input features as a whole; trainable scalar offset parameter (denoted as b) refers to a globally shared real-number parameter that can be learned and adjusted through backpropagation, used to translate the scaled input features as a whole.

[0070] By using single scalar parameters w and b to uniformly adjust all feature dimensions, the network is forced to learn a globally consistent scaling and translation pattern, solving the problem of inconsistent learning pace caused by traditional channel-level vector parameters. Simultaneously, a learnable gating mechanism is provided for information balance between residual branches and backbone branches. For example, using the aforementioned acoustic model structure (8 layers of neural network units, input features are 40-dimensional FBank features, backbone processing includes an input mapping layer and a bottleneck layer), a global linear transformation step is added before the input features enter the input mapping layer. Assuming the scalar scaling parameter w = 0.7 and the scalar offset parameter b = 0.2 in a certain training stage, for the input feature X^l_t (40 dimensions), the transformation result for any dimension i (i = 1, 2, ..., 40) is S_i = w × X^l_t(i) + b. For example, the 8th feature value of X^l_t is 1.3, and after transformation, S_8 = 0.7 × 1.3 + 0.2 = 1.11; the 30th feature value is 0.9, and after transformation, S_30 = 0.7 × 0.9 + 0.2 = 0.83. All 40 features are transformed using the same w and b to ensure consistency in global adjustment. The transformed feature S_t then enters the input mapping layer.

[0071] Global linear transformation avoids the inconsistency in learning pace caused by channel-level vector parameters by using a single scalar parameter, reducing the difficulty of model optimization and playing a role similar to regularization, thus reducing the risk of overfitting. The scalar parameter w can dynamically balance the information contribution of the residual branch (identity mapping) and the backbone processing branch (complex feature transformation). The network can autonomously adjust the value of w according to the needs of different depth layers (smaller w for shallow layers to retain original information, and larger w for deep layers to enhance feature transformation), optimizing the gradient propagation path. There are only 2 scalar parameters, far fewer than channel-level vector parameters (40-dimensional features require 80 parameters), significantly reducing the number of model parameters and improving training efficiency.

[0072] This embodiment addresses the gradient instability and data distribution drift issues encountered during deep training of existing DFSMN models. It balances the information contributions of residual branches and main branches through a global linear transformation, optimizing the gradient propagation path and supporting the construction of deeper, high-performance models. The design of scalar parameters reduces the number of model parameters, lowers training overhead, and simultaneously acts as a regularizer, improving the model's generalization ability and reducing overfitting. A globally consistent linear transformation avoids inconsistent learning paces between channels, reducing the difficulty of model optimization and accelerating training convergence. Only a few computational steps are added (global scalar multiplication and addition), without affecting model inference efficiency, balancing training performance and deployment requirements. In conjunction with a dynamic context modeling scheme, it forms a complete unit structure of "input optimization + dynamic modeling," further improving speech recognition accuracy and model robustness.

[0073] In one embodiment, the trainable scalar scaling parameters and trainable scalar offset parameters are fused into the weights and biases of the fully connected layer immediately following the neural network unit during the model inference phase.

[0074] The model inference phase refers to the process of recognizing new input speech using the trained parameters after the model training is completed. The core goal is to efficiently output the recognition result. The following fully connected layer refers to the first fully connected layer in the neural network unit after the global linear transformation (i.e., the input mapping layer mentioned above, denoted as layer U). Parameter fusion refers to integrating the effects of the scalar scaling parameter w and the offset parameter b into the weight matrix W_U and bias vector b_U of the fully connected layer U through mathematical transformation, so that the global linear transformation does not need to be performed separately during inference.

[0075] During training, the global linear transformation and subsequent fully connected layers are computationally independent steps to ensure effective learning of the scalar parameters w and b. During inference, to eliminate the additional computational overhead of the global linear transformation, w and b are mathematically fused into W_U and b_U. The fused fully connected layer can simultaneously perform the global linear transformation and the original input mapping. During inference, the fused parameters are used directly without altering the network structure or computational accuracy. For example, the weight matrix W_U of the fully connected layer U is 256×40 (40-dimensional input, 256-dimensional output), and the bias vector b_U is 256-dimensional. After training, the scalar scaling parameter w = 0.8, and the scalar bias parameter b = 0.15. The mathematical transformation for parameter fusion is: the fused weight matrix W'_U = w × W_U (i.e., each element of W_U is multiplied by 0.8); the fused bias vector b'_U = w × b_U + b (i.e., each element of b_U is multiplied by 0.8 and then incremented by 0.15). For example, if an element in W_U is 0.6, the corresponding element in W'_U after fusion is 0.6 × 0.8 = 0.48; if an element in b_U is 0.9, the corresponding element in b'_U after fusion is 0.9 × 0.8 + 0.15 = 0.87. During inference, the input feature X^l_t is directly input into the fused fully connected layer, and the calculation result is W'_U × X^l_t + b'_U = 0.8 × (W_U × X^l_t + b_U) + 0.15, which is completely consistent with the calculation result of "global linear transformation + fully connected layer" in the training phase, and there is no need to perform the global linear transformation step separately.

[0076] The parameter fusion process is completed offline before inference, without affecting inference efficiency. After fusion, there is no need to perform a separate global linear transformation during inference, eliminating the computational overhead of this step and achieving the effect of "improving performance during training and zero overhead during inference". The fusion process only involves simple matrix and vector operations, without losing computational accuracy, ensuring that the model's inference performance is completely consistent with that of the training phase. There is no need to modify the inference framework and network structure, only the parameters of the fully connected layers need to be replaced, which is compatible with the existing deployment process and reduces the deployment difficulty.

[0077] This embodiment achieves zero-overhead optimization in the inference stage. By eliminating the extra computational steps of global linear transformation through parameter fusion, it improves the model's inference speed and meets the low-latency requirements of terminal devices. The fusion process does not lose computational accuracy, ensuring that the inference performance is completely consistent with that in the training stage, avoiding a decrease in recognition accuracy due to optimization. It is compatible with existing inference frameworks and deployment processes, requiring no modification to the network structure or inference code, only updating the parameters of the fully connected layers, reducing deployment difficulty and cost. It further improves the technical solution for input preprocessing, making the design of "global linear transformation" more practical and solving the problem of balancing performance and efficiency in existing technologies. In conjunction with dynamic context modeling, deep training optimization, and other solutions, it forms a complete technical solution of "stable training, efficient inference, and accurate recognition," significantly enhancing the practical value and market competitiveness of the model.

[0078] In one embodiment, generating the output features of the neural network unit based on the context enhancement features includes: S51: Add the residuals of the context enhancement features and the current input features to obtain residual fusion features.

[0079] Residual addition refers to the element-wise addition of the context-enhanced features (output of the main processing branch) and the current input features (features processed by the above input mapping layer, denoted as S_t), which is the core operation of the residual network; residual fusion features refer to the feature vector after residual addition, which integrates the context-enhanced information of the main processing branch and the original information of the input features.

[0080] For example, using the acoustic model structure described above, the current input feature S_t after processing by the input mapping layer is 256-dimensional, the context enhancement feature F_t is 256-dimensional, and the residual fusion feature F_fuse_t = F_t + S_t. For instance, the 60th feature value of S_t is 0.8, the 60th feature value of F_t is 0.25, and the 60th feature value of F_fuse_t after adding the residuals is 1.05; the 150th feature value of S_t is 0.4, the 150th feature value of F_t is -0.08, and the 150th feature value of F_fuse_t after adding the residuals is 0.32.

[0081] Residual summation effectively alleviates the vanishing gradient problem in deep networks, enabling the model to train deeper network structures (such as expanding from 8 layers to 12 layers); it integrates the original information of input features, avoiding overfitting of the main processing branches and improving the model's generalization ability; it autonomously adjusts the information flow pattern, allowing the network to flexibly allocate the weights of contextual enhancement information and original information according to learning needs, optimizing the learning process and improving training stability and convergence efficiency.

[0082] S52: Perform layer normalization on the residual fusion features to obtain the unit output features.

[0083] Layer normalization (LayerNorm) refers to the operation of calculating the mean and variance of all dimensions of the feature vector and then normalizing the feature vector to a mean of 0 and a variance of 1. It can effectively eliminate the internal covariate bias in deep networks. Unit output features refer to the feature vector after layer normalization, which is the final feature output by the neural network unit to the next layer.

[0084] Step S52, located after residual summation and before unit output, performs distribution calibration on the residual fusion features. This stabilizes the input feature distribution of each layer, preventing data distribution drift caused by depth stacking, and constrains the dynamic range of the feature vectors, providing support for subsequent quantization deployment. For example, the 256-dimensional feature values ​​of the residual fusion feature F_fuse_t are distributed in the interval [-0.4, 1.7], with a calculated mean μ = 0.5 and variance... Perform layer normalization on each dimension of F_fuse_t: For example, the 90th eigenvalue of F_fuse_t is 1.1, and after normalization, X^(l+1)_t(90) = (1.1 - 0.5) / 0.6 = 1.0; the 220th eigenvalue of F_fuse_t is 0.2, and after normalization, X^(l+1)_t(220) = (0.2 - 0.5) / 0.6 = -0.5. The normalized unit output feature X^(l+1)_t follows a distribution with a mean of 0 and a variance of 1 in all dimensions, and is subsequently input into the next layer of the neural network or the output mapping layer.

[0085] Layer normalization addresses the internal covariate shift problem in deep networks, enabling models to use larger learning rates (e.g., increasing from 0.001 to 0.005) and accelerating training convergence (improving convergence speed by over 30%). Stable feature distribution enhances training stability, supports the construction of deeper networks, and further improves recognition accuracy. Constraining the dynamic range of feature vectors concentrates feature values ​​in the [-2,2] interval, significantly reducing accuracy loss after INT8 quantization and enhancing the model's quantization robustness. Layer normalization computation can be accelerated via hardware instructions without significantly increasing inference overhead.

[0086] The residual addition and layer normalization in this embodiment work together to solve the gradient instability and data distribution drift problems in the deep training of existing DFSMN, improving the training convergence efficiency and stability of the model and supporting the construction of deeper, high-performance networks. Layer normalization constrains the dynamic range of feature vectors, significantly improving the robustness of the model to fixed-point quantization, enabling the model to maintain high recognition accuracy even after INT8 quantization, meeting the deployment requirements of server and terminal devices. Residual fusion preserves the original information of input features, and layer normalization improves the discriminative power of features. The combination of the two further improves the recognition accuracy and generalization ability of the model. The computational complexity is low, as residual addition and layer normalization are both efficient vector operations that can be accelerated by hardware without significantly increasing inference overhead. In conjunction with the dynamic context modeling scheme mentioned above, a complete technical system of "dynamic modeling + stable training + efficient deployment" is formed, significantly improving the overall performance and practical value of the speech recognition model.

[0087] Reference Figure 2The present invention also provides a speech recognition device for performing the speech recognition method described in any of the above embodiments, including: Acquisition unit 10 is used to acquire the input speech signal and extract the frame-level acoustic feature sequence; Input unit 20 is used to input the acoustic feature sequence into an acoustic model for processing, the acoustic model including at least one neural network unit with a memory module; The computing unit 30 is used in the neural network unit to generate the filter weights of the memory module in real time through the parameter generation network based on the intermediate features of the current frame. Aggregation unit 40 is used to perform weighted aggregation of historical and future frames in the acoustic feature sequence using the filter weights to obtain context-enhanced features; Output unit 50 is used to generate the output features of the neural network unit based on the context enhancement features; The recognition unit 60 is used to decode the speech signal to obtain the recognition result based on the output features of the neural network unit.

[0088] Reference Figure 3 The present invention also provides a computer device, the internal structure of which can be as follows: Figure 3 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor is designed to provide computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores operating devices, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores acoustic models and other related data. The network interface is used to communicate with external terminals via a network connection. Furthermore, the computer device may also include an input device and a display screen. When the computer program is executed by the processor, it implements the speech recognition method described in any of the above embodiments. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.

[0089] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the speech recognition method described in any of the above embodiments. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0090] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in this application and in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0091] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0092] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A speech recognition method, characterized in that, The method includes: Acquire the input speech signal and extract frame-level acoustic feature sequences; The acoustic feature sequence is input into an acoustic model for processing, the acoustic model containing at least one neural network unit with a memory module; In the neural network unit, the filter weights of the memory module are generated in real time through a parameter generation network based on the intermediate features of the current frame. The filter weights are used to perform weighted aggregation of historical and future frames in the acoustic feature sequence to obtain context-enhanced features. Based on the context enhancement features, the output features of the neural network unit are generated; The recognition result of the speech signal is obtained by decoding based on the output features of the neural network unit.

2. The speech recognition method according to claim 1, characterized in that, The parameter generation network is a multilayer perceptron, which includes two parallel output heads.

3. The speech recognition method according to claim 2, characterized in that, The two parallel output heads include a first output head and a second output head, wherein the first output head outputs a filter coefficient vector and the second output head outputs a structure mask vector. The method further includes: multiplying the filter coefficient vector element-wise with the structure mask vector to obtain the filter weights.

4. The speech recognition method according to claim 3, characterized in that, The structure mask vector is obtained by applying the Sigmoid function to the original output of the second output head, and the values ​​of its elements are located in the interval [0, 1].

5. The speech recognition method according to claim 1, characterized in that, In the neural network unit, before performing backbone processing on the current input features, a global linear transformation is performed on the current input features. The global linear transformation employs a trainable scalar scaling parameter and a trainable scalar offset parameter.

6. The speech recognition method according to claim 5, characterized in that, The trainable scalar scaling parameters and trainable scalar offset parameters are fused into the weights and biases of the fully connected layers immediately following the neural network unit during the model inference phase.

7. The speech recognition method according to claim 1, characterized in that, The step of generating the output features of the neural network unit based on the context enhancement features includes: The residuals of the context-enhanced features and the current input features are added together to obtain the residual fusion features; A layer normalization operation is performed on the residual fusion features to obtain the unit output features.

8. A speech recognition apparatus for performing the speech recognition method as described in any one of claims 1-7, characterized in that, include: The acquisition unit is used to acquire the input speech signal and extract the frame-level acoustic feature sequence. An input unit is used to input the acoustic feature sequence into an acoustic model for processing, the acoustic model containing at least one neural network unit with a memory module; A computing unit is used in the neural network unit to generate the filter weights of the memory module in real time through a parameter generation network based on the intermediate features of the current frame. An aggregation unit is used to perform weighted aggregation of historical and future frames in the acoustic feature sequence using the filter weights to obtain context-enhanced features. An output unit is used to generate the output features of the neural network unit based on the context enhancement features; The recognition unit is used to decode the speech signal to obtain the recognition result based on the output features of the neural network unit.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the speech recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the speech recognition method as described in any one of claims 1 to 7.

Images