Speech recognition method based on language embedding fusion and hybrid expert-convolution transform
By integrating language embeddings and acoustic features into the MoE-Conformer architecture, dynamically selecting expert subnets, and designing CTC alignment and Attention decoding task branches, the problems of low accuracy and low model efficiency in multilingual speech recognition are solved, achieving efficient and low-cost multilingual speech recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PACHIRA TIMES (ZHUHAI HENGQIN) INFORMATION TECH CO LTD
- Filing Date
- 2026-02-12
- Publication Date
- 2026-05-29
AI Technical Summary
Existing multilingual speech recognition technologies suffer from insufficient accuracy and robustness, difficulty in adapting to language differences, low fault tolerance in mixed scenarios, weak anti-interference in complex environments, low model efficiency, high deployment costs, large number of parameters and computational load, large latency in switching between multiple models, poor language adaptation in low-resource environments, reliance on large amounts of data, limited transfer effects, complex training and optimization, and difficulty in reconciling conflicts between multilingual features.
We adopt a speech recognition method based on the MoE-Conformer architecture, which integrates language embedding and acoustic features. We dynamically select the appropriate expert subnet through a gating network and design two task branches: CTC alignment and Attention decoding. This achieves language-expert mapping and adaptive loss weights, simplifying the training and optimization process. By relying on language embedding transfer adaptation and dedicated expert subnets, we enhance the adaptation of low-resource languages.
It improves the accuracy and robustness of multilingual speech recognition, reduces computational cost and deployment cost, simplifies the training process, enhances the model's generalization ability and low-resource language adaptation ability, achieves a synergy between high recognition accuracy and low computational cost, and improves speech recognition efficiency.
Smart Images
Figure CN122116883A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of automatic speech recognition technology; in particular, it relates to a multilingual and multi-dialect speech processing scheme; specifically, it relates to a speech recognition method based on language embedding fusion and hybrid expert-convolutional transform. Background Technology
[0002] With globalization and the widespread application of speech recognition technology, the demand for multilingual and multi-dialect speech recognition is increasing. Currently, multilingual speech recognition technologies can be mainly divided into the following four categories:
[0003] 1. Multilingual joint modeling: integrates multilingual data with a single model to adapt to randomly switching scenarios;
[0004] 2. Language adaptive design: Fine-tuning the general model with a small amount of data to optimize low-resource language recognition;
[0005] 3. Dynamic language detection + multi-model switching, with fixed alternating languages processed through "detection-switching-recognition";
[0006] 4. Multi-task collaborative modeling, combined with auxiliary tasks to improve scenario robustness.
[0007] However, the existing multilingual speech recognition technologies mentioned above have the following drawbacks:
[0008] 1. Insufficient accuracy and robustness, difficulty in adapting to language differences, low fault tolerance in mixed scenarios, and weak resistance to interference in complex environments;
[0009] 2. The model is inefficient, has high deployment costs, a large number of parameters and computational load, and suffers from long latency when switching between multiple models, as well as high storage consumption.
[0010] 3. Low-resource language compatibility is poor, it relies on a large amount of data, and the migration effect is limited;
[0011] 4. Training and optimization are complex, and conflicts between multilingual features are difficult to reconcile. These multilingual feature conflicts manifest themselves in the following technical shortcomings:
[0012] (1) Acoustic feature conflict: The phoneme system and spectral features of different languages are significantly different. The core acoustic features of tonal languages and stressed languages are easily confused, which leads to a decrease in the accuracy of feature extraction.
[0013] (2) Conflicts in prosodic features: The rules for speech rate, pauses and syllable duration differ among languages, and the unified prosodic modeling module cannot adapt to the prosodic rules of multiple languages;
[0014] (3) Data resource conflict: The imbalance of data volume between high-resource languages and low-resource languages, the excessive dominance of high-resource language features in model training, and the coverage of unique features of low-resource languages, resulting in the degradation of multilingual recognition performance. Summary of the Invention
[0015] Therefore, the purpose of this invention is to propose a speech recognition method based on language embedding fusion and hybrid expert-convolutional transform. Addressing the pain points of low recognition accuracy and high computational cost in current multilingual scenarios, this invention proposes to fuse language embedding information into the MoE-Conformer (hybrid expert-convolutional transform) architecture, integrating language embedding with acoustic features to improve model accuracy and robustness. It employs MoE-Conformer sparse computation and single-model multilingual coverage to improve efficiency and reduce deployment costs. A joint optimization of two task branches—CTC alignment and Attention decoding—simplifies the training and optimization process. Language embedding transfer adaptation and a dedicated expert subnet enhance low-resource language adaptation. Automatic language-expert mapping and adaptive loss weights are implemented to achieve multilingual speech recognition with high accuracy and low computational cost.
[0016] This invention provides a speech recognition method based on language embedding fusion and hybrid expert-convolutional transform, comprising:
[0017] In the feature layer of speech recognition models (such as ASR), pre-trained language embeddings (vector representations that encode a language type) are fused with acoustic features (such as spectrograms or phoneme representations) to construct a unified language-acoustic representation and capture language specificity in advance.
[0018] The model layer of the speech recognition model adopts the MoE-Conformer Encoder architecture, which dynamically selects the expert subnet adapted to the current language through a gating network, eliminating the need for full parameter calculation.
[0019] In the task layer of the speech recognition model, two task branches, CTC alignment and Attention decoding, are designed in parallel. Through backpropagation of multiple losses, including CTC loss and Attention loss, the gating network can automatically learn the language-expert mapping relationship without external language supervision.
[0020] Joint optimization of CTC alignment and Attention decoding tasks, through L CTC L Attn The collaborative backpropagation allows the gating network to automatically learn the "language-expert" mapping, improving the robustness of the speech recognition model to fuzzy speech.
[0021] Gated network parameters are obtained through L total Unified backpropagation allows for the acquisition of implicit language-expert correspondences without the need for additional language-specific supervisory signals.
[0022] Specifically, CTC alignment is a mechanism in Connected Temporal Classification (CTC) algorithms that handles the non-strict alignment between the input sequence and the output label sequence. It allows the model to be trained and inferred without manual fine-tuning. CTC alignment does not require frame-level annotation and automatically learns phoneme-character boundaries.
[0023] The process of CTC alignment task is as follows: acoustic feature sequence X=(x1,...,xT)→conformer→Softmax→posterior distribution P(zt|X), and P(zt|X) is mapped to the target sequence Y through dynamic programming to generate frame-level label distribution.
[0024] The attention-based decoding task computes and decodes the input acoustic feature sequence, gradually generating a text sequence. The decoding in the two branches is as follows:
[0025] Branch A: CTC Decoder (linear mapping), loss L ctc ;
[0026] Branch B: Attention Decoder (Transformer), loss L Attn .
[0027] Furthermore, the expression for the joint multi-loss of the CTC loss and the Attention loss is as follows:
[0028] L total = β L CTC + γ L Attn ;
[0029] Among them, L total It is a joint loss, L CTC It is CTC loss, L Attn β and γ are the Attention loss, respectively, and β and γ are the weights of the CTC loss and the Attention loss.
[0030] With respect to parameter θ, the gradient of backpropagation is:
[0031] ;
[0032] The parameter θ represents the collective term for all trainable parameters in the speech recognition model during the backpropagation process.
[0033] Furthermore, the method for fusing pre-trained language embeddings with acoustic features includes:
[0034] A lightweight neural network (such as Transformer) is used to pre-train a language classification model on multilingual corpora; after training, the feature vector of the penultimate layer is extracted as the language embedding, which represents the language attribute of the current speech.
[0035] A streaming training method is employed to process speech data frame-by-frame or segment-by-segment, updating the parameters of the language classification model to improve its real-time adaptability to language changes. For example, after processing each frame of speech data, the language model parameters are updated based on the language label of that frame. The streaming training method includes: using the Fbank acoustic feature (filter bank feature) extraction method to perform frame-level feature extraction processing on the input multilingual speech signals.
[0036] A gated adaptive fusion strategy is adopted to achieve dynamic adaptation of language embedding and acoustic features.
[0037] Furthermore, the method for dynamically adapting language embedding and acoustic features using a gated adaptive fusion strategy includes:
[0038] The input multilingual speech signals are synchronously processed through a dual-task branch for feature extraction and fusion, generating a low-level feature vector (frame-level joint feature vector) x∈R that combines acoustic representation and language identification. d , where d is the feature dimension after fusion;
[0039] The gated network takes the fused low-level feature vector x as input, and outputs the weight distribution of the N expert subnets of the gated network through linear transformation and Softmax normalization, thereby achieving the selection of suitable experts.
[0040] The linear transformation and Softmax normalization expressions for gated networks are as follows:
[0041] g(x) = Softmax(W g x+b g ),
[0042] ω=[ω1,ω2,...ω i ,ω N ];
[0043] Where g(x) is the output mapping function of the gated network, and g(x) is the normalized weight distribution of N expert subnets by a combination of linear transformation and Softmax normalization of the feature vector x; W g ∈R N×d W g b is the weight matrix of the gated network; g ∈R N b gω is the bias vector of the gating network; ω is the gating weight, and ω is the similarity metric between the input speech features and the preset language feature templates of each expert subnetwork. i Let be the weight of the i-th expert subnet, satisfying ∑ N i=1 ω i =1, the larger the weight value, the higher the compatibility between the expert and the current input language;
[0044] A Top-K sparse selection strategy (preferably K=2) is applied based on gated weights, activating only the top K expert subnets by weight for subsequent feature encoding, while the remaining NK expert subnets are not included in the calculation; the output feature fusion formula of the activated expert subnets is:
[0045] h=∑ i∈Top-2 ω i •E i (x);
[0046] Among them, E i (x) is the encoded output of the i-th activated expert subnet for the input feature x; h is the final encoded feature after fusion, which is input to the subsequent layers of the Conformer encoder (MoE-Conformer Encoder) to complete multilingual speech recognition.
[0047] Specifically, the MoE-Conformer Encoder has a total of L layers, each containing a "multi-head self-attention module + convolutional module + MoE feedforward subnet". The gating network takes the current frame's fused features as input, outputs expert weights, selects the outputs of the Top-K expert subnets, and sums them according to their weights to obtain the final output of the MoE feedforward subnet of that layer.
[0048] This invention utilizes the MoE-Conformer sparse computation mechanism. MoE-Conformer sparse computation is based on the dynamic activation and sparse scheduling of expert subnets with gating mechanism. Combined with the acoustic feature modeling capability of Conformer, in multilingual speech recognition scenarios, only a small number of expert subnets adapted to the current input language are activated to participate in the computation, and the parameter updates and forward propagation of most irrelevant subnets are shielded. Based on the Top-K expert selection strategy, this invention reduces the computational load of feedforward subnets while ensuring feature representation capability, thereby improving the reduction of computing power consumption and model efficiency.
[0049] Furthermore, the method for constructing a unified language-acoustic representation includes:
[0050] Construct a gating unit, input frame-level acoustic features and frame-level language features, calculate the adaptive gating weights through an activation function, and use the adaptive gating weights to dynamically adjust the fusion ratio of the acoustic features and the language features;
[0051] Based on the adaptive gating weights, perform weighted fusion on the frame-level acoustic features and the frame-level language features, and output frame-level fusion features; the gating unit dynamically adjusts the weight allocation according to the semantic relevance of the input features, so that the fusion features adaptively retain the effective information in the acoustic and language dimensions.
[0052] Through gated adaptive fusion rather than simple splicing, the present invention realizes the precise matching of language information and acoustic features, and enhances the sensitivity of the speech recognition model to language differences.
[0053] Further, the method for generating the text sequence includes:
[0054] When training text annotation, add a language label to each character, and use the text with language labels for the training of a speech recognition model (such as ASR), so that the speech recognition model can accurately learn multi-language features.
[0055] Examples of language labels are as follows:
[0056] The Mandarin "How's the weather today" is labeled as "今_c天_c天_c气_c怎_c样_c";
[0057] The Cantonese "How's the weather today" is labeled as "今_y天_y天_y气_y怎_y样_y".
[0058] Further, the method for pre-training a language classification model on a multi-language corpus includes:
[0059] Fine-tune the pre-trained language classification model again, use the training data with language labels, and adjust the parameters of the language classification model according to the actual language labels of the speech signals to optimize the language embedding vectors.
[0060] Further, the method for performing the Top-K sparse selection strategy based on the gating weights includes:
[0061] Based on the gating weights output by the gating network, sort the gating weights ω1, ω2,..., ω N in descending order, and select the expert subnets corresponding to the top K weights as the activation objects.
[0062] The magnitude of the gating weight ω directly reflects the adaptability of the corresponding expert subnetwork to the current input language. The advantage of choosing Top-K instead of a single expert is that it can cover cross-language mixed segments and low-resource language scenarios. By fusing the outputs of two highly adaptable expert subnetworks, it can make up for the insufficient feature modeling of a single expert and improve the robustness of multilingual recognition.
[0063] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the speech recognition method based on language embedding fusion and hybrid expert-convolutional transform as described above.
[0064] The present invention also provides a computer device, the computer device including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the steps of the speech recognition method based on language embedding fusion and hybrid expert-convolutional transform as described above.
[0065] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0066] This invention provides a speech recognition method based on language embedding fusion and hybrid expert-convolutional transform. It integrates language embedding information within the MoE-Conformer (hybrid expert-convolutional transform) architecture, fusing language embedding with acoustic features to improve model accuracy and robustness. Employing MoE-Conformer sparse computation and single-model multilingual coverage improves efficiency, reduces computational load while maintaining model performance, enhances generalization ability, and lowers deployment costs. The method designs two task branches—CTC alignment and Attention decoding—for joint optimization, simplifying the training process and improving efficiency. It enhances low-resource language adaptation by relying on language embedding transfer adaptation and dedicated expert subnets. It achieves automatic language-expert mapping and adaptive loss weights. Through the language embedding fusion mechanism, the model can effectively capture language characteristics, improving multilingual recognition accuracy. Attached Figure Description
[0067] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.
[0068] In the attached diagram:
[0069] Figure 1 This is a schematic diagram of the speech recognition process based on language embedding fusion and hybrid expert-convolutional transform according to an embodiment of the present invention;
[0070] Figure 2This is a schematic diagram of the configuration of a computer device according to an embodiment of the present invention. Detailed Implementation
[0071] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and products consistent with some aspects of this disclosure as detailed in the appended claims.
[0072] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.
[0073] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."
[0074] The embodiments of the present invention will be described in further detail below.
[0075] This invention provides a speech recognition method based on language embedding fusion and hybrid expert-convolutional transform, including:
[0076] In the feature layer of the speech recognition model (ASR), pre-trained language embeddings are fused with acoustic features to construct a unified language-acoustic representation and capture language specificity in advance.
[0077] The method for fusing pre-trained language embeddings with acoustic features includes:
[0078] A lightweight neural network (Transformer) is used to pre-train a language classification model on multilingual corpora; after training, the feature vector of the penultimate layer is extracted as the language embedding, which represents the language attribute of the current speech.
[0079] The method for pre-training language classification models on multilingual corpora includes:
[0080] The pre-trained language classification model is fine-tuned again. Using the training data with language labels, the parameters of the language classification model are adjusted according to the actual language labels of the speech signals, and the language embedding vector is optimized.
[0081] A streaming training method is employed to process speech data frame-by-frame or segment-by-segment, updating the parameters of the language classification model to improve its real-time adaptability to language changes. For example, after processing each frame of speech data, the language model parameters are updated based on the language label of that frame. The streaming training method includes: using the Fbank acoustic feature (filter bank feature) extraction method to perform frame-level feature extraction processing on the input multilingual speech signals.
[0082] A gated adaptive fusion strategy is adopted to achieve dynamic adaptation of language embedding and acoustic features.
[0083] The method for dynamically adapting language embedding and acoustic features using a gated adaptive fusion strategy includes:
[0084] The input multilingual speech signals are synchronously processed through a dual-task branch for feature extraction and fusion, generating a low-level feature vector (frame-level joint feature vector) x∈R that combines acoustic representation and language identification. d , where d is the feature dimension after fusion;
[0085] The gated network takes the fused low-level feature vector x as input, and outputs the weight distribution of the N expert subnets of the gated network through linear transformation and Softmax normalization, thereby achieving the selection of suitable experts.
[0086] The linear transformation and Softmax normalization expressions for gated networks are as follows:
[0087] g(x) = Softmax(W g x+b g ),
[0088] ω=[ω1,ω2,...ω i ,ω N ];
[0089] Where g(x) is the output mapping function of the gated network, and g(x) is the normalized weight distribution of N expert subnets by a combination of linear transformation and Softmax normalization of the feature vector x; W g ∈R N×d W g b is the weight matrix of the gated network; g ∈R N b gω is the bias vector of the gating network; ω is the gating weight, and ω is the similarity metric between the input speech features and the preset language feature templates of each expert subnetwork. i Let be the weight of the i-th expert subnet, satisfying ∑ N i=1 ω i =1, the larger the weight value, the higher the compatibility between the expert and the current input language;
[0090] A Top-K sparse selection strategy is implemented based on gated weights (K=2 in this embodiment). Only the top K expert subnets by weight are activated to participate in subsequent feature encoding, while the remaining NK expert subnets are not included in the calculation. The output feature fusion formula of the activated expert subnets is as follows:
[0091] h=∑ i∈Top-2 ω i •E i (x);
[0092] Among them, E i (x) is the encoded output of the i-th activated expert subnet for the input feature x; h is the final encoded feature after fusion, which is input to the subsequent layers of the Conformer encoder (MoE-Conformer Encoder) to complete multilingual speech recognition.
[0093] The MoE-Conformer Encoder has a total of L layers, each containing a multi-head self-attention module, a convolutional module, and a MoE feedforward subnet. The gating network takes the fused features of the current frame as input, outputs expert weights, selects the outputs of the Top-2 expert subnets, and sums them according to their weights to obtain the final output of the MoE feedforward subnet of that layer.
[0094] This embodiment utilizes the MoE-Conformer sparse computation mechanism. MoE-Conformer sparse computation is based on the dynamic activation and sparse scheduling of expert subnets with gating mechanism. Combined with the acoustic feature modeling capability of Conformer, in multilingual speech recognition scenarios, only a small number of expert subnets adapted to the current input language are activated to participate in the computation, and the parameter updates and forward propagation of most irrelevant subnets are shielded. In this embodiment, based on the Top-2 expert selection strategy, the computation of feedforward subnets is reduced by 75% while ensuring feature representation capability, thereby improving the reduction of computing power consumption and model efficiency.
[0095] The method for executing the Top-K sparse selection strategy based on gated weights includes:
[0096] Based on the gating weights output by the gating network, the gating weights ω1, ω2, ..., ω are... NSort the data in descending order and select the expert subnets corresponding to the top K weights as the activation objects.
[0097] The magnitude of the gating weight ω directly reflects the adaptability of the corresponding expert subnetwork to the current input language. Choosing Top-K instead of a single expert can cover cross-language mixed segments and low-resource language scenarios. By fusing the outputs of two highly adaptable expert subnetworks, the shortcomings of feature modeling by a single expert can be compensated for, thus improving the robustness of multilingual recognition.
[0098] The method for constructing a unified language-acoustic representation includes:
[0099] A gating unit is constructed, and frame-level acoustic features and frame-level language features are input. An adaptive gating weight is calculated through an activation function. The adaptive gating weight is used to dynamically adjust the fusion ratio of acoustic features and language features.
[0100] Based on the adaptive gating weights, frame-level acoustic features and frame-level language features are weighted and fused to output frame-level fused features; the gating unit dynamically adjusts the weight allocation according to the semantic relevance of the input features, so that the fused features adaptively retain effective information in the acoustic and language dimensions.
[0101] This embodiment achieves accurate matching of language information and acoustic features through gated adaptive fusion, rather than simple splicing, thereby enhancing the sensitivity of the speech recognition model to language differences.
[0102] The speech recognition model adopts a MoE-Conformer Encoder architecture at the model layer. It dynamically selects the expert subnetwork adapted to the current language through a gating network, eliminating the need for full parameter calculation. At the task layer of the speech recognition model, two task branches, CTC alignment and Attention decoding, are designed in parallel. Through backpropagation of multiple losses, including CTC loss and Attention loss, the gating network automatically learns the language-expert mapping relationship without external language supervision. [Experts specifically refer to the subnetworks in the MoE (Mixture of Experts) layer, where each expert is an independent feedforward neural network.]
[0103] The expression for the joint loss of CTC loss and Attention loss is as follows:
[0104] L total = β L CTC + γ L Attn ;
[0105] Among them, L total It is a joint loss, L CTC It is CTC loss, L AttnIt is the Attention loss, and β and γ are the weights of the CTC loss and the Attention loss respectively;
[0106] For the parameter θ, the gradient of backpropagation is:
[0107] ;
[0108] The parameter θ represents the collective term of all trainable parameters in the model during the backpropagation process of the speech recognition model.
[0109] This embodiment jointly optimizes the two branch tasks of CTC alignment and Attention decoding. Through the collaborative backpropagation of L CTC and L Attn , the gating network automatically learns the "language-expert" mapping, effectively solving the pain point of the traditional MoE that requires external supervision signals, and at the same time improving the robustness of the speech recognition model to fuzzy speech. The parameters of the gating network are backpropagated uniformly through L total , and the implicit correspondence relationship between languages and experts can be learned without additional language supervision signals.
[0110] CTC alignment is a mechanism for dealing with the non-strict alignment between the input sequence and the output label sequence. CTC alignment does not require frame-level annotation and automatically learns the phoneme-character boundaries. The process of the CTC alignment task is as follows: the acoustic feature sequence X=(x1,...,xT) → conformer → Softmax → posterior distribution P(zt∣X), and P(zt∣X) is mapped to the target sequence Y through dynamic programming to generate the frame-level label distribution.
[0111] The Attention decoding task calculates and decodes the input acoustic feature sequence to gradually generate the text sequence.
[0112] The method for generating the text sequence includes:
[0113] When training the text annotation, add a language label to each character, and use the text with the language label for the training of the speech recognition model (ASR), so that the speech recognition model can accurately learn the multi-language features.
[0114] The language labels include:
[0115] The Mandarin "What's the weather like today" is labeled as "今_c天_c天_c气_c怎_c样_c";
[0116] The Cantonese "What's the weather like today" is labeled as "今_y天_y天_y气_y怎_y样_y".
[0117] Figure 1 Shows the basic process of the speech recognition based on language embedding fusion and hybrid expert-convolution transformation of this embodiment.
[0118] This embodiment presents a speech recognition method based on language embedding fusion and hybrid expert-convolutional transform. It integrates language embedding information within the MoE-Conformer (hybrid expert-convolutional transform) architecture, fusing language embedding with acoustic features to improve model accuracy and robustness. Through the language embedding fusion mechanism, the model can effectively capture language characteristics, enhancing multilingual recognition accuracy. The design of two task branches—CTC alignment and Attention decoding—for joint optimization simplifies the training process and improves training efficiency. Employing MoE-Conformer sparse computation and single-model multilingual coverage reduces computational load while maintaining model performance, enhancing generalization ability and lowering deployment costs. By relying on language embedding transfer adaptation and dedicated expert subnets, it enhances low-resource language adaptation, achieving language-expert mapping and adaptive loss weights, thereby improving speech recognition efficiency.
[0119] This invention also provides a computer device. Figure 2 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention; see the accompanying drawings. Figure 2 As shown, the computer device includes: an input device 23, an output device 24, a memory 22, and a processor 21; the memory 22 is used to store one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the speech recognition method based on language embedding fusion and hybrid expert-convolutional transform as provided in the above embodiments; wherein the input device 23, the output device 24, the memory 22, and the processor 21 can be connected via a bus or other means. Figure 2 Taking the example of a connection between China and Israel via a bus.
[0120] The memory 22, as a read / write storage medium for a computing device, can be used to store software programs and computer-executable programs, such as the program instructions corresponding to the speech recognition method based on language embedding fusion and hybrid expert-convolutional transform described in this embodiment of the invention. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device. Furthermore, the memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 22 may further include memory remotely located relative to the processor 21, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0121] Input device 23 can be used to receive input digital or character information, and generate key signal inputs related to user settings and function control of the device; output device 24 may include display devices such as a display screen.
[0122] The processor 21 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 22, thereby realizing the above-mentioned speech recognition method based on language embedding fusion and hybrid expert-convolutional transform.
[0123] The computer equipment provided above can be used to execute the speech recognition method based on language embedding fusion and hybrid expert-convolutional transform provided in the above embodiments, and has corresponding functions and beneficial effects.
[0124] This invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the speech recognition method based on language embedding fusion and hybrid expert-convolutional transform as provided in the above embodiments. The storage medium can be any type of memory device or storage device, including: mounting media such as CD-ROM, floppy disk, or magnetic tape; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements; the storage medium may also include other types of memory or combinations thereof; furthermore, the storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet); the second computer system can provide program instructions to the first computer for execution. The storage medium includes two or more storage media that can reside in different locations (e.g., in different computer systems connected via a network). The storage medium can store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.
[0125] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the speech recognition method based on language embedding fusion and hybrid expert-convolutional transform as described in the above embodiments, but can also perform related operations in the speech recognition method based on language embedding fusion and hybrid expert-convolutional transform provided in any embodiment of the present invention.
[0126] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.
[0127] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A speech recognition method based on language embedding fusion and hybrid expert-convolutional transform, characterized in that, include: In the feature layer of the speech recognition model, the pre-trained language embeddings are fused with acoustic features to construct a unified language-acoustic representation and capture language specificity in advance. The model layer of the speech recognition model adopts the MoE-Conformer Encoder architecture, and the expert subnet adapted to the current language is dynamically selected through a gating network; In the task layer of the speech recognition model, two task branches, CTC alignment and Attention decoding, are designed in parallel. Through backpropagation of multiple losses, including CTC loss and Attention loss, the gating network can automatically learn the language-expert mapping relationship without external language supervision. The Attention decoding task computes and decodes the input acoustic feature sequence to gradually generate a text sequence.
2. The speech recognition method based on language embedding fusion and hybrid expert-convolutional transform according to claim 1, characterized in that, The expression for the joint loss of CTC loss and Attention loss is as follows: L total = β L CTC + γ L Attn ; Among them, L total It is a joint loss, L CTC It is CTC loss, L Attn β and γ are the Attention loss, respectively, and β and γ are the weights of the CTC loss and the Attention loss. With respect to parameter θ, the gradient of backpropagation is: ； The parameter θ represents the collective term for all trainable parameters in the speech recognition model during the backpropagation process.
3. The speech recognition method based on language embedding fusion and hybrid expert-convolutional transform according to claim 1, characterized in that, The method for fusing pre-trained language embeddings with acoustic features includes: A lightweight neural network is used to pre-train a language classification model on multilingual corpora; after training, the feature vector of the penultimate layer is extracted as the language embedding, which represents the language attribute of the current speech. A streaming training method is used to process speech data frame by frame or segment by segment, update the parameters of the language classification model, and improve the real-time adaptability of the language classification model to language changes; the streaming training method includes: using the Fbank acoustic feature extraction method to perform frame-level feature extraction processing on the input multilingual speech signals; A gated adaptive fusion strategy is adopted to achieve dynamic adaptation of language embedding and acoustic features.
4. The speech recognition method based on language embedding fusion and hybrid expert-convolutional transform according to claim 3, characterized in that, The method for dynamically adapting language embedding and acoustic features using a gated adaptive fusion strategy includes: The input multilingual speech signals are simultaneously processed through a dual-task branch for feature extraction and fusion, generating a low-level feature vector x∈R that combines acoustic representation and language identification. d , where d is the feature dimension after fusion; The gated network takes the fused low-level feature vector x as input, and through linear transformation and Softmax normalization, outputs the weight distribution of the N expert subnets of the gated network to achieve the selection of suitable experts. The linear transformation and Softmax normalization expressions for gated networks are as follows: g(x)=Softmax(W g x+b g ) , ω=[ω1,ω2,...ω i ,oh N ]; Where g(x) is the output mapping function of the gated network, and g(x) is the normalized weight distribution of N expert subnets by a combination of linear transformation and Softmax normalization of the feature vector x; W g ∈R N×d W g b is the weight matrix of the gated network; g ∈R N b g ω is the bias vector of the gating network; ω is the gating weight, and ω is the similarity metric between the input speech features and the preset language feature templates of each expert subnetwork. i Let be the weight of the i-th expert subnet, satisfying ∑ N i=1 ω i =1, the larger the weight value, the higher the compatibility between the expert and the current input language; A Top-K sparse selection strategy is applied based on gated weights, activating only the top K expert subnets by weight for subsequent feature encoding, while the remaining NK expert subnets are not included in the calculation; the output feature fusion formula of the activated expert subnets is: h=∑ i∈Top-2 oh i •E i (x); Among them, E i (x) is the encoded output of the i-th activated expert subnet for the input feature x; h is the final encoded feature after fusion, which is input to the subsequent layers of the Conformer encoder to complete multilingual speech recognition.
5. The speech recognition method based on language embedding fusion and hybrid expert-convolutional transform according to claim 1, characterized in that, The method for constructing a unified language-acoustic representation includes: A gating unit is constructed, and frame-level acoustic features and frame-level language features are input. An adaptive gating weight is calculated through an activation function. The adaptive gating weight is used to dynamically adjust the fusion ratio of acoustic features and language features. Based on the adaptive gating weights, frame-level acoustic features and frame-level language features are weighted and fused to output frame-level fused features; the gating unit dynamically adjusts the weight allocation according to the semantic relevance of the input features, so that the fused features adaptively retain effective information in the acoustic and language dimensions.
6. The speech recognition method based on language embedding fusion and hybrid expert-convolutional transform according to claim 3, characterized in that, The method for generating text sequences includes: When training text annotation, a language label is added to each character. The text with language labels is used to train the speech recognition model, so that the speech recognition model can accurately learn multilingual features.
7. The speech recognition method based on language embedding fusion and hybrid expert-convolutional transform according to claim 6, characterized in that, The method for pre-training language classification models on multilingual corpora includes: The pre-trained language classification model is fine-tuned again. Using the training data with language labels, the parameters of the language classification model are adjusted according to the actual language labels of the speech signals, and the language embedding vector is optimized.
8. The speech recognition method based on language embedding fusion and hybrid expert-convolutional transform according to claim 4, characterized in that, The method for executing the Top-K sparse selection strategy based on gated weights includes: Based on the gating weights output by the gating network, the gating weights ω1, ω2, ..., ω are... N Sort the data in descending order and select the expert subnets corresponding to the top K weights as the activation objects.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the speech recognition method based on language embedding fusion and hybrid expert-convolutional transform as described in any one of claims 1-8.
10. A computer device, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the speech recognition method based on language embedding fusion and hybrid expert-convolutional transform as described in any one of claims 1-8.