Speech recognition method and apparatus, electronic device, and storage medium

Abstract

The present application relates to the technical field of speech recognition, and provides a speech recognition method and device, electronic equipment and storage medium, wherein the method comprises: performing feature extraction on a speech signal to be recognized, inputting an extracted acoustic feature sequence into a speech recognition model to obtain target recognition text that has been optimized by text; wherein the speech recognition model comprises an encoder and a decoder, and a hybrid expert module for a text optimization task is embedded in the encoder; the encoder is used for performing layer-by-layer encoding processing on the acoustic feature sequence, and performing feature transformation corresponding to the text optimization task on intermediate level features through the hybrid expert module in the encoding process to obtain encoding features containing text optimization semantics; and the decoder is used for decoding the encoding features to obtain the target recognition text, effectively solving the problems of bloated architecture and error accumulation caused by separation of recognition and text optimization in a traditional speech recognition system, and achieving dual improvement of inference efficiency and recognition quality.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and more particularly to a speech recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of deep learning technology, automatic speech recognition technology has become a key interface for human-computer interaction and is widely used in many fields such as intelligent customer service and meeting recording. Current speech recognition models mostly rely on unified neural networks to improve recognition accuracy, achieving significant results in accuracy, but still have obvious shortcomings in terms of presentation.

[0003] To improve text presentation, enhance user reading experience, and meet downstream task requirements, current approaches often employ plug-in or large model cascading strategies, such as linking independent modules or rewriting with a large model. However, these solutions all have drawbacks: cascading solutions suffer from fragmented architecture, error accumulation, and high maintenance costs; large model solutions have high computational costs, large inference latency, and are prone to excessive content rewriting, violating speech-to-text principles and making them unsuitable for practical applications. Summary of the Invention

[0004] This invention provides a speech recognition method, device, electronic device, and storage medium to solve the problems of bloated system architecture, error accumulation, high computational cost, and large inference latency in existing text optimization schemes, which make it difficult to meet the needs of practical applications.

[0005] This invention provides a speech recognition method, comprising:

[0006] Acquire the speech signal to be recognized;

[0007] Feature extraction is performed on the speech signal to obtain an acoustic feature sequence;

[0008] The acoustic feature sequence is input into the speech recognition model to obtain the target recognition text that has been optimized by the speech recognition model.

[0009] The speech recognition model includes an encoder and a decoder, wherein the encoder has a hybrid expert module embedded in it for text optimization tasks.

[0010] The encoder is used to encode the acoustic feature sequence layer by layer, and during the encoding process, the hybrid expert module performs feature transformation on the intermediate level features corresponding to the text optimization task to obtain encoded features containing text optimization semantics; the decoder is used to decode the encoded features to obtain the target recognition text.

[0011] According to a speech recognition method provided by the present invention, there are multiple text optimization tasks, and each text optimization task has a corresponding task expert network in the hybrid expert module;

[0012] The step of inputting the acoustic feature sequence into a speech recognition model to obtain the text-optimized target recognition text output by the speech recognition model includes:

[0013] Obtain the text optimization instructions corresponding to the speech signal;

[0014] The acoustic feature sequence and the text optimization instructions are input into the speech recognition model to obtain the target recognition text that has been optimized by the speech recognition model.

[0015] The encoder is used to determine the activation status of each text optimization task based on the text optimization instructions, control the activation status of the corresponding task expert network in the hybrid expert module based on the activation status, and perform feature transformation on the intermediate-level features through the activated task expert network to obtain the encoded features containing text optimization semantics.

[0016] According to a speech recognition method provided by the present invention, the hybrid expert module further includes multiple recognition expert networks for maintaining basic speech recognition capabilities;

[0017] The encoder is also used to determine the routing weight of each recognition expert network based on the intermediate-level features, determine the target expert network from the plurality of recognition expert networks based on the routing weight, control the activation state of the target expert network, and perform feature transformation on the intermediate-level features through the activated target expert network.

[0018] The encoder is used to fuse the basic feature representation output by the activated target expert network with the optimized feature representation output by the activated task expert network to obtain the encoded features containing text optimization semantics.

[0019] According to a speech recognition method provided by the present invention, the text optimization instruction is a multi-dimensional vector, and each dimension of the multi-dimensional vector corresponds to a text optimization task;

[0020] The encoder is used to parse the multidimensional vector to obtain the value of each dimension. When the value of any dimension is a first preset value, the activation status of the text optimization task corresponding to the any dimension is determined to be enabled, and the activation status of the corresponding task expert network in the hybrid expert module is controlled to be activated.

[0021] The encoder is also used to determine that the activation status of the text optimization task corresponding to any dimension is disabled when the value of any dimension is a second preset value, and to control the activation status of the corresponding task expert network in the hybrid expert module to be dormant, and to skip the parameter calculation of the corresponding task expert network during the encoding process.

[0022] According to a speech recognition method provided by the present invention, the encoder adopts a network architecture with a parameter scale larger than that of the decoder, and the encoder includes a bottom layer network, a middle layer network, and a top layer network; the hybrid expert module is embedded in multiple layers of the top layer network in a distributed manner.

[0023] The bottom-layer network is used to encode the acoustic feature sequence to obtain an acoustic feature representation; the middle-layer network is used to perform language information fusion on the acoustic feature representation to obtain the intermediate-level features containing semantic and syntactic information; the top-layer network is used to perform progressive feature transformation on the intermediate-level features through the hybrid expert modules embedded in the multiple layers to obtain the encoded features containing text-optimized semantics.

[0024] According to a speech recognition method provided by the present invention, the speech recognition model is trained based on the following steps:

[0025] An initial recognition model is determined, which includes an initial encoder and an initial decoder, wherein the initial encoder embeds an initial expert module for the text optimization task;

[0026] Based on the first training sample, the initial recognition model is pre-trained to obtain a basic model; the first training sample includes a first sample speech and a corresponding first sample text; the parameters of the initial expert module in the initial encoder are not updated during the pre-training process;

[0027] Based on the second training samples, the base model is jointly trained to obtain the speech recognition model; the second training samples include second samples corresponding to various text optimization tasks, each second sample including second sample speech, and second text optimization instructions and second sample optimized text corresponding to the second sample speech; during the joint training process, the parameters of the initial expert module in the initial encoder are updated.

[0028] According to a speech recognition method provided by the present invention, the step of jointly training the base model based on a second training sample to obtain the speech recognition model includes:

[0029] According to the initial sampling ratio, target second samples are extracted from the second samples corresponding to various text optimization tasks in the second training samples;

[0030] Based on the target second sample, the base model is jointly trained;

[0031] During the joint training process, the performance of the trained base model is periodically verified to obtain the current convergence index for each text optimization task. Based on the current convergence index, the sampling ratio of each text optimization task in the initial sampling ratio is adjusted.

[0032] Based on the adjusted sampling ratio, new target second samples are extracted from the second samples corresponding to various text optimization tasks in the second training samples.

[0033] Based on the new target second sample, the trained base model is jointly trained to jointly update the parameters of the initial encoder, the initial expert module and the initial decoder, thereby obtaining the speech recognition model.

[0034] The present invention also provides a voice recognition device, comprising:

[0035] Acquisition unit, used to acquire the speech signal to be recognized;

[0036] An extraction unit is used to extract features from the speech signal to obtain an acoustic feature sequence;

[0037] The recognition unit is used to input the acoustic feature sequence into the speech recognition model to obtain the target recognition text that has been optimized by the speech recognition model.

[0038] The speech recognition model includes an encoder and a decoder, wherein the encoder has a hybrid expert module embedded in it for text optimization tasks.

[0039] The encoder is used to encode the acoustic feature sequence layer by layer, and during the encoding process, the hybrid expert module performs feature transformation on the intermediate level features corresponding to the text optimization task to obtain encoded features containing text optimization semantics; the decoder is used to decode the encoded features to obtain the target recognition text.

[0040] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the speech recognition method as described above.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method as described above.

[0042] The speech recognition method, apparatus, electronic device, and storage medium provided by this invention effectively solve the problems of bloated architecture and error accumulation caused by the separation of recognition and text format optimization in traditional speech recognition systems by deeply embedding a hybrid expert module for text optimization tasks into the encoder of the end-to-end speech recognition model. By utilizing the hybrid expert module to perform deep feature transformation on intermediate-level features, the model integrates advanced text optimization semantics such as punctuation, capitalization, number normalization, and removal of colloquialisms during the encoding stage. Thus, without the need for an external post-processing model and without significantly increasing inference latency, it can directly output high-quality, highly readable target recognition text, achieving a dual improvement in inference efficiency and recognition quality, and greatly enhancing the usability of speech recognition technology in practical scenarios such as meeting recording and real-time captioning. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the speech recognition method provided by the present invention;

[0045] Figure 2 This is an overall architecture diagram of the speech recognition method provided by the present invention;

[0046] Figure 3 This is a schematic diagram of the structure of the hybrid expert module provided by the present invention;

[0047] Figure 4 This is a schematic diagram of the structure of the speech recognition device provided by the present invention;

[0048] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0050] With the rapid development of deep learning technology, Automatic Speech Recognition (ASR) technology has become a key interface for human-computer interaction and is widely used in many fields such as intelligent customer service, meeting recording, real-time captioning, medical transcription, and legal document transcription. Current end-to-end speech recognition models, especially those based on autoregressive architectures, directly establish the mapping from speech signals to text sequences through a unified neural network, greatly improving the recognition accuracy.

[0051] However, although existing end-to-end speech recognition technologies perform excellently in "hearing accurately", they still lack in "writing well". That is, the transcribed texts output by traditional end-to-end models often lack a good presentation form, usually manifested as no punctuation marks, chaotic capitalization formats (such as the first letters of proper nouns not being capitalized), all numbers being in Chinese form (such as "two thousand and twenty-three"), and containing a large number of spoken non-fluency phenomena (such as filler words like "um", "ah", etc.). Such raw texts not only seriously affect people's reading experience but also lead to blurred semantic boundaries of long sentences, low information acquisition efficiency, and are difficult to meet the requirements of downstream semantic tasks such as machine translation and text summarization for high-quality input data.

[0052] To obtain transcribed texts that meet specific presentation requirements (such as having punctuation, standard format, and being de-spokenized), existing technologies mainly adopt strategies of external plug-ins or large model cascades. Specifically, the first common solution is to adopt a cascaded post-processing architecture, that is, after the speech recognition model, several independent modules such as a punctuation restoration model, a capitalization restoration model, and a text inverse normalization model are sequentially connected in series; the second solution is to use a Large Language Model (LLM) for rewriting, inputting the original recognized text into the large model, and using its powerful language ability for polishing and format adjustment; the third solution is to try to directly replace the decoder of speech recognition with a large model and use the large model to generate formatted text.

[0053] However, the aforementioned existing technical solutions all have significant drawbacks in practical applications. Specifically, for the first cascaded post-processing solution, its core flaw lies in the fragmented system architecture and error accumulation. Since speech recognition and text optimization are two completely independent stages, errors generated in the basic recognition stage are unconditionally propagated and amplified in subsequent text optimization modules, resulting in severely compromised final output quality. Simultaneously, maintaining multiple independent models leads to bloated system deployment and high maintenance costs. For the second and third solutions that introduce large models, the main problems are excessively high computational costs and large inference latency. The large number of parameters in a large model results in slow inference speeds, often exhibiting linear or even superlinear time growth, making it difficult to apply to scenarios with extremely high real-time requirements, such as voice assistants and real-time captions. Furthermore, the uncertainty generated by large models can easily lead to excessive rewriting of the original speech content, violating the fundamental principle that speech transcription must remain faithful to the original meaning.

[0054] To address these issues, this invention provides a speech recognition method that aims to solve the problems of bloated system architecture and error accumulation caused by the use of cascaded architectures in existing technologies, as well as the problems of high computational resource consumption and high inference latency caused by reliance on large-scale language models. This method achieves deep and efficient integration of speech recognition with multi-dimensional text optimization functions. In this embodiment, a hybrid expert module for text optimization tasks is embedded in the encoder layer of the end-to-end speech recognition model. This allows for feature transformation and fusion of acoustic features and text optimization semantics during the feature encoding stage, thereby directly outputting the optimized target recognition text while maintaining a simple model architecture and efficient inference.

[0055] Figure 1 This is a flowchart illustrating the speech recognition method provided by this invention. This method achieves integrated transcription from speech to high-quality, standardized text by deeply embedding a hybrid expert module with text optimization capabilities into the encoder of an end-to-end architecture. The execution entity of this method can be various devices with data processing capabilities, such as smartphones, in-vehicle terminals, smart conferencing systems, cloud servers, etc. Figure 1 As shown, the method includes:

[0056] Step 110: Obtain the speech signal to be recognized;

[0057] Step 120: Extract features from the speech signal to obtain an acoustic feature sequence;

[0058] Step 130: Input the acoustic feature sequence into the speech recognition model to obtain the target recognition text that has been optimized by the speech recognition model.

[0059] The speech recognition model includes an encoder and a decoder, with the encoder embedding a hybrid expert module for text optimization tasks.

[0060] The encoder is used to encode the acoustic feature sequence layer by layer. During the encoding process, the hybrid expert module performs feature transformation on the intermediate level features corresponding to the text optimization task to obtain the encoded features containing text optimization semantics. The decoder is used to decode the encoded features to obtain the target recognition text.

[0061] Specifically, before performing speech recognition, it is first necessary to determine the speech signal to be recognized, which is the audio data stream carrying the user's language information. In practical applications, the source of this signal is widely adaptable. It can be an audio data stream recorded in real time by the user through a microphone array during a meeting or conversation, or an audio file, such as a recording document in WAV or MP3 format, pre-stored in local storage or a cloud database. Furthermore, to adapt to recognition needs in different environments, after acquiring the raw speech signal, it can be preprocessed, such as removing background interference through echo cancellation and noise reduction algorithms, and extracting valid speech segments through speech activity detection, thereby obtaining the speech signal to be recognized. It should be noted that regardless of the language (e.g., Chinese, English), accent, or duration of the speech signal to be recognized, it can be used as an input signal for subsequent speech recognition.

[0062] After obtaining the speech signal to be identified, in this embodiment of the invention, feature extraction can be performed on the speech signal, that is, the acoustic features of the speech signal can be extracted to obtain an acoustic feature sequence.

[0063] Here, the purpose of acoustic feature extraction of speech signals is to convert the speech waveform in the time domain into frequency domain feature vectors that the model can understand. Specifically, acoustic feature extraction algorithms, such as those for extracting F-bank (Filter Bank) features, MFCC (Mel-frequency cepstral coefficients) features, and spectrogram features, can be used to perform framing, windowing, and Fourier transform processing on the speech signal. After this series of transformations, the continuous speech signal is converted into a vector sequence arranged according to time steps, i.e., the acoustic feature sequence. This sequence, while preserving the dynamic changes in the temporal sequence of speech, characterizes the acoustic texture of speech through frequency domain energy distribution.

[0064] After this, the extracted acoustic feature sequence can be input into the speech recognition model to perform speech recognition, thereby obtaining the target recognition text output by the model after text optimization. The speech recognition model here is a specially designed end-to-end deep learning network, which mainly consists of an encoder and a decoder; the encoder embeds a hybrid expert module for text optimization tasks, through which multi-dimensional text optimization can be achieved, and finally the optimized target recognition text can be obtained.

[0065] In detail, the processing of acoustic feature sequences within a speech recognition model can be divided into the following two stages:

[0066] Figure 2 This is an overall architecture diagram of the speech recognition method provided by the present invention, as shown below. Figure 2 As shown, during the encoding stage, the acoustic feature sequence can be encoded layer by layer using an encoder. This encoder can employ a multi-layer Conformer structure, which has a large parameter scale. Specifically, the acoustic feature sequence first flows through the bottom and middle layers of the encoder, extracting intermediate-level features representing semantic and syntactic information through each layer. Subsequently, when these intermediate-level features are passed to the high / top layers of the encoder, they enter the Mixture of Experts (MoE) module because the top layer embeds a MoE module. For example, in the high-level part of the encoder, the feedforward network portion of each layer can be replaced with a MoE structure. This module integrates various expert networks for different text optimization tasks, such as punctuation restoration, case restoration, text inverse normalization, and non-fluency removal. These expert networks have the same network structure (e.g., standard feedforward neural networks) and a similar parameter scale, allowing them to work collaboratively in the same feature space.

[0067] During the encoding process, the hybrid expert module performs feature transformations on these intermediate-level features for the corresponding task, injecting optimized semantics such as text formatting specifications and syntactic logic into the features that originally only contained acoustic information. After this transformation, the encoder's final output is no longer a simple acoustic representation, but an encoded feature that integrates rich information such as "a pause and comma are needed here," "this proper noun should be capitalized," and "colloquial words should be removed here," i.e., encoded features containing optimized text semantics.

[0068] In the decoding stage, a decoder is used to decode the encoded features containing text optimization semantics to obtain the target recognition text. This decoder can be a lightweight Transformer decoder, such as an Attention-based Transformer structure, which has a small parameter size. Specifically, the decoder can utilize a cross-attention mechanism to focus on the encoded features containing rich text optimization semantics. Since the encoded features implicitly include the planning of the text presentation format, the decoder, in the process of autoregressively generating tokens and mapping them to text, can directly output results conforming to written language norms, i.e., the target recognition text. This not only accurately restores the content of the speech signal but also automatically completes optimization operations such as adding punctuation, standardizing number formats, and removing colloquialisms, directly meeting the user's reading and usage needs.

[0069] The speech recognition method provided by this invention effectively solves the problems of bloated architecture and error accumulation caused by the separation of recognition and text format optimization in traditional speech recognition systems by deeply embedding a hybrid expert module for text optimization tasks into the encoder of the end-to-end speech recognition model. By utilizing the hybrid expert module to perform deep feature transformation on intermediate-level features, the model integrates advanced text optimization semantics such as punctuation, capitalization, number normalization, and removal of colloquialisms during the encoding stage. Thus, without the need for an external post-processing model and without significantly increasing inference latency, it can directly output high-quality, highly readable target recognition text, achieving a dual improvement in inference efficiency and recognition quality, and greatly enhancing the usability of speech recognition technology in practical scenarios such as meeting recording and real-time captioning.

[0070] Based on the above embodiments, there are various text optimization tasks, and each text optimization task has a corresponding task expert network in the hybrid expert module.

[0071] The acoustic feature sequence is input into the speech recognition model to obtain the target recognition text that has been optimized by the speech recognition model, including:

[0072] Obtain the text optimization instructions corresponding to the speech signal;

[0073] The acoustic feature sequence and text optimization instructions are input into the speech recognition model to obtain the target recognition text that has been optimized by the speech recognition model.

[0074] The encoder is used to determine the activation status of each text optimization task based on the text optimization instructions. Based on the activation status, it controls the activation status of the corresponding task expert network in the hybrid expert module. The activated task expert network performs feature transformation on the intermediate-level features to obtain encoded features containing text optimization semantics.

[0075] Specifically, in practical applications, users' text optimization needs are often diverse and dynamically changing. For example, in instant messaging scenarios, users may want to retain interjections in spoken language to convey emotion; while in formal document writing scenarios, it is necessary to strictly remove colloquialisms and standardize numerical expressions. To meet these differentiated needs, in this embodiment of the invention, the text optimization task is not a single-dimensional processing, but rather multiple, covering several specific optimization directions. Correspondingly, the hybrid expert module embedded in the encoder is not a general whole, but includes task expert networks corresponding to various text optimization tasks. Specifically, for each specific text optimization task such as punctuation restoration, case restoration, text denormalization, and non-fluency removal, the hybrid expert module deploys at least one corresponding task expert network, such as a punctuation restoration expert network, a case restoration expert network, a text denormalization expert network, and a non-fluency removal expert network. These expert networks each perform their own functions and possess different dimensions of text optimization capabilities.

[0076] It should be noted here that the expert networks for each task in the hybrid expert module adopt standard feedforward neural networks, consistent with the structure of conventional feedforward neural networks in the Conformer architecture, which include linear transformation layers and activation functions. Furthermore, the parameters of each task expert network can be pre-trained independently, enabling them to learn specialized feature transformation patterns for specific text optimization functions. For example, the punctuation restoration expert network learns the prosodic and semantic feature patterns required to insert punctuation marks in appropriate positions; the case restoration expert network focuses on linguistic phenomena requiring capitalization, such as proper nouns and sentence beginnings; the text denormalization expert network processes the feature transformation rules from colloquial expressions such as numbers, dates, and times to standard written forms; and the non-fluency removal expert network identifies the acoustic and linguistic features of non-fluency phenomena such as filler words, repetitions, and self-corrections.

[0077] However, despite their different text optimization functions, the task experts share the same network structure (such as standard feedforward neural networks) and a similar parameter scale. This unified design not only simplifies the model implementation and training process, but also enables the task expert networks to work in coordination within the same feature space.

[0078] Based on this, in this embodiment of the invention, a key control dimension is added during the process of inputting the acoustic feature sequence into the speech recognition model for processing. The specific process is as follows:

[0079] First, it is necessary to obtain the text optimization instructions corresponding to the speech signal. These instructions are a direct digital representation of the user's intent, and can originate from user-selected settings on the client interface, such as selecting "Enable punctuation restoration" or "Disable inverse text normalization," or from preset configuration parameters based on the application scenario. Here, the text optimization instructions explicitly indicate which dimensions of text optimization processing the user currently desires for the speech signal to be recognized.

[0080] Subsequently, the acoustic feature sequence and text optimization instructions can be input into the speech recognition model. Unlike traditional models that only receive acoustic features, in this embodiment of the invention, the speech recognition model has multimodal input processing capabilities, enabling it to simultaneously parse acoustic signals and control instructions.

[0081] Here, after the acoustic feature sequence and text optimization instructions are input into the speech recognition model, the encoder, as the main control and processing center, executes fine-grained control logic based on the instructions. Specifically, when data flows to the top-level network embedded with the hybrid expert module, the encoder first parses the input text optimization instructions to determine the activation status of each text optimization task. This activation status reflects the business logic requirement, i.e., whether the user wants the function. Next, the encoder maps this logical activation status to physical computational control signals, controlling the activation status of the corresponding task expert network in the hybrid expert module according to the activation status. This control mechanism embodies the idea of ​​on-demand computation; if the instruction indicates that the user needs a certain text optimization function, such as "enable punctuation restoration," the encoder sets the corresponding task expert network, such as the punctuation restoration expert network, to an active state, allowing data to flow through that network for computation. Conversely, if the instruction indicates that a certain text optimization function is not needed, such as "disable non-fluency removal," the corresponding task expert network, such as the non-fluency removal expert network, will be set to a dormant or inactive state, skipping computation. Ultimately, the activated task expert network will perform feature transformation on the intermediate-level features. That is, only the activated task expert network will integrate the optimized semantics it has mastered (such as punctuation position information) into the features, thereby obtaining encoded features containing optimized text semantics. This means that the generated encoded features are tailored to the user's instructions, neither redundant nor missing.

[0082] In this embodiment of the invention, a one-to-one correspondence control mechanism between text optimization instructions and task expert networks is introduced, significantly improving the flexibility and computational efficiency of speech recognition. On the one hand, this explicit instruction control gives users absolute control over the output text format, enabling a single model to handle multiple application scenarios and solving the problem of unstable output from large models. On the other hand, by transforming the enabled state at the business level into the active state at the network level, true on-demand computing is achieved. When users do not need certain text optimization functions, the corresponding task expert network automatically goes into sleep mode, thereby significantly saving the model's inference computing power and power consumption, and avoiding the resource waste caused by enabling all functions.

[0083] Based on the above embodiments, the hybrid expert module also includes multiple recognition expert networks for maintaining basic speech recognition capabilities;

[0084] The encoder is also used to determine the routing weight of each recognition expert network based on intermediate-level features, determine the target expert network from multiple recognition expert networks based on the routing weight, control the activation state of the target expert network, and perform feature transformation on the intermediate-level features through the activated target expert network.

[0085] The encoder is used to fuse the basic feature representation output by the activated target expert network with the optimized feature representation output by the activated task expert network to obtain encoded features that contain optimized text semantics.

[0086] Specifically, within the hybrid expert module, in addition to the task expert network that is explicitly controlled by instructions, there is another crucial type of network: multiple recognition expert networks used to improve the model's basic speech recognition capabilities.

[0087] Understandably, the task expert network is responsible for text optimization, while the recognition expert network is responsible for basic speech recognition, ensuring accuracy. In other words, these recognition expert networks focus on capturing acoustic details, phoneme variations, and basic linguistic features in the speech signal, aiming to provide the decoder with the most robust recognition basis.

[0088] Figure 3 This is a schematic diagram of the structure of the hybrid expert module provided by the present invention, as shown below. Figure 3As shown, unlike task expert networks which obey external instructions, recognition expert networks operate in a more intelligent and dynamic manner. This embodiment of the invention employs a dynamic routing mechanism based on data content to manage these recognition expert networks. Specifically, when intermediate-level features reach the hybrid expert module, the routing weights of each recognition expert network can be calculated using these intermediate-level features. This process is typically accomplished by a built-in gating network or routing network, which analyzes the current speech segment features (intermediate-level features), such as whether it is a clear male voice, a noisy background, or a specific accent segment, and calculates the matching degree between this feature and the capabilities of each recognition expert network, i.e., the routing weight.

[0089] Subsequently, based on the calculated routing weights, the target expert network can be determined from among the various expert networks, and its activation state can be controlled. This process typically employs a Top-K strategy, where only the top K expert networks by routing weight (e.g., K=2) are activated as target expert networks, while the remaining expert networks with lower weights, considered less adept at processing the current speech segment features, remain dormant. Next, the activated target expert networks perform feature transformation on the current speech segment features. These selected "elite forces" utilize their expertise in parameter space analysis to refine the speech segment features, extracting high-quality acoustic language features.

[0090] At this point, the hybrid expert module actually forms two parallel data streams: one is the optimized feature representation output by the task expert network activated by the text optimization instructions (carrying information such as punctuation and capitalization), and the other is the basic feature representation output by the target expert network selected by the routing mechanism (carrying core recognition information). The encoder fuses the basic feature representation and the optimized feature representation, such as through weighted summation, concatenation, or other linear / nonlinear fusion, to bring the two information streams together and ultimately obtain encoded features containing optimized text semantics.

[0091] In this embodiment of the invention, by introducing a recognition expert network into the hybrid expert module and combining it with a dynamic routing mechanism, a balance between accurate recognition and on-demand optimization is achieved. Specifically, on the one hand, the Top-K dynamic routing mechanism enables the model to automatically schedule the most suitable recognition expert network to participate in the computation based on the specific content of the speech, ensuring recognition accuracy in complex acoustic environments while maintaining efficient inference speed through sparse activation (activating only K experts). On the other hand, by fusing the basic feature representation of the recognition expert network with the optimized feature representation of the task expert network, the final generated encoded features are ensured to contain both accurate speech recognition content and rich text optimization semantics, enabling a single model to simultaneously achieve high-precision recognition and high-quality text representation.

[0092] Based on the above embodiments, the text optimization instruction is a multi-dimensional vector, and each dimension of the multi-dimensional vector corresponds to a text optimization task.

[0093] The encoder is used to parse multidimensional vectors to obtain the value of each dimension. When the value of any dimension is the first preset value, it determines that the text optimization task corresponding to that dimension is enabled and controls the activation state of the corresponding task expert network in the hybrid expert module to be activated.

[0094] The encoder is also used to determine that the text optimization task corresponding to any dimension is disabled when the value of any dimension is a second preset value, and to control the activation state of the corresponding task expert network in the hybrid expert module to be dormant, and to skip the parameter calculation of the corresponding task expert network during the encoding process.

[0095] Specifically, to achieve independent on / off control of multiple parallel text optimization functions, in this embodiment of the invention, the text optimization instruction can be a structured multi-dimensional vector, such as a multi-dimensional binary vector. Each dimension of this vector corresponds to a text optimization task. For example, in a four-dimensional binary vector [d1, d2, d3, d4], the first dimension d1 might correspond to a "punctuation restoration task," the second dimension d2 to a "case restoration task," the third dimension d3 to a "text denormalization task," and the fourth dimension d4 to a "non-fluency removal task." This design allows a single, concise text optimization instruction to fully cover the configuration of all text optimization functions.

[0096] The following example uses a four-dimensional binary vector to illustrate the configuration of text optimization. The vector contains four dimensions corresponding to four text optimization tasks, each with a value of 0 or 1, indicating the enabled status of the corresponding task. For example, a text optimization command of [1, 1, 0, 0] indicates that punctuation restoration and case restoration tasks are enabled, while text de-normalization and non-fluency removal tasks are disabled. Similarly, a command of [1, 1, 1, 1] indicates that punctuation restoration, case restoration, text de-normalization, and non-fluency removal tasks are enabled. Finally, a command of [0, 0, 0, 0] indicates that only basic speech recognition is performed, without any text optimization, effectively disabling punctuation restoration, case restoration, text de-normalization, and non-fluency removal tasks. Different combinations of values ​​can achieve various output formats to meet the needs of different application scenarios.

[0097] During model inference, the encoder can parse the multidimensional vector to obtain the value of each dimension. In this embodiment of the invention, a protocol is predefined to correspond the value of each dimension to the activation status of the text optimization task. A first preset value (usually a value "1" or a Boolean true value) is set to represent "enabled", and a second preset value (usually a value "0" or a Boolean false value) is set to represent "disabled".

[0098] Based on this protocol, the encoder executes the following judgment and control logic: when the value of any dimension in the vector is parsed to the first preset value, such as d1=1, the encoder immediately determines that the text optimization task corresponding to that dimension is enabled, that is, the punctuation restoration task is enabled; then, the encoder sends a signal to the hybrid expert module to control the activation state of the corresponding task expert network in the hybrid expert module to be activated, which means that the punctuation restoration expert network corresponding to the punctuation restoration task is allowed to access the data stream and participate in the calculation and transformation of the current feature.

[0099] Conversely, when the value of any dimension in the vector is parsed to a second preset value, such as d3=0, the encoder determines that the text optimization task corresponding to that dimension is disabled, i.e., the text inverse normalization task is disabled. At this time, the encoder implements a strict resource management strategy, controlling the activation state of the text inverse normalization expert network corresponding to the text inverse normalization task in the hybrid expert module to be dormant. This dormancy is not merely about not outputting results, but rather, at the physical computation level, skipping the parameter calculations of the task's expert network during the encoding process. In other words, the current feature will not enter the matrix multiplication operation of the task's expert network at all; the floating-point operations in this part are completely eliminated, thus saving computational resources and avoiding unnecessary interference from the text optimization task on the current feature.

[0100] In this embodiment of the invention, multi-dimensional vectors are used as instruction carriers, achieving a simplified expression and precise control of complex function combinations. That is, the multi-dimensional vector format allows users to arbitrarily arrange and combine various optimization functions, offering extremely high flexibility. More importantly, by directly associating instruction values ​​with the underlying computational control, it clarifies under what circumstances computation is skipped. This on-demand control mechanism not only shields unnecessary functions, reducing interference with current features, but also reduces computational load and memory consumption. Compared to large model solutions running with all parameters, this mechanism delivers significant speed improvements and energy savings when some functions are disabled, making it particularly suitable for resource-sensitive mobile or edge computing scenarios. Furthermore, the independence of each dimension allows different text optimization tasks to be arbitrarily combined without requiring separate model training for each combination, greatly enhancing flexibility and practicality.

[0101] Based on the above embodiments, the encoder adopts a network architecture with a parameter scale larger than that of the decoder. The encoder includes a bottom layer network, a middle layer network, and a top layer network. The hybrid expert module is embedded in multiple layers of the top layer network in a distributed manner.

[0102] The bottom layer network is used to encode the acoustic feature sequence to obtain acoustic feature representation; the middle layer network is used to fuse language information into the acoustic feature representation to obtain intermediate-level features containing semantic and syntactic information; the top layer network is used to perform progressive feature transformation on the intermediate-level features through hybrid expert modules embedded in multiple layers to obtain encoded features containing text-optimized semantics.

[0103] Specifically, in terms of overall architecture design, this embodiment of the invention abandons the traditional symmetrical design where the encoder and decoder have roughly equal parameter sizes, and instead adopts an asymmetric network architecture where the encoder has a larger parameter size than the decoder. Specifically, most of the model's parameters, such as over 80%, are concentrated on the encoder side, giving it powerful feature extraction and context modeling capabilities; while the decoder adopts a lightweight design, consisting of only a small number of Transformer layers. The rationale behind this "top-heavy" design is that the encoder can be computed in parallel, while the decoder is typically generated serially via autoregression during inference. Therefore, the lightweight decoder reduces the decoding burden, significantly lowers inference latency, and enables the overall inference speed to meet the requirements of real-time applications, making it suitable for scenarios with extremely high real-time requirements, such as voice assistants and real-time captions. Furthermore, the number of parameters in the hybrid expert module of the encoder is much smaller than that of a large-scale language model, making it particularly suitable for deployment on resource-constrained edge devices, while also possessing good scalability.

[0104] In order to process speech signals more precisely, in this embodiment of the invention, the massive encoder is logically divided into a bottom layer network, a middle layer network, and a top layer network, in which data flows from bottom to top, completing the transformation from acoustic signals to high-level semantics.

[0105] Specifically, the acoustic feature sequence first enters the bottom-level network (usually composed of convolutional layers or early Conformer layers), where it encodes the acoustic feature sequence to obtain an acoustic feature representation. At this stage, the model primarily focuses on the physical properties of the speech signal, such as timbre, pitch, and formants; the extracted features largely reflect the texture of the sound itself. Subsequently, the data flows to the middle-level network, where acoustic and linguistic information are integrated. That is, the middle-level network uses a self-attention mechanism to capture long-range contextual dependencies, transforming the pure acoustic signal into intermediate-level features containing semantic and syntactic information. At this point, the features begin to incorporate linguistic knowledge such as vocabulary, phrases, and syntactic structures, laying the foundation for subsequent text optimization.

[0106] Following this, the data enters the top-level network. In the top-level network, hybrid expert modules are embedded in multiple layers in a distributed manner, and the task expert networks are evenly distributed across multiple layers and coexist within the same layer. Unlike inserting hybrid expert modules only at a single layer, in this embodiment of the invention, the hybrid expert modules are distributed across several consecutive or spaced layers of the top-level network; for example, each of the last six layers embeds a MoE. When intermediate-level features flow through these layers, the hybrid expert modules embedded in these layers perform progressive feature transformations on the intermediate-level features according to the corresponding task.

[0107] Here, the progressive feature transformation can be understood as follows: for punctuation recovery tasks, the first layer of MoE might first perform sentence segmentation, injecting basic sentence segmentation information. Subsequent MoE layers then further determine the punctuation marks at each sentence segmentation. This multi-layered, iterative refinement leverages the advanced semantic understanding capabilities already possessed by the top-level network. Text optimization requires deep contextual understanding and linguistic knowledge; therefore, embedding a hybrid expert module into the top-level network of the encoder fully utilizes the existing language representations, resulting in more accurate and natural encoded features containing optimized text semantics. This lays the foundation for the subsequent decoder to generate target recognition text that meets the requirements.

[0108] In this embodiment of the invention, an asymmetric architecture of a large-scale encoder and a lightweight decoder, combined with a distributed embedding strategy of MoE at the top layer of the encoder, achieves global optimization of model performance and efficiency. On the one hand, the lightweight decoder design significantly reduces the time required to generate each character, ensuring real-time response capability. On the other hand, placing the hybrid expert module at the top layer of the encoder leverages the characteristic that higher neural network layers have stronger semantics, ensuring that the text optimization task is performed based on a thorough understanding of the semantics, rather than shallow acoustic matching. Simultaneously, the distributed embedding method makes feature transformation a gradual process, avoiding drastic oscillations in the feature space caused by single-layer mutations, enabling the final generated encoded features to smoothly and stably fuse both recognition and optimization semantics.

[0109] Based on the above embodiments, the speech recognition model is trained using the following steps:

[0110] The initial recognition model is determined, which includes an initial encoder and an initial decoder. The initial encoder contains an initial expert module for text optimization tasks.

[0111] Based on the first training sample, the initial recognition model is pre-trained to obtain the basic model; the first training sample includes the first sample speech and the corresponding first sample text; the parameters of the initial expert module in the initial encoder are not updated during the pre-training process;

[0112] Based on the second training samples, the base model is jointly trained to obtain the speech recognition model. The second training samples include second samples corresponding to various text optimization tasks. Each second sample includes second sample speech, as well as second text optimization instructions and second sample optimized text corresponding to the second sample speech. During the joint training process, the parameters of the initial expert module in the initial encoder are updated.

[0113] Specifically, in order to enable the model to maintain high-precision speech recognition capabilities while flexibly mastering various text optimization techniques, this embodiment of the invention adopts a phased training strategy, which mainly includes two phases: pre-training and joint training.

[0114] The specific process of the pre-training phase is as follows:

[0115] First, an initial recognition model needs to be determined. This model architecturally includes an initial encoder and an initial decoder, with the initial encoder embedding an initial expert module for text optimization tasks. At this stage, the parameters of each part of the model are usually randomly initialized, and it does not yet possess actual recognition capabilities.

[0116] Subsequently, the initial recognition model can be pre-trained using the first training sample to obtain the basic model. The first training sample here is mainly used to train the model's basic ability to "recognize characters by sound," and its data consists of speech-text pairs, that is, containing the first sample speech and its corresponding first sample text. The first sample text is usually the original transcribed text, that is, plain text without punctuation or formatting, or standard text that has been processed according to rules.

[0117] During this phase, a special parameter freezing strategy was adopted, meaning that the parameters of the initial expert modules in the initial encoder were not updated during pre-training. Specifically, in architectures containing only task expert networks, these task expert networks were temporarily bypassed or frozen; in hybrid architectures containing recognition expert networks and task expert networks, only the recognition expert network (using a Top-K dynamic routing mechanism to select activations) was allowed to participate in updates, while the task expert networks remained silent. The purpose of this strategy was to prevent the introduction of complex text optimization tasks prematurely before the model had mastered basic acoustic modeling capabilities, which could lead to difficulties in model convergence or distraction.

[0118] During training, the initial encoder learns general acoustic-to-language feature transformations, while the initial decoder focuses on the output of the initial encoder through cross-attention, autoregressively generating text sequences. The training objective at this stage is to minimize the difference between the output of the initial decoder and the first sample text. Specifically, this difference can be measured using the cross-entropy loss function to obtain the model loss. Based on this loss, the model parameters are adjusted, allowing the model to fully learn basic speech recognition capabilities through continuous parameter updates, thus laying the foundation for the subsequent introduction of text optimization functions. The basic model obtained at this stage already possesses excellent speech recognition capabilities in various scenarios, but it does not yet know how to optimize text.

[0119] After this, the next stage of training can begin. The multi-task instruction-driven joint training process specifically includes the following steps:

[0120] Once the basic model is built, it can be jointly trained using the second training samples to obtain the speech recognition model. The data at this stage is richer and more complex, including second samples corresponding to multiple text optimization tasks, second samples corresponding to a single text optimization task, second samples corresponding to multiple text optimization tasks, and second samples containing only the basic recognition task.

[0121] To enable the model to understand instructions, each second sample includes the second sample speech, the corresponding second text optimization instruction, and the second sample optimized text. For example, multiple second samples can be constructed for the same second sample speech:

[0122] The second sample corresponding to a single text optimization task: the second sample speech, the second text optimization instruction [1, 0, 0, 0] (only the punctuation recovery task is enabled), and the corresponding second sample optimized text (optimized text with punctuation marks).

[0123] Second samples corresponding to various text optimization tasks: second sample speech, second text optimization instructions [1, 1, 0, 0] (enable punctuation restoration task and case restoration task), and corresponding second sample optimized text (optimized text with punctuation and case).

[0124] The second sample of the basic recognition task: the second sample speech, the second text optimization instruction [0, 0, 0, 0] (no optimization, i.e. all text optimization tasks are disabled), and the corresponding second sample optimized text (the original transcribed text, i.e. plain text without punctuation and without formatting).

[0125] In this stage, the training mechanism changes. During joint training, the parameters of the initial expert module in the initial encoder are updated, meaning the task expert network in the initial expert module is activated. Specifically, during training, each batch of second samples activates the corresponding task expert network based on the second text optimization instructions they carry. When the initial encoder in the base model propagates forward to the initial expert module layer, the recognition expert network continues to select activation through a Top-K dynamic routing mechanism, while the task expert network decides whether to participate in the computation based on the instructions it carries. The initial decoder in the base model receives the feature representation output by the initial encoder and generates a text sequence. The training loss in this stage is measured using the standard cross-entropy loss function to calculate the difference between the output of the initial decoder and the corresponding optimized text of the second sample. However, although the loss function has a uniform form, different second samples correspond to different optimization needs, allowing a single loss function to drive multi-task learning simultaneously.

[0126] During this training phase, through extensive joint training with explicit instructions, the individual task expert networks in the initial expert module gradually learn the feature transformation patterns they are responsible for. For example, the punctuation recovery expert network learns to inject semantics such as commas, pauses, and semicolons at pauses, while the model as a whole learns how to accurately route and fuse features according to instructions.

[0127] It's important to note here that, because the instructions for different second samples vary, the task expert networks or combinations of networks activated during each training session are also different. This dynamic activation mechanism allows the model to learn the independent characteristics of each text optimization task, as well as the collaborative relationships between tasks. When multiple text optimization tasks are activated simultaneously, the corresponding task expert networks need to coordinate their work to jointly optimize feature representations to satisfy multiple text optimization objectives.

[0128] In this embodiment of the invention, pre-training ensures that the model first establishes a solid acoustic language alignment capability, avoiding gradient conflicts or performance degradation caused by simultaneous learning of recognition and optimization during the model's "unclear" period. On this basis, joint training is carried out, using rich data augmentation samples to enable the model to efficiently learn multi-dimensional text optimization skills while maintaining a basic recognition rate, greatly improving the model's convergence speed and performance.

[0129] Based on the above embodiments, the base model is jointly trained using the second training sample to obtain a speech recognition model, including:

[0130] According to the initial sampling ratio, target second samples are extracted from the second samples corresponding to various text optimization tasks in the second training samples;

[0131] Based on the target second sample, the base model is jointly trained;

[0132] During the joint training process, the performance of the trained base model is periodically verified to obtain the current convergence index for each text optimization task. Based on the current convergence index, the sampling ratio of each text optimization task in the initial sampling ratio is adjusted.

[0133] Based on the adjusted sampling ratio, new target second samples are extracted from the second samples corresponding to various text optimization tasks in the second training samples.

[0134] Based on the new target second sample, the trained base model is jointly trained to jointly update the parameters of the initial encoder, initial expert module and initial decoder, thus obtaining the speech recognition model.

[0135] Specifically, during the joint training phase, the model needs to simultaneously master multiple skills, including punctuation restoration, case restoration, text inverse normalization, and non-fluency removal. However, the learning difficulty and data sparsity of different text optimization tasks often vary significantly. For example, punctuation restoration is relatively easy, while complex text inverse normalization may be difficult to converge. If a fixed data ratio is always used for training, it can easily lead to overfitting on simple tasks and underfitting on difficult tasks. Therefore, this embodiment of the invention employs a dynamic sampling mechanism, the specific process of which is as follows:

[0136] At the start of training, target second samples are first extracted from the second samples corresponding to various text optimization tasks in the second training samples, according to the initial sampling ratio. Here, the initial sampling ratio can be a uniform distribution or an empirical ratio preset based on the total amount of data for each text optimization task. These extracted target second samples constitute the main content of the current training batch.

[0137] Then, the pre-trained base model can be jointly trained using the target second sample. During this process, the model updates its parameters through backpropagation and begins to learn various text optimization tasks.

[0138] To monitor the learning status of each text optimization task in real time, in this embodiment of the invention, the base model after phased training (after several batches of training) is periodically validated during joint training. For example, performance validation is performed every N epochs or steps. That is, the performance of the phased-trained base model on each individual text optimization task, such as punctuation recovery accuracy and digit conversion accuracy, is tested on a pre-prepared validation set to obtain the current convergence metric for each text optimization task. This metric objectively reflects the current mastery of each skill by the phased-trained base model.

[0139] Next, based on this feedback, the sampling ratio of each text optimization task in the initial sampling ratio can be adjusted. The specific adjustment logic follows the principle of "strengthening the weak," that is, for text optimization tasks with low convergence index (i.e., not yet learned well enough), their sampling weight is appropriately increased; while for text optimization tasks with high convergence index (i.e., already learned well enough), their sampling weight is appropriately decreased.

[0140] Subsequently, the next round of joint training can begin, which involves extracting new target second samples from the second samples corresponding to various text optimization tasks in the second training samples, based on the adjusted sampling ratio. This means that the number of training samples corresponding to those text optimization tasks that are "not yet well learned" has increased in the new training samples.

[0141] Finally, the new target second sample can be used to jointly train the trained base model. That is, in this continuous iteration process, the parameters of the initial encoder, initial expert module, and initial decoder in the trained base model are jointly updated. Through this closed-loop training mechanism of continuous verification, adjustment, and retraining, until the model as a whole reaches the preset convergence criterion, a well-trained speech recognition model can be obtained.

[0142] In this embodiment of the invention, an adaptive allocation of training resources is achieved through a feedback adjustment mechanism based on convergence indices. This breaks the limitations of traditional static data allocation and can dynamically allocate computing power and data resources to the weakest task based on the real-time performance of the model during training. This not only effectively avoids the phenomenon that the gradient direction of a certain task dominates in multi-task learning, causing other tasks to be suppressed, but also significantly improves the overall balance and final performance of the model across all text optimization dimensions, ensuring that the final trained speech recognition model is an "all-rounder" high-quality model.

[0143] The speech recognition device provided by the present invention is described below. The speech recognition device described below and the speech recognition method described above can be referred to in correspondence.

[0144] Figure 4 This is a schematic diagram of the structure of the voice recognition device provided by the present invention, as shown below. Figure 4 As shown, the device includes:

[0145] Acquisition unit 410 is used to acquire the speech signal to be recognized;

[0146] Extraction unit 420 is used to extract features from the speech signal to obtain an acoustic feature sequence;

[0147] Recognition unit 430 is used to input the acoustic feature sequence into the speech recognition model to obtain the target recognition text that has been optimized by the speech recognition model.

[0148] The speech recognition model includes an encoder and a decoder, wherein the encoder has a hybrid expert module embedded in it for text optimization tasks.

[0149] The encoder is used to encode the acoustic feature sequence layer by layer, and during the encoding process, the hybrid expert module performs feature transformation on the intermediate level features corresponding to the text optimization task to obtain encoded features containing text optimization semantics; the decoder is used to decode the encoded features to obtain the target recognition text.

[0150] The speech recognition device provided by this invention effectively solves the problems of bloated architecture and error accumulation caused by the separation of recognition and text format optimization in traditional speech recognition systems by deeply embedding a hybrid expert module for text optimization tasks into the encoder of the end-to-end speech recognition model. By utilizing the hybrid expert module to perform deep feature transformation on intermediate-level features, the model integrates advanced text optimization semantics such as punctuation, capitalization, number normalization, and removal of colloquialisms during the encoding stage. Thus, without the need for an external post-processing model and without significantly increasing inference latency, it can directly output high-quality, highly readable target recognition text, achieving a dual improvement in inference efficiency and recognition quality, and greatly enhancing the usability of speech recognition technology in practical scenarios such as meeting recording and real-time captioning.

[0151] Based on the above embodiments, there are multiple text optimization tasks, and each text optimization task has a corresponding task expert network in the hybrid expert module;

[0152] The identification unit 430 is used for:

[0153] Obtain the text optimization instructions corresponding to the speech signal;

[0154] The acoustic feature sequence and the text optimization instructions are input into the speech recognition model to obtain the target recognition text that has been optimized by the speech recognition model.

[0155] The encoder is used to determine the activation status of each text optimization task based on the text optimization instructions, control the activation status of the corresponding task expert network in the hybrid expert module based on the activation status, and perform feature transformation on the intermediate-level features through the activated task expert network to obtain the encoded features containing text optimization semantics.

[0156] Based on the above embodiments, the hybrid expert module further includes multiple recognition expert networks for maintaining basic speech recognition capabilities;

[0157] The encoder is also used to determine the routing weight of each recognition expert network based on the intermediate-level features, determine the target expert network from the plurality of recognition expert networks based on the routing weight, control the activation state of the target expert network, and perform feature transformation on the intermediate-level features through the activated target expert network.

[0158] The encoder is used to fuse the basic feature representation output by the activated target expert network with the optimized feature representation output by the activated task expert network to obtain the encoded features containing text optimization semantics.

[0159] Based on the above embodiments, the text optimization instruction is a multi-dimensional vector, and each dimension of the multi-dimensional vector corresponds to a text optimization task;

[0160] The encoder is used to parse the multidimensional vector to obtain the value of each dimension. When the value of any dimension is a first preset value, the activation status of the text optimization task corresponding to the any dimension is determined to be enabled, and the activation status of the corresponding task expert network in the hybrid expert module is controlled to be activated.

[0161] The encoder is also used to determine that the activation status of the text optimization task corresponding to any dimension is disabled when the value of any dimension is a second preset value, and to control the activation status of the corresponding task expert network in the hybrid expert module to be dormant, and to skip the parameter calculation of the corresponding task expert network during the encoding process.

[0162] Based on the above embodiments, the encoder adopts a network architecture with a parameter scale larger than that of the decoder, and the encoder includes a bottom layer network, a middle layer network, and a top layer network; the hybrid expert module is embedded in multiple layers of the top layer network in a distributed manner;

[0163] The bottom-layer network is used to encode the acoustic feature sequence to obtain an acoustic feature representation; the middle-layer network is used to perform language information fusion on the acoustic feature representation to obtain the intermediate-level features containing semantic and syntactic information; the top-layer network is used to perform progressive feature transformation on the intermediate-level features through the hybrid expert modules embedded in the multiple layers to obtain the encoded features containing text-optimized semantics.

[0164] Based on the above embodiments, the device further includes a training unit for:

[0165] An initial recognition model is determined, which includes an initial encoder and an initial decoder, wherein the initial encoder embeds an initial expert module for the text optimization task;

[0166] Based on the first training sample, the initial recognition model is pre-trained to obtain a basic model; the first training sample includes a first sample speech and a corresponding first sample text; the parameters of the initial expert module in the initial encoder are not updated during the pre-training process;

[0167] Based on the second training samples, the base model is jointly trained to obtain the speech recognition model; the second training samples include second samples corresponding to various text optimization tasks, each second sample including second sample speech, and second text optimization instructions and second sample optimized text corresponding to the second sample speech; during the joint training process, the parameters of the initial expert module in the initial encoder are updated.

[0168] Based on the above embodiments, the training unit is used for:

[0169] According to the initial sampling ratio, target second samples are extracted from the second samples corresponding to various text optimization tasks in the second training samples;

[0170] Based on the target second sample, the base model is jointly trained;

[0171] During the joint training process, the performance of the trained base model is periodically verified to obtain the current convergence index for each text optimization task. Based on the current convergence index, the sampling ratio of each text optimization task in the initial sampling ratio is adjusted.

[0172] Based on the adjusted sampling ratio, new target second samples are extracted from the second samples corresponding to various text optimization tasks in the second training samples.

[0173] Based on the new target second sample, the trained base model is jointly trained to jointly update the parameters of the initial encoder, the initial expert module and the initial decoder, thereby obtaining the speech recognition model.

[0174] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a speech recognition method, which includes: acquiring a speech signal to be recognized; extracting features from the speech signal to obtain an acoustic feature sequence; inputting the acoustic feature sequence into a speech recognition model to obtain text-optimized target recognition text output by the speech recognition model; wherein the speech recognition model includes an encoder and a decoder, the encoder embedding a hybrid expert module for the text optimization task; the encoder is used to perform layer-by-layer encoding processing on the acoustic feature sequence, and during the encoding process, the hybrid expert module performs feature transformation on intermediate-level features corresponding to the text optimization task to obtain encoded features containing text optimization semantics; the decoder is used to decode the encoded features to obtain the target recognition text.

[0175] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0176] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the speech recognition method provided by the above methods, the method comprising: acquiring a speech signal to be recognized; extracting features from the speech signal to obtain an acoustic feature sequence; inputting the acoustic feature sequence into a speech recognition model to obtain a text-optimized target recognition text output by the speech recognition model; wherein the speech recognition model comprises an encoder and a decoder, the encoder embedding a hybrid expert module for a text optimization task; the encoder is used to perform layer-by-layer encoding processing on the acoustic feature sequence, and during the encoding process, the hybrid expert module performs feature transformation on intermediate-level features corresponding to the text optimization task to obtain encoded features containing text optimization semantics; the decoder is used to decode the encoded features to obtain the target recognition text.

[0177] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the speech recognition method provided by the methods described above. The method includes: acquiring a speech signal to be recognized; extracting features from the speech signal to obtain an acoustic feature sequence; inputting the acoustic feature sequence into a speech recognition model to obtain text-optimized target recognition text output by the speech recognition model; wherein the speech recognition model includes an encoder and a decoder, the encoder embedding a hybrid expert module for a text optimization task; the encoder is used to perform layer-by-layer encoding processing on the acoustic feature sequence, and during the encoding process, the hybrid expert module performs feature transformation on intermediate-level features corresponding to the text optimization task to obtain encoded features containing text optimization semantics; the decoder is used to decode the encoded features to obtain the target recognition text.

[0178] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0179] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech recognition method, characterized in that, include: Acquire the speech signal to be recognized; Feature extraction is performed on the speech signal to obtain an acoustic feature sequence; The acoustic feature sequence is input into the speech recognition model to obtain the target recognition text that has been optimized by the speech recognition model. The speech recognition model includes an encoder and a decoder, wherein the encoder has a hybrid expert module embedded in it for text optimization tasks. The encoder is used to encode the acoustic feature sequence layer by layer, and during the encoding process, the hybrid expert module performs feature transformation on the intermediate level features corresponding to the text optimization task to obtain encoded features containing text optimization semantics; the decoder is used to decode the encoded features to obtain the target recognition text; The encoder adopts a network architecture with a larger parameter scale than the decoder, and the encoder includes a bottom layer network, a middle layer network, and a top layer network; the hybrid expert module is embedded in multiple layers of the top layer network in a distributed manner. The bottom-layer network is used to encode the acoustic feature sequence to obtain an acoustic feature representation; the middle-layer network is used to perform language information fusion on the acoustic feature representation to obtain the intermediate-level features containing semantic and syntactic information; the top-layer network is used to perform progressive feature transformation on the intermediate-level features through the hybrid expert modules embedded in the multiple layers to obtain the encoded features containing text-optimized semantics.

2. The speech recognition method according to claim 1, characterized in that, There are various text optimization tasks, and each text optimization task has a corresponding task expert network in the hybrid expert module. The step of inputting the acoustic feature sequence into a speech recognition model to obtain the text-optimized target recognition text output by the speech recognition model includes: Obtain the text optimization instructions corresponding to the speech signal; The acoustic feature sequence and the text optimization instructions are input into the speech recognition model to obtain the target recognition text that has been optimized by the speech recognition model. The encoder is used to determine the activation status of each text optimization task based on the text optimization instructions, control the activation status of the corresponding task expert network in the hybrid expert module based on the activation status, and perform feature transformation on the intermediate-level features through the activated task expert network to obtain the encoded features containing text optimization semantics.

3. The speech recognition method according to claim 2, characterized in that, The hybrid expert module also includes multiple recognition expert networks for maintaining basic speech recognition capabilities; The encoder is also used to determine the routing weight of each recognition expert network based on the intermediate-level features, determine the target expert network from the plurality of recognition expert networks based on the routing weight, control the activation state of the target expert network, and perform feature transformation on the intermediate-level features through the activated target expert network. The encoder is used to fuse the basic feature representation output by the activated target expert network with the optimized feature representation output by the activated task expert network to obtain the encoded features containing text optimization semantics.

4. The speech recognition method according to claim 2, characterized in that, The text optimization instructions are multi-dimensional vectors, and each dimension of the multi-dimensional vector corresponds to a text optimization task. The encoder is used to parse the multidimensional vector to obtain the value of each dimension. When the value of any dimension is a first preset value, the activation status of the text optimization task corresponding to the any dimension is determined to be enabled, and the activation status of the corresponding task expert network in the hybrid expert module is controlled to be activated. The encoder is also used to determine that the activation status of the text optimization task corresponding to any dimension is disabled when the value of any dimension is a second preset value, and to control the activation status of the corresponding task expert network in the hybrid expert module to be dormant, and to skip the parameter calculation of the corresponding task expert network during the encoding process.

5. The speech recognition method according to any one of claims 1 to 4, characterized in that, The speech recognition model is trained based on the following steps: An initial recognition model is determined, which includes an initial encoder and an initial decoder, wherein the initial encoder embeds an initial expert module for the text optimization task; Based on the first training sample, the initial recognition model is pre-trained to obtain the basic model; the first training sample includes the first sample speech and the corresponding first sample text. During the pre-training process, the parameters of the initial expert module in the initial encoder are not updated; Based on the second training samples, the base model is jointly trained to obtain the speech recognition model; the second training samples include second samples corresponding to various text optimization tasks, each second sample including second sample speech, and second text optimization instructions and second sample optimized text corresponding to the second sample speech; during the joint training process, the parameters of the initial expert module in the initial encoder are updated.

6. The speech recognition method according to claim 5, characterized in that, The step of jointly training the base model based on the second training samples to obtain the speech recognition model includes: According to the initial sampling ratio, target second samples are extracted from the second samples corresponding to various text optimization tasks in the second training samples; Based on the target second sample, the base model is jointly trained; During the joint training process, the performance of the trained base model is periodically verified to obtain the current convergence index for each text optimization task. Based on the current convergence index, the sampling ratio of each text optimization task in the initial sampling ratio is adjusted. Based on the adjusted sampling ratio, new target second samples are extracted from the second samples corresponding to various text optimization tasks in the second training samples. Based on the new target second sample, the trained base model is jointly trained to jointly update the parameters of the initial encoder, the initial expert module and the initial decoder, thereby obtaining the speech recognition model.

7. A voice recognition device, characterized in that, include: Acquisition unit, used to acquire the speech signal to be recognized; An extraction unit is used to extract features from the speech signal to obtain an acoustic feature sequence; The recognition unit is used to input the acoustic feature sequence into the speech recognition model to obtain the target recognition text that has been optimized by the speech recognition model. The speech recognition model includes an encoder and a decoder, wherein the encoder has a hybrid expert module embedded in it for text optimization tasks. The encoder is used to encode the acoustic feature sequence layer by layer, and during the encoding process, the hybrid expert module performs feature transformation on the intermediate level features corresponding to the text optimization task to obtain encoded features containing text optimization semantics; the decoder is used to decode the encoded features to obtain the target recognition text; The encoder adopts a network architecture with a larger parameter scale than the decoder, and the encoder includes a bottom layer network, a middle layer network, and a top layer network; the hybrid expert module is embedded in multiple layers of the top layer network in a distributed manner. The bottom-layer network is used to encode the acoustic feature sequence to obtain an acoustic feature representation; the middle-layer network is used to perform language information fusion on the acoustic feature representation to obtain the intermediate-level features containing semantic and syntactic information; the top-layer network is used to perform progressive feature transformation on the intermediate-level features through the hybrid expert modules embedded in the multiple layers to obtain the encoded features containing text-optimized semantics.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the speech recognition method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech recognition method as described in any one of claims 1 to 6.

Images