Speech recognition method and apparatus, electronic device, and storage medium

Abstract

The application provides a speech recognition method and device, electronic equipment and storage medium, wherein the method comprises: inputting an acoustic feature sequence to be recognized into a first decoding branch for decoding to obtain a first decoding text and a first feature parameter; inputting the acoustic feature sequence into a second decoding branch for decoding to obtain a second decoding text and a second feature parameter; the second decoding branch is used for extracting a multilingual phoneme sequence of the acoustic feature sequence and performing finite state transition decoding based on the multilingual phoneme sequence; and based on the first feature parameter and the second feature parameter, a target recognition text is determined from the first decoding text and the second decoding text. The method can realize efficient and lightweight customization of specific field vocabulary without changing huge main model network parameters, solves the problems of long customization period, large resource occupation and mutual interference of different languages, and improves the landing adaptation efficiency and recognition accuracy of the model in complex vertical business scenarios.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and more particularly to a speech recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of artificial intelligence, general-purpose end-to-end multilingual speech recognition models have been widely used. These models are usually trained on large-scale general corpora and can meet recognition needs well in general scenarios, such as daily conversations. However, in practical applications, especially in specific vertical industries or when there are personalized needs, general-purpose models suffer from insufficient domain adaptability.

[0003] Currently, end-to-end speech recognition customization is mostly carried out from three dimensions: data engineering, model optimization, and decoding strategy. However, existing customization solutions usually involve heavy data processing and large-scale model parameter adjustment and training, resulting in slow customization speed and long cycle. Moreover, directly fine-tuning or modifying the parameters of general large models can easily destroy the original general recognition performance of the model when giving customization capabilities, making it difficult to balance the effectiveness of customization and general effect. Summary of the Invention

[0004] This invention provides a speech recognition method, device, electronic device, and storage medium to solve the problems of long customization cycles, large resource consumption, and easy interference between languages ​​in the prior art.

[0005] This invention provides a speech recognition method, comprising: Obtain the acoustic feature sequence of the multilingual speech to be identified; The acoustic feature sequence is input into the first decoding branch for decoding to obtain the first decoded text and its corresponding first feature parameter; The acoustic feature sequence is input into the second decoding branch for decoding to obtain the second decoded text and its corresponding second feature parameter; the second decoding branch is used to extract the multilingual phoneme sequence of the acoustic feature sequence and perform finite state transition decoding based on the multilingual phoneme sequence; Based on the first feature parameter and the second feature parameter, the target recognition text corresponding to the multilingual speech is determined from the first decoded text and the second decoded text.

[0006] According to a speech recognition method provided by the present invention, the second decoding branch includes an acoustic model and a finite state transition network; The step of inputting the acoustic feature sequence into the second decoding branch for decoding to obtain the second decoded text and its corresponding second feature parameters includes: Determine the language embedding features of the target language corresponding to the multilingual speech; The acoustic feature sequence and the language embedding feature are input into the acoustic model for sequence modeling to obtain a multilingual phoneme sequence; Based on the multilingual phoneme sequence, a path search is performed in the finite state transition network to obtain the second decoded text and its corresponding second feature parameter.

[0007] According to a speech recognition method provided by the present invention, the acoustic model is used to perform time-dimensional downsampling processing on the acoustic feature sequence to obtain a dimensionality-reduced feature sequence, and to extract temporal features from the dimensionality-reduced feature sequence to obtain a temporal feature sequence. Based on the temporal feature sequence and the language embedding features, a fusion mapping is performed to obtain the multilingual phoneme sequence.

[0008] According to a speech recognition method provided by the present invention, the acoustic model is trained based on the following steps: Obtain the sample speech feature sequence carrying phoneme labels; The sample speech feature sequence is masked according to the time masking parameter to obtain the masked speech feature sequence; the time masking parameter is determined by the pronunciation duration of the phoneme unit in the sample speech feature sequence. The acoustic model is trained based on the masked speech feature sequence and the phoneme labels.

[0009] According to a speech recognition method provided by the present invention, the finite state transition network is constructed based on a preset phoneme catalog and pronunciation dictionary, as well as a language model; the language model includes a general language model and a customized language model, and the language model is trained based on the following steps: Obtain customized and general corpus texts; An initial language model is trained based on the customized corpus text, and based on the initial language model, the perplexity score of each general statement in the general corpus text is determined. Based on the perplexity score, target general text is selected from the general corpus text, and the customized language model is trained based on the customized corpus text and the target general text; The general language model is trained based on the general corpus text.

[0010] According to a speech recognition method provided by the present invention, determining the target recognition text corresponding to the multilingual speech from the first decoded text and the second decoded text based on the first feature parameter and the second feature parameter includes: Based on the first decoded text and the second decoded text, a score evaluation is performed to obtain the first language score corresponding to the first decoded text and the second language score corresponding to the second decoded text; The first feature parameter and the second feature parameter are subjected to parameter alignment processing to obtain the first aligned parameter and the second aligned parameter. The feature parameters include acoustic score, word length and frame number. The first alignment parameter and the first language score, as well as the second alignment parameter and the second language score, are input into the classification decision model to make a classification decision, thereby obtaining a classification result. Based on the classification result, the target recognition text is determined from the first decoded text and the second decoded text.

[0011] According to a speech recognition method provided by the present invention, the classification decision model is trained based on the following steps: Identify multiple sample speech texts, as well as the first sample decoded text, the second sample decoded text, and the sample annotation text corresponding to each sample speech text; For each sample speech, determine the matching degree between the first sample decoded text and the second sample decoded text and the sample annotation text, respectively; If the matching degree of the first sample decoded text is higher than that of the second sample decoded text, the classification label of the corresponding sample speech is determined as the first identifier; If the matching degree of the second sample decoded text is higher than that of the first sample decoded text, the classification label of the corresponding sample speech is determined as the second identifier; If the matching degree of the first sample decoded text is equal to the matching degree of the second sample decoded text, the classification label of the corresponding sample speech is determined as the third identifier; Based on the multiple sample speech and their corresponding classification labels, the initial decision model is trained to obtain the classification decision model; the initial decision model is constructed on the basis of gradient boosting decision tree.

[0012] The present invention also provides a voice recognition device, comprising: The acquisition unit is used to acquire the acoustic feature sequence of the multilingual speech to be identified; The first decoding unit is used to input the acoustic feature sequence into the first decoding branch for decoding to obtain the first decoded text and its corresponding first feature parameter; The second decoding unit is used to input the acoustic feature sequence into the second decoding branch for decoding to obtain the second decoded text and its corresponding second feature parameter; the second decoding branch is used to extract the multilingual phoneme sequence of the acoustic feature sequence and perform finite state transition decoding based on the multilingual phoneme sequence; The decision unit is configured to determine the target recognition text corresponding to the multilingual speech from the first decoded text and the second decoded text based on the first feature parameter and the second feature parameter.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the speech recognition method as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method as described above.

[0015] The speech recognition method, device, electronic device, and storage medium provided by this invention, by adding a lightweight customized branch based on multilingual phoneme sequences and finite state transition decoding to the general decoding branch, and using the feature parameters output by the two branches for decision-making, can achieve efficient and lightweight customization of vocabulary in specific domains without changing the parameters of the large main model network, while ensuring the powerful general speech recognition capabilities of the original large model. It effectively solves the problems of long multilingual customization cycle, large resource consumption, and easy interference between languages ​​in the prior art, and greatly improves the efficiency and recognition accuracy of end-to-end speech recognition model in complex vertical business scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the speech recognition method provided by the present invention; Figure 2 This is a structural example diagram of the acoustic model provided by the present invention; Figure 3 This is an example diagram of the second decoding branch provided by the present invention; Figure 4 This is a flowchart of the overall speech recognition method provided by the present invention; Figure 5 This is a schematic diagram of the structure of the speech recognition device provided by the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] With the development of artificial intelligence technology, general-purpose end-to-end multilingual speech recognition models, such as Whisper, have been widely used. These models are usually trained on large-scale general-purpose corpora and can well meet the recognition needs of general scenarios such as daily conversations and news broadcasts.

[0020] However, in practical applications, especially in specific vertical industries or when facing personalized needs, general-purpose models often suffer from insufficient domain adaptability. To address this issue, current end-to-end speech recognition customization typically involves three dimensions: data engineering, model optimization, and decoding strategies. Specifically, the core of data engineering is to construct high-quality, scenario-matched speech-text aligned data to solve the problem of insufficient coverage of general-purpose corpora in specific domains; the core of model optimization is to enhance the model's feature extraction capabilities for the customized data through incremental fine-tuning, model distillation, and other methods; and the core of decoding strategies is to introduce external language models for joint decoding to supplement the contributions of the language model level.

[0021] However, existing customization solutions of this kind have obvious drawbacks in practical applications. Specifically, because the customization process inevitably involves heavy data processing and the adjustment and retraining of numerous model parameters, existing customization solutions are typically slow and have long customization cycles. Furthermore, directly fine-tuning or modifying the parameters of a general-purpose model, while endowing the model with domain-specific customization capabilities, often damages the model's original general recognition performance, making it difficult to ensure both effective customization and general-purpose performance.

[0022] In response, this invention provides a speech recognition method that aims to introduce an acoustic language separation architecture into an end-to-end recognition model, construct a lightweight customized branch independent of the general decoding branch, and accelerate the customization cycle of the end-to-end speech recognition model through dual-branch parallel decoding and feature parameter evaluation decisions. While ensuring effective customization in vertical domains, it also effectively guarantees general recognition performance.

[0023] Figure 1 This is a flowchart illustrating the speech recognition method provided by the present invention, which can be applied to a speech recognition system. Figure 1 As shown, the method includes: Step 110: Obtain the acoustic feature sequence of the multilingual speech to be identified; Step 120: Input the acoustic feature sequence into the first decoding branch for decoding to obtain the first decoded text and its corresponding first feature parameter; Step 130: Input the acoustic feature sequence into the second decoding branch for decoding to obtain the second decoded text and its corresponding second feature parameter; the second decoding branch is used to extract the multilingual phoneme sequence of the acoustic feature sequence and perform finite state transition decoding based on the multilingual phoneme sequence; Step 140: Based on the first feature parameter and the second feature parameter, determine the target recognition text corresponding to the multilingual speech from the first decoded text and the second decoded text.

[0024] Specifically, in practical speech recognition scenarios, the system first needs to determine the multilingual speech to be recognized. This can be achieved by acquiring audio data in real time using audio acquisition devices such as microphones or pickups, or by reading pre-recorded audio files. Here, the multilingual speech to be recognized not only covers single-language speech data but also broadly includes speech data from complex speech scenarios where multiple different languages ​​are mixed, such as smart home commands mixed with Chinese and English, or academic presentations containing specialized foreign language vocabulary. After acquiring the multilingual speech, the system inputs it into the front-end feature extraction module of a general end-to-end speech recognition model, such as a shared encoder, for signal processing and feature extraction. After this processing, the original continuous analog or digital audio signal is converted into a series of data matrices or vector sequences that objectively reflect the physical characteristics of the speech, i.e., acoustic feature sequences, such as Log-MelFilterbank features and deep acoustic hidden layer features.

[0025] Next, the system inputs the extracted acoustic feature sequence into two different decoding channels for processing. Specifically, the acoustic feature sequence can be input into the first decoding branch for decoding, yielding the first decoded text and its corresponding first feature parameters. The first decoding branch serves as the main decoding channel ensuring the system's basic general recognition capabilities, corresponding to the original end-to-end general master model. Within this branch, the system performs autoregressive or non-autoregressive decoding calculations according to the original massive network weights, mapping and converting the input one-dimensional or multi-dimensional acoustic features into human-readable natural language text. After decoding, this branch outputs the first decoded text as a candidate recognition result, such as the top-1 candidate text.

[0026] Simultaneously, the system also extracts and records one or more sets of parameters that are generated during the generation of the first decoded text, reflecting the multi-dimensional confidence or decoding status of the decoded text; these are known as the first feature parameters. Specifically, the first feature parameters can cover information such as acoustic score (reflecting pronunciation matching degree) in the acoustic dimension, word length in the text dimension, and frame number in the time dimension. It should be noted that this embodiment of the invention does not limit the specific dimensions of the first feature parameters, as long as they can objectively and comprehensively reflect the decoding status of the first decoding branch.

[0027] In parallel with the first decoding branch, the system inputs the acoustic feature sequence into the second decoding branch for decoding to obtain the second decoded text and its corresponding second feature parameters. The second decoding branch here refers to a lightweight customized decoding channel that is mounted separately outside the general main model. Its core function is to achieve rapid adaptation to new vocabulary or specific industry terms without changing the large number of parameters of the general main model.

[0028] In practice, after receiving the acoustic feature sequence, the second decoding branch first uses its internally constructed acoustic model or acoustic modeling network to extract features and outputs a sequence of basic pronunciation units across languages, i.e., a multilingual phoneme sequence, which is usually represented as a phoneme probability distribution. During this process, to achieve lightweight customization, the phoneme sequences of multiple languages ​​requiring customization can be modeled together uniformly, thus eliminating the need to configure redundant resource packages for each language separately and avoiding resource waste between languages.

[0029] After extracting the multilingual phoneme sequences, the second decoding branch uses them as acoustic input and employs finite-state transition decoding for the final text conversion. Specifically, this process utilizes a finite-state converter to fuse the phoneme probability distribution output by the acoustic model with a pre-set pronunciation dictionary and an external language model, and then uses network graph search, such as the shortest path search method, to find the optimal text path.

[0030] Because the finite state transition decoding architecture decouples acoustics from the language model, this branch only needs to update the dictionary and the external language model to achieve accurate recognition of domain-specific vocabulary. Finally, the second decoding branch outputs a set of recognition results belonging to the customized domain, namely the second decoded text, and simultaneously outputs the accompanying reference parameters, namely the second feature parameters, such as the acoustic score, word length, and frame count under this customized branch.

[0031] After both branches of parallel decoding are completed, the system determines the target recognition text corresponding to the multilingual speech from the first and second decoded texts based on the first and second feature parameters obtained from the decoding. This process is essentially an intelligent decision-making process following parallel decoding. That is, after obtaining the first feature parameter reflecting general recognition confidence and the second feature parameter reflecting customized recognition confidence, the system performs a deep comprehensive arbitration on these two sets of data. The specific arbitration logic can be based on pre-set empirical rule thresholds, or preferably, the two sets of feature parameters are normalized and aligned to obtain a comprehensive feature parameter, which is then input into a decision model, such as a binary classification machine learning model, for intelligent adjudication. The core purpose of the decision is to determine whether the multilingual speech is more likely to be customized speech for a specific domain or ordinary everyday speech.

[0032] Furthermore, if the comprehensive feature parameters indicate a higher matching degree of specialized vocabulary in the multilingual speech, the system tends to select the second decoded text as the only accurate text that fits the current scenario, i.e., the target recognition text; conversely, if it is determined to be non-customized Mandarin, the first decoded text is selected as the target recognition text. Through this dynamic decision-making mechanism, the system can intelligently and smoothly select the optimal recognition result based on different inputs during operation.

[0033] The speech recognition method provided by this invention adds a lightweight customized branch based on multilingual phoneme sequences and finite state transition decoding to the general decoding branch, and uses the feature parameters output by the two branches for decision-making. This not only enables efficient and lightweight customization of vocabulary in specific domains without changing the parameters of the large main model network, but also ensures the powerful general speech recognition capabilities of the original large model. It effectively solves the problems of long multilingual customization cycle, large resource consumption, and easy interference between languages ​​in the existing technology, and greatly improves the efficiency and recognition accuracy of end-to-end speech recognition model in complex vertical business scenarios.

[0034] Based on the above embodiments, the second decoding branch includes an acoustic model and a finite state transition network; The acoustic feature sequence is input into the second decoding branch for decoding, resulting in the second decoded text and its corresponding second feature parameters, including: Determine the language embedding features of the target language corresponding to the multilingual speech; The acoustic feature sequence and language embedding features are input into the acoustic model for sequence modeling to obtain multilingual phoneme sequences; Based on multilingual phoneme sequences, path search is performed in a finite state transition network to obtain the second decoded text and its corresponding second feature parameters.

[0035] Specifically, the second decoding branch includes an acoustic model and a finite state transition network. Here, the acoustic model refers to a neural network module used to transform the acoustic features extracted from the front end into probability distributions of basic phonetic units (such as phonemes); the finite state transition network serves as the core bridge connecting the phoneme probability distributions output by the acoustic model and the final decoded text. It typically integrates the phoneme probability distributions output by the acoustic model, a pre-built pronunciation dictionary, and an external language model to achieve efficient text decoding in subsequent stages.

[0036] In this architecture, the process of inputting the acoustic feature sequence into the second decoding branch for decoding can be specifically described as follows: When entering the customized branch for decoding, the system first needs to determine the language embedding features of the target language corresponding to the multilingual speech. That is, since the customized branch is designed to support lightweight operation, the phoneme sequences of multiple languages ​​that need to be customized are directly put together for unified modeling, thus avoiding the cumbersome process of configuring independent resource packages for each language. However, multilingual mixed modeling can lead to language crosstalk between different languages. To avoid this problem, the system determines the target language information corresponding to the multilingual speech before decoding and converts it into specific language embedding features, i.e., language embedding. This feature is essentially a dense vector that can clearly indicate the language currently being processed. It is equivalent to providing the acoustic model with a powerful auxiliary prior condition, explicitly telling the model which language's phoneme space should be focused on.

[0037] Next, the system inputs the acoustic feature sequence and language embedding features into the acoustic model for sequence modeling, resulting in a multilingual phoneme sequence. During this process, the acoustic feature sequence and language embedding features are concatenated or fused and then fed together as input into a customized branch of the acoustic model. The acoustic model performs deep sequence modeling on these input data to capture the temporal dependencies of the speech sequence. Thanks to the explicit guidance of the language embedding features, the model can effectively avoid interference from other language phonemes, thus accurately outputting the multilingual phoneme sequence corresponding to the multilingual speech. In practice, this sequence is typically represented as a probability distribution matrix of multilingual phonemes at each time step.

[0038] Following this, the system can perform path search in a finite state transition network based on multilingual phoneme sequences to obtain the second decoded text and its corresponding second feature parameters. Specifically, the generated phoneme probability distribution is directly fed into the finite state transition network, which integrates a pronunciation dictionary and a language model. Therefore, the system can map pronunciation phonemes to specific words without changing the acoustic model parameters. Based on this, the system employs path search algorithms, such as the shortest path search algorithm and Viterbi decoding search, to quickly find a globally optimal path from phoneme to text in the vast state transition graph. Through this efficient decoding process, the system ultimately outputs the second decoded text for this customized domain, such as the top-1 candidate text of the customized branch, and simultaneously compiles the second feature parameters along the decoding path, such as the acoustic score, word length, and total number of frames corresponding to the path.

[0039] In this embodiment of the invention, the language embedding features of the target language are introduced in the acoustic sequence modeling stage. This not only inherits the flexibility of finite state transition networks in quickly customizing vocabulary without changing acoustic parameters, but also fundamentally solves the language crosstalk problem that is very easy to occur when modeling multiple languages ​​in a unified manner. This greatly improves the purity of the phoneme sequence output by the multilingual customization branch in a specific domain and the accuracy of the final decoded text.

[0040] Based on the above embodiments, the acoustic model is used to perform time-dimension downsampling on the acoustic feature sequence to obtain a dimensionality-reduced feature sequence, and to extract temporal features from the dimensionality-reduced feature sequence to obtain a temporal feature sequence. Based on the temporal feature sequence and language embedding features, a fusion mapping is performed to obtain a multilingual phoneme sequence.

[0041] Specifically, Figure 2 This is a structural example diagram of the acoustic model provided by the present invention, such as... Figure 2 As shown, acoustic models (such as ctc_block) first perform temporal downsampling on the input acoustic feature sequence during feature processing. For example, specific convolutional network layers (such as a single conv1d convolutional layer) are used to compress dense speech feature frames along the time axis, such as by a factor of 2. This preserves core acoustic information while eliminating redundant frames, resulting in a shorter, more concise dimensionality-reduced feature sequence. This dimensionality reduction significantly reduces the computational parameters and overall computational cost of subsequent network components, laying the foundation for fast and lightweight custom branches.

[0042] Next, the system continues to perform deep temporal feature extraction on the dimensionality-reduced feature sequence. In this step, the acoustic model employs a network structure with lightweight memory capabilities, such as a two-layer feedforward sequential memory network (FSMN) structure, to deeply model the contextual relationships between speech frames. Through this network operation that captures the dependencies between speech sequences, the system can further transform the dimensionality-reduced acoustic feature sequence into a temporal feature sequence containing rich contextual information and dynamic temporal changes.

[0043] It should be noted that, compared to traditional large and complex recurrent neural networks, the lightweight structure used in this embodiment of the invention for extracting temporal features not only has low computational overhead, but is also extremely friendly to practical engineering engine integration.

[0044] Following this, to generate the final acoustic modeling result, the system needs to fuse and map the temporal feature sequence and language embedding features to obtain a multilingual phoneme sequence. Specifically, the system combines the temporal feature sequence containing temporal information obtained in the previous step with the language embedding features used to indicate a specific language at the feature level, such as by concatenation or addition. Then, through the output layer at the end of the acoustic model, i.e., the mapping layer, such as a conv1d layer, the fused features are directly projected onto the preset multilingual phonetic unit space, thereby calculating the probability distribution of each phoneme at each time step. Thanks to the prior guidance of the language embedding features, this mapping process can ultimately accurately output a multilingual phoneme sequence that avoids crosstalk between different languages, which can be used by the backend finite state transition network for efficient path search.

[0045] In this embodiment of the invention, a cascaded processing logic of downsampling dimensionality reduction, temporal feature extraction, and language fusion mapping is designed within the acoustic model. This not only significantly reduces the overall computational load of the customized branch by utilizing downsampling operations, ensuring the system's extreme response speed and lightweight characteristics, but also effectively captures the temporal dependencies of the speech data. Furthermore, by combining language embedding features in the final mapping stage, it greatly ensures the high purity and high accuracy of phoneme output in multilingual mixed modeling scenarios.

[0046] Based on the above embodiments, the acoustic model is trained using the following steps: Obtain the sample speech feature sequence carrying phoneme labels; The sample speech feature sequence is masked according to the time masking parameter to obtain the masked speech feature sequence; the time masking parameter is determined by the pronunciation duration of the phoneme unit in the sample speech feature sequence. An acoustic model is trained based on masked speech feature sequences and phoneme labels.

[0047] Specifically, to further improve the feature extraction capability and generalization performance of the acoustic model in the customized branch for multilingual pronunciation units, this embodiment of the invention introduces an optimization algorithm for the CTC (connectionist temporal classification) structure, such as the CRCTC (consistency-regularized connectionist temporal classification) algorithm, during the training process of the acoustic model to improve model performance. The training process of the acoustic model may specifically include: In detail, the training process begins with acquiring sample speech feature sequences carrying phoneme labels. That is, during the offline training or fine-tuning phase of the acoustic model, the system needs to construct or load a large amount of supervised training data. Here, the sample speech feature sequence refers to a data matrix or vector sequence reflecting the physical characteristics of speech, formed after front-end processing of a large batch of sound samples used for training (consistent with the input format in the inference phase mentioned above). The corresponding phoneme labels are correct pronunciation sequences generated manually or through alignment tools; they serve as the benchmark for model training, providing the acoustic model with a precise input-output mapping.

[0048] Next, to improve the model's robustness to local feature interference, the system masks the sample speech feature sequence according to the temporal masking parameters, obtaining a masked speech feature sequence. Specifically, before inputting the sample speech feature sequence into the model for processing, the system masks consecutive feature frames along the time dimension, that is, it masks several consecutive frames of features, such as setting feature values ​​to zero or replacing them with specific constants, thereby generating a masked speech feature sequence. This forces the model not to rely excessively on local short-term features during training, but rather to infer the current pronunciation more through contextual dependencies, such as the role of the FSMN structure.

[0049] However, it is worth noting that since the acoustic model under the custom branch uses phoneme units as the basic modeling units (phonemes are the smallest sounding units in language that can distinguish semantics), and the actual sounding duration of each such sounding unit on the time axis, i.e., the number of feature frames occupied, is usually small, the time masking parameter that determines the size of the mask span, such as the time masking parameter tmask (time masking) in the CRCTC algorithm, cannot be set blindly, but needs to be strictly determined according to the sounding duration of the phoneme unit. It usually tends to take a small value, so as to effectively prevent the mask window from being too large and erasing all the information of multiple consecutive complete phonemes, which would cause the model to fall into the problem of not being able to converge or learning incorrect features.

[0050] After this, the system can train an acoustic model using the masked speech feature sequence and phoneme labels. Specifically, after generating masked data with a reasonable occlusion range, the system inputs it into the acoustic model to be trained, prompting the model to output a predicted phoneme probability distribution for the current sample speech feature sequence. Then, it calculates the loss function using the actual phoneme labels and continuously iterates and updates the model's internal weight parameters through backpropagation. After extensive iterative optimization, the system finally solidifies and outputs an acoustic model with strong fault tolerance and high-precision phoneme prediction capabilities, which can then be directly deployed to the second decoding branch.

[0051] In this embodiment of the invention, by introducing a mask training strategy based on dynamic matching of time mask parameters for phoneme pronunciation duration during the training process of the acoustic model, not only is the problem of excessive loss of local key semantic features caused by excessively large mask scale effectively avoided, but the ability of customized branches to capture multilingual pronunciation details, as well as the generalization and robustness performance to complex and incomplete acoustic environments, is also greatly enhanced, thus consolidating the precision foundation of multilingual lightweight customization from the source of training.

[0052] Based on the above embodiments, the finite state transition network is constructed based on a preset phoneme catalog and pronunciation dictionary, as well as a language model; the language model includes a general language model and a customized language model, and the language model is trained based on the following steps: Obtain customized and general corpus texts; An initial language model is obtained by training on a customized corpus of texts, and based on the initial language model, the perplexity score of each general statement in a general corpus of texts is determined. Based on the perplexity score, target general texts are selected from general corpus texts. Based on the customized corpus texts and the target general texts, a customized language model is trained. A general language model is trained based on a general corpus of text.

[0053] Specifically, Figure 3 This is an example diagram of the second decoding branch provided by the present invention, such as... Figure 3 As shown, finite state transition networks, such as TLG.fst (Token, Lexicon, Grammar . finite state transducer), are constructed based on a pre-defined phoneme catalog and pronunciation dictionary, as well as an external language model.

[0054] In detail, when constructing a finite state transition network, the system needs to integrate three core components: first, a phoneme index token, which defines the set of basic phonetic units that the system can recognize; second, a pronunciation dictionary lexicon, which serves as the core bridge connecting the phoneme probability distribution output by the acoustic model and the final decoded text, recording the mapping relationship from words to phonemes and supporting all words in a customized sentence; and third, a language model grammar, which provides the transition probabilities of word sequences in a specific context. Under this architecture, the system can achieve the customization function of new vocabulary or domain-specific slang by quickly customizing and training the language model without changing the large number of acoustic model parameters.

[0055] To balance the richness of general scenarios with the accuracy of customized scenarios, the language model in this embodiment of the invention may include a general language model and a customized language model. The general language model, such as the main N-gram, is mainly used to provide broader coverage of general vocabulary and is not limited to customized corpora; the customized language model, such as the customized N-gram, is specifically optimized for vertical domains or specific utterances to improve the recognition effect on customized sentences.

[0056] To ensure the lightweight nature of the customized resource package, the language model is trained based on the following steps: First, the system acquires both customized and general-purpose text corpora. Specifically, when preparing training data, the system collects customized text corpora highly relevant to specific vertical scenarios, such as specific instructions for smart homes and sets of professional report scripts, as well as general-purpose text corpora containing massive amounts of daily conversations and news broadcasts. However, due to the sheer volume of the general-purpose corpus, incorporating it entirely into the customized branches would result in an overly bloated resource package.

[0057] Therefore, the system first trains an initial language model based on a customized corpus of text, and then uses this model to calculate the perplexity score of each general statement in a general corpus of text. In practice, the system first uses the customized corpus of text alone. Train a basic initial language model, such as a 5-gram model. Subsequently, the model was used. For massive general-purpose text corpus Sentence by sentence is scored, and the perplexity score of each general statement is calculated. The score directly reflects the degree of fit between the corresponding general statement and the customized scenario. That is, the lower the score, the more similar the grammatical logic or word usage of the general statement is to the customized corpus.

[0058] Next, the system filters target general text from the general corpus based on perplexity scores, and trains a customized language model based on the customized corpus and the target general text. Specifically, in this filtering step, the system sets a perplexity threshold. For example, setting a perplexity threshold equal to the initial threshold This threshold can be referenced from the initial language model. Customized corpus text The average or highest calculated above Scoring is set. The system will assign a perplexity score below the perplexity threshold. High-quality general-purpose sentences are selected and used as target general-purpose text. This refers to a concise and general corpus that is highly relevant to customized scenarios.

[0059] Subsequently, the system utilizes the filtered target general text Train a temporary language model, such as a lightweight 3-gram model. and use the model Combined with end-to-end speech recognition model (e.g., Whisper), for customized corpus texts Perform simulated joint decoding to obtain the simulated recognition effect under the current filtered data. Next, the system will display the simulated recognition results. The expected model recognition performance Perform comparative verification to determine if it meets expectations. If the simulated recognition effect... If the expected results are not met, the system will automatically adjust the set perplexity threshold. If the amount of data currently filtered is very small, increase the perplexity threshold. Introduce more data; if the data volume is very large, reduce the perplexity threshold. To streamline the data and return it to a general corpus text for reprocessing. Filtering and temporary language models Verification; if the simulated recognition effect If the expected outcome is achieved, the system stops iterating and displays the target generic text that meets the conditions at this point. The final output is a streamlined, high-quality general-purpose corpus.

[0060] Finally, the system will use the original customized corpus text. This part of the target general text has been rigorously validated and is highly relevant. By merging these datasets, an expanded, high-quality dataset is obtained. Joint training with this dataset ultimately yields a customized language model for practical decoding. This validated closed-loop iterative approach significantly enriches sentence diversity while ensuring domain-specific tailoring.

[0061] Meanwhile, to ensure the system maintains strong basic recognition capabilities when processing non-customized speech, it also trains a general language model based on a general corpus of text. That is, a general language model containing extensive linguistic knowledge can be trained using unedited general corpus text, serving as a powerful supplement to the customized language model.

[0062] In this embodiment of the invention, by introducing a general corpus selection strategy based on perplexity scoring into a finite state transition network, it is possible not only to achieve rapid customization of a language model for a specific domain without changing the acoustic model parameters, but also to accurately extract target general texts highly relevant to the customized scenario from a massive general corpus of texts for knowledge supplementation. This greatly enriches the vocabulary and sentence diversity of the customized branch while strictly controlling the size of the customized resource package and achieving lightweight deployment, thus avoiding computational redundancy caused by irrelevant corpus.

[0063] Based on the above embodiments, determining the target recognition text corresponding to multilingual speech from the first decoded text and the second decoded text based on the first feature parameter and the second feature parameter includes: Based on the first decoded text and the second decoded text, a score evaluation is performed to obtain the first language score corresponding to the first decoded text and the second language score corresponding to the second decoded text. The first feature parameter and the second feature parameter are aligned to obtain the first aligned parameter and the second aligned parameter. The feature parameters include acoustic score, word length and number of frames. The first alignment parameter and the first language score, as well as the second alignment parameter and the second language score, are input into the classification decision model to make a classification decision, and the classification result is obtained. Based on the classification result, the target recognition text is determined from the first decoded text and the second decoded text.

[0064] Specifically, the process of determining the target recognition text from the first decoded text and the second decoded text based on the first feature parameter and the second feature parameter includes: Figure 4 This is a flowchart of the overall speech recognition method provided by the present invention, as follows: Figure 4As shown, in practical applications, the general decoding branch (ed end) and the customized branch (ce end) run in parallel and output decoding results respectively. Next, the system evaluates the scores of the first decoded text output by the ed end and the second decoded text output by the ce end, obtaining the first language score for the first decoded text and the second language score for the second decoded text. That is, to judge which decoding result from the two branches better conforms to the expression habits of a specific customized scenario from the perspectives of syntax, logic, and semantics, the system introduces a dedicated two-pass language model (2-pass language model) for text scoring. Specifically, the first and second decoded texts are input into this model for scoring evaluation. Through the model's probability calculation, the system can obtain a first language score (ed end language score) reflecting the coherence and rationality of the first decoded text, and a second language score (ce end language score) reflecting the fit of the second decoded text in the customized context.

[0065] It's important to note that the purpose of the second-pass language model is to accurately output the language score of the decoded text. The actual training corpus configuration can be flexibly adjusted according to the specific needs of the customized scenario. For example, when customized recognition faces resource package size limitations and the customized fixed phrases don't change much, such as multilingual expressions like "turn on the air conditioner," "turn on the air conditioner," and "all climatisation" in smart home scenarios, the training corpus for the second-pass language model can directly use customized sentence patterns summarized from various expressions. However, when the customized fixed phrases may change, such as in professional report scenarios, the training corpus for the second-pass language model can adopt the aforementioned perplexity-based general corpus selection strategy, supplementing the customized sentence patterns with potentially relevant general corpus. In this case, the amount of supplementation can be strictly constrained by the resource package size limit; if there is no resource package size limit, the cleaned general corpus can also be used directly for model training.

[0066] Subsequently, to eliminate the differences in underlying model structure and parameter scale between different decoding branches, the system also needs to align the feature parameters in the decoding results output by the two branches to obtain aligned feature parameters, namely the first aligned parameters and the second aligned parameters. Specifically, during dual-end decoding, in addition to the decoded text itself, each branch will also output corresponding feature parameters. These parameters include acoustic scores (ed-end acoustic scores and ce-end acoustic scores) reflecting the acoustic model's confidence in the matching of input acoustic features with output phonemes, word length (decoding length) reflecting the length feature of the output text, and the number of frames reflecting the speech time dimension. Furthermore, since the scale space of the model in the general decoding branch and the model in the customized branch may be different when calculating the acoustic score, and the word segmentation method of the output text may also be different, the system needs to perform parameter alignment processing on these original features, that is, normalization and scale unification processing. For example, scale unification is performed on the acoustic score, and word segmentation unification is performed on the word length of the decoded text. This alignment operation eliminates dimensional and scale differences, resulting in first and second alignment parameters under the same comparison benchmark, providing standardized data input for subsequent fair adjudication.

[0067] Finally, the system inputs the first alignment parameter and the first language score, as well as the second alignment parameter and the second language score, into a classification decision model (PK model) for classification decision-making, obtains the classification result, and determines the target recognition text from the first and second decoded texts based on the classification result. That is, the first alignment parameter and the first language score, representing the performance of the general decoding branch, and the second alignment parameter and the second language score, representing the performance of the customized branch, are uniformly aggregated, such as by concatenation, to obtain a comprehensive feature parameter. This parameter is input into a pre-trained classification decision model, such as a decision tree model built based on a lightweight gradient boosting machine (LGBM). This model traverses through high-dimensional information features to find the optimal split point for classification decision-making, thereby predicting whether the current speech tends towards a specific fixed utterance or ordinary non-customized utterance. The model ultimately outputs a binary classification result, such as recommending whether to use the decoding result from the CE end or the decoding result from the ED end. The system directly chooses between the first and second decoded texts based on the classification result to select the final target recognition text.

[0068] In this embodiment of the invention, the decision-making mechanism based on the dual-end decoding results not only comprehensively and objectively evaluates the recognition quality of the two branches from multiple dimensions, but also achieves intelligent diversion of fixed customized scripts and general scripts through a lightweight decision model. This ensures that when using the customization function, specific professional terms can accurately follow the customized branch, while ordinary non-customized scripts still maintain the extremely high recognition level of the original general branch. The two do not interfere with each other, achieving the best balance between customized and general effects.

[0069] Based on the above embodiments, the classification decision model is trained using the following steps: Identify multiple sample speech texts, as well as the first sample decoded text, the second sample decoded text, and the sample annotation text corresponding to each sample speech text; For each sample speech, determine the matching degree between the first sample decoded text and the second sample decoded text and the sample annotation text, respectively; If the matching degree of the first sample decoded text is higher than that of the second sample decoded text, the classification label of the corresponding sample speech is determined as the first label; If the matching degree of the second sample decoded text is higher than that of the first sample decoded text, the classification label of the corresponding sample speech is determined as the second label; If the matching degree of the first sample decoded text is equal to the matching degree of the second sample decoded text, the classification label of the corresponding sample speech is determined as the third label; Based on multiple sample speech and their corresponding classification labels, the initial decision model is trained to obtain a classification decision model; the initial decision model is built on the basis of gradient boosting decision tree.

[0070] Specifically, in order to enable the classification decision model to make the optimal choice between the decoding results output by the general decoding branch (ed end) and the customized branch (ce end) in practical applications, in this embodiment of the invention, the classification decision model can be pre-trained. The training process of the model specifically includes the following steps: First, in the data preparation phase, the system needs to acquire multiple sample speech texts, as well as the first sample decoded text, second sample decoded text, and sample annotation text for each sample speech text. Specifically, in this stage, the system collects a massive amount of sample speech texts covering diverse scenarios (including general scenarios and specific customized scenarios). For each sample speech text, the system inputs it into a dual-end architecture for forward inference decoding, thereby obtaining the first sample decoded text output by the general decoding branch and the second sample decoded text output by the customized branch. Simultaneously, the system also acquires sample annotation text (i.e., baseline text), which represents the actual pronunciation content of the sample speech text and is precisely verified or cleaned by humans, serving as an absolute benchmark for subsequent evaluation of model performance.

[0071] Subsequently, to determine which branch performs better in different speech scenarios, the system compares the first and second sample decoded texts of each speech sample with the sample annotation text to determine the matching degree between the inference result and the baseline text. This matching degree aims to objectively quantify the accuracy of the decoding result, such as by calculating the word error rate or edit distance. The system rigorously compares the dual-output sample decoded texts with the baseline text to clearly evaluate the recognition accuracy of the general decoding branch and the customized branch on the current speech sample.

[0072] After obtaining an accurate match, the system enters an automated data annotation process. That is, based on the comparison results, a guiding label is assigned to each sample speech. This process specifically includes the following three scenarios: Firstly, if the matching degree of the first sample decoded text is higher than that of the second sample decoded text, the corresponding sample speech is designated as the first identifier. In other words, if the current sample speech belongs to non-customized Mandarin, and the general decoding branch decodes it more accurately (i.e., the ed is more accurate), the system will label it as the first identifier representing the selection of the general decoding branch, such as the value "0".

[0073] Secondly, if the matching degree of the second sample decoded text is higher than that of the first sample decoded text, the classification label of the corresponding sample speech will be determined as the second identifier. That is, when the current sample speech contains a large number of professional terms or fixed phrases, and the decoding result of the customized branch is better (i.e., CE is more accurate), the system will mark its classification label as the second identifier representing the preference for the customized branch, such as the value "1".

[0074] Third, if the matching degree of the first sample decoded text is equal to that of the second sample decoded text, the classification label of the corresponding sample speech will be determined as the third label. That is, when the output results of both ends are basically the same or completely the same, or when the accuracy is the same (ce and ed outputs are the same), the system will label its classification label as the third label, such as the value "-1", indicating that either one can be chosen or special weighting is performed in binary classification training.

[0075] After the classification labels are completed, the system can use the labeled sample speech to train the initial decision model and obtain the classification decision model.

[0076] In practice, the system uses the first and second sample feature parameters (corresponding to the first and second feature parameters in the inference stage) obtained after forward inference decoding of the sample speech as input features, and the automatically generated classification labels (0, 1, or -1) as supervision targets, uniformly feeding them into the pre-built initial decision model. To ensure the efficiency and lightweight nature of the decision-making process, the initial decision model adopts an algorithm architecture built on gradient boosting decision trees, such as the IGBM framework. During training, the initial decision model continuously traverses the complex information features in the general decoding branch and the customized branch, finding the optimal feature segmentation point by calculating information gain. After multiple rounds of iteration and gradient boosting, the system finally solidifies and outputs a classification decision model with strong generalization ability, used for efficient selection and decision-making during online inference.

[0077] In this embodiment of the invention, three-state classification labels are automatically generated by comparing the matching degree between the dual-end decoded text and the real labeled text. Multidimensional feature parameters are used to learn on the basis of gradient boosting decision tree. This not only greatly reduces the cost of manual feature labeling, but also gives full play to the advantages of tree model in terms of low computational overhead and strong ability to find the optimal split point when processing structured features. This enables the final classification decision model to accurately capture the feature boundary between general and customized language and make the optimal arbitration at runtime with extremely low latency.

[0078] The speech recognition device provided by the present invention is described below. The speech recognition device described below and the speech recognition method described above can be referred to in correspondence.

[0079] Figure 5 This is a schematic diagram of the structure of the voice recognition device provided by the present invention, as shown below. Figure 5 As shown, the device includes: Acquisition unit 510 is used to acquire the acoustic feature sequence of the multilingual speech to be identified; The first decoding unit 520 is used to input the acoustic feature sequence into the first decoding branch for decoding to obtain the first decoded text and its corresponding first feature parameter; The second decoding unit 530 is used to input the acoustic feature sequence into the second decoding branch for decoding to obtain the second decoded text and its corresponding second feature parameter; the second decoding branch is used to extract the multilingual phoneme sequence of the acoustic feature sequence and perform finite state transition decoding based on the multilingual phoneme sequence. Decision unit 540 is used to determine the target recognition text corresponding to the multilingual speech from the first decoded text and the second decoded text based on the first feature parameter and the second feature parameter.

[0080] The speech recognition device provided by this invention, by adding a lightweight customized branch based on multilingual phoneme sequences and finite state transition decoding to the general decoding branch, and using the feature parameters output by the two branches for decision-making, can achieve efficient and lightweight customization of vocabulary in specific domains without changing the parameters of the large main model network, while ensuring the powerful general speech recognition capabilities of the original large model. It effectively solves the problems of long multilingual customization cycle, large resource consumption, and easy interference between languages ​​in the existing technology, and greatly improves the efficiency and recognition accuracy of end-to-end speech recognition model in complex vertical business scenarios.

[0081] Based on the above embodiments, the second decoding branch includes an acoustic model and a finite state transition network; the second decoding unit 530 is used for: Determine the language embedding features of the target language corresponding to the multilingual speech; The acoustic feature sequence and the language embedding feature are input into the acoustic model for sequence modeling to obtain a multilingual phoneme sequence; Based on the multilingual phoneme sequence, a path search is performed in the finite state transition network to obtain the second decoded text and its corresponding second feature parameter.

[0082] Based on the above embodiments, the acoustic model is used to perform time-dimension downsampling processing on the acoustic feature sequence to obtain a dimensionality-reduced feature sequence, and to extract temporal features from the dimensionality-reduced feature sequence to obtain a temporal feature sequence. Based on the temporal feature sequence and the language embedding features, a fusion mapping is performed to obtain the multilingual phoneme sequence.

[0083] Based on the above embodiments, the device further includes a model training unit, used for: Obtain the sample speech feature sequence carrying phoneme labels; The sample speech feature sequence is masked according to the time masking parameter to obtain the masked speech feature sequence; the time masking parameter is determined by the pronunciation duration of the phoneme unit in the sample speech feature sequence. The acoustic model is trained based on the masked speech feature sequence and the phoneme labels.

[0084] Based on the above embodiments, the finite state transition network is constructed based on a preset phoneme catalog and pronunciation dictionary, as well as a language model; the language model includes a general language model and a customized language model, and the model training unit is further used for: Obtain customized and general corpus texts; An initial language model is trained based on the customized corpus text, and based on the initial language model, the perplexity score of each general statement in the general corpus text is determined. Based on the perplexity score, target general text is selected from the general corpus text, and the customized language model is trained based on the customized corpus text and the target general text; The general language model is trained based on the general corpus text.

[0085] Based on the above embodiments, the decision unit 540 is used for: Based on the first decoded text and the second decoded text, a score evaluation is performed to obtain the first language score corresponding to the first decoded text and the second language score corresponding to the second decoded text; The first feature parameter and the second feature parameter are subjected to parameter alignment processing to obtain the first aligned parameter and the second aligned parameter. The feature parameters include acoustic score, word length and frame number. The first alignment parameter and the first language score, as well as the second alignment parameter and the second language score, are input into the classification decision model to make a classification decision, thereby obtaining a classification result. Based on the classification result, the target recognition text is determined from the first decoded text and the second decoded text.

[0086] Based on the above embodiments, the model training unit is also used for: Identify multiple sample speech texts, as well as the first sample decoded text, the second sample decoded text, and the sample annotation text corresponding to each sample speech text; For each sample speech, determine the matching degree between the first sample decoded text and the second sample decoded text and the sample annotation text, respectively; If the matching degree of the first sample decoded text is higher than that of the second sample decoded text, the classification label of the corresponding sample speech is determined as the first identifier; If the matching degree of the second sample decoded text is higher than that of the first sample decoded text, the classification label of the corresponding sample speech is determined as the second identifier; If the matching degree of the first sample decoded text is equal to the matching degree of the second sample decoded text, the classification label of the corresponding sample speech is determined as the third identifier; Based on the multiple sample speech and their corresponding classification labels, the initial decision model is trained to obtain the classification decision model; the initial decision model is constructed on the basis of gradient boosting decision tree.

[0087] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a speech recognition method, which includes: acquiring an acoustic feature sequence of multilingual speech to be recognized; inputting the acoustic feature sequence into a first decoding branch for decoding to obtain a first decoded text and its corresponding first feature parameter; inputting the acoustic feature sequence into a second decoding branch for decoding to obtain a second decoded text and its corresponding second feature parameter; the second decoding branch is used to extract a multilingual phoneme sequence from the acoustic feature sequence and perform finite state transition decoding based on the multilingual phoneme sequence; and based on the first feature parameter and the second feature parameter, determining the target recognition text corresponding to the multilingual speech from the first decoded text and the second decoded text.

[0088] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the speech recognition method provided by the above methods, the method comprising: acquiring an acoustic feature sequence of multilingual speech to be recognized; inputting the acoustic feature sequence into a first decoding branch for decoding to obtain a first decoded text and its corresponding first feature parameter; inputting the acoustic feature sequence into a second decoding branch for decoding to obtain a second decoded text and its corresponding second feature parameter; the second decoding branch being used to extract a multilingual phoneme sequence of the acoustic feature sequence, and performing finite state transition decoding based on the multilingual phoneme sequence; and determining the target recognition text corresponding to the multilingual speech from the first decoded text and the second decoded text based on the first feature parameter and the second feature parameter.

[0090] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the speech recognition method provided by the methods described above. The method includes: acquiring an acoustic feature sequence of multilingual speech to be recognized; inputting the acoustic feature sequence into a first decoding branch for decoding to obtain a first decoded text and its corresponding first feature parameter; inputting the acoustic feature sequence into a second decoding branch for decoding to obtain a second decoded text and its corresponding second feature parameter; the second decoding branch is used to extract a multilingual phoneme sequence from the acoustic feature sequence and perform finite state transition decoding based on the multilingual phoneme sequence; and based on the first feature parameter and the second feature parameter, determining the target recognition text corresponding to the multilingual speech from the first decoded text and the second decoded text.

[0091] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech recognition method, characterized in that, include: Obtain the acoustic feature sequence of the multilingual speech to be identified; The acoustic feature sequence is input into the first decoding branch for decoding to obtain the first decoded text and its corresponding first feature parameter; The acoustic feature sequence is input into the second decoding branch for decoding to obtain the second decoded text and its corresponding second feature parameter; The second decoding branch is used to extract the multilingual phoneme sequence of the acoustic feature sequence and perform finite state transition decoding based on the multilingual phoneme sequence; Based on the first feature parameter and the second feature parameter, the target recognition text corresponding to the multilingual speech is determined from the first decoded text and the second decoded text.

2. The speech recognition method according to claim 1, characterized in that, The second decoding branch includes an acoustic model and a finite state transition network; The step of inputting the acoustic feature sequence into the second decoding branch for decoding to obtain the second decoded text and its corresponding second feature parameters includes: Determine the language embedding features of the target language corresponding to the multilingual speech; The acoustic feature sequence and the language embedding feature are input into the acoustic model for sequence modeling to obtain a multilingual phoneme sequence; Based on the multilingual phoneme sequence, a path search is performed in the finite state transition network to obtain the second decoded text and its corresponding second feature parameter.

3. The speech recognition method according to claim 2, characterized in that, The acoustic model is used to perform time-dimensional downsampling on the acoustic feature sequence to obtain a dimensionality-reduced feature sequence, and to extract temporal features from the dimensionality-reduced feature sequence to obtain a temporal feature sequence. Based on the temporal feature sequence and the language embedding features, a fusion mapping is performed to obtain the multilingual phoneme sequence.

4. The speech recognition method according to claim 2, characterized in that, The acoustic model is trained based on the following steps: Obtain the sample speech feature sequence carrying phoneme labels; The sample speech feature sequence is masked according to the time masking parameter to obtain the masked speech feature sequence; the time masking parameter is determined by the pronunciation duration of the phoneme unit in the sample speech feature sequence. The acoustic model is trained based on the masked speech feature sequence and the phoneme labels.

5. The speech recognition method according to any one of claims 2 to 4, characterized in that, The finite state transition network is constructed based on a pre-defined phoneme catalog and pronunciation dictionary, as well as a language model. The language model includes a general language model and a customized language model, which is trained based on the following steps: Obtain customized and general corpus texts; An initial language model is trained based on the customized corpus text, and based on the initial language model, the perplexity score of each general statement in the general corpus text is determined. Based on the perplexity score, target general text is selected from the general corpus text, and the customized language model is trained based on the customized corpus text and the target general text; The general language model is trained based on the general corpus text.

6. The speech recognition method according to any one of claims 1 to 4, characterized in that, The step of determining the target recognition text corresponding to the multilingual speech from the first decoded text and the second decoded text based on the first feature parameter and the second feature parameter includes: Based on the first decoded text and the second decoded text, a score evaluation is performed to obtain the first language score corresponding to the first decoded text and the second language score corresponding to the second decoded text; The first feature parameter and the second feature parameter are subjected to parameter alignment processing to obtain the first aligned parameter and the second aligned parameter. The feature parameters include acoustic score, word length and frame number. The first alignment parameter and the first language score, as well as the second alignment parameter and the second language score, are input into the classification decision model to make a classification decision, thereby obtaining a classification result. Based on the classification result, the target recognition text is determined from the first decoded text and the second decoded text.

7. The speech recognition method according to claim 6, characterized in that, The classification decision model is trained based on the following steps: Identify multiple sample speech texts, as well as the first sample decoded text, the second sample decoded text, and the sample annotation text corresponding to each sample speech text; For each sample speech, determine the matching degree between the first sample decoded text and the second sample decoded text and the sample annotation text, respectively; If the matching degree of the first sample decoded text is higher than that of the second sample decoded text, the classification label of the corresponding sample speech is determined as the first identifier; If the matching degree of the second sample decoded text is higher than that of the first sample decoded text, the classification label of the corresponding sample speech is determined as the second identifier; If the matching degree of the first sample decoded text is equal to the matching degree of the second sample decoded text, the classification label of the corresponding sample speech is determined as the third identifier; Based on the multiple sample speech and their corresponding classification labels, the initial decision model is trained to obtain the classification decision model; the initial decision model is constructed on the basis of gradient boosting decision tree.

8. A voice recognition device, characterized in that, include: The acquisition unit is used to acquire the acoustic feature sequence of the multilingual speech to be identified; The first decoding unit is used to input the acoustic feature sequence into the first decoding branch for decoding to obtain the first decoded text and its corresponding first feature parameter; The second decoding unit is used to input the acoustic feature sequence into the second decoding branch for decoding, to obtain the second decoded text and its corresponding second feature parameter; The second decoding branch is used to extract the multilingual phoneme sequence of the acoustic feature sequence and perform finite state transition decoding based on the multilingual phoneme sequence; The decision unit is configured to determine the target recognition text corresponding to the multilingual speech from the first decoded text and the second decoded text based on the first feature parameter and the second feature parameter.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the speech recognition method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech recognition method as described in any one of claims 1 to 7.

Images