Speech recognition model inference method and apparatus, device, and medium

Abstract

The application provides a speech recognition model inference method and device, equipment and medium, the speech recognition model includes a small speech model and a large speech model, comprising: obtaining speech data; input the speech data into the small speech model, obtain the inference lattice output by the small speech model, input the speech data into the large speech model, obtain the word token probability distribution of each time output by the large speech model, the inference lattice includes multiple lattice time slots, each lattice time slot includes multiple candidate word tokens and the probability of multiple candidate word tokens at a time; fuse the lattice time slot at the first time and the word token probability distribution at the first time to determine the speech recognition result of the speech data at the first time.Combining the word token probability distribution of the large speech model and the inference lattice of the small speech model to determine the speech recognition result can effectively improve the speech recognition accuracy of the speech recognition model and reduce the frequency of recognition errors, recognition loss and other problems.

Description

Technical Field

[0001] Embodiments of this application relate to a reasoning method for a speech recognition model, a reasoning device for a speech recognition model, an electronic device, and a non-transitory computer-readable storage medium. Background Technology

[0002] Speech recognition refers to the process of converting speech data into text, and the text corresponding to the speech data can also be the speech recognition result. Speech recognition is required in many scenarios; for example, in teaching scenarios, especially those with a high degree of language mixing, accurate and fast speech recognition is particularly important. Summary of the Invention

[0003] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] At least one embodiment of this application provides an inference method for a speech recognition model, the speech recognition model including a small speech model and a large speech model, the method comprising: acquiring speech data, wherein the speech data includes speech data mixed with multiple languages; inputting the speech data into the small speech model to acquire an inference grid output by the small speech model; and inputting the speech data into the large speech model to acquire the lexical probability distribution at each time step output by the large speech model, wherein the inference grid includes multiple grid time slots, each grid time slot including multiple candidate lexical units and the probabilities of the multiple candidate lexical units at a time step; fusing the grid time slot at a first time step and the lexical probability distribution at the first time step to determine the speech recognition result of the speech data at the first time step, thereby obtaining the speech recognition result of the speech data, wherein the first time step is any time step in the inference process of the speech recognition model.

[0005] At least one embodiment of this application provides an inference apparatus for a speech recognition model, the speech recognition model including a small speech model and a large speech model, the apparatus comprising: an acquisition module configured to acquire speech data; an inference module configured to: input the speech data into the small speech model to acquire an inference grid output by the small speech model, and input the speech data into the large speech model to acquire the lexical probability distribution at each time step output by the large speech model, wherein the inference grid includes multiple grid time slots, each grid time slot including multiple candidate lexical units and the probabilities of the multiple candidate lexical units at a time step; and a determination module configured to: fuse the grid time slot at a first time step and the lexical probability distribution at the first time step to determine the speech recognition result of the speech data at the first time step, thereby obtaining the speech recognition result of the speech data, wherein the first time step is any time step in the inference process of the speech recognition model.

[0006] At least one embodiment of this application provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute a reasoning method for a speech recognition model provided in at least one embodiment of this application.

[0007] At least one embodiment of this application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute a reasoning method of a speech recognition model provided in at least one embodiment of this application.

[0008] At least one embodiment of this application provides a computer program product, including a computer program that, when executed by a processor, implements the inference method of the speech recognition model provided in at least one embodiment of this application. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this application, and are not intended to limit this application.

[0010] Figure 1 This illustration schematically depicts an application scenario of a reasoning method for a speech recognition model provided in at least one embodiment of this application;

[0011] Figure 2 This illustration schematically depicts an application scenario of a reasoning method for a speech recognition model provided in at least one embodiment of this application;

[0012] Figure 3The illustration shows a flowchart of an inference method for a speech recognition model provided in at least one embodiment of this application;

[0013] Figure 4 The illustration shows a schematic diagram of a reasoning grid provided in at least one embodiment of this application;

[0014] Figure 5 The illustration shows a flowchart of determining a fusion probability distribution provided by at least one embodiment of this application;

[0015] Figure 6 The illustration shows a schematic diagram of a model distillation principle provided in at least one embodiment of this application;

[0016] Figure 7 This illustration schematically shows a distillation principle diagram of a self-attention layer provided in at least one embodiment of this application;

[0017] Figure 8 This illustration schematically shows a distillation principle diagram of an activation layer provided in at least one embodiment of this application;

[0018] Figure 9 This schematic diagram illustrates the structure of an inference device for a speech recognition model provided in at least one embodiment of this application;

[0019] Figure 10 The schematic diagram illustrates the structure of an inference electronic device that implements a speech recognition model according to at least one embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the described embodiments of this application without creative effort are within the scope of protection of this application.

[0021] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0022] To keep the following description of the embodiments of this application clear and concise, detailed descriptions of some known functions and known components have been omitted.

[0023] Speech recognition refers to the process of converting speech data into text, and the text corresponding to the speech data can also be the speech recognition result. Speech recognition is required in many scenarios; for example, in teaching scenarios, especially those with a high degree of language mixing, accurate and fast speech recognition is particularly important.

[0024] The industry typically uses large speech models for speech recognition. However, the inference process of large speech models often encounters problems such as repeated recognition of text (insertion problem), missing recognition of text (deletion problem), and incorrect recognition of text (substitution problem). The reason for these problems is that during the inference process of large speech models, the features and information carried by the speech data have been flattened by the speech encoder, losing temporal information. Furthermore, if an erroneous word is output at a certain temporal position during the inference process of a large speech model, that erroneous word will generate more erroneous words.

[0025] In view of the above, at least one embodiment of this application provides an inference method for a speech recognition model, the speech recognition model including a small speech model and a large speech model, the method including: acquiring speech data, the speech data including speech data mixed from multiple languages; inputting the speech data into the small speech model to acquire an inference grid output by the small speech model; and inputting the speech data into the large speech model to acquire the word probability distribution at each time step output by the large speech model, the inference grid including multiple grid time slots, each grid time slot including multiple candidate words and the probabilities of multiple candidate words at a time step; fusing the grid time slots at a first time step and the word probability distribution at the first time step to determine the speech recognition result of the speech data at the first time step, so as to obtain the speech recognition result of the speech data, wherein the first time step is any time step in the inference process of the speech recognition model.

[0026] In the embodiments of this application, a large speech model and a small speech model are used together for speech recognition. Since the inference grid output by the small speech model contains temporal information, and the small speech model can also incorporate domain knowledge, combining the word probability distribution of the large speech model and the inference grid of the small speech model... Figure 1 Determining the speech recognition result can effectively improve the speech recognition accuracy of the speech recognition model, such as in multilingual mixed scenarios, and reduce the frequency of recognition errors and loss of recognition.

[0027] Embodiments of this application also provide an inference device for a speech recognition model, an electronic device, a computer-readable storage medium, and a computer program product. The inference method for the speech recognition model described above can be applied to the inference device for the speech recognition model provided in the embodiments of this application, and the inference device for the speech recognition model can be configured on an electronic device. The electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a mobile phone, headset, tablet computer, in-vehicle equipment, etc.

[0028] The embodiments of this application will be described in detail below with reference to the accompanying drawings, but this application is not limited to these specific embodiments.

[0029] Figure 1 The diagram illustrates an application scenario of a reasoning method for a speech recognition model provided in at least one embodiment of this application.

[0030] like Figure 1 As shown, in at least one embodiment of this application, speech data 101 is input into a large speech model 102 and a small speech model 103. The small speech model 103 is obtained by model distillation of the large speech model 102. Then, the inference lattice 105 output by the small speech model 103 is combined with the output of the large speech model 102 by a probability fusion algorithm 104, and then a beam search algorithm 106 is performed to obtain the speech recognition result 107.

[0031] Figure 2 The diagram illustrates an application scenario of a reasoning method for a speech recognition model provided in at least one embodiment of this application.

[0032] like Figure 2 As shown, the speech recognition model includes a small speech model 103 and a large speech model 102. The small speech model 103 can be a small speech model optimized by domain knowledge (e.g., domain knowledge related to multilingual teaching scenarios), and the large speech model 102 can be a large speech model of a general domain.

[0033] In some embodiments, the small speech model 103 can receive speech data 101, perform feature calculation on the speech data 101, such as FBank feature calculation 201, obtain FBank feature 202, and then input the FBank feature 202 into the second encoder 203. After the encoding processing of the second encoder 203, it is input into the CTC decoder 204 to obtain the inference grid 105. The inference grid 105 includes the word probability distribution 210 of the small speech model.

[0034] The large speech model 102 can input the FBank features 202 into the first encoder 207, input the output of the first encoder 207 into the speech adapter 208, and the large speech model 102 can also perform word vectorization 206 on the decoded words 205 output by the CTC decoder 204, and input the output of the speech adapter 208 and the output after word vectorization 206 into the large language model 209 to obtain the word probability distribution 211 of the large speech model.

[0035] In some embodiments, a probability fusion algorithm 104 is performed on the word probability distribution 210 of the small speech model and the word probability distribution 211 of the large speech model to obtain a speech recognition result 107.

[0036] In some embodiments, a bundle search algorithm 106 may also be performed, such as a lexical-level dynamic pruning algorithm 212 and a vocabulary-level online pruning algorithm 213.

[0037] Figure 3 This illustration schematically shows a flowchart of an inference method for a speech recognition model provided in at least one embodiment of this application, such as... Figure 3 As shown, the method specifically includes:

[0038] S301: Acquire voice data.

[0039] In one or more embodiments of this application, voice data can be understood as data that has a voice recognition requirement and is to be recognized. For example, voice data can be a voice waveform file.

[0040] In different speech recognition scenarios, the sources of speech data can vary. For example, in a teaching scenario, speech data can be the teacher's speech data during the teaching process, and the speech data can include speech data from multiple languages.

[0041] S302: Input the speech data into the small speech model to obtain the inference trellis output by the small speech model, and input the speech data into the large speech model to obtain the word probability distribution at each time step output by the large speech model.

[0042] The training and inference processes of the small speech model will be introduced below.

[0043] Considering that it is difficult to inject domain knowledge into large speech models and that large speech models are prone to problems such as the curse of forgetting, in one or more embodiments of this application, a small speech model that integrates specific domain knowledge (such as a domain language model) is used as an auxiliary module of the large speech model to perform speech recognition together, thereby enhancing the speech recognition accuracy of the speech recognition model in a specific domain.

[0044] During the training of the small speech model, the original speech with text annotations, such as historical speech data from teaching scenarios with manual annotations, is input into the encoding layer of the small speech model, namely the second encoder (e.g., the Confomer-Encoder), to complete the alignment of word temporal positions. Then, it passes through the softmax layer and combines the text annotations corresponding to the original speech with the loss function to achieve frame-level classification training of the small speech model.

[0045] In one or more embodiments of this application, a small language model is injected with a specific domain knowledge source. For example, a domain language model is used to generate specific domain knowledge, such as lecture texts for multilingual teaching scenarios, to construct a domain knowledge source, enabling the small speech model to perform reasoning with the help of domain knowledge.

[0046] During the inference process of the small speech model, the small speech model can perform Viterbi search on the speech data. During the Viterbi search, multiple candidate lexical units are retained and the inference trellis is output.

[0047] In one or more embodiments of this application, the inference grid includes multiple grid time slots, each grid time slot including multiple candidate tokens and their probabilities at a given time. That is, in the Viterbi search process based on dynamic programming, not only the optimal path but also the suboptimal path and its probability are retained, forming multiple paths including multiple possible token sequences and their probabilities (also known as confidence levels). The inference grid is formed by clustering the time intervals and tokens of adjacent nodes in the optimal and suboptimal paths.

[0048] Figure 4 The illustration shows a schematic diagram of a reasoning lattice provided in at least one embodiment of this application.

[0049] like Figure 4 As shown, the reasoning graph can be a directed graph composed of nodes and edges. Each node represents a time node, and two adjacent nodes constitute a time point (e.g., a time point can correspond to one reasoning step). Each edge includes a candidate word and its probability. Each edge can also include the start and end time nodes of the candidate word. For example, each edge can include a tuple. , Indicates candidate word elements, This represents the probability of the candidate word (i.e., the probability of outputting the candidate word in this inference). This represents the time corresponding to the candidate lexical unit (i.e., the corresponding inference). Thus, by using multiple time slots in the inference grid, which consist of adjacent nodes and edges connecting adjacent nodes, multiple inference results obtained sequentially along the time path during the inference process of the small speech model can be represented.

[0050] In addition to inputting the speech data into the small speech model to obtain the inference grid output by the small speech model, the speech data is also input into the large speech model to obtain the word probability distribution at each time step output by the large speech model (that is, the inference process of the large speech model for speech recognition can include multiple inferences, one inference corresponds to one time step, and the word probability distribution at that time step can be understood as the probability of each candidate word step output in that inference). In other words, during the inference process of the large speech model for speech data, an additional inference process of the small speech model for speech data is added, so that the large speech model can use the inference grid output by the small speech model to update the word probability distribution at each time step and improve the speech recognition effect.

[0051] S303: By fusing the grid time slots and the word probability distribution at the first moment, the speech recognition result of the speech data at the first moment is determined, so as to obtain the speech recognition result of the speech data.

[0052] In one or more embodiments of this application, by fusing the inference lattice of the small speech model and the word probability distribution at each time step of the large speech model, the word probability distribution is reconstructed, thereby improving the inference performance of the large speech model and curbing the problems of repetition and excessively short pauses.

[0053] Figure 5 The illustration shows a flowchart of determining a fusion probability distribution provided by at least one embodiment of this application.

[0054] The first moment can be any moment in the inference process of the speech recognition model. That is, at each inference moment of the speech recognition model, the inference results of the small speech model and the large speech model are integrated, so that the final inference result includes both the inference result with high accuracy obtained by the large speech model with a large number of model parameters and the inference result obtained by the small speech model with injected domain knowledge and temporal information, ensuring that the fusion can improve the speech recognition effect.

[0055] like Figure 5As shown, the speech recognition result of the speech data at the first moment is determined by fusing the grid time slot 501 and the word probability distribution 502 at the first moment, including: fusing the grid time slot 501 and the word probability distribution 502 at the first moment, determining the fused probability distribution 503, and determining the speech recognition result of the speech data at the first moment based on the fused probability distribution 503.

[0056] In other words, since the time slots of each grid in the inference grid output by the small speech model include the probabilities of multiple candidate words, and the word probability distribution output by the large speech model also includes the probabilities of multiple candidate words, the probability fusion algorithm 104 is used to fuse the probabilities of multiple candidate words output by the small speech model and the large speech model at the same time, and update the probability of candidate words at that time. This makes the fused probability distribution include both the inference results of the small speech model and the inference results of the large speech model. By utilizing the temporal information and domain knowledge carried by the small speech model, the overall performance of speech recognition is improved.

[0057] For example, the fusion probability distribution is determined by fusing the first time slot of the graph and the word probability distribution at the first time, including: determining a first probability random vector based on the probability of multiple candidate words in the first time slot of the graph in the inference graph; determining a second probability random vector based on the word probability distribution at the first time; determining a fusion weight based on the first distance between the first probability random vector and the second probability random vector; and fusing the first time slot of the graph and the word probability distribution at the first time based on the fusion weight to determine the fusion probability distribution.

[0058] In other words, the probabilities of multiple candidate words in the first time slot and the word probability distribution at the first time are converted into vectors. This vector (i.e., the first probability random vector or the second probability random vector) can represent the probability of each candidate word among the multiple candidate words at that time in vector form. By calculating the distance between the vectors, the distance is used to represent the difference between the inference results of the small speech model and the large speech model.

[0059] Continue as Figure 5 As shown, the probability corresponding to the candidate word in the first time slot 501 of the grid is defined as follows: The word probability distribution at the first moment is 502. ,right and Probability normalization is performed using the softmax formula, resulting in a first probability random vector. Second probability random vector .

[0060] By combining the distance between the first probability random vector and the second probability random vector, different fusion weights are assigned to the probabilities of candidate words output by the small speech model and the large speech model. The fusion weights are used to fuse the word probability distributions to obtain a fused probability distribution that is related to the inference results of both the large and small speech models. That is, the word probability distribution of the large speech model at this moment is updated using the inference trellis of the small speech model.

[0061] For example, considering that simplex manifolds are suitable for handling the probability distributions of the same random variable described from different perspectives, simplex manifolds are used to fuse the probability distributions of two word segments in small speech models and large speech models.

[0062] In this case, the definition Since it is a simplex manifold, and because this manifold is a set of points in a d-dimensional positive real vector space whose components sum to 1, it fits the normalization constraint of the probability distribution. Therefore, it satisfies the normalization condition. and It can be directly embedded into the above simplex manifold, i.e. .

[0063] In some possible implementations, the first distance can be the Wasserstein distance, that is, using the Wasserstein distance metric. and The difference between them. The Wasserstein distance can be:

[0064]

[0065] in, All transportation plans The transportation plan for the resulting set must satisfy the marginal distribution constraint: In other words, Indicates from The i-th component is "transported" to The "probability mass" of the j-th component is ensured by marginal distribution constraints to keep the probability distribution unchanged before and after transportation.

[0066] Distance via Wasserstein Define fusion weights to achieve weighted fusion on the simplex manifold, for example, let Representing the Wasserstein distance, the fusion weights can be... ,satisfy Furthermore, the fusion weights are dynamically adjusted based on the first distance: the smaller the first distance, the higher the fusion weights. The closer it is to 1, the more likely it is to be retained. The characteristic is that the greater the first distance, The closer it is to 1, the more likely it is to be retained. feature.

[0067] On a simplex manifold, for and The weighted average (i.e., geometric fusion) yields the following fusion probability distribution: Since the simplex is closed under linear combinations, the sum of the components after weighted averaging is still 1, and the components are non-negative, therefore... This ensures the rationality of the fusion probability distribution.

[0068] Thus, by leveraging the differences in inference results between the large and small speech models at various time points, fusion weights are rationally allocated, resulting in a more reasonable fusion probability distribution. Decoding is performed on the fusion probability distribution (e.g., Viterbi search) to obtain the inference result at that time point. The inference result of the small speech model is used to improve the inference process of the large speech model. The inference graph of the small speech model provides auxiliary functionality to the inference process of the large speech model, adjusting the probability of candidate words at various time points during the inference process. This significantly alleviates inference problems such as text repetition and text loss in speech recognition, improves the accuracy of speech recognition at each inference time point, and thus enhances the robustness of the speech recognition model.

[0069] As mentioned above, in one or more embodiments of this application, lexical-level pruning and vocabulary-level pruning can also be performed to ensure the efficient execution of the speech recognition process.

[0070] In some embodiments, lexical-level pruning can be dynamic lexical-level pruning based on conditional entropy. Based on the fusion probability distribution, a beam search is performed. At each time step of the beam search, the speech recognition model retains a corresponding number of candidate lexical units. At the second time step of the beam search, a first number of candidate lexical units corresponding to the second time step is determined based on the conditional entropy of the historical path. The first number is adjusted to a second number based on the conditional entropy and the historical conditional entropy of at least one historical time step. The second time step can be any time step of the beam search, and the conditional entropy is used to measure the uncertainty of the first number of candidate lexical units.

[0071] For example, in a given historical path At time 1, assuming the first quantity is M, the first quantity of candidate words corresponding to the second time step is... The conditional entropy based on historical paths is: ,in, It is a fusion probability distribution.

[0072] Thus, at each moment of the beam search (i.e., Viterbi search), the beam size is dynamically adjusted based on the conditional entropy at the current moment and the uncertainty of speech recognition at the current moment, thereby balancing search efficiency and search accuracy.

[0073] In some embodiments, in response to the conditional entropy being greater than the historical conditional entropy of at least one historical moment satisfying a first condition, the first quantity is increased to a second quantity; or in response to the conditional entropy being less than the historical conditional entropy of at least one historical moment satisfying a second condition, the first quantity is decreased to a second quantity.

[0074] The first condition can be understood as the conditional entropy at the second moment having higher uncertainty compared to the historical stage, while the second condition can be understood as the conditional entropy at the second moment having lower uncertainty compared to the historical stage. The logic of dynamically adjusting the number of candidate lexical units at the second moment (i.e., the size of the bundle constraint) can be understood as follows: when uncertainty is high, increase the first number to the second number, thereby increasing the bundle constraint width and retaining more candidates; when uncertainty is low, decrease the first number to the second number, thereby decreasing the bundle constraint width and accelerating the search.

[0075] For example, the dynamic range of the number of candidate lexical units (i.e., the bundle constraint width) at the second time step is defined as follows: This is mapped to the dynamic range via entropy normalization:

[0076]

[0077] in: It is the conditional entropy at the second moment; and These are the minimum and maximum values ​​of the conditional entropy at historical moments. Using a sliding window, we take the minimum and maximum values ​​of the conditional entropy from the N moments prior to the second moment. Thus, after entropy normalization mapping, we have... This ensures that the bundle constraint width is dynamic and controllable.

[0078] By combining the changes in conditional entropy, the "criticality" of the second time step is determined, and the expansion strategy for increasing or decreasing the bundle constraint width is adaptively adjusted. For example, a sliding window of length W is used to store the conditional entropy of the W most recent historical time steps. Calculate the statistical values ​​of the conditional entropy for W historical moments. ,in, Let W be the mean of the conditional entropy over W historical moments. Let be the standard deviation of the conditional entropy over W historical time points. Thus, by using the statistical values ​​of the conditional entropy over W historical time points, we can determine whether the conditional entropy at the current time point (i.e., the second time point) significantly deviates from the conditional entropy at the historical time points. The sliding window method described above can significantly reduce the computational load.

[0079] For example, by calculating the rate of change of conditional entropy Identify decision-sensitive time intervals where conditional entropy rapidly increases or decreases.

[0080] Thus, the first condition can be (That is, the conditional entropy is significantly higher than the historical conditional entropy, indicating high uncertainty), or, the first condition can be... (i.e., the conditional entropy rises rapidly). In this case, increase the first quantity to the second quantity by increasing the width of the constraint, for example, , It's a step size, retaining more candidates.

[0081] The second condition can be... (That is, the conditional entropy is significantly lower than the historical conditional entropy, and the uncertainty is small), or, the second condition can be... (i.e., the conditional entropy decreases rapidly). In this case, the first quantity is reduced to the second quantity by decreasing the bundle constraint width, for example... This speeds up the search.

[0082] In the process of adjusting the number of candidate terms, considering issues such as incomplete windows in the early stages of inference and potential path loss due to sudden changes in bundle constraint width, in order to ensure search stability, before inference... At time , the fixed bundle constraint width is Once the window stabilizes, switch to dynamically adjusting the bundle constraint width.

[0083] Furthermore, a smooth transition algorithm is employed when the bundle constraint width is dynamically adjusted. For example, a maximum step size limit is introduced when the bundle constraint width changes. ,in It is the maximum step size of change.

[0084] Furthermore, after adjusting the first quantity to the second quantity based on the conditional entropy and the historical conditional entropy of at least one historical moment, multiple inference paths corresponding to the second moment can be obtained, and multiple completed words included in the multiple inference paths can be determined. For each completed word among the multiple completed words, the word probability of the completed word is determined based on the conditional probability of the completed word in each inference path including the completed word. Based on the word probabilities corresponding to the multiple completed words respectively, the second quantity is adjusted to the third quantity.

[0085] In other words, after performing lexical-level dynamic pruning based on conditional entropy, lexical-level pruning can also be performed. Considering that multiple inference paths are retained simultaneously during the beam search process of the speech recognition model, and each inference path includes a different sequence of lexical terms, the completed lexical boundaries are determined for each inference path at each time step during the inference process. Based on the word probability of each completed word in each inference path, the number of candidate lexical terms at each time step is dynamically adjusted to retain inference paths containing more completed words as much as possible and discard inference paths containing incomplete words, thereby improving inference efficiency.

[0086] For example, the inference path includes the word sequence "I", "love", and "prob". The inference path is input into the word segmenter, which converts the word sequence of the inference path into words, determines the boundaries of completed words, and maintains a status flag for the inference path to record completed words. For example, when the inference path generates a new word, if the current word ends (i.e. becomes a completed word), such as when a space character, an end character, or the word segmenter determines it to be a completed word, then a new completed word is obtained; otherwise, the status flag of the inference path is marked as "incomplete".

[0087] For each completed word, the word probability of a completed word can be the logarithmic sum of the conditional probabilities of that completed word in each inference path that includes it. For example, the completed word "love" in... If any of these three reasoning paths exist, then the probability of the completed word "love" can be: For example, the perfected word "probability" in... If either of these two reasoning paths exists, then the word probability of the completed word "probability" can be... .

[0088] Thus, based on the word probabilities corresponding to multiple completed words, the number of candidate lexical units at each time step is adjusted to achieve lexical-level pruning. For example, based on the word probabilities corresponding to multiple completed words, the entropy change feature of the lexical distribution at the second time step is determined. In response to the entropy change feature indicating an increase in uncertainty at the second time step, the second number is increased to the third number; in response to the entropy change feature indicating a decrease in uncertainty at the second time step, the second number is decreased to the third number; or in response to the entropy change feature indicating that the uncertainty at the second time step is in a stable state, the second number is maintained.

[0089] In other words, when uncertainty increases sharply, the number of words to retain is increased; when uncertainty decreases sharply, aggressive pruning methods are adopted, such as retaining only the top-1 words; and when in a stable state, the pruning strategy from the previous time step is maintained. Thus, by combining the entropy changes at different times during the inference process, the bundle constraint width is dynamically adjusted.

[0090] For example, performing probability distribution normalization calculations at the vocabulary level:

[0091]

[0092] in, It represents the set of all completed words at the current moment.

[0093] Next, Shannon entropy is calculated, and the window entropy is defined as... Based on word probability Calculate the word distribution entropy at the current time:

[0094]

[0095] Define the instantaneous change of entropy as The instantaneous volatility of entropy is defined as ,in, This represents the mean window entropy at historical moments corresponding to the sliding window. This represents the standard deviation of the window entropy at any given historical moment corresponding to the sliding window.

[0096] Entropy change characteristics can include instantaneous changes in entropy, when When this occurs, it indicates that the entropy change characteristic represents an increase in uncertainty at that moment; that is, uncertainty increases sharply, with the second increase becoming the third increase. When this occurs, it indicates that the entropy change characteristic represents a decrease in uncertainty at that moment; that is, the uncertainty drops sharply, decreasing the second quantity to the third quantity. When the entropy change characteristic represents the uncertainty at that moment, it indicates that the uncertainty is in a stable state and maintains the second quantity.

[0097] Furthermore, entropy change characteristics can also include the instantaneous fluctuation rate of entropy, when When the entropy change feature indicates that there is a local high fluctuation at that moment, the current vocabulary diversity is maintained, that is, no vocabulary layer pruning is performed.

[0098] Thus, the number of words retained by the large speech model at each time step (i.e., the threshold value of the bundle constraint width at each time step) can be:

[0099]

[0100] in, , and For learnable scaling factors and pre-defined parameters, utilize Implement a reasoning path selection mechanism, that is, select The most probable reasoning path, As a threshold value for real-time lexical pruning of large speech models, it enables lexical layer pruning processing.

[0101] Thus, in the reasoning method of the speech recognition model provided in one or more embodiments of this application, considering that the reasoning of large speech models lacks specificity, if the reasoning accuracy is to be improved, it can only be trained with a large amount of speech data, which is extremely costly. Furthermore, the reasoning strategy of large speech models is simple, containing only text-level restriction information or simple restriction strategies, making it difficult to improve the reasoning accuracy in specific scenarios (such as teaching scenarios with a high degree of multilingualism). Therefore, by combining a small speech model with domain knowledge (such as domain knowledge of teaching scenarios with a high degree of multilingualism), and fusing the reasoning grid output by the small speech model with the word probability distribution output by the large speech model, a more accurate speech recognition reasoning process can be achieved.

[0102] Figure 6 The illustration shows a schematic diagram of a model distillation principle provided in at least one embodiment of this application.

[0103] like Figure 6 As shown, the small speech model 103 includes a second encoder 203 and a CTC decoder 204, and the large speech model 102 includes a first encoder 207, a CTC decoder 204, a speech adapter 208, and a large language model 209. The speech adapter 208 outputs a speech context word sequence, and the CTC decoder 204 in the large speech model 102 outputs a CTC predicted context word sequence. The speech context word sequence and the CTC predicted context word sequence together form the actual input context word sequence. The actual input context word sequence is input into the large language model 209 to obtain the word probability distribution at each time step.

[0104] In at least one embodiment of this application, the second encoder 203 is obtained by performing a first distillation process on the first encoder 207, the first distillation process using a first distillation loss function 601; the second activation layer is obtained by performing a second distillation process on the first activation layer, the second distillation process using a second distillation loss function 602 and a third distillation loss function 603.

[0105] By distilling the speech recognition capabilities of a large speech model in a specific domain into a small speech model, the speech recognition capabilities of the small speech model are improved. Considering that traditional hard distillation or soft distillation methods have poor distillation effects in speech recognition scenarios, such as speech recognition scenarios which belong to sequence learning, including not only information from the physical audio layer but also information from the text structure layer, and that the information capacity and reasoning ability of the large speech model far exceeds that of the small speech model, the large speech model will generate a lot of information redundancy during the prediction process, therefore, in at least one embodiment of this application, a multi-strategy distillation method can be adopted to enable the small speech model to learn the modeling information of the large speech model for audio and text layers more effectively, and to prevent redundant information from being propagated to the small speech model as much as possible.

[0106] Figure 7 The illustration shows a schematic diagram of the distillation principle of a self-attention layer provided in at least one embodiment of this application.

[0107] The distillation process can include distillation of the self-attention layer of a small speech model. For example, such as... Figure 7 As shown, the large speech model may include a first encoder 207, and the small speech model may include a second encoder 203. The second encoder 203 is obtained by performing a first distillation process on the first encoder 207. The distillation position corresponding to the first distillation process is located in the first numerical layer of the first encoder 207 and the second encoder 203. The first numerical value is less than half of the total number of layers of the first encoder 207 and the second encoder 203. The first distillation loss function used in the first distillation process is used to minimize the difference between the self-attention matrix of the first encoder 207 and the self-attention matrix of the second encoder 203.

[0108] In other words, on the one hand, the distillation position is selected in the first 50% of the encoder layers of the two speech models, close to the input. Since the shallow network is mainly used to process the acoustic and speech features of the lower layer, the above distillation position can effectively improve the small speech model's ability to encode the original speech information.

[0109] On the other hand, the first distillation loss function is used to distill the self-attention matrix of the shallow layer of the first encoder input of the large speech model. For example, the first distillation loss function can be: .

[0110] in, This represents the top-level features (i.e., the self-attention matrix) of a large speech model. This represents the top-level features (i.e., the self-attention matrix) of the small speech model. Expressing expectations, express and The square of the L2 norm of the difference between these two matrices (i.e., the square of the Frobenius norm).

[0111] By minimizing This allows the self-attention matrix of a small speech model to gradually "imitate" the self-attention matrix of a large speech model, enabling the small speech model to inherit the attention pattern of the large speech model and better encode the contextual dependencies in speech data with a smaller number of parameters.

[0112] Figure 8 The illustration shows a schematic diagram of the distillation principle of an activation layer provided in at least one embodiment of this application.

[0113] The distillation process may include distilling the activation layer (softmax layer) of a small speech model. For example, a large speech model may include a first activation layer, and a small speech model may include a second activation layer, which is obtained by performing a second distillation on the first activation layer.

[0114] like Figure 8 As shown, the second distillation process may include the following steps: simultaneously inputting the first speech data 801 into the large speech model 102 and the small speech model 103; receiving the first sequence label 802 output by the first activation layer and the second sequence label 803 output by the second activation layer; dividing the first sequence label 802 and the second sequence label 803 into a first part (the remaining parts excluding the second part, not shown in the figure) and a second part according to the entropy value of the first sequence label 802; the entropy value of the first part in the first sequence label 802 is less than the entropy value of the second part in the first sequence label; and distilling the second activation layer using the second distillation loss function 602 for the first part, the second distillation loss function 602 being used to minimize the entropy value of the first sequence label 802. The distribution of the first part in the second sequence label 803 differs from the distribution of the first part in the second sequence label 803. For the second part, the second activation layer is distilled using a third distillation loss function 603. The third distillation loss function includes a first sub-function, a second sub-function, and a second distillation loss function 602. The first sub-function is used to control the output of the second part in the second sequence label 803 to be the same as that in the second part in the first sequence label 802 at the same time and to control the output of the second part in the second sequence label 803 to be different from that in the second part in the first sequence label 802 at different times. The second sub-function is used to control the change trend of the second part in the second sequence label 803 to be the same as that in the second part in the first sequence label 802 between consecutive time points.

[0115] In other words, after simultaneously inputting the same speech data (i.e., the first speech data) into both the large speech model and the small speech model, the time series output by the large speech model (i.e., the first sequence label) and the time series output by the small speech model (i.e., the second sequence label) are obtained. Based on the entropy value of the first sequence label, the entire time interval is divided into a low-entropy segment (i.e., the first part) and a high-entropy segment (i.e., the second part). For the low-entropy segment, the second distillation loss function is used to directly distill the first sequence label of the large speech model. For the high-entropy segment, multiple distillation loss functions (i.e., the third distillation loss function) are superimposed to perform enhanced distillation, thereby strengthening the small speech model's learning of the causal patterns and complex contexts of the large speech model.

[0116] For example, the first sequence label can be The second sequence label can be t is a certain moment in the time series (i.e., time step), c is the number of categories, and T is the total length of the first sequence label and the second sequence label.

[0117] In this case, calculate the average entropy value of the first sequence label:

[0118]

[0119] Then, based on the average entropy value of the first sequence label, the first part and the second part are determined. For example, the time set (i.e., the time step set) of the second part (i.e., the high entropy segment) can be: .

[0120] For example, , Let be the entropy value of the first sequence label at time t. The minimum entropy value of the first sequence label.

[0121] The second distillation loss function can be:

[0122]

[0123]

[0124] in, This is the second distillation loss function. Let KL divergence be the KL divergence. It is a minimum value used to avoid the denominator being 0.

[0125] Thus, for the low-entropy segment, the second activation layer is distilled using the second distillation loss function, directly allowing the output distribution of the small speech model (i.e., the second sequence label) to be directly adjusted. Approximates the output distribution of a large speech model (i.e., the first sequence label). ), learn the predictive capabilities of large speech models.

[0126] The third distillation loss function can be... , As the first sub-function, This is the second sub-function.

[0127] The first sub-function can be:

[0128]

[0129] in, The softmax output of the second part of the first sequence label and the second sequence label at the same time t. Representing temperature values, for example, .

[0130] The second sub-function can be:

[0131]

[0132] The first sub-function makes the outputs of the small speech model and the large speech model as similar as possible at the same time step, while making the outputs of the small speech model and the large speech model as different as possible at different time steps. This strengthens the small speech model's learning of time step specificity and avoids confusion between the contexts of different time steps.

[0133] The second sub-function compares the output changes of the large and small speech models between adjacent time steps, ensuring that the output change trend of the small speech model is consistent with that of the large speech model, thereby enabling the small speech model to learn temporal causal relationships.

[0134] Therefore, considering that large speech models are very sensitive to strong local contextual relationships and key phonemes when outputting word sequences, while small speech models often cannot distinguish these subtle structural relationships, the distillation method described above allows small speech models to not only learn the output probability distribution of large speech models, but also to learn "causal relationships," that is, how the output distribution should change accordingly when the input changes. At the same time, it can also solve the problem of poor distillation effect of high-entropy speech segments (such as noise, interference, and difficult-to-pronounce segments), and improve the performance of small speech models in complex speech recognition scenarios.

[0135] It is understood that before using the technical solutions disclosed in the embodiments of this application, users should be informed of the type, scope of use, and usage scenarios of the personal information involved in this application (e.g., user profile features, historical dialogue text, etc.) and their authorization obtained in accordance with relevant laws and regulations through appropriate means. For example, in response to receiving a user's active request, a prompt message can be sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media executing the technical solutions of this application based on the prompt message.

[0136] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0137] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this application. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this application.

[0138] It is understood that the data involved in the technical solution of this application (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0139] The above text combined Figures 1 to 8 The reasoning method of the speech recognition model provided in the embodiments of this application has been described in detail. The apparatus and electronic equipment provided in the embodiments of this application will be described below with reference to the accompanying drawings. Figure 9 This is a schematic diagram of the structure of an inference device for a speech recognition model provided in at least one embodiment of this application.

[0140] like Figure 9 As shown, the speech recognition model in the inference device 900 of the speech recognition model in this embodiment includes a small speech model and a large speech model. The speech recognition model inference device 900 includes an acquisition module 901, an inference module 902, and a determination module 903. For example, these units or modules can be implemented by hardware (e.g., circuit) modules or software modules, etc. The following embodiments are the same and will not be described again. For example, these units or modules can be implemented by a central processing unit (CPU), a general-purpose graphics processor (GPGPU), a graphics processing unit (GPU), a tensor processor (TPU), a field-programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities, as well as corresponding computer instructions.

[0141] The acquisition module 901 is configured to acquire voice data, wherein the voice data includes voice data in multiple languages.

[0142] The inference module 902 is configured to: input the speech data into the small speech model to obtain the inference grid output by the small speech model, and input the speech data into the large speech model to obtain the word probability distribution at each time step output by the large speech model, wherein the inference grid includes multiple grid time slots, and each grid time slot includes multiple candidate words at a time step and the probability of the multiple candidate words;

[0143] The determining module 903 is configured to: fuse the trellis time slot at the first moment and the word probability distribution at the first moment to determine the speech recognition result of the speech data at the first moment, so as to obtain the speech recognition result of the speech data, wherein the first moment is any moment in the inference process of the speech recognition model.

[0144] For example, the acquisition module 901 can be configured to execute step S301 described above. Its specific implementation principle can be referred to the relevant description of step S301. The reasoning module 902 can be configured to execute step S302 described above. Its specific implementation principle can be referred to the relevant description of step S302. The determination module 903 can be configured to execute step S303 described above. Its specific implementation principle can be referred to the relevant description of step S303, and will not be repeated here.

[0145] In at least one embodiment of this application, the small speech model outputs the inference graph in the following manner: performing a Viterbi search on the speech data, retaining multiple candidate lexical units during the Viterbi search process, and outputting the inference graph.

[0146] In at least one embodiment of this application, the determining module 903 is further configured to: fuse the trellis time slot at the first moment and the word probability distribution at the first moment to determine the fused probability distribution; and determine the speech recognition result of the speech data at the first moment based on the fused probability distribution.

[0147] In at least one embodiment of this application, the determining module 903 is further configured to: determine a first probability random vector based on a plurality of candidate words in a time slot of the inference grid at a first time and the probabilities of the plurality of candidate words; and determine a second probability random vector based on the word probability distribution at the first time; determine a fusion weight based on a first distance between the first probability random vector and the second probability random vector; and determine a fusion probability distribution by fusing the time slot of the grid at the first time and the word probability distribution at the first time based on the fusion weight.

[0148] In at least one embodiment of this application, the determining module 903 is further configured to: perform a beam search based on the fusion probability distribution, wherein the speech recognition model retains a corresponding number of candidate words at each time point of the beam search; determine, at a second time point of the beam search, a first number of candidate words corresponding to the second time point based on the conditional entropy of the historical path, wherein the second time point is any time point of the beam search, and the conditional entropy is used to measure the uncertainty of the first number of candidate words; and adjust the first number to a second number based on the conditional entropy and the historical conditional entropy of at least one historical time point.

[0149] In at least one embodiment of this application, the determining module 903 is further configured to: increase the first quantity to the second quantity in response to the conditional entropy being greater than the historical conditional entropy of at least one historical moment satisfying a first condition; or decrease the first quantity to the second quantity in response to the conditional entropy being less than the historical conditional entropy of at least one historical moment satisfying a second condition.

[0150] In at least one embodiment of this application, the determining module 903 is further configured to: acquire multiple inference paths corresponding to the second time step; determine multiple completed words included in the multiple inference paths; for each completed word among the multiple completed words, determine the word probability of the completed word based on the conditional probability of the completed word in each inference path including the completed word; and adjust the second quantity to a third quantity based on the word probabilities corresponding to the multiple completed words respectively.

[0151] In at least one embodiment of this application, the determining module 903 is further configured to: determine the entropy change feature of the word distribution at the second time step based on the word probabilities corresponding to the plurality of completed words respectively; increase the second quantity to the third quantity in response to the entropy change feature indicating an increase in uncertainty at the second time step; decrease the second quantity to the third quantity in response to the entropy change feature indicating a decrease in uncertainty at the second time step; or maintain the second quantity in response to the entropy change feature indicating that the uncertainty at the second time step is in a stable state.

[0152] In at least one embodiment of this application, the large speech model includes a first encoder, and the small speech model includes a second encoder, the second encoder being obtained by performing a first distillation process on the first encoder; wherein the distillation position corresponding to the first distillation process is located in a first numerical layer of the first encoder and the second encoder, the first numerical value being less than half of the total number of layers of the first encoder and the second encoder; wherein the first distillation loss function used in the first distillation process is used to minimize the difference between the self-attention matrix of the first encoder and the self-attention matrix of the second encoder.

[0153] In at least one embodiment of this application, the large speech model includes a first activation layer, and the small speech model includes a second activation layer, wherein the second activation layer is obtained by performing a second distillation process on the first activation layer; wherein the second distillation process includes the following steps: simultaneously inputting first speech data into the large speech model and the small speech model; receiving a first sequence label output by the first activation layer and a second sequence label output by the second activation layer; dividing the first sequence label and the second sequence label into a first part and a second part according to the entropy value of the first sequence label, wherein the entropy value of the first part in the first sequence label is less than the entropy value of the second part in the first sequence label; and distilling the second activation layer using a second distillation loss function for the first part, wherein the second distillation... The loss function is used to minimize the difference between the distribution of the first part in the first sequence label and the distribution of the first part in the second sequence label; for the second part, the second activation layer is distilled using a third distillation loss function, wherein the third distillation loss function includes a first sub-function, a second sub-function and a second distillation loss function, the first sub-function is used to control the output of the second part in the second sequence label to be the same as that of the second part in the first sequence label at the same time and to control the output of the second part in the second sequence label to be different from that of the second part in the first sequence label at different times, and the second sub-function is used to control the change trend of the second part in the second sequence label to be the same as that of the second part in the first sequence label between consecutive time points.

[0154] It should be noted that, for clarity and brevity, this application embodiment does not show all the constituent units of the speech recognition model inference device 900. To realize the necessary functions of the speech recognition model inference device, those skilled in the art can provide and set other constituent units (not shown) according to specific needs, and this application embodiment does not limit this.

[0155] This application also provides an electronic device. This electronic device is specifically used to implement, as described above. Figure 9 The inference device 900 of the speech recognition model in the illustrated embodiment has the following functions.

[0156] Figure 10 A structural schematic diagram of an electronic device 1000 is provided, such as... Figure 10 As shown, the electronic device 1000 includes a bus 1001, a processor 1002, a communication interface 1003, and a memory 1004. The processor 1002, the memory 1004, and the communication interface 1003 communicate with each other via the bus 1001.

[0157] Bus 1001 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0158] The processor 1002 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0159] The communication interface 1003 is used for communication with external devices. For example, the communication interface 1003 can be used to communicate with a terminal.

[0160] Memory 1004 may include volatile memory, such as random access memory (RAM). Memory 1004 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0161] The memory 1004 stores executable code, and the processor 1002 executes the executable code to perform the inference method of the aforementioned speech recognition model.

[0162] Specifically, in achieving Figure 10 In the case of the illustrated embodiment, and Figure 10 When the modules or units of the inference device 900 of the speech recognition model described in the embodiment are implemented by software, the following steps are performed: Figure 10 The software or program code required for the functions of each module / unit can be partially or entirely stored in the memory 1004. The processor 1002 executes the program code corresponding to each unit stored in the memory 1004 and executes the inference method of the aforementioned speech recognition model.

[0163] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the inference method of the speech recognition model applied to the inference device 900 of the speech recognition model described above.

[0164] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute a reasoning method for a speech recognition model.

[0165] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a reasoning method for a speech recognition model.

[0166] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

[0167] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0168] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An inference method for a speech recognition model, wherein, The speech recognition model includes a small speech model and a large speech model, and the method includes: Acquire voice data, wherein the voice data includes voice data in a mixture of multiple languages; The speech data is input into the small speech model to obtain the inference grid output by the small speech model, and the speech data is input into the large speech model to obtain the word probability distribution at each time step output by the large speech model. The inference grid includes multiple grid time slots, and each grid time slot includes multiple candidate words at a time step and the probability of the multiple candidate words. By fusing the trellis time slot at the first moment and the word probability distribution at the first moment, the speech recognition result of the speech data at the first moment is determined to obtain the speech recognition result of the speech data, wherein the first moment is any moment in the inference process of the speech recognition model.

2. The method according to claim 1, wherein, The small speech model outputs the inference grid in the following manner: A Viterbi search is performed on the speech data. During the Viterbi search, multiple candidate lexical units are retained, and the inference lattice is output.

3. The method according to claim 1, wherein, The process of fusing the lattice time slots of the first time step and the word probability distribution of the first time step to determine the speech recognition result of the speech data at the first time step includes: The fused probability distribution is determined by fusing the grid time slots at the first time point and the word probability distribution at the first time point. Based on the fusion probability distribution, the speech recognition result of the speech data at the first moment is determined.

4. The method according to claim 3, wherein, The process of fusing the lattice time slots at the first time point and the word probability distribution at the first time point to determine the fused probability distribution includes: Based on the multiple candidate words and their probabilities in the time slot of the first time in the inference grid, a first probability random vector is determined, and based on the word probability distribution at the first time, a second probability random vector is determined. The fusion weights are determined based on the first distance between the first probability random vector and the second probability random vector. Based on the fusion weights, the lattice time slots at the first time point and the word probability distribution at the first time point are fused to determine the fusion probability distribution.

5. The method according to claim 3, wherein, Determining the speech recognition result of the speech data at the first time step based on the fusion probability distribution includes: Based on the fusion probability distribution, a beam search is performed, wherein the speech recognition model retains a corresponding number of candidate word units at each time point of the beam search; At a second moment of the beam search, the conditional entropy of a first number of candidate lexical units based on the historical path is determined, wherein the second moment is any moment of the beam search, and the conditional entropy is used to measure the uncertainty of the first number of candidate lexical units; The first quantity is adjusted to the second quantity based on the conditional entropy and the historical conditional entropy of at least one historical moment.

6. The method according to claim 5, wherein, The step of adjusting the first quantity to a second quantity based on the conditional entropy and the historical conditional entropy of at least one historical moment includes: In response to the conditional entropy being greater than the historical conditional entropy at at least one historical moment satisfying a first condition, the first quantity is increased to the second quantity; or In response to the conditional entropy being less than the historical conditional entropy of at least one historical moment satisfying the second condition, the first quantity is reduced to the second quantity.

7. The method according to claim 5, wherein, After adjusting the first quantity to a second quantity based on the conditional entropy and the historical conditional entropy of at least one historical moment, the method further includes: Obtain the multiple inference paths corresponding to the second time step; Identify multiple completed words included in the multiple reasoning paths; For each of the plurality of completed words, the word probability of the completed word is determined based on the conditional probability of the completed word in each inference path that includes the completed word; Based on the word probabilities corresponding to the multiple completed words, the second quantity is adjusted to the third quantity.

8. The method according to claim 7, wherein, The step of adjusting the second quantity to a third quantity based on the word probabilities corresponding to the plurality of completed words includes: Based on the word probabilities corresponding to the multiple completed words, the entropy change characteristics of the word distribution at the second time step are determined; In response to the increase in uncertainty at the second moment, as characterized by the entropy change feature, the second quantity is increased to the third quantity; In response to the decrease in uncertainty at the second moment, as indicated by the entropy change characteristic, the second quantity is reduced to the third quantity; or In response to the entropy change characteristic indicating that the uncertainty at the second moment is in a stable state, the second quantity is maintained.

9. The method according to any one of claims 1 to 8, wherein, The large speech model includes a first encoder, and the small speech model includes a second encoder, which is obtained by performing a first distillation process on the first encoder. Wherein, the distillation position corresponding to the first distillation process is located in the first numerical layer of the first encoder and the second encoder, and the first value is less than half of the total number of layers of the first encoder and the second encoder; The first distillation loss function used in the first distillation process is used to minimize the difference between the self-attention matrix of the first encoder and the self-attention matrix of the second encoder.

10. The method according to any one of claims 1 to 8, wherein, The large speech model includes a first activation layer, and the small speech model includes a second activation layer, which is obtained by performing a second distillation process on the first activation layer. The second distillation process includes the following steps: The first speech data is simultaneously input into the large speech model and the small speech model, and the first sequence label output by the first activation layer and the second sequence label output by the second activation layer are received. Based on the entropy value of the first sequence tag, the first sequence tag and the second sequence tag are divided into a first part and a second part, wherein the entropy value of the first part of the first sequence tag is less than the entropy value of the second part of the first sequence tag. For the first part, the second activation layer is distilled using a second distillation loss function, wherein the second distillation loss function is used to minimize the difference between the distribution of the first part in the first sequence label and the distribution of the first part in the second sequence label; For the second part, the second activation layer is distilled using a third distillation loss function, wherein the third distillation loss function includes a first sub-function, a second sub-function, and a second distillation loss function. The first sub-function is used to control the output of the second part in the second sequence label to be the same as that in the second part in the first sequence label at the same time and to control the output of the second part in the second sequence label to be different from that in the second part in the first sequence label at different times. The second sub-function is used to control the change trend of the second part in the second sequence label to be the same as that in the second part in the first sequence label between consecutive time points.

11. An inference device for a speech recognition model, wherein, The speech recognition model includes a small speech model and a large speech model, and the device includes: The acquisition module is configured to acquire voice data. The inference module is configured to: input the speech data into the small speech model to obtain the inference grid output by the small speech model, and input the speech data into the large speech model to obtain the lexical probability distribution at each time step output by the large speech model, wherein the inference grid includes multiple grid time slots, and each grid time slot includes multiple candidate lexical units at a time step and the probabilities of the multiple candidate lexical units. The determination module is configured to: fuse the trellis time slot at the first moment and the word probability distribution at the first moment to determine the speech recognition result of the speech data at the first moment, so as to obtain the speech recognition result of the speech data, wherein the first moment is any moment in the inference process of the speech recognition model.

12. An electronic device, comprising: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 10.

Images