Speech recognition method, related device and computer program product
By using a shallow and deep encoder structure of a speech recognition model, the problem of timing asynchrony between small and large models is solved, achieving timing consistency between real-time word output and the final recognition result, reducing system maintenance costs and improving recognition accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-24
- Publication Date
- 2026-03-27
AI Technical Summary
In existing speech recognition solutions, the word-splitting process of the small model and the recognition results of the large model in the main network are out of sync, resulting in a complex system structure and high maintenance costs.
A speech recognition model is adopted, which includes a shallow encoder and a deep encoder. The shallow encoder decodes in real time to obtain the streaming recognition result, and the deep encoder further decodes to obtain the final recognition result. The training objectives and parameters are shared to realize the temporal relationship between the streaming recognition result and the final recognition result.
This achieves temporal consistency between real-time character recognition results and final recognition results, reduces system maintenance costs, improves the accuracy of final recognition results, and reduces memory usage and computational latency.
Smart Images
Figure CN121747535A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of speech recognition technology, and more specifically, to a speech recognition method, related equipment, and computer program product. Background Technology
[0002] In online speech recognition scenarios, it is necessary to output streaming characters in real time during the speech recognition process (referred to as real-time character output), and output the final recognition result after the current streaming speech segment has been decoded. Traditional methods such as Figure 1 As shown, the speech to be recognized is first preprocessed to extract FB (Filter Bank) features, which are then copied into two copies:
[0003] The first FB feature is fed into the small character model for real-time decoding. This small model can generally adopt a relatively simple structure such as RNNT (Recurrent Neural Network Transducer) to obtain the decoding result in real time, so that the character jumping will be smoother and the stuttering phenomenon will be reduced. The decoding result of the small model will be sent to the post-processing component to perform a series of conversion operations such as capitalization and number normalization, as the result of the real-time character jumping process.
[0004] The second set of FB features is fed into the main network model for decoding. The main network can generally adopt a more complex structure such as Transformer / Conformer, mainly to obtain higher recognition accuracy, not real-time performance, in order to improve the final recognition effect. The recognition results of the main network are also sent to the post-processing component to perform operations such as adding punctuation and number normalization to obtain the final recognition result.
[0005] Existing speech recognition solutions employ two models, a small one and a large one, which have the following drawbacks: the word-reading process of the small model and the recognition result of the main network large model are independent of each other. It is possible that the main network large model has already given the final recognition result, but the small model is still reading words, resulting in a time-staggered problem. In addition, the system structure is complex and requires time-consuming and labor-intensive maintenance of two sets of models simultaneously. Summary of the Invention
[0006] In view of the above problems, this application is proposed to provide a speech recognition method, related equipment, and computer program product to ensure the temporal relationship between real-time word recognition and the final recognition result, thereby reducing the maintenance cost of the speech recognition system. The specific solution is as follows:
[0007] In a first aspect, this application provides a speech recognition method, including:
[0008] Acquire the streaming speech segment to be identified and extract the audio features of the streaming speech segment;
[0009] The audio features are processed by a shallow encoder of a speech recognition model to obtain shallow encoded features. Based on the shallow encoded features, the streaming recognition result is obtained and output in real time as an intermediate process result.
[0010] The shallow coding features are further encoded by the deep encoder of the speech recognition model to obtain deep coding features, and the final recognition result of the streaming speech segment is obtained by decoding based on the deep coding features.
[0011] In one possible design, in another implementation of the first aspect of the embodiments of this application, the shallow encoder includes a first shallow encoder and a second shallow encoder;
[0012] The process of processing the audio features using a shallow encoder to obtain shallow coded features, and then decoding these shallow coded features to obtain streaming recognition results, includes:
[0013] The audio features are processed by the first shallow encoder to obtain the first shallow encoded features. CTC decoding is performed based on the first shallow encoded features to obtain the original token sequence containing the blank token and the character token.
[0014] The audio features are processed by the second shallow encoder to obtain the second shallow encoded features. Based on the second shallow encoded features, the probability of each decoding position belonging to the blank tag is predicted, and target decoding positions whose probability of belonging to blank exceeds a set probability threshold are filtered.
[0015] The markers at the target decoding positions in the original marker sequence are replaced with blanks to obtain a replaced marker sequence. The blanks are then removed and duplicate characters are merged in the replaced marker sequence to obtain the streaming recognition result.
[0016] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of further encoding the shallow encoded features by a deep encoder to obtain deep encoded features includes:
[0017] The first shallow coding feature and the second shallow coding feature are fused to obtain the fused shallow coding feature;
[0018] The fused shallow coding features are further encoded by a deep encoder to obtain deep coding features.
[0019] In one possible design, in another implementation of the first aspect of the embodiments of this application, the first shallow encoder includes: a first downsampling module and an N1 layer Conformer module connected in series;
[0020] The process of processing the audio features using the first shallow encoder to obtain the first shallow encoded features includes:
[0021] The audio features are downsampled by the first downsampling module, and the downsampled audio features are sent to the N1 layer Conformer module for inference calculation to obtain the first shallow-layer encoded features output by the last Conformer module.
[0022] In one possible design, in another implementation of the first aspect of the embodiments of this application, the second shallow encoder includes: a second downsampling module and an N2 layer Conformer module connected in series, where N2 is less than N1;
[0023] The process of processing the audio features using the second shallow encoder to obtain the second shallow encoded features includes:
[0024] The audio features are downsampled by the second downsampling module, and the downsampled audio features are sent to the N2 layer Conformer module for inference calculation to obtain the second shallow-layer encoded features output by the last Conformer module.
[0025] In one possible design, in another implementation of the first aspect of the embodiments of this application, before processing the audio features by the shallow encoder of the speech recognition model, the method further includes:
[0026] Based on the audio features, the VAD (Voice Activity Detection) of the streaming speech segment is performed to obtain the audio features corresponding to the valid speech segment, which are then used as the input to the speech recognition model.
[0027] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of predicting the probability that each decoding position belongs to the blank tag based on the second shallow coding features includes:
[0028] CTC decoding is performed based on the second shallow coding features to obtain the label probability distribution at each decoding position, and the probability of belonging to the blank label is obtained from the label probability distribution at each decoding position.
[0029] In one possible design, in another implementation of the first aspect of the embodiments of this application, the training process of the speech recognition model includes:
[0030] Obtain sample speech segments labeled with streaming recognition result tags and final recognition result tags;
[0031] The sample audio features of the sample speech segment are fed into the speech recognition model to obtain the first CTC decoding result output by the first CTC decoder, the second CTC decoding result output by the second CTC decoder, and the final prediction recognition result output by the deep decoder. The first CTC decoder is used to perform CTC decoding based on the output features of the first shallow encoder, the second CTC decoder is used to perform CTC decoding based on the output features of the second shallow encoder, and the deep decoder is used to perform decoding based on the output features of the deep encoder.
[0032] A first CTC loss is calculated based on the first CTC decoding result and the streaming identification result label; a second CTC loss is calculated based on the second CTC decoding result and the streaming identification result label; and the first CTC loss and the second CTC loss are weighted and fused to obtain the CTC weighted loss.
[0033] The final identification result loss is calculated based on the final predicted identification result and the final identification result label;
[0034] The total loss is determined based on the CTC weighted loss and the final identification result loss, and the model parameters are updated according to the total loss.
[0035] Secondly, this application provides an electronic device, including: a memory and a processor;
[0036] The memory is used to store programs;
[0037] The processor is configured to execute the program to implement the various steps of the speech recognition method described in any of the first aspects of this application.
[0038] Thirdly, this application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the various steps of the speech recognition method described in any of the first aspects of this application.
[0039] Fourthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the various steps of the speech recognition method described in any of the first aspects of this application.
[0040] By employing the aforementioned technical solution, this application designs a speech recognition model that can simultaneously achieve real-time word output during the decoding process and obtain the final recognition result after decoding. Specifically, the speech recognition model includes a shallow encoder and a deep encoder. The shallow encoder processes the audio features of the streaming speech segment to obtain shallow encoded features, which can then be used to decode the streaming recognition result, outputting it in real-time as an intermediate process result (real-time word output process). The deep encoder further encodes the shallow encoded features to extract higher-level semantic features (deep encoded features), and decoding based on these deep encoded features yields the final recognition result of the streaming speech segment. In this application's speech recognition model, the real-time word output result and the final recognition result have a temporal relationship; the real-time word output result is output before the final recognition result, solving the temporal synchronization problem of traditional solutions and eliminating the need to maintain two different models, thus reducing system maintenance costs.
[0041] In addition, the shallow coding features extracted by the shallow encoder are fed into the deep encoder for further encoding to extract higher-level semantic features, which are then used for decoding to obtain the final recognition result, thus improving the decoding accuracy of the final recognition result. Attached Figure Description
[0042] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0043] Figure 1 This example illustrates a flowchart of a traditional speech recognition method.
[0044] Figure 2 A schematic diagram of an implementation system architecture for the speech recognition method provided in this application embodiment;
[0045] Figure 3 This is a schematic flowchart of a speech recognition method provided in an embodiment of this application;
[0046] Figure 4 A schematic diagram of the processing flow of a speech recognition model provided in an embodiment of this application;
[0047] Figure 5 A schematic diagram illustrating the processing flow of another speech recognition model provided in an embodiment of this application;
[0048] Figure 6 A schematic diagram of a shallow encoder network structure provided in an embodiment of this application;
[0049] Figure 7A schematic diagram of the processing flow of a second shallow encoder and a second CTC decoder provided for embodiments of this application;
[0050] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0052] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0053] This application proposes a speech recognition scheme that employs an Encoder+Decoder network structure to balance real-time word processing with the accuracy of the final recognition result. Furthermore, the use of a single network structure establishes a temporal relationship between word processing and the final recognition result, simplifying and streamlining the entire system maintenance.
[0054] The speech recognition solution provided in this application can be applied to a variety of recognition scenarios, including but not limited to voice input methods, voice assistants, and conference systems, which require real-time output of the speech recognition results.
[0055] This application provides a speech recognition method that can be applied to, for example... Figure 2 The system architecture shown may include a terminal 100 and a server 200. The server 200 may include one or more servers (…). Figure 2 (This example uses a server as an illustration).
[0056] Either terminal 100 or server 200 can be used independently to execute the speech recognition method provided in the embodiments of this application. Alternatively, terminal 100 and server 200 can also be used collaboratively to execute the speech recognition method provided in the embodiments of this application.
[0057] The following description Figure 2 The product form of the mid-terminal 100;
[0058] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, vehicle-mounted device, conference terminal, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.
[0059] This application provides a speech recognition method, which is illustrated by applying the method to a computer device. Specifically, the computer device may be... Figure 2 The system consists of terminal 100 or a combination of terminal 100 and server 200. (Refer to...) Figure 3 The speech recognition method specifically includes the following steps:
[0060] Step S100: Obtain the streaming speech segment to be identified and extract the audio features of the streaming speech segment.
[0061] This application addresses real-time speech recognition scenarios by acquiring streaming speech segments to be recognized. To facilitate processing by the speech recognition model, this step extracts audio features from the streaming speech segments. Audio features can be MFCC (Mel frequency cepstrum coefficient) or FB (Filter Bank), among others.
[0062] Optionally, after obtaining the streaming speech segment to be identified and extracting audio features, a process of performing voice activity detection (VAD) on the streaming speech segment can be further added. That is, VAD detection is performed on the streaming speech segment based on audio features to determine the start and end points of the valid speech segment, thereby obtaining the audio features corresponding to the valid speech segment, which are used as input to the next speech recognition model for forward inference calculation by the shallow encoder.
[0063] VAD detection can filter out invalid noise and remove invalid calculations.
[0064] Step S110: The audio features are processed by the shallow encoder of the speech recognition model to obtain shallow encoded features. The streaming recognition result is obtained by decoding based on the shallow encoded features and output as an intermediate process result in real time.
[0065] Reference Figure 4As shown, the shallow encoder encodes audio features to obtain shallow encoded features. Further, the shallow decoder can decode based on these shallow encoded features to obtain streaming recognition results, which serve as the real-time output of the decoding process.
[0066] Optionally, the streaming recognition results can be further post-processed, such as conversion operations like case sensitivity and number normalization, and the post-processed streaming recognition results can be output.
[0067] Step S120: The shallow coding features are further encoded by the deep encoder of the speech recognition model to obtain deep coding features. The final recognition result of the streaming speech segment is obtained by decoding based on the deep coding features.
[0068] The shallow-encoded features are further fed into a deep encoder to extract high-level semantic features, resulting in deep-encoded features. The final recognition result of the streaming speech segment can then be obtained by decoding based on these deep-encoded features using a deep decoder.
[0069] The deep decoder can interact with BeamSearch using a self-looping structure, continuously decoding the deep encoded features fed in from the deep encoder, starting from the beginning. <begin>Decoding continues until it is complete. After the deep decoder finishes decoding, it outputs the Top N recognition results with the highest probability of the whole sentence. The Top 1 result with the highest probability is sent to the post-processing component for operations such as number normalization and adding punctuation. Finally, the whole sentence result is output, which is the final recognition result.
[0070] The speech recognition method provided in this application has the following advantages:
[0071] 1. Reduce memory overhead and improve computing efficiency
[0072] Only one speech recognition model needs to be loaded, reducing memory usage (especially for edge devices or large-scale deployment scenarios).
[0073] Using shallow coding features as input to deep coding eliminates the need to re-extract audio features, thus improving computational efficiency.
[0074] 2. Reduce the latency difference between flow cytometry and the final result.
[0075] Shallow coding can output streaming recognition results quickly, while deep coding can further optimize the results, resulting in smoother delay gradients for both.
[0076] Deep coding benefits from shallow features: shallow coding features have captured local temporal patterns. Deep encoders perform deep coding based on shallow coding features, which can focus on learning long-distance dependencies and semantic information, thereby obtaining higher quality deep coding features, which is conducive to improving the accuracy of the final recognition results.
[0077] 3. Improve the consistency between streaming cytometry and the final result.
[0078] Shared underlying features: Since shallow and deep encoders share training objectives and parameters, streaming results usually have higher consistency with the final results (such as pronunciation, sentence segmentation, etc.), reducing contradictory outputs caused by model differences.
[0079] End-to-end optimization: The model can jointly optimize the streaming recognition and the final recognition target (such as multi-task learning), making shallow encoded features more suitable for subsequent deep optimization.
[0080] 4. Simplify system complexity
[0081] This solution only requires training, updating, and deploying a single model, reducing project complexity. Traditional dual-model solutions require coordinating the interfaces, versions, and resource configurations of the two models, while this single-model solution is easier to manage.
[0082] 5. More suitable for end-to-end training
[0083] Joint optimization: The streaming recognition (shallow decoding) and final recognition (deep decoding) can be optimized simultaneously through a multi-task loss function, so that the model learns features that are beneficial to the global task in the early encoding stage.
[0084] Avoiding information loss: In traditional solutions, small streaming models may lose information due to capacity limitations, while large models need to reprocess the original audio. In this case, deep coding in the single model can directly utilize shallow coding features, reducing information loss.
[0085] In some embodiments of this application, several optional structures for shallow encoders are described.
[0086] In one alternative implementation, the shallow encoder can be set to a single encoder, and the corresponding shallow decoder can also be a single decoder.
[0087] Shallow encoders can employ various network structures, such as a network consisting of a downsampling module and a Conformer module connected in series.
[0088] Shallow decoders can employ CTC decoders, for example, consisting of linear layers and softmax layers.
[0089] like Figure 6 It exemplifies a shallow encoder structure consisting of a downsampling module and a Conformer module connected in series.
[0090] The downsampling module can include several layers of VGGBlock, and the ConformerBlock can include several layers. Figure 6 The following explanation uses a 2-layer VGGBlock and an 8-layer ConformerBlock as an example.
[0091] use Let t represent the frame-level audio features of the input shallow encoder, where t represents the t-th frame of the audio in the time dimension. The audio features are first downsampled to half their original value using the first VGGBlock layer, and then downsampled again using the second VGGBlock layer to a quarter of their original value.
[0092] The downsampled audio features are fed into an 8-layer ConformerBlock for inference computation. The inference computation process is the same for each layer, starting with FFN (Feed-Forward Network) computation. Next, we will perform the calculation of multi-head self-attention (MHSA). The more detailed calculation process within MHSA is as follows:
[0093] First input Perform a linear transformation to obtain the query ,key Sum ,in It is a learnable weight matrix. ,in This is the dimension of the features; it's divided by a scaling factor to make the model more stable. Next, we will perform convolution calculations. After calculating the convolution, LayerNorm is further calculated. layerNorm is the last computation in a ConformerBlock. There are a total of 8 ConformerBlocks, and the computation process of each layer is similar until the inference computation of these 8 ConformerBlocks is completed. The output hidden state (shallow encoded features) is then fed into the shallow decoder and deep encoder, respectively.
[0094] Shallow encoded features obtained by shallow encoder The data is then fed into a shallow decoder for decoding. A CTC decoder can be used as the shallow decoder; the CTC decoding process includes:
[0095]
[0096] shallow coding features First, a linear transformation is performed, where It is a learnable weight matrix, then a softmax operation is performed, and the output is... It includes the probabilities of all tokens, sorts all token probabilities from largest to smallest, and performs a BeamSearch operation on the top K tokens, or only selects the top 1 token, i.e., Greedy Search. This case uses the Greedy Search strategy, which is sufficient to meet the performance metrics of the process, with a time complexity of O(n log n). , It is the length of the time frame. It represents the total number of token categories, including (N-1) character tokens and 1 blank token.
[0097] In another optional implementation, this embodiment considers that traditional small models lack the ability to reject noise and silence, and are prone to generating meaningless results such as "ah ah" or "noise". To further improve the rejection capability of streaming recognition results for noise and silence, this embodiment provides another optional implementation of a shallow encoder and a shallow decoder. Specifically:
[0098] A shallow encoder can be configured with two encoders, see reference. Figure 5 As shown, these are defined as the first shallow encoder Encoder1 and the second shallow encoder Encoder2, respectively. Correspondingly, the shallow decoder may include the first CTC decoder and the second CTC decoder.
[0099] The aforementioned step S110, which processes audio features using a shallow encoder to obtain shallow coded features and then decodes these features to obtain streaming recognition results, may include:
[0100] S1. The audio features are processed by the first shallow encoder to obtain the first shallow encoded features. CTC decoding is performed based on the first shallow encoded features to obtain the original token sequence containing the blank token and the character token.
[0101] The first shallow encoder's first shallow encoded features are fed into the first CTC decoder for CTC decoding, resulting in an original token sequence containing blank and character tokens, such as: c, -, a, a, b, -, t. Here, "-" represents the blank token. The original token sequence may contain duplicate characters, which will be merged in subsequent processing stages.
[0102] S2. The audio features are processed by the second shallow encoder to obtain the second shallow coding features. Based on the second shallow coding features, the probability of each decoding position belonging to the blank tag is predicted, and the target decoding positions whose probability of belonging to blank exceeds the set probability threshold are selected.
[0103] Examples of setting probability thresholds include values such as 0.7 or other values.
[0104] In this embodiment, considering the potential for misidentification of noise and silence in the CTC decoding result of the first shallow encoder, the streaming recognition result is not directly determined based on the CTC decoding result of the first shallow encoder. Instead, the audio features are processed by a second shallow encoder to obtain second shallow encoded features. Based on the second shallow encoded features, the probability of each decoding position (each decoding time step) belonging to blank is predicted, and target decoding positions whose probability of belonging to blank exceeds a set probability threshold are selected. In this embodiment, the blank result at the target decoding position is considered meaningless, belonging to noise or silence, and can be used to reject the result at the corresponding position in the original tag sequence.
[0105] In some possible implementations, binary classification decoding can be performed based on the second shallow coding features, that is, the decoding space includes only two types of labels: blank and non-blank. In this way, the probability of each decoding position belonging to blank can be obtained.
[0106] In other possible implementations, CTC decoding can be performed based on the second shallow coding features to obtain the tag probability distribution at each decoding position (including the probability distribution of each type of character tag and the blank tag), and the probability of belonging to the blank tag can be obtained from the tag probability distribution at each decoding position, such as... Figure 7 As shown, the audio features (x1, x2, x3, x4…) are fed into the second shallow encoder for encoding to obtain the second shallow encoded features (h1, h2, h3, h4…), which are then fed into the second CTC decoder for softmax classification to obtain the label probability distribution of each decoding position, from which the probability value belonging to the blank label is obtained individually.
[0107] Reference Figure 5 As shown, CTC decoding can be performed using the second CTC decoder based on the second shallow coding features to obtain the label probability distribution at each decoding position, and then the probability of belonging to blank can be selected from it.
[0108] S3. Replace the markers at the target decoding positions in the original marker sequence with blanks to obtain the replaced marker sequence. Then, remove blanks and merge duplicate characters in the replaced marker sequence to obtain the streaming recognition result.
[0109] The result obtained in S2 above can be used for blank rejection. Specifically, for the marker at the target decoding position in the original marker sequence (which may be a blank or a misidentified character marker), this step selects to trust the decoding result of the second CTC decoder, replaces the marker at the target decoding position in the original marker sequence with a blank, obtains the replaced marker sequence, and removes the blank and merges duplicate characters in the replaced marker sequence to obtain the streaming recognition result.
[0110] Taking the original tag sequence from the previous example: c, -, a, a, b, -, t as an example. Assuming that the target decoding position belonging to blank is determined to be the 5th tag position according to step S2, the tag "b" at the 5th tag position in the original tag sequence can be replaced with "-", resulting in the replaced tag sequence: c, -, a, a, -, -, t. Removing blank and merging duplicate characters from this replaced tag sequence yields the streaming recognition result: cat.
[0111] The shallow encoder network structure provided in this embodiment can reduce false triggering of characters in noisy and silent scenarios, thereby improving the accuracy of streaming recognition results.
[0112] Further integration Figure 5 As shown, the first shallow coding feature obtained by the first shallow encoder and the second shallow coding feature obtained by the second shallow encoder can be fused to obtain the fused shallow coding feature.
[0113] The first shallow coding feature focuses more on the real-time text content, while the second shallow coding feature focuses more on the validity of the content (i.e., whether it belongs to a character or blank). The two features complement each other, and their fusion yields a more comprehensive and richer fused shallow coding feature. This fused shallow coding feature is then fed into a deep encoder for further encoding, resulting in deep coding features. The deep decoder then decodes based on these deep coding features to obtain the final recognition result, which can improve the accuracy of the final recognition result.
[0114] In some embodiments of this application, optional structures of a first shallow encoder and a second shallow encoder are provided.
[0115] The first shallow encoder includes a first downsampling module and an N1-layer Conformer module connected in series.
[0116] The process of processing audio features through a first shallow encoder to obtain first shallow encoded features includes:
[0117] The audio features are downsampled by the first downsampling module, and the downsampled audio features are sent to the N1 layer Conformer module for inference calculation to obtain the first shallow-layer encoded features output by the last Conformer module.
[0118] The second shallow encoder includes: a second downsampling module and an N2 layer Conformer module connected in series;
[0119] The process of processing audio features through a second shallow encoder to obtain second shallow encoded features includes:
[0120] The audio features are downsampled by the second downsampling module, and the downsampled audio features are sent to the N2 layer Conformer module for inference calculation to obtain the second shallow-layer encoded features output by the last Conformer module.
[0121] The values of N1 and N2 can be the same or different. In some implementations of this application, considering that the encoding features of the second shallow encoder only need to predict the probability of belonging to blank, the task difficulty is relatively lower, so N2 can be set to be less than N1. For example, N1 is 8 and N2 is 4.
[0122] In some embodiments of this application, the training process of the speech recognition model is further described.
[0123] by Figure 5 Taking the speech recognition model structure shown as an example, the model training process can include the following steps:
[0124] S1. Obtain sample speech segments labeled with streaming recognition result tags and final recognition result tags.
[0125] S2. Input the sample audio features of the sample speech segment into the speech recognition model to obtain the first CTC decoding result output by the first CTC decoder, the second CTC decoding result output by the second CTC decoder, and the final prediction recognition result output by the deep decoder.
[0126] S3. Calculate the first CTC loss based on the first CTC decoding result and the streaming recognition result label, calculate the second CTC loss based on the second CTC decoding result and the streaming recognition result label, and perform weighted fusion of the first CTC loss and the second CTC loss to obtain the CTC weighted loss.
[0127] S4. Calculate the final identification result loss based on the final predicted identification result and the final identification result label.
[0128] S5. Determine the total loss based on the CTC weighted loss and the final identification result loss, and update the model parameters according to the total loss.
[0129] The first CTC loss calculated in S3 above can be expressed as:
[0130]
[0131] Where Y is the label of the streaming recognition result, and X represents the input frame-level audio features. Represents all paths of length T that are mapped to Y. The set, T, represents the number of frames of the input audio features. Indicates the output flag for time step t The probability of.
[0132] The first CTC loss mentioned above uses the standard CTC loss calculation formula.
[0133] The second CTC loss in S3 can be defined as the blank-enhanced CTC loss, which improves upon the standard CTC loss and can be expressed as:
[0134]
[0135]
[0136]
[0137] in, It is a blank enhancement factor greater than 1, where c represents the total number of all types of tags in the decoding space (including multiple character tags and one blank tag).
[0138] The second CTC loss mentioned above can be understood as, for a path of length T that can be mapped to Y... When calculating its probability, if the type of the label with the highest probability obtained at a certain time step j is blank (i.e., in the formula above) If we assign a weight factor greater than 1 to the highest probability value of time step j, we can enhance the confidence that the decoding result is blank.
[0139] Guided by the improved second CTC loss described above, the accuracy of speech recognition models in blank prediction can be enhanced.
[0140] Furthermore, regarding the first CTC loss Second CTC loss Perform weighted fusion to obtain the CTC weighted loss. :
[0141]
[0142] In step S5, based on CTC weighted loss and the loss of the final recognition result Determine the total loss :
[0143]
[0144] Updating the parameters of the entire speech recognition model based on the total loss can help improve the accuracy of the speech recognition model in recognizing streaming results and the final recognition result.
[0145] This application also provides an electronic device in its embodiments. (See reference...) Figure 8 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, tablets, large-screen teaching displays, wearable devices, etc. Figure 8 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0146] like Figure 8 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 2 or a program loaded from a storage device 8 into a random access memory (RAM) 3, to implement the voice recognition method of the foregoing embodiments of this application. When the electronic device is powered on, the RAM 3 also stores various programs and data required for the operation of the electronic device. The processing unit 1, ROM 2, and RAM 3 are interconnected via a bus 4. An input / output (I / O) interface 5 is also connected to the bus 4.
[0147] Typically, the following devices can be connected to I / O interface 5: input devices 6 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 7 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 8 including, for example, memory cards, hard drives, etc.; and communication devices 9. Communication device 9 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.
[0148] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the speech recognition methods provided in this application.
[0149] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the speech recognition methods provided in this application.
[0150] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.
[0151] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0152] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.
[0153] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).
[0154] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.< / begin>
Claims
1. A speech recognition method, characterized in that, include: Acquire the streaming speech segment to be identified and extract the audio features of the streaming speech segment; The audio features are processed by a shallow encoder of a speech recognition model to obtain shallow encoded features. Based on the shallow encoded features, the streaming recognition result is obtained and output in real time as an intermediate process result. The shallow coding features are further encoded by the deep encoder of the speech recognition model to obtain deep coding features, and the final recognition result of the streaming speech segment is obtained by decoding based on the deep coding features.
2. The method according to claim 1, characterized in that, The shallow encoder includes a first shallow encoder and a second shallow encoder; The process of processing the audio features using a shallow encoder to obtain shallow coded features, and then decoding these shallow coded features to obtain streaming recognition results, includes: The audio features are processed by the first shallow encoder to obtain the first shallow encoded features. CTC decoding is performed based on the first shallow encoded features to obtain the original token sequence containing the blank token and the character token. The audio features are processed by the second shallow encoder to obtain the second shallow encoded features. Based on the second shallow encoded features, the probability of each decoding position belonging to the blank tag is predicted, and target decoding positions whose probability of belonging to blank exceeds a set probability threshold are filtered. The markers at the target decoding positions in the original marker sequence are replaced with blanks to obtain a replaced marker sequence. The blanks are then removed and duplicate characters are merged in the replaced marker sequence to obtain the streaming recognition result.
3. The method according to claim 2, characterized in that, The process of further encoding the shallow-level encoded features using a deep encoder to obtain deep-level encoded features includes: The first shallow coding feature and the second shallow coding feature are fused to obtain the fused shallow coding feature; The fused shallow coding features are further encoded by a deep encoder to obtain deep coding features.
4. The method according to claim 2, characterized in that, The first shallow encoder includes: a first downsampling module and an N1-layer Conformer module connected in series; The process of processing the audio features using the first shallow encoder to obtain the first shallow encoded features includes: The audio features are downsampled by the first downsampling module, and the downsampled audio features are sent to the N1 layer Conformer module for inference calculation to obtain the first shallow-layer encoded features output by the last Conformer module.
5. The method according to claim 4, characterized in that, The second shallow encoder includes: a second downsampling module and an N2-layer Conformer module connected in series, where N2 is less than N1; The process of processing the audio features using the second shallow encoder to obtain the second shallow encoded features includes: The audio features are downsampled by the second downsampling module, and the downsampled audio features are sent to the N2 layer Conformer module for inference calculation to obtain the second shallow-layer encoded features output by the last Conformer module.
6. The method according to any one of claims 1-5, characterized in that, Before processing the audio features using a shallow encoder of a speech recognition model, the process further includes: Based on the audio features, the VAD (Voice Activity Detection) of the streaming speech segment is performed to obtain the audio features corresponding to the valid speech segment, which are then used as the input of the speech recognition model.
7. The method according to claim 2, characterized in that, The process of predicting the probability that each decoding position belongs to the blank tag based on the second shallow coding features includes: CTC decoding is performed based on the second shallow coding features to obtain the label probability distribution at each decoding position, and the probability of belonging to the blank label is obtained from the label probability distribution at each decoding position.
8. The method according to claim 7, characterized in that, The training process of the speech recognition model includes: Obtain sample speech segments labeled with streaming recognition result tags and final recognition result tags; The sample audio features of the sample speech segment are fed into the speech recognition model to obtain the first CTC decoding result output by the first CTC decoder, the second CTC decoding result output by the second CTC decoder, and the final prediction recognition result output by the deep decoder. The first CTC decoder is used to perform CTC decoding based on the output features of the first shallow encoder, the second CTC decoder is used to perform CTC decoding based on the output features of the second shallow encoder, and the deep decoder is used to perform decoding based on the output features of the deep encoder. A first CTC loss is calculated based on the first CTC decoding result and the streaming identification result label; a second CTC loss is calculated based on the second CTC decoding result and the streaming identification result label; and the first CTC loss and the second CTC loss are weighted and fused to obtain the CTC weighted loss. The final identification result loss is calculated based on the final predicted identification result and the final identification result label; The total loss is determined based on the CTC weighted loss and the final identification result loss, and the model parameters are updated according to the total loss.
9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the speech recognition method as described in any one of claims 1 to 8.
10. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the speech recognition method as described in any one of claims 1 to 8.
11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the various steps of the speech recognition method as described in any one of claims 1 to 8.