A speech recognition method, device, system and storage medium

Abstract

This application relates to a speech recognition method, apparatus, system, and storage medium. The method includes: acquiring acoustic features of the speech to be recognized and character vectors and phoneme vectors of hot words; obtaining a first encoder output based on the acoustic features; obtaining a first decoder output based on the character vectors; iteratively updating the character vectors and phoneme vectors using optimized transmission to obtain context word vectors; obtaining a second decoder output based on the first decoder output and the context word vectors; obtaining a second encoder output based on the first encoder output and the context word vectors; and obtaining a posterior probability distribution based on the second decoder output and the second encoder output to obtain a speech recognition result. This application has the effect of reducing information loss and improving accuracy.

Description

Technical Field

[0001] This application relates to the technical field of speech recognition, and more specifically to a speech recognition method, apparatus, system and storage medium. Background Technology

[0002] With the development of end-to-end (E2E) technology, the transducer has become one of the mainstream frameworks for automatic speech recognition (ASR). It consists of an encoder, a decoder, and a joint network. The transducer does not require an additional language model (LM) during inference and can achieve good recognition results and response speed in general recognition. However, for commercial speech recognition, it often needs to be customized according to customer needs.

[0003] There is a problem with the mapping method of the transformer to obtain context word vectors in the related technology, which also leads to the limitation of the domain-specific customization of the context-aware transformer. Summary of the Invention

[0004] This application is proposed to address at least one of the aforementioned problems. According to one aspect of this application, a speech recognition method is provided, the method comprising: acquiring acoustic features of the speech to be recognized and character vectors and phoneme vectors of hot words; obtaining a first encoder output based on the acoustic features; obtaining a first decoder output based on the character vectors; iteratively updating the character vectors and phoneme vectors using optimized transmission to obtain context word vectors; obtaining a second decoder output based on the first decoder output and the context word vectors; obtaining a second encoder output based on the first encoder output and the context word vectors; and obtaining a posterior probability distribution based on the second decoder output and the second encoder output to obtain a speech recognition result.

[0005] In one embodiment of this application, the step of iteratively updating the context word vector based on the character vector and the phoneme vector using optimized transmission to obtain the context word vector includes: modeling the character vector and the phoneme vector respectively to obtain high-level representations of the character vector and the phoneme vector; defining a mapping set based on the high-level representation, the mapping set including a transmission matrix that maps the high-level representation to the context word vector; and mapping the high-level representation to the context word vector based on the transmission matrix, wherein the information loss cost of mapping from the high-level representation to the context word vector is minimized.

[0006] In one embodiment of this application, the method, when obtaining the output of the second encoder, is also based on the character vector and the phoneme vector.

[0007] In one embodiment of this application, obtaining a second encoder output based on the first encoder output, the context word vector, the character vector, and the phoneme vector includes: calculating a first similarity between the first encoder output and the context word vector; obtaining a stretching matrix of the character vector and the phoneme vector; calculating a second similarity between the stretching matrix of the first encoder output and the character vector; calculating a third similarity between the stretching matrix of the first encoder output and the phoneme vector; and obtaining the second encoder output based on the first similarity, the second similarity, and the third similarity.

[0008] In one embodiment of this application, obtaining the second encoder output based on the first similarity, the second similarity, and the third similarity includes: summing the first similarity, the second similarity, and the third similarity to obtain a summation result; and fusing the summation result with the first encoder output to obtain the second encoder output.

[0009] In one embodiment of this application, obtaining the first encoder output based on the acoustic features includes: downsampling the acoustic features to obtain a sampling result; and obtaining the first encoder output based on the sampling result.

[0010] In one embodiment of this application, obtaining the second decoder output based on the first decoder output and the context word vector includes: calculating a fourth similarity between the first decoder output and the context word vector; and fusing the fourth similarity with the first decoder output to obtain the second decoder output.

[0011] According to another aspect of this application, a speech recognition device is provided, the device comprising an encoder, a decoder, a combiner, a first bias attention module, a second bias attention module, and a joint network, wherein: the encoder is used to acquire acoustic features of the speech to be recognized, and obtain a first encoder output based on the acoustic features; the decoder is used to acquire character vectors of hot words, and obtain a first decoder output based on the character vectors; the combiner is used to acquire character vectors and phoneme vectors of the hot words, and perform iterative updates using optimized transmission based on the character vectors and the phoneme vectors to obtain context word vectors; the first bias attention module is used to obtain a second decoder output based on the first decoder output and the context word vectors; the second bias attention module is used to obtain a second encoder output based on the first encoder output and the context word vectors; the joint network is used to obtain a posterior probability distribution based on the second decoder output and the second encoder output to obtain a speech recognition result.

[0012] According to another aspect of this application, a speech recognition system is provided, the system including a memory and a processor, the memory storing a computer program executed by the processor, the computer program, when executed by the processor, causing the processor to perform the speech recognition method described above.

[0013] According to another aspect of this application, a storage medium is provided, on which a computer program is stored, wherein the above-described speech recognition method is executed when the computer program is running.

[0014] According to the speech recognition method in this application, the context word vector is obtained by iteratively updating through optimized transmission. This enables the mapping result to more accurately calculate the attention score of the corresponding hot word based on the output of the encoder and decoder at each moment of the transformer. It also ensures that the information loss after mapping is minimized. This can improve the problem of limited domain personalization of the variable context-aware transformer speech recognition method, meet the user's need for fast response, and ensure the effect of streaming speech recognition. Attached Figure Description

[0015] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 A flowchart of a context-aware transducer speech recognition method in related technologies is shown.

[0017] Figure 2 A flowchart of the speech recognition method in an embodiment of this application is shown.

[0018] Figure 3 A flowchart of the speech recognition method in an embodiment of this application is shown.

[0019] Figure 4 A structural block diagram of the speech recognition device in an embodiment of this application is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of this application, and not all of the embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0021] In related technologies, a common approach to address the personalized domain customization problem in speech recognition methods is shallow fusion (SF). This requires introducing an additional language model based on a neural network (NN) or a weighted finite state transducer (WFST). While NN language models can meet domain customization requirements, they degrade general recognition performance. WFST language models, while meeting both general and domain customization needs, suffer from high memory usage, hindering deployment on low- to medium-powered computing devices. Recording is another common strategy. It reorders the phoneme recognition results from the transducer acoustic model and transcribes them into Chinese / English characters. Since recording models commonly use WFST, this approach heavily relies on pronunciation dictionaries and fixed sentence structures. During inference, limited device computing power leads to pruning of the decoding path, resulting in poorer recognition performance. Personalized domain customization of transducers is an urgent and pressing problem that needs to be solved.

[0022] In existing solutions, introducing a bias encoder is a good strategy for personalized transducer speech recognition. It primarily determines whether to use hotword information based on different attention scores. This strategy not only handles general recognition but also allows for customization of hotwords within the domain, without relying on fixed sentence structures. Therefore, our existing technical solution is a context-aware transducer speech recognition method. Figure 1As shown, unlike traditional converters, the network is divided into six parts: encoder, decoder, bias encoder, combiner, bias attention, and joint network.

[0023] In related technologies, context-aware transducer speech recognition methods, such as... Figure 1 As shown, the encoder output and decoder output are obtained. A high-level representation is obtained based on the phoneme vector and character vector of the hot words. The high-level representation is used as the input of the combiner to obtain the context word vector. The first similarity between the decoder output and the context word vector is obtained through the first bias attention module. The second similarity between the encoder output and the context word vector is obtained through the second bias attention module. The first and second similarities are then used as the input of the joint network. Finally, the posterior probability distribution of the transformer is obtained through the function layer for speech recognition.

[0024] like Figure 1 As shown, the specific process of the speech recognition method is as follows:

[0025] In the encoder section, the input acoustic features X are downsampled by a factor of four using the VGG module, and the sampled result is then input into layer N. To ensure the encoder's streaming nature, the convolutional module employs causal convolution, and self-attention uses a mask to implement contextual information for the left visual field 16 and the right visual field 0. Finally, the encoder output h at frame t. t As shown in formula (1):

[0026] h t =encoder(x t-16:t (1)

[0027] Where X = (x1, x2, ..., x T ), h t ∈ 1×d d is the vector dimension.

[0028] In the decoder section, the grapheme vector Y is used as the input to the M-th convolutional layer. Here, the u-th grapheme vector h... u The decoder output is shown in equation (2):

[0029] h u =decoder(y u-2:u (2)

[0030] Where Y = (y1, y2, ..., y U ), h u ∈ 1×d Since the decoder has little impact on the transformer, a one-dimensional convolution with a kernel of 3 is used for computation.

[0031] For the pixel vector g of the k-th hot word respectively k and phoneme vector p k The model is built using an L-layer BERT, and the character vector corresponding to the k-th hot word is... and phoneme vectors The senior management stated:

[0032]

[0033]

[0034] in, s g and s p Let be the lengths of the character vector and phoneme vector of the k-th hot word. For K hot words, and Let be the lengths of the longest character vector and phoneme vector among the K hot words. Here, considering that general recognition does not have a corresponding hot word during training and inference, one of the K words is needed.<no_bias> The hot words represent the word vectors corresponding to general recognition.

[0035] Next, the high-level representations of the character vectors and factor vectors are used as input to the combiner module to calculate context embeddings (CE), i.e., CE information. There are two commonly used calculation methods. The first is to add a prefix to each hot word... <cls>Symbol, then take <cls>The corresponding CE information. The second method is to follow along... and The CE information is obtained by calculating the mean of the d-dimensional values. Then, the CE information is fused along the d-dimensional plane and mapped to h through a fully connected (FC) and LayerNorm (LN) layer. CE Therefore, the combiner module mainly consists of formulas (5) to (7), as shown below:

[0036] h CE,g =f(h g (5)

[0037] h CE,p =f(h p (6)

[0038] h CE =LN(FC([h CE,g h CE,p ])) (7)

[0039] Where f() is <cls>The two methods are mean and mean, h CE,g ,h CE,p ,h CE ∈ K×1×d h CE The CE information represents the hidden state of the Lth layer BERT for the character vector and phoneme vector.

[0040] The bias attention module's function is to sum the output of the encoder and decoder at each time step with h. CE The similarity is calculated and fused. It mainly consists of two steps. The first step is to calculate the attention score using Multi-Head Attention (MHA). This score represents the similarity between the encoder / decoder output and the CE information. The encoder / decoder output is used as the query, h... CE The attention score is calculated using MHA as the key and value. The second step is to fuse the attention score with the encoder / decoder output through concatenation and mapping operations.

[0041] For encoder output h t The attention score between the t-th speech frame and the K hot words is calculated using a biased attention module. Then with h t Along d-dimensional fusion to obtain

[0042]

[0043]

[0044] For decoder output h u It can calculate the attention score between the u-th element vector and the K hot words through the bias attention module. Then with h u Along d-dimensional fusion to obtain

[0045]

[0046]

[0047] in, In MHA, we use an attention module that contains only one head.

[0048] Ultimately, and As input to the joint network, the result is then passed through a softmax layer to obtain the posterior probability distribution of the transformer.

[0049]

[0050] P(y|t,u)=softamx(z t,u (13)

[0051] Next, during the training phase, 30,000 hours of commercial data were used for training. First, a part-of-speech tagging tool was used to label nouns. Then, in each batch, P was applied to each sample. sent The probability selection is as follows: If the sample contains multiple nouns, then P is applied to each noun. noun The probability determines whether to include a keyword in the K hot words. Where P... sent >P noun For personalized, domain-specific customization, general recognition performance also needs to be considered. Besides...<no_bias> In addition to trending keywords, we will randomly select N from non-current batches. phrase One counterexample enhances the converter's resistance to crosstalk in general identification. Where N phrase With P sent Proportional.

[0052] During the inference phase, tests were conducted on three different datasets. The first dataset only loaded...<no_bias> The first type is a general recognition of 60,000 hot words, denoted as S1. The second type is personalized domain customization, taking video search scenarios as an example, with test sets of 200, 1000, 3000, and 5000 hot words, denoted as S2, S3, S4, and S5. The third type is a general recognition of 60,000 hot words with 200, 1000, 3000, and 5000 hot words, denoted as S6, S7, S8, and S9. For the first and third test sets, word accuracy (%) is used as the metric; for the second test set, slot accuracy (%) is used. See Table 1 below for details.

[0053]

[0054] Table 1: Comparison of speech recognition performance between traditional transformers and context-aware transformers

[0055] In the table above, the first row represents traditional transformer models, while the second and third rows represent context-aware transformer models. The second row uses the CLS-based method, and the third row uses the MAIN-based method. The MAIN-based context-aware transformer performs better than the CLS-based one. (The last sentence appears to be incomplete and possibly refers to loading only...)<no_bias> In case S1 (hot words), the traditional and context-aware transformers perform similarly in general recognition. Comparisons from S2 to S5 show that the slot accuracy of the context-aware transformer model gradually deteriorates as the number of hot words increases, but overall it still outperforms the traditional transformer model. For results S6 to S9, the degradation in general recognition becomes increasingly severe as the number of hot words loaded increases. Currently, the existing context-aware transformer technology shows good positive improvement in slot accuracy when the number of hot words is 200, while having no impact on general recognition.

[0056] Based on this, the speech recognition method of the context-aware transformer in the related technology has the problem of limited domain personalization customization. One of the reasons is that its method of calculating context word vectors (CE) is not reasonable enough. To address this problem, the speech recognition method of the embodiment of this application is proposed.

[0057] like Figure 2 As shown in the figure, this application discloses a speech recognition method, which includes:

[0058] S11. Obtain the acoustic features of the speech to be recognized and the character vectors and phoneme vectors of hot words.

[0059] S12. Obtain the output of the first encoder based on acoustic features.

[0060] S13. Obtain the output of the first decoder based on the character vector.

[0061] S14. Based on the character vector and the phoneme vector, iterative updates are performed using optimal transmission to obtain the context word vector.

[0062] S15. Obtain the output of the second decoder based on the output of the first decoder and the context word vectors.

[0063] S16. Obtain the output of the second encoder based on the output of the first encoder and the context word vectors.

[0064] S17. Obtain the posterior probability distribution based on the output of the second decoder and the output of the second encoder to obtain the speech recognition result.

[0065] According to the speech recognition method of this application embodiment, the context word vector is obtained by iteratively updating through optimized transmission. The iterative update based on optimized transmission enables the mapping result to more accurately calculate the attention score of the corresponding hot word based on the output of the encoder and decoder at each moment. It can also ensure that the information loss after mapping is minimized. This can improve the problem of limited domain personalization customization of the variable context-aware transformer speech recognition method, meet the user's need for rapid response, and ensure the effect of streaming speech recognition.

[0066] For example, the first encoder output is obtained based on acoustic features, including downsampling the acoustic features to obtain a sampling result, and obtaining the first encoder output based on the sampling result. For example, the downsampling process can use a four-fold downsampling to obtain the sampling result, which is then input into the N-layer configuration of the encoder. To ensure the encoder's streaming nature, the convolutional module uses causal convolution, and self-attention uses a mask to implement the contextual information of the left field of view 16 and the right field of view 0. Finally, the encoder output h of the t-th frame is obtained. t .

[0067] For example, context word vectors are obtained by iteratively updating based on character vectors and phoneme vectors using optimized transfer, including: modeling character vectors and phoneme vectors respectively to obtain high-level representations of character vectors and phoneme vectors; defining a mapping set based on the high-level representations, wherein the mapping set includes a transfer matrix that maps the high-level representations to context word vectors; and mapping the high-level representations to context word vectors based on the transfer matrix, wherein the information loss cost of mapping from high-level representations to context word vectors is minimized.

[0068] For example, the above mapping method optimizes f() in equations (5) and (6) into an optimal transport mapping.

[0069] In the f() optimization section, the size needs to be... The matrix is ​​mapped to a matrix of size K×1×d. This mapping path is not unique, and the set of possible mappings is defined as follows:

[0070]

[0071] Where a is b is h CE,g ,h CE,p ∈K×1×d, n is m is 1. P is the transfer matrix that maps a to b. I m and I n It is a vector of length m and n consisting entirely of 1s.

[0072] Since m≠n, the optimal transmission problem needs to be transformed into a Kantorovich relaxation optimal transmission problem to ensure that the cost C of information loss is minimized throughout the entire transmission process.

[0073]

[0074] Wherein, the cost matrix, C∈ n×m <> represents the inner product of two matrices of the same size.

[0075] In most cases, finding the standard Kondratiev solution is unnecessary. If regularization is used to find an approximate solution, the computational cost will be significantly reduced. The entropy-based regularization cost function is defined as shown in equation (16), and substituting it into equation (15) yields equation (17):

[0076] H(P)=-∑ i,j P i,j (log(P i,j -1)) (16)

[0077]

[0078] Where ε is the regularization coefficient, and f∈ n ,g∈ m With two dual variables, formula (17) is obtained by the Lagrange multiplier method:

[0079] L(P,f,g)=<C,P>-εH(P)-<f,PI m -a>-<g,P T I n -b> (18)

[0080] The inner product of the last two terms is 0, so it will not affect the result.

[0081] Find the first derivative of formula (18) with respect to P:

[0082]

[0083] Therefore,

[0084]

[0085] in,

[0086] Formula (20) can be written in matrix form as shown in formula (21). Substituting PI... m =a,P T I n =b yields formula (22):

[0087] P=diag(u)Kdiag(v) (21)

[0088] diag(u)Kdiag(v)I m =a,diag(v)Kdiag(u)I n =b (22)

[0089] Furthermore, it can be written in the form of matrix multiplication and dot product:

[0090] u(Kv)=a,v(K T u)=b (23)

[0091] Formula (23) can be written in iterative form:

[0092]

[0093] Among them, v (0) =I m Let l be the number of iterations, typically 10. For K, let K represent the cost matrix, which consists of a learnable matrix and h. g ,h p The computation is generated and minimized during the backpropagation phase. Finally, the high-level representations corresponding to the character vectors and phoneme vectors are mapped into new CE information using the OT method.

[0094] The above method is used to obtain context word vectors (CE information). Through iterative updates, the method is used to find words of size [missing information]. The matrix is ​​mapped to K×1×d. The optimal mapping method for the K×1×d matrix, compared with the methods of related technologies, not only can more accurately calculate the attention score of the corresponding hot words based on the output of the transformer encoder and decoder at each time step, but also ensures that the information loss after mapping is minimized.

[0095] For example, when obtaining the output of the second encoder, this method is based not only on the output of the first encoder and the context word vectors, but also on the character vectors and phoneme vectors. That is, the output of the first encoder needs to calculate the attention score not only with the K hot words, but also with the character and phoneme of each hot word.

[0096] For example, such as Figure 3 As shown, the second encoder output is obtained based on the first encoder output, context word vectors, character vectors, and phoneme vectors. This includes: calculating a first similarity between the first encoder output and the context word vectors, and obtaining the stretching matrices of the character vectors and phoneme vectors. Then, a second similarity is calculated between the first encoder output and the stretching matrices of the character vectors, and a third similarity is calculated between the first encoder output and the stretching matrices of the phoneme vectors. Finally, the second encoder output is obtained based on the first, second, and third similarities.

[0097] Please refer to Figure 3 The process of obtaining the output of the second encoder is as follows: In order to introduce the relationship between the element vectors of each hot word, the output matrix h of formulas (3) and (4) is used. g ,h p Stretch into At this point, the model treats each character / phoneme as a hot word. In character form, the number of hot words is... In phoneme form, the number of hot words is Since the benefits of CE information interacting with the encoder output are greater, only equations (8) and (9) are optimized without modifying the second decoder output. Figure 3 The biased attention module introduces character-based attention units and phoneme-based attention units, used to calculate the second and third similarities, respectively. The calculations for the first, second, and third similarities are as follows:

[0098]

[0099]

[0100]

[0101]

[0102] in, It is the output of the character vectors and phoneme vectors corresponding to K hot words after passing through a combiner based on Optimal Transport (OT). For each frame h of the encoder, t Not only do we need to calculate the similarity with K hot words, but we also need to calculate the similarity with Calculate the similarity. Finally, Equation (28) represents the attention score of the similarity. and Summing, then dividing by h t By splicing along the d-dimensional axis, a new encoder output is obtained.

[0103] For the decoder part, the second decoder output is obtained based on the first decoder output and the context word vectors, including: calculating the similarity between the first decoder output and the context word vectors to obtain a fourth similarity, and fusing the fourth similarity with the first decoder output to obtain the second decoder output, as shown in formulas (10)-(11). Because the interaction between the context word vectors and the decoder yields less benefit, in this embodiment, the similarity between the phoneme vector stretching matrix and the character vector stretching matrix and the first decoder output is not considered.

[0104] After obtaining the outputs of the second encoder and the second decoder, the posterior probability distribution is obtained based on the second decoder output and the second encoder output to obtain the speech recognition result. In the calculation, the formula (12) can be used... Replace with

[0105] Table 2 below shows a comparison of the recognition performance of the context-aware transducer speech recognition method based on optimized transmission.

[0106]

[0107] Table 2 shows that, from S2 to S5, the slot accuracy remains above 91% as the number of hot keywords increases. Notably, with 5000 hot keywords loaded (the bolded portion of Table 2), the slot accuracy is 91.04%, meeting the requirements of current commercial versions. This compares to S1 (with only 5000 hot keywords loaded).<no_bias> S6 to S9 (loading different numbers of hot words) have little impact on the crosstalk of the general recognition effect, and the general effect does not deteriorate significantly with the number of hot words loaded.

[0108] Based on the above method of the embodiments of this application, the context word vector is obtained by iteratively updating through optimized transmission. The iterative update method based on optimized transmission enables the mapping result to more accurately calculate the attention score of the corresponding hot word based on the output of the encoder and decoder at each moment. It can also ensure that the information loss after mapping is minimized. It can improve the problem that the domain personalization customization of the variable context-aware transformer speech recognition method is limited, so as to meet the user's need for rapid response and ensure the effect of streaming speech recognition.

[0109] like Figure 4 As shown, this application also discloses a speech recognition device, which includes an encoder, a decoder, a combiner, a first bias attention module, a second bias attention module, and a joint network.

[0110] The system consists of: an encoder to acquire acoustic features of the speech to be recognized, and a first encoder output based on these features; a decoder to acquire character vectors of hot words, and a first decoder output based on these character vectors; a combiner to acquire character vectors and phoneme vectors of hot words, and iteratively updates these vectors using optimized transmission to obtain context word vectors; a first bias attention module to obtain the second decoder output based on the first decoder output and the context word vectors; and a second bias attention module to obtain the second encoder output based on the first encoder output and the context word vectors. A joint network to obtain the posterior probability distribution based on the second decoder output and the second encoder output to obtain the speech recognition result.

[0111] For example, the combiner iteratively updates context word vectors based on character vectors and phoneme vectors using optimized transfer, including: modeling character vectors and phoneme vectors respectively to obtain high-level representations of character vectors and phoneme vectors; defining a mapping set based on the high-level representations, the mapping set including a transfer matrix that maps the high-level representations to context word vectors; and mapping the high-level representations to context word vectors based on the transfer matrix, where the information loss cost of mapping from high-level representations to context word vectors is minimized.

[0112] For example, when obtaining the output of the second encoder, the second bias attention module also bases it on the character vector and the phoneme vector. That is, the second bias attention module obtains the second encoder output based on the first encoder output, the context word vector, the character vector, and the phoneme vector. For example, the process of obtaining the second encoder output based on the first encoder output, the context word vector, the character vector, and the phoneme vector includes: calculating a first similarity between the first encoder output and the context word vector; obtaining a stretching matrix of the character vector and the phoneme vector; calculating a second similarity between the first encoder output and the stretching matrix of the character vector; calculating a third similarity between the first encoder output and the stretching matrix of the phoneme vector; and obtaining the second encoder output based on the first similarity, the second similarity, and the third similarity. The calculation of the first similarity, the second similarity, and the third similarity is as follows:

[0113]

[0114]

[0115]

[0116] The second bias attention module obtains the second encoder output based on the first similarity, second similarity, and third similarity, including: summing the first similarity, second similarity, and third similarity to obtain a summation result, and fusing the summation result with the first encoder output to obtain the second encoder output, as shown in the following formula.

[0117]

[0118] According to the speech recognition device of this application embodiment, the context word vector is obtained by iteratively updating through optimized transmission. The iterative update method based on optimized transmission enables the mapping result to more accurately calculate the attention score of the corresponding hot word based on the output of the encoder and decoder at each moment. It can also ensure that the information loss after mapping is minimized. This can improve the problem of limited domain personalization of the variable context-aware transformer speech recognition method, meet the user's need for rapid response, and ensure the effect of streaming speech recognition.

[0119] Furthermore, in the scheme of this application embodiment, it is necessary not only to calculate attention scores with K hot words, but also to calculate attention scores with the character elements and phonemes of each hot word. Through this method, the output of the converter encoder / decoder at each moment can calculate attention scores between and within hot words, thereby solving the problem that the character element recognition effect is better at the beginning and end of longer hot words, but worse in the middle.

[0120] This application also discloses a speech recognition system, which includes a memory and a processor. The memory stores a computer program that is run by the processor. When the computer program is run by the processor, it causes the processor to execute the speech recognition method described above.

[0121] This application also discloses a storage medium storing a computer program, which executes the above-described speech recognition method when running.

[0122] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0123] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0125] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0126] Similarly, it should be understood that, in order to streamline this application and aid in understanding one or more of the various inventive aspects, features of this application may sometimes be grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with features fewer than all features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0127] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0128] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0129] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0130] It should be noted that the above embodiments are illustrative of this application and not limiting of it, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0131] The above description is merely a specific embodiment or illustration of the embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.< / cls> < / cls> < / cls>

Claims

1. A speech recognition method, characterized in that, The method includes: Acquire the acoustic features of the speech to be recognized and the character vectors and phoneme vectors of hot words; The first encoder output is obtained based on the acoustic features; The first decoder output is obtained based on the character vector; Based on the character vector and the phoneme vector, the context word vector is obtained by iteratively updating using optimal transmission. The second decoder output is obtained based on the first decoder output and the context word vectors; The second encoder output is obtained based on the first encoder output and the context word vectors; The posterior probability distribution is obtained based on the output of the second decoder and the output of the second encoder to obtain the speech recognition result; The process of iteratively updating the context word vector based on the character vector and the phoneme vector using optimized transmission includes: The character vector and the phoneme vector are modeled separately to obtain high-level representations of the character vector and the phoneme vector; Based on the high-level representation, a mapping set is defined, the mapping set including a transfer matrix that maps the high-level representation to the context word vectors; The high-level representation is mapped to the context word vector based on the transfer matrix, wherein the information loss cost of mapping from the high-level representation to the context word vector is minimized.

2. The method according to claim 1, characterized in that, When obtaining the output of the second encoder, the method also relies on the character vector and the phoneme vector.

3. The method according to claim 2, characterized in that, The second encoder output is obtained based on the first encoder output, the context word vector, the character vector, and the phoneme vector, including: A first similarity is obtained by calculating the similarity between the output of the first encoder and the context word vectors. Obtain the stretching matrices of the character vector and the phoneme vector; A second similarity is obtained by performing a similarity calculation on the stretching matrix of the first encoder output and the character vector; A third similarity is obtained by performing a similarity calculation on the stretching matrix of the first encoder output and the phoneme vector; The second encoder output is obtained based on the first similarity, the second similarity, and the third similarity.

4. The method according to claim 3, characterized in that, The process of obtaining the second encoder output based on the first similarity, the second similarity, and the third similarity includes: The summation result is obtained by summing the first similarity, the second similarity, and the third similarity. The summation result is fused with the output of the first encoder to obtain the output of the second encoder.

5. The method according to any one of claims 1-4, characterized in that, The process of obtaining the first encoder output based on the acoustic features includes: The acoustic features are downsampled to obtain the sampling results; The first encoder output is obtained based on the sampling results.

6. The method according to any one of claims 1-4, characterized in that, The process of obtaining the second decoder output based on the first decoder output and the context word vectors includes: A fourth similarity is obtained by calculating the similarity between the output of the first decoder and the context word vectors. The fourth similarity is fused with the output of the first decoder to obtain the output of the second decoder.

7. A voice recognition device, characterized in that, The device includes an encoder, a decoder, a combiner, a first bias attention module, a second bias attention module, and a joint network, wherein: The encoder is used to acquire the acoustic features of the speech to be recognized, and the first encoder output is obtained based on the acoustic features; The decoder is used to obtain the character vectors of hot words, and the output of the first decoder is obtained based on the character vectors; The combiner is used to obtain the character vector and phoneme vector of the hot word, and iteratively update the context word vector using optimized transmission based on the character vector and phoneme vector. The iterative update using optimized transmission to obtain the context word vector includes: modeling the character vector and phoneme vector respectively to obtain high-level representations of the character vector and phoneme vector; defining a mapping set based on the high-level representation, the mapping set including a transmission matrix that maps the high-level representation to the context word vector; and mapping the high-level representation to the context word vector based on the transmission matrix, wherein the information loss cost of mapping from the high-level representation to the context word vector is minimized. The first bias attention module is used to obtain the output of the second decoder based on the output of the first decoder and the context word vectors; The second bias attention module is used to obtain the second encoder output based on the first encoder output and the context word vectors; The joint network is used to obtain the posterior probability distribution based on the output of the second decoder and the output of the second encoder, so as to obtain the speech recognition result.

8. A speech recognition system, characterized in that, The system includes a memory and a processor, the memory storing a computer program executed by the processor, the computer program, when executed by the processor, causing the processor to perform the speech recognition method as described in any one of claims 1-6.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when running, executes the speech recognition method as described in any one of claims 1-6.

Images