Speech recognition method, device, equipment, medium and product

Abstract

The invention provides a speech recognition method and device, equipment, a medium and a product, and the method comprises the steps: obtaining a first prediction result and a second prediction result of a to-be-recognized speech, the first prediction result comprises a prediction result obtained based on a candidate text recognition result of the to-be-recognized speech, and the second prediction result comprises a prediction result obtained based on a candidate text recognition result of the to-be-recognized speech; the second prediction result comprises a prediction result obtained based on the to-be-recognized voice; determining a first weight corresponding to the first prediction result, and determining a second weight corresponding to the second prediction result; and performing feature fusion on the first prediction result and the second prediction result based on the first weight and the second weight, and determining a recognition result of the to-be-recognized voice based on the obtained fusion feature. The voice recognition effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method, device, equipment, medium and product. Background Art

[0002] With the continuous development of science and technology, speech recognition technology has been widely used in various industries. Speech recognition technology refers to the conversion of human speech into text or instructions that can be understood by machines, so that users can interact with machines through voice. In real life, speech signals are easily interfered by various noises, which affects the speech recognition effect.

[0003] In order to improve the speech recognition effect in a noisy environment, an acoustic model specially trained for a noisy environment is usually used to extract acoustic features, and then speech recognition is performed based on the acoustic features. Alternatively, speech recognition is performed in combination with other modal data in the hope of improving the speech recognition effect in a noisy environment.

[0004] However, the current processing method still has poor effect on speech recognition in noisy environments and cannot meet user needs. Summary of the invention

[0005] Based on the above technical status, the present application proposes a speech recognition method, device, equipment, medium and product, which can improve the accuracy of speech recognition in noisy environments.

[0006] In order to achieve the above technical objectives, this application specifically proposes the following technical solutions:

[0007] In a first aspect, the present application proposes a speech recognition method, comprising: obtaining a first prediction result and a second prediction result of a speech to be recognized, the first prediction result including a prediction result obtained based on a candidate text recognition result of the speech to be recognized, and the second prediction result including a prediction result obtained based on the speech to be recognized; determining a first weight corresponding to the first prediction result, and determining a second weight corresponding to the second prediction result; based on the first weight and the second weight, performing feature fusion on the first prediction result and the second prediction result, and determining a recognition result of the speech to be recognized based on the obtained fusion feature.

[0008] In some implementations, determining the first weight corresponding to the first prediction result and determining the second weight corresponding to the second prediction result include: determining the first weight corresponding to the first prediction result based on the predicted posterior entropy of the frame prediction results corresponding to each speech frame in the speech to be recognized; and determining the second weight corresponding to the second prediction result based on the first weight corresponding to the first prediction result.

[0009] In some implementations, the predicted a posteriori entropy of the frame prediction results corresponding to each speech frame in the speech to be recognized is determined by the following steps: based on the prediction probability of the frame prediction results corresponding to each speech frame in the speech to be recognized by the large language model, the predicted a posteriori entropy of the frame prediction results corresponding to each speech frame in the speech to be recognized is determined.

[0010] In some implementations, determining the first weight corresponding to the first prediction result based on the predicted a posteriori entropy of the frame prediction results corresponding to each of the speech frames includes: processing the predicted a posteriori entropy of the frame prediction results corresponding to each of the speech frames through an activation function to determine the first weight corresponding to the first prediction result; or determining the first weight corresponding to the first prediction result based on a comparison result of the predicted a posteriori entropy of the frame prediction results corresponding to each of the speech frames with a preset entropy.

[0011] In some implementations, the first prediction result and the second prediction result are feature fused based on the first weight and the second weight, and the recognition result of the speech to be recognized is determined based on the obtained fused features, including: based on the first weight corresponding to the first prediction result and the second weight corresponding to the second prediction result, a first eigenvalue corresponding to the first prediction result and a second eigenvalue corresponding to the second prediction result are weightedly summed to obtain a fused eigenvalue; and the fused eigenvalue is processed by a softmax function to determine the recognition result of the speech to be recognized.

[0012] In some implementations, the first prediction results include multiple first prediction results, each of which corresponds to a first weight; based on the first weight corresponding to the first prediction result and the second weight corresponding to the second prediction result, the first feature value corresponding to the first prediction result and the second feature value corresponding to the second prediction result are weighted summed to obtain a fused feature value, including: for each first prediction result in the multiple first prediction results, based on the first weight corresponding to the first prediction result and the second weight corresponding to the second prediction result, the first feature value corresponding to the first prediction result and the second feature value corresponding to the second prediction result are weighted summed to obtain a fused feature value corresponding to the first prediction result; the fused feature value is processed by a softmax function to determine the recognition result of the speech to be recognized, including: for the fused feature value corresponding to each first prediction result, the fused feature value is processed by a softmax function to determine the initial recognition result of the speech to be recognized; based on the initial recognition results corresponding to each of the multiple first prediction results, the final speech recognition result is determined.

[0013] In some implementations, determining the final speech recognition result based on the recognition results corresponding to each of the multiple first prediction results includes: determining the scores of the initial recognition results corresponding to each of the multiple first prediction results; and selecting the initial recognition result corresponding to the highest score to determine it as the final speech recognition result.

[0014] In some implementations, the candidate text recognition result is the candidate recognition result output by the end-to-end speech recognition model for recognizing the speech to be recognized, the first prediction result is the prediction result obtained by rescoring the candidate text recognition result through a large language model; and the second prediction result is the output result of the last layer of the decoder when the end-to-end speech recognition model processes the speech to be recognized.

[0015] The second aspect of the present application proposes a speech recognition device, comprising: an acquisition unit, used to acquire a first prediction result and a second prediction result of a speech to be recognized, the first prediction result including a prediction result obtained based on a candidate text recognition result of the speech to be recognized, and the second prediction result including a prediction result obtained based on the speech to be recognized; a weight determination unit, used to determine a first weight corresponding to the first prediction result, and to determine a second weight corresponding to the second prediction result; a fusion unit, used to perform feature fusion on the first prediction result and the second prediction result based on the first weight and the second weight, and determine the recognition result of the speech to be recognized based on the obtained fusion feature.

[0016] A third aspect of the present application provides an electronic device, including a memory and a processor;

[0017] The memory is connected to the processor and is used to store programs;

[0018] The processor is used to implement the speech recognition method described in any one of the first aspect and the implementation manner of the first aspect by running the program in the memory.

[0019] In a fourth aspect, the present application proposes a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the speech recognition method described in the first aspect and any one of the implementation methods of the first aspect is implemented.

[0020] In a fifth aspect, the present application proposes a computer program product, comprising computer program instructions, which, when executed by a processor, enable the processor to implement the speech recognition method described in the first aspect and any one of the implementation methods of the first aspect.

[0021] The speech recognition method, device, equipment, medium and product of the embodiment of the present application, the method obtains a first prediction result obtained based on the candidate text recognition result of the speech to be recognized, and a second prediction result obtained based on the speech to be recognized, and determines a first weight corresponding to the first prediction result and a second weight corresponding to the second prediction result, and based on the first weight and the second weight, performs feature fusion on the first prediction result and the second prediction result, and determines the recognition result of the speech to be recognized based on the obtained fusion feature. On the one hand, since the second prediction result obtained based on the speech to be recognized is introduced as information in the acoustic mode in the speech recognition process, and the first prediction result in the text mode and the second prediction result in the acoustic mode are jointly decoded, the information in the acoustic mode is used to compensate for the information in the text mode to improve the speech recognition performance. On the other hand, weights are determined for the first prediction result and the second prediction result respectively to adjust the importance of the first prediction result and the second prediction result for the joint decoding process, and the first prediction result and the second prediction result are jointly decoded based on the different importance of the joint decoding process, which can further improve the speech recognition performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0023] Figure 1 A schematic diagram of the principle of optimizing speech recognition effect based on a large language model in the related art;

[0024] Figure 2 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of a first prediction result of a speech to be recognized provided in an embodiment of the present application;

[0026] Figure 4 A flowchart of an implementation process of determining a first weight and a second weight provided in an embodiment of the present application;

[0027] Figure 5 A flowchart of the feature fusion process provided in the embodiment of the present application;

[0028] Figure 6 A framework diagram of the speech recognition method provided in the embodiment of the present application;

[0029] Figure 7A schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solution proposed in the embodiment of the present application is applicable to various speech recognition application scenarios. For example, in the intelligent customer service scenario, speech recognition technology allows users to quickly communicate with customer service personnel to solve user problems, thereby improving the efficiency and quality of customer service. In the smart home scenario, speech recognition technology allows users to control home appliances through voice, improving the convenience of life. In the in-car entertainment scenario, speech recognition technology can help users control the operation of music playback, navigation and other functions through voice, improving driving safety.

[0032] The above examples are some application scenarios of speech recognition technology. In practice, the application scenarios of speech recognition technology are very wide, and they are not listed one by one here. The speech recognition technology of the embodiment of the present application can be applied to any speech recognition application scenario to improve the accuracy of speech recognition in any application scenario.

[0033] The technical solution provided in the embodiments of the present application can be exemplarily applied to hardware devices such as processors, electronic devices, servers (including cloud servers), or packaged into software programs to be run. When the hardware device executes the processing of the technical solution of the embodiments of the present application, or the above-mentioned software program is run, the automatic splitting of the target task and the automatic calling of the application program interface required for the task can be achieved to complete the purpose of the target task. The embodiments of the present application only exemplarily introduce the specific processing of the technical solution of the present application, and do not limit the specific implementation form of the technical solution of the present application. Any technical implementation form that can execute the processing of the technical solution of the present application can be adopted by the embodiments of the present application.

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] Before introducing this application solution, the relevant technologies are first introduced:

[0036] With the rapid development of artificial intelligence technology, automatic speech recognition (ASR) has become a popular technology. ASR technology can convert the speaker's voice into text, helping people communicate more efficiently and conveniently.

[0037] The traditional ASR technology was initially developed. It uses a multi-stage speech signal processing algorithm, including speech feature extraction, acoustic model training, language model training and decoding. The traditional ASR technology not only has the problem of complex system design, but also has delays and errors between the processing processes of each stage, resulting in unsatisfactory final speech recognition results.

[0038] In order to improve the speech recognition effect, end-to-end automatic speech recognition technology has emerged. Unlike traditional ASR technology, end-to-end speech recognition technology does not require multiple processing of speech signals, multiple stages of feature extraction, noise reduction and speech recognition operations. After pre-processing the speech signal, end-to-end speech recognition technology directly inputs it into a neural network, and the output of the neural network is the corresponding speech recognition result. This reduces errors and delays in the recognition process and improves the speech recognition effect.

[0039] However, in practical applications, speech signals are often subject to many interferences, such as environmental noise, diversity of languages ​​and dialects, changes in accents and speaking speed, and individual differences, which make the speech signal less clear and thus affect the accuracy of end-to-end speech recognition results. Therefore, how to improve the end-to-end speech recognition effect has become a problem that needs to be solved.

[0040] Taking the noise scene as an example, in order to improve the end-to-end speech recognition effect, one way to deal with it is to train an acoustic model specifically for the noise scene so that it can better learn the speech characteristics in the noise environment, adapt to the acoustic characteristics of the noise scene, and improve the speech recognition effect. However, in actual applications, when the noise type in the application scene does not match the noise type in the training scene, the speech recognition effect will drop sharply.

[0041] In another processing method, the end-to-end speech recognition effect can be optimized through other modal data. For example, for the N-best prediction results output by the end-to-end speech recognition model, the N-best prediction results are re-scored in combination with language models such as n-gram models or neural network language models (LM), and the final speech recognition result is determined based on the re-scoring results. However, the speech recognition effect is still not ideal.

[0042] In recent years, large language models (LLM, hereinafter referred to as large models) have been gradually widely used in the field of natural language processing (NLP). Through their huge knowledge base and powerful generation capabilities, they have achieved significant performance improvements in text tasks. Therefore, a speech recognition optimization method has emerged that uses a large language model (LLM) to learn the mapping relationship between N-best prediction results and transcribed texts. Specifically, the N-best prediction results are corrected by the trained large language model, and the final speech recognition result is determined based on the error correction result. The decoding process of the large language model is explained below in conjunction with the accompanying drawings:

[0043] Figure 1 The figure is a schematic diagram of the principle of optimizing speech recognition effect based on a large language model in the related art. Figure 1 As shown, assuming that for a given speech signal X∈R m The goal of the language recognition task is to use an end-to-end speech recognition model to predict the corresponding text transcription result, which has T consecutive characters Y T =(y1,y2,…y T ). The end-to-end speech recognition model obtains the N-best set based on the speech signal, which includes the n candidate text recognition results generated by decoding. Where n is a positive integer. The goal of the generative error correction method (GER) is to learn the mapping relationship F between n candidate prediction results and transcriptions using autoregression:

[0044]

[0045] In formula (2), L m is a large language model; Y <t is the historical candidate text recognition result (i.e., y1, y2, ...y t-1 ).

[0046] The meaning of the above formula is: Based on the large language model L m The probability distribution of the re-prediction result obtained by re-scoring the n candidate text recognition results is obtained by multiplying the probability distributions of each speech frame in the T speech frames corresponding to the T time steps of the speech signal; the probability distribution of each time step is obtained by the large language model L m It is obtained by re-scoring n candidate text recognition results based on the historical candidate text recognition results at each time step.

[0047] The speech recognition technology enhanced by the large language model is called generative error correction (GER). It can provide compensation information from the semantic level for each of the n candidate text recognition results when the speech information is corrupted by noise, so as to optimize the speech recognition effect in noisy scenes. However, due to the diversity of human expressions when speaking, there may be sentences that do not conform to semantics in some scenes, but the pronunciation in the real speech signal is indeed the case. This generative error correction method will introduce greater uncertainty, resulting in poor end-to-end speech recognition effect.

[0048] In view of this, the embodiments of the present application are dedicated to providing a speech recognition method, apparatus, device and storage medium, which fuses the information of the acoustic modality and the information of the text modality to solve the noise interference problem in the speech recognition process, thereby improving the end-to-end speech recognition accuracy. The following embodiments are described in detail one by one.

[0049] Exemplary Methods

[0050] Figure 2 A flow chart of a speech recognition method provided in an embodiment of the present application. Figure 2 As shown, the speech recognition method provided in this embodiment includes steps S201-S203:

[0051] S201. Obtain a first prediction result and a second prediction result of a speech to be recognized.

[0052] In this embodiment, the speech to be recognized refers to the speech data to be subjected to speech recognition. The speech to be recognized can be speech data in any speech recognition application scenario. The speech to be recognized includes various speech frames. The speech recognition method of this embodiment performs speech recognition using a speech frame as the minimum processing unit.

[0053] Among them, the first prediction result of the speech to be recognized includes a prediction result obtained based on the candidate text recognition result of the speech to be recognized. The candidate text recognition result is a candidate recognition result output by the end-to-end speech recognition model for speech recognition of the speech to be recognized. The first prediction result can be obtained by re-scoring the candidate text recognition result by the large language model. The following describes the process of obtaining the first prediction result:

[0054] In some embodiments, obtaining a first prediction result of the speech to be recognized includes: obtaining N-best candidate text recognition results output by the end-to-end speech recognition model; re-predicting the N-best candidate text recognition results respectively through a large language model (LLM) to obtain a first prediction result. The large language model has the ability to provide an independent probability distribution based on pure text modal information. The large language model can be any large model, such as the LLaMA-7B open source model.

[0055] Among them, the first prediction result refers to the output result of the output layer of the large language model (LLM). The output layer of the large language model (LLM) is used to map each speech frame in the speech to be recognized to each word category in the word vector space, and obtain the word classification result of each speech frame in the word vector space. For example, the speech to be recognized includes T speech frames, and the word vector space includes M word categories. The output result of the output layer of the large language model (LLM) includes the word classification results of each of the T speech frames under the M word categories. The output layer of the large language model (LLM) can be the previous layer of the Softmax layer of the large language model. The first prediction result is introduced below in conjunction with the accompanying drawings:

[0056] Figure 3 Schematic diagram of the first prediction result of the speech to be recognized provided in the embodiment of the present application. Figure 3 As shown in the figure, the horizontal axis represents T speech frames to be recognized, and the vertical axis represents M word categories. Each small square in the figure represents the word classification result of one speech frame under one word category. 11 Represents the word classification result of the first speech frame under the first word category.

[0057] The word classification result is the output result of the output layer of the large language model (LLM), which is represented by a set of real numbers. By processing the set of real numbers corresponding to the speech frame through the softmax function, the probability distribution of the speech frame under each word category can be obtained. This set of real numbers is a set of logical values ​​Logits, hereinafter referred to as the first eigenvalue. The larger the first eigenvalue, the greater the possibility that the corresponding speech frame is in the corresponding word category. Conversely, the smaller the first eigenvalue, the smaller the possibility that the speech frame corresponding to the first eigenvalue is in the corresponding word category. Wherein, T and M are both positive integers.

[0058] The second prediction result includes a prediction result obtained based on the speech to be recognized. The second prediction result is an output result of the last layer of the decoder when the end-to-end speech recognition model processes the speech to be recognized. The end-to-end speech recognition model can be an end-to-end speech recognition model based on a conformer structure.

[0059] In some embodiments, obtaining a second prediction result of the speech to be recognized includes: obtaining an output result of a last layer of a decoder when an end-to-end speech recognition model processes the speech to be recognized.

[0060] The process of obtaining the first prediction result is similar. The second prediction result is the output result of the output layer of the end-to-end speech recognition model. The output layer of the end-to-end speech recognition model can be the previous layer of the Softmax layer of the end-to-end speech recognition model. The output layer of the end-to-end speech recognition model maps each speech frame of the speech to be recognized to each word category in the word vector space, and obtains the word classification result of each speech frame in the word vector space.

[0061] It should be noted that in order to ensure the speech recognition effect, the word vector space of the first prediction result and the second prediction result must be consistent, so as to ensure that the two have the same decoding space, which is convenient for subsequent fusion processing.

[0062] In this embodiment, the first prediction result is obtained by re-predicting the N-best candidate text recognition results output by the large language model based on the end-to-end speech recognition model. The N-best candidate text recognition results are information under the text modality. In other words, the first prediction result is obtained based on the text modality information. The second prediction result is obtained by recognizing the speech to be recognized by the end-to-end speech recognition model. The speech to be recognized is the original speech signal under the acoustic modality, which contains rich acoustic information. Therefore, it can be understood that the second prediction result is obtained based on the acoustic modality information.

[0063] Continue reading Figure 2 , after step S201, step S202 may also be included.

[0064] S202: Determine a first weight corresponding to the first prediction result, and determine a second weight corresponding to the second prediction result.

[0065] Referring to the introduction of the above formula (2), it can be seen that in the above generative error correction method, when the large language model re-predicts based on the N-best candidate prediction results, there is a lack of reference to acoustic information, resulting in uncertainty in the prediction results, thereby failing to guarantee the speech recognition effect.

[0066] In practice, although the speech to be recognized is disturbed by noise, the voice information therein still contains important information. Therefore, this embodiment integrates the large language model into the end-to-end speech recognition model and uses a multimodal fusion method to implement it in a speech recognition automatic regression decoding, thereby dynamically integrating information from the acoustic modality and the text modality. For example, when the large language model has a high confidence in the first prediction result, the final speech recognition result mainly depends on the first prediction result from the large language model; and when the end-to-end speech recognition model has a high confidence in the second prediction result (such as a speech segment with less noise and clear pronunciation), the final speech recognition result mainly depends on the second prediction result obtained by decoding the end-to-end speech recognition model.

[0067] Therefore, in the generative error correction process, this embodiment introduces the second prediction result under the acoustic mode into the decoding process, so that the first prediction result under the text mode and the second prediction result under the acoustic mode are jointly decoded to improve the speech recognition effect.

[0068] In the joint decoding process, in each speech frame of the speech to be recognized, the confidence of the first prediction result under the text mode and the confidence of the second prediction result under the acoustic mode are different. The confidence of the first prediction result and the confidence of the second prediction result can be determined based on the degree to which each speech frame is covered by noise (also known as the degree of noise). For example, if the coverage of the noise energy of the speech in a certain speech frame is greater than the preset coverage, the confidence of the second prediction result corresponding to the speech frame is low. At this time, it mainly relies on the first prediction result of the large model; when the speech in the speech frame is clear and easy to understand, that is, the coverage of the noise energy of the speech in the speech frame is less than or equal to the preset coverage, it can mainly rely on the first prediction result. Therefore, this embodiment can dynamically adjust the first weight corresponding to the first prediction result and the second weight corresponding to the second prediction result for each speech frame. Among them, the weight corresponding to the text mode is the first weight. The weight corresponding to the acoustic mode is the second weight. The following describes the determination process of the first weight and the second weight:

[0069] Figure 4 The flowchart of the implementation process of determining the first weight and the second weight provided in the embodiment of the present application is as follows. Figure 4 As shown, in some embodiments, determining a first weight corresponding to a first prediction result and determining a second weight corresponding to a second prediction result include step S401 and step S402:

[0070] S401. Determine a first weight corresponding to a first prediction result based on the predicted posterior entropy of the prediction results corresponding to each speech frame of the speech to be recognized.

[0071] In some embodiments, step S401 includes: determining the predicted a posteriori entropy of the prediction results corresponding to each speech frame of the speech to be recognized; and determining a first weight corresponding to the first prediction result based on the predicted a posteriori entropy of the prediction results corresponding to each speech frame of the speech to be recognized.

[0072] The step of determining the predicted posterior entropy of the prediction result corresponding to each speech frame of the speech to be recognized includes: determining the predicted posterior entropy of the frame prediction result corresponding to each speech frame in the speech to be recognized based on the prediction probability of the frame prediction result corresponding to each speech frame in the speech to be recognized by the large language model. The process can be expressed as the following formula (3):

[0073]

[0074] In formula (3), represents the predicted posterior entropy corresponding to the frame prediction result of the t-th speech frame of the speech to be recognized; P(y t ) represents the probability distribution of the frame prediction result corresponding to the t-th speech frame, which can be determined by the output of the softmax layer of the large language model.

[0075] Among them, the predicted posterior entropy of the frame prediction result of the large model for the t-th speech frame is It is negatively correlated with the confidence of the frame prediction result of the large model for the tth speech frame. That is to say, when The larger the value of is, the higher the uncertainty of the frame prediction result of the large language model for the t-th speech frame, and the lower the confidence, indicating that the coverage of the t-th speech frame by the noise energy is greater than the preset coverage; when The smaller the value of , the lower the uncertainty of the frame prediction result of the large language model for the t-th speech frame, and the higher the confidence, indicating that the coverage of the t-th speech frame by noise energy is less than or equal to the preset coverage.

[0076] For example, when When it is close to 0, it indicates that the large language model is very confident about the frame prediction result of the t-th speech frame, which means that the coverage of the t-th speech frame by noise energy is low and the speech signal is clear and easy to understand.

[0077] After determining the predicted posterior entropy of the prediction result corresponding to each speech frame of the speech to be recognized, the first weight corresponding to the first prediction result can be determined based on the predicted posterior entropy. There are multiple ways to implement the determination of the first weight of the first prediction result, which can be as follows:

[0078] In some embodiments, the predicted posterior entropy of the frame prediction results corresponding to each speech frame can be processed by an activation function to determine the first weight corresponding to the first prediction result. The process can be specifically expressed as the following formula (4):

[0079]

[0080] In formula (4), represents the first weight corresponding to the frame prediction result of the large language model for the t-th speech frame; sigmoid() is the activation function; The predicted posterior entropy corresponding to the frame prediction result of the t-th speech frame of the speech to be recognized can be determined by formula (3).

[0081] In some embodiments, the first weight corresponding to the first prediction result may also be determined based on a comparison result between the predicted posterior entropy of the frame prediction result corresponding to each speech frame and a preset entropy.

[0082] For example, for the t-th speech frame in each speech frame, when the predicted a posteriori entropy of the frame prediction result corresponding to the t-th speech frame is greater than or equal to the preset entropy, the first weight corresponding to the frame prediction result corresponding to the t-th speech frame is determined to be a first value; or, when the predicted a posteriori entropy of the frame prediction result corresponding to the t-th speech frame is less than the preset entropy, the first weight corresponding to the frame prediction result corresponding to the t-th speech frame is determined to be a second value.

[0083] For another example, multiple preset entropies can be set to divide the predicted posterior entropy of the frame prediction results of each speech frame into multiple intervals, thereby dividing the first weight corresponding to the frame prediction result corresponding to the tth speech frame into multiple different weight value intervals.

[0084] For example, the plurality of preset entropies include a first preset entropy and a second preset entropy, and the second preset entropy is greater than the first preset entropy. For the t-th speech frame in each speech frame, when the predicted posterior entropy of the frame prediction result corresponding to the t-th speech frame is less than or equal to the first preset entropy, the first weight corresponding to the frame prediction result corresponding to the t-th speech frame is determined to be a first value.

[0085] When the predicted posterior entropy of the frame prediction result corresponding to the t-th speech frame is greater than the first preset entropy and less than or equal to the second preset entropy, the first weight corresponding to the frame prediction result corresponding to the t-th speech frame is determined to be a second value.

[0086] When the predicted posterior entropy of the frame prediction result corresponding to the t-th speech frame is greater than the second preset entropy, the first weight corresponding to the frame prediction result corresponding to the t-th speech frame is determined to be a third value.

[0087] It should be understood that the above two preset entropies are exemplary and do not limit the number of preset entropies. Those skilled in the art can set the number of preset entropies according to actual needs, and this embodiment does not limit this.

[0088] Continue reading Figure 4 , after step S401, step S402 may also be included.

[0089] S402: Determine a second weight corresponding to the second prediction result based on the first weight corresponding to the first prediction result.

[0090] Among them, for the t-th speech frame in each speech frame of the speech to be recognized, the sum of the first weight and the second weight corresponding to the frame prediction result of the t-th speech frame is 1. The frame prediction result of the t-th speech frame can be understood as the logical value distribution result or the characteristic value distribution result of the t-th speech frame under M word categories.

[0091] After determining the first weight corresponding to the frame prediction result of the t-th speech frame, the second weight corresponding to the frame prediction result of the t-th speech frame can be determined based on the first weight corresponding to the frame prediction result of the t-th speech frame. This process can be expressed as the following formula (5):

[0092]

[0093] In formula (5), represents a second weight corresponding to the frame prediction result of the end-to-end speech recognition model for the t-th speech frame; Represents the first weight corresponding to the frame prediction result of the large language model for the t-th speech frame.

[0094] According to the above formula (4), formula (5) can also be converted into: According to the converted formula, if the large language model is very confident (i.e. is close to 0), then Close to 0.5, and then If the value of is close to 0, is close to 1. The final speech recognition result depends entirely on the prediction result of the large language model. If the uncertainty of the large language model is high, then greater than 0, is greater than 0 and less than 0.5, that is, the second weight of the speech recognition model increases with If it increases with the increase of , it is necessary to combine the two to decode them together.

[0095] This embodiment can dynamically adjust the first weight of the first prediction result and the second weight corresponding to the second prediction result through the predicted posterior entropy of the frame prediction results corresponding to each speech frame in the speech to be recognized in the large language model, which is in line with the human perception logic, that is, when the speech is clear, the prediction result at the speech level is more inclined; when the speech is poor but the linguistic logic is smooth, the linguistic prediction result is more inclined, thereby realizing dynamic information compensation and improving the speech recognition effect in a noisy environment.

[0096] Continue reading Figure 2 , after step S202, the method may further include step S203:

[0097] S203: Based on the first weight and the second weight, perform feature fusion on the first prediction result and the second prediction result, and determine the recognition result of the speech to be recognized based on the obtained fusion feature.

[0098] As mentioned above, the first prediction result and the second prediction result are the feature values ​​of each speech frame of the speech to be recognized under each word category in the word vector space. That is to say, the first prediction result includes the feature values ​​of each speech frame under each word category in the word vector space. The second prediction result includes the feature values ​​of each speech frame under each word category in the word vector space. In this embodiment, the speech frame is used as the fusion unit to perform feature fusion on the first prediction result and the second prediction result. The specific implementation method of the feature fusion process is introduced below in conjunction with the accompanying drawings:

[0099] Figure 5 The following is a flow chart of the feature fusion process provided in the embodiment of the present application. Figure 5 As shown, step S203 specifically includes step S501 and step S502:

[0100] S501. Based on a first weight corresponding to the first prediction result and a second weight corresponding to the second prediction result, a first eigenvalue corresponding to the first prediction result and a second eigenvalue corresponding to the second prediction result are weightedly summed to obtain a fused eigenvalue.

[0101] As mentioned above, the speech to be recognized includes various speech frames. In the first prediction result, each speech frame corresponds to its first eigenvalue under M word categories. For the t-th speech frame, its corresponding first eigenvalue represents the possibility of the large language model predicting the t-th speech frame under the M word categories in the preset word vector space.

[0102] Similarly, each speech frame in the second prediction result corresponds to its second eigenvalue under the M word categories. For the t-th speech frame, its corresponding second eigenvalue represents the possibility of the end-to-end speech recognition model predicting the t-th speech frame under the M word categories in the preset word vector space.

[0103] Therefore, step S501 is to take the speech frame as the minimum processing unit, perform weighted summation on the first eigenvalue, the first weight, the second eigenvalue and the second weight corresponding to each speech frame, and obtain a fused eigenvalue.

[0104] Specifically, weighted summation is performed on the first eigenvalue, the first weight, the second eigenvalue and the second weight corresponding to each speech frame to obtain a fused eigenvalue, including: for the t-th speech frame in each speech frame, multiplying the first eigenvalue and the first weight corresponding to the t-th speech frame, and multiplying the second eigenvalue and the second weight corresponding to the t-th speech frame, and the cumulative result of the two multiplication results is determined as the fused eigenvalue of the t-th speech frame. The feature fusion process can be expressed as the following formula (6):

[0105]

[0106] In formula (6), g t represents the fusion feature value of the tth speech frame; The second weight corresponding to the t-th speech frame can be determined according to the above formula (5); The second eigenvalue corresponding to the t-th speech frame can be determined according to the last layer output of the decoder of the end-to-end speech recognition model; represents the first weight corresponding to the t-th speech frame, which can be determined according to the above formula (4); represents the first eigenvalue corresponding to the t-th speech frame, which can be determined according to the output result of the output layer of the large language model; X represents the speech to be recognized; Represents the candidate text recognition result.

[0107] It should be noted that the fused feature value of the t-th speech frame mentioned above is the distribution result of the fused feature values ​​of the t-th speech frame under M word categories.

[0108] S502: Process the fused feature value through a softmax function to determine a recognition result of the speech to be recognized.

[0109] In some embodiments, step S502 includes: processing the fused feature value through a softmax function to determine an initial recognition result of the speech to be recognized; and determining a final speech recognition result based on the initial recognition result of the speech to be recognized.

[0110] By processing the fused feature values ​​corresponding to each speech frame of the speech to be recognized through the softmax function, the probability distribution of each speech frame under M word categories can be obtained, that is, the initial recognition result of the speech to be recognized.

[0111] Among them, the goal of speech recognition is to predict the probability distribution of each speech frame in each word category. However, in practical applications, the fused feature value may not be in a standard probability form. For example, the final output result of the fused feature value of the t-th speech frame under M word categories may be a set of real numbers such as [230, 40, 50, 10, 45, 980, 780, 440, 9]. Such output results are less interpretable. Therefore, this embodiment processes the fused feature value through the softmax function, normalizes the fused feature value to the value range of [0, 1], and then obtains the initial recognition result of the speech to be recognized.

[0112] After that, the final speech recognition result of the speech to be recognized can be obtained based on the initial recognition results corresponding to each speech frame. Among them, the initial recognition result includes the probability distribution of each speech frame of the speech to be recognized under each word category. Based on the initial recognition results corresponding to each speech frame, the final speech recognition result of the speech to be recognized is obtained, including: for each speech frame in each speech frame of the speech to be recognized, determine the maximum probability in the probability distribution corresponding to each word category corresponding to the speech frame, and select the word category corresponding to the maximum probability as the frame recognition result corresponding to the speech frame. By analogy, the frame recognition results corresponding to each speech frame can be obtained, and the frame recognition results corresponding to each speech frame are the final speech recognition results.

[0113] It should be noted that in this embodiment, the autoregressive decoding method is used to decode the speech to be recognized. The principle of the autoregressive decoding method is as follows: softmax processing is performed on the fused feature value corresponding to the t-th speech frame to obtain the probability distribution of the t-th speech frame under each word category, and then the probability distribution of the t-th speech frame under each word category is input into the decoder, so that the decoder determines the probability distribution of the t+1-th speech frame under each word category based on the probability distribution of the t-th speech frame under each word category. The autoregressive decoding process can be expressed as the following formula (7):

[0114]

[0115] In formula (7), P(Y T ) represents the probability distribution of each speech frame in the speech to be recognized under each word category; T represents the total number of speech frames.

[0116] The above embodiment introduces the implementation process of speech recognition based on one first prediction result. As introduced in step S201, the N-best candidate text recognition results can also be multiple candidate text recognition results. Correspondingly, the first prediction results also include multiple. Each of the multiple first prediction results corresponds to a first weight.

[0117] For multiple first prediction results, step S202 may further include: for each first prediction result in the multiple first prediction results, determining a first weight corresponding to the first prediction result. Accordingly, step S501 may further include: for each first prediction result in the multiple first prediction results, based on the first weight corresponding to the first prediction result and the second weight corresponding to the second prediction result, weighted summing the first feature value corresponding to the first prediction result and the second feature value corresponding to the second prediction result to obtain a fused feature value corresponding to the first prediction result.

[0118] Correspondingly, step S502 may also include: for each first prediction result, the fused feature value corresponding to the fused feature value is processed by a softmax function to determine the recognition result of the speech to be recognized; based on the recognition results corresponding to each of the multiple first prediction results, the final speech recognition result is determined.

[0119] Among them, the specific implementation method of determining the corresponding fusion feature value for each first prediction result among the multiple first prediction results can refer to the introduction of the relevant content in the aforementioned embodiment, which will not be repeated here.

[0120] Here, the focus is on how to determine the final speech recognition result after obtaining the initial recognition results corresponding to each of the multiple first prediction results. Specifically, based on the recognition results corresponding to each of the multiple first prediction results, the final speech recognition result is determined, including: determining the scores of the initial recognition results corresponding to each of the multiple first prediction results; selecting the initial recognition result corresponding to the highest score and determining it as the final speech recognition result.

[0121] In some embodiments, determining the scores of the initial recognition results corresponding to each of the multiple first prediction results includes: for each of the initial recognition results of the multiple first prediction results, determining the score of the initial recognition result based on the product of the maximum probabilities corresponding to each of the speech frames in the initial recognition result.

[0122] After the scores of the initial recognition results corresponding to the multiple first prediction results are determined respectively, the initial recognition result corresponding to the highest score can be selected and determined as the final speech recognition result corresponding to the speech to be recognized.

[0123] In summary, the speech recognition method of this embodiment, in order to solve the speech recognition problem in complex noise scenarios, builds a dynamic perception fusion framework based on the end-to-end speech recognition model and the large language model. Figure 6 As shown, the framework includes: an end-to-end speech recognition model and a large language model.

[0124] The end-to-end speech recognition model includes an encoder and a decoder. The encoder is used to encode the noisy speech to be recognized, obtain an encoding result, and input the encoding result to the decoder so that the decoder decodes based on the encoding result.

[0125] The framework's processing flow includes:

[0126] (1) Determine the first prediction result and the second prediction result.

[0127] The first prediction result can be obtained directly from the output layer of the decoder.

[0128] After the first prediction result is subjected to softmax processing, a text candidate result can be obtained. The text candidate result is input into the large language model to obtain a second prediction result.

[0129] It should be noted that the first prediction result and the second prediction result here are both in the form of logical values.

[0130] (2) Determine a first weight corresponding to each speech frame in the first prediction result, and determine a second weight corresponding to each speech frame in the second prediction result.

[0131] Among them, the first weight corresponding to each speech frame in the first prediction result can be determined based on the prediction uncertainty of the large language model for each speech frame.

[0132] Since the sum of the first weight and the second weight corresponding to each speech frame is 1, after determining the first weight corresponding to each speech frame, the second weight corresponding to each speech frame can be determined based on the first weight corresponding to each speech frame.

[0133] (3) A fusion feature value is obtained based on the first weight and the first prediction result corresponding to each speech frame, and the weighted sum of the second weight and the second prediction result.

[0134] The specific fusion process can be found in the introduction of the above embodiment, which will not be described in detail here.

[0135] (4) The fused feature values ​​are processed through the softmax function to obtain the initial speech recognition result.

[0136] The initial speech recognition result includes the probability distribution of each speech frame under each word category.

[0137] (5) Based on the initial speech recognition result, determine the final speech recognition result.

[0138] Wherein, the initial speech recognition result includes one or more. When the initial speech recognition result includes one, for each speech frame in each speech frame, based on the word category corresponding to the maximum probability in the probability distribution of the speech frame under each word category, the frame recognition result corresponding to the speech frame is determined.

[0139] By performing the above steps on each speech frame, the frame recognition result corresponding to each speech frame can be obtained. The frame recognition results corresponding to each speech frame are connected in time to obtain the final speech recognition result.

[0140] When the initial speech recognition results include multiple ones, for each speech recognition result, the probabilities corresponding to the frame recognition results of each speech frame in the speech recognition result are multiplied to obtain the score corresponding to the speech recognition result; and the speech recognition result corresponding to the maximum score is selected as the final speech recognition result.

[0141] It should be noted that during the decoding process of this embodiment, after decoding each frame of speech to obtain the probability distribution of the speech frame under each word category, it is necessary to input it into the decoder again so that the decoder can decode the next speech frame based on the historical decoding results.

[0142] The solution of this embodiment is based on the large language model to perform gradual late fusion in the autoregressive decoding process. The large language model is used to predict the uncertainty of the current decoding process, and the information of the acoustic mode and the text mode is dynamically integrated to perform information compensation to solve the speech recognition problem in noisy scenarios. It has the following beneficial effects:

[0143] (1) Since the second prediction result based on the speech to be recognized is introduced as the information in the acoustic mode in the speech recognition process, and the first prediction result in the text mode and the second prediction result in the acoustic mode are jointly decoded, the information in the acoustic mode is used to compensate for the information in the text mode, so as to improve the speech recognition performance.

[0144] (2) Taking the speech frame as the smallest processing unit, the weights of each speech frame in the acoustic mode and text mode are dynamically adjusted according to the uncertainty of the decoding process of each speech frame by the large language model. This method of dynamically adjusting the weights is consistent with the human multimodal perception method, that is, when the information uncertainty brought by the main mode in the multimodality is high, humans actively seek compensatory information from other modes. This dynamic information compensation method can further improve speech recognition performance.

[0145] (3) This embodiment performs gradual late fusion in the autoregressive decoding process, and only needs to modify the decoding process. This late fusion method avoids the subsequent training process after the fusion information in the autoregressive decoding, and does not require additional training processes for noise types. It has good noise generalization ability and is also convenient and fast to integrate and deploy.

[0146] Corresponding to the above-mentioned speech recognition method, the present application embodiment also proposes a speech recognition device, such as Figure 7 As shown, the device comprises:

[0147] An acquisition unit 701 is used to acquire a first prediction result and a second prediction result of a speech to be recognized, wherein the first prediction result includes a prediction result obtained based on a candidate text recognition result of the speech to be recognized, and the second prediction result includes a prediction result obtained based on the speech to be recognized;

[0148] A weight determination unit 702, configured to determine a first weight corresponding to the first prediction result, and determine a second weight corresponding to the second prediction result;

[0149] The fusion unit 703 is used to perform feature fusion on the first prediction result and the second prediction result based on the first weight and the second weight, and determine the recognition result of the speech to be recognized based on the obtained fusion feature.

[0150] In some embodiments, the weight determination unit 702 determines a first weight corresponding to the first prediction result, and determines a second weight corresponding to the second prediction result, including: determining a first weight corresponding to the first prediction result based on the predicted posterior entropy of the frame prediction results corresponding to each speech frame in the speech to be recognized; and determining a second weight corresponding to the second prediction result based on the first weight corresponding to the first prediction result.

[0151] In some embodiments, when the weight determination unit 702 determines the predicted a posteriori entropy of the frame prediction results corresponding to each speech frame in the speech to be recognized, the following steps are adopted: based on the prediction probability of the frame prediction results corresponding to each speech frame in the speech to be recognized by the large language model, the predicted a posteriori entropy of the frame prediction results corresponding to each speech frame in the speech to be recognized is determined.

[0152] In some embodiments, the weight determination unit 702 determines the first weight corresponding to the first prediction result based on the predicted a posteriori entropy of the frame prediction results corresponding to each of the speech frames, including: processing the predicted a posteriori entropy of the frame prediction results corresponding to each of the speech frames through an activation function to determine the first weight corresponding to the first prediction result; or determining the first weight corresponding to the first prediction result based on a comparison result of the predicted a posteriori entropy of the frame prediction results corresponding to each of the speech frames with a preset entropy.

[0153] In some embodiments, the fusion unit 703 performs feature fusion on the first prediction result and the second prediction result based on the first weight and the second weight, and determines the recognition result of the speech to be recognized based on the obtained fusion feature, including: based on the first weight corresponding to the first prediction result and the second weight corresponding to the second prediction result, weighted summing up the first feature value corresponding to the first prediction result and the second feature value corresponding to the second prediction result to obtain a fusion feature value; processing the fusion feature value through a softmax function to determine the recognition result of the speech to be recognized.

[0154] In some embodiments, the first prediction results include multiple first prediction results, each of which corresponds to a first weight; the fusion unit 703 performs weighted summation on the first feature value corresponding to the first prediction result and the second feature value corresponding to the second prediction result based on the first weight corresponding to the first prediction result and the second weight corresponding to the second prediction result to obtain a fused feature value, including: for each first prediction result in the multiple first prediction results, based on the first weight corresponding to the first prediction result and the second weight corresponding to the second prediction result, perform weighted summation on the first feature value corresponding to the first prediction result and the second feature value corresponding to the second prediction result to obtain a fused feature value corresponding to the first prediction result; process the fused feature value through a softmax function to determine the recognition result of the speech to be recognized, including: for the fused feature value corresponding to each first prediction result, process the fused feature value through a softmax function to determine the initial recognition result of the speech to be recognized; determine the final speech recognition result based on the initial recognition results corresponding to each of the multiple first prediction results.

[0155] In some embodiments, the fusion unit 703 determines the final speech recognition result based on the recognition results corresponding to each of the multiple first prediction results, including: determining the scores of the initial recognition results corresponding to each of the multiple first prediction results; selecting the initial recognition result corresponding to the highest score and determining it as the final speech recognition result.

[0156] In some embodiments, the candidate text recognition result is the candidate recognition result output by the end-to-end speech recognition model for recognizing the speech to be recognized, and the first prediction result is the prediction result obtained by rescoring the candidate text recognition result through a large language model; the second prediction result is the output result of the last layer of the decoder when the end-to-end speech recognition model processes the speech to be recognized.

[0157] The speech recognition device provided in this embodiment belongs to the same application concept as the speech recognition method provided in the above embodiments of this application, can execute the speech recognition method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the speech recognition method. For technical details not fully described in this embodiment, please refer to the specific processing content of the speech recognition method provided in the above embodiments of this application, which will not be repeated here.

[0158] The functions implemented by the various units in the above speech recognition device can be implemented by the same or different processors, which is not limited in the embodiments of the present application.

[0159] It should be understood that the units in the above speech recognition device can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the unit in the device can be implemented in the form of a hardware circuit, and the functions of some or all units can be realized by designing the hardware circuit. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of some or all of the above units. All units of the above device can be implemented in the form of a processor calling software, or in the form of a hardware circuit, or in the form of a processor calling software, and the remaining part can be implemented in the form of a hardware circuit.

[0160] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0161] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0162] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0163] An embodiment of the present application further provides a control device, which includes a processor and an interface circuit. The processor in the control device is connected to a data input component via the interface circuit of the control device.

[0164] The data input component specifically refers to a functional component capable of inputting or collecting voice data, such as a microphone, etc.

[0165] The above-mentioned interface circuit can be any interface circuit that can realize the data communication function, for example, it can be a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIE circuit, etc.

[0166] The processor in the control device is also a circuit with signal processing capability, which executes the speech recognition method introduced in the above embodiment. The specific implementation of the processor can refer to the above processor implementation, and the embodiment of the present application is not strictly limited.

[0167] Another embodiment of the present application also provides an electronic device, see Figure 8 As shown, the device includes:

[0168] Memory 200 and processor 210;

[0169] The memory 200 is connected to the processor 210 and is used to store programs;

[0170] The processor 210 is used to implement the model training method or speaker recognition method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0171] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0172] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0173] A bus may include a pathway that transfers information between components of a computer system.

[0174] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0175] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0176] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0177] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0178] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0179] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0180] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any speech recognition method provided in the above embodiments of the present application.

[0181] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the speech recognition method introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the introduction to the above-mentioned embodiment of the speech recognition method.

[0182] In addition to the above methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the speech recognition method described in any of the above embodiments of this specification.

[0183] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0184] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is used by a processor to execute the steps of the speech recognition method described in any of the above embodiments of this specification.

[0185] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0186] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0187] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0188] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0189] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0190] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0192] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0193] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0194] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0195] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech recognition method, characterized in that: include: Obtaining a first prediction result and a second prediction result of the speech to be recognized, wherein the first prediction result includes a prediction result obtained based on a candidate text recognition result of the speech to be recognized, and the second prediction result includes a prediction result obtained based on the speech to be recognized; Determining a first weight corresponding to the first prediction result, and determining a second weight corresponding to the second prediction result; Based on the first weight and the second weight, feature fusion is performed on the first prediction result and the second prediction result, and a recognition result of the speech to be recognized is determined based on the obtained fusion feature.

2. The method according to claim 1, characterized in that The determining a first weight corresponding to the first prediction result, and determining a second weight corresponding to the second prediction result, includes: Determining a first weight corresponding to the first prediction result based on the predicted posterior entropy of the frame prediction result corresponding to each speech frame in the speech to be recognized; Based on the first weight corresponding to the first prediction result, a second weight corresponding to the second prediction result is determined.

3. The method according to claim 2, characterized in that The predicted posterior entropy of the frame prediction result corresponding to each speech frame in the speech to be recognized is determined by the following steps: Based on the prediction probability of the large language model for the frame prediction results corresponding to each speech frame in the speech to be recognized, the predicted posterior entropy of the frame prediction results corresponding to each speech frame in the speech to be recognized is determined.

4. The method according to claim 2 or 3, characterized in that: The determining, based on the predicted posterior entropy of the frame prediction results corresponding to each of the speech frames, a first weight corresponding to the first prediction result comprises: Processing the predicted posterior entropy of the frame prediction results corresponding to each of the speech frames through an activation function to determine a first weight corresponding to the first prediction result; or, Based on the comparison result between the predicted posterior entropy of the frame prediction result corresponding to each of the speech frames and the preset entropy, a first weight corresponding to the first prediction result is determined.

5. The method according to any one of claims 1 to 3, characterized in that: The step of fusing the first prediction result and the second prediction result based on the first weight and the second weight, and determining the recognition result of the speech to be recognized based on the obtained fusion feature, includes: Based on a first weight corresponding to the first prediction result and a second weight corresponding to the second prediction result, weighted summing a first eigenvalue corresponding to the first prediction result and a second eigenvalue corresponding to the second prediction result to obtain a fused eigenvalue; The fused feature value is processed by a softmax function to determine a recognition result of the speech to be recognized.

6. The method according to claim 5, characterized in that The first prediction results include a plurality of first prediction results, each of which corresponds to a first weight; Based on a first weight corresponding to the first prediction result and a second weight corresponding to the second prediction result, a first eigenvalue corresponding to the first prediction result and a second eigenvalue corresponding to the second prediction result are weightedly summed to obtain a fused eigenvalue, including: For each first prediction result among the multiple first prediction results, based on a first weight corresponding to the first prediction result and a second weight corresponding to the second prediction result, a first eigenvalue corresponding to the first prediction result and a second eigenvalue corresponding to the second prediction result are weighted summed to obtain a fused eigenvalue corresponding to the first prediction result; Processing the fused feature value by a softmax function to determine a recognition result of the speech to be recognized includes: For each fused feature value corresponding to the first prediction result, the fused feature value is processed by a softmax function to determine an initial recognition result of the speech to be recognized; Based on the initial recognition results corresponding to each of the multiple first prediction results, a final speech recognition result is determined.

7. The method according to claim 6, characterized in that The determining of a final speech recognition result based on the recognition results corresponding to each of the plurality of first prediction results comprises: Determining scores of initial recognition results corresponding to each of the plurality of first prediction results; The initial recognition result corresponding to the highest score is selected and determined as the final speech recognition result.

8. The method according to any one of claims 1 to 3, characterized in that: The candidate text recognition result is a candidate recognition result output by an end-to-end speech recognition model for recognizing the speech to be recognized, and the first prediction result is a prediction result obtained by rescoring the candidate text recognition result through a large language model; The second prediction result is the output result of the last layer of the decoder when the end-to-end speech recognition model processes the speech to be recognized.

9. A speech recognition device, characterized in that: include: an acquisition unit, configured to acquire a first prediction result and a second prediction result of the speech to be recognized, wherein the first prediction result includes a prediction result obtained based on a candidate text recognition result of the speech to be recognized, and the second prediction result includes a prediction result obtained based on the speech to be recognized; a weight determination unit, configured to determine a first weight corresponding to the first prediction result, and determine a second weight corresponding to the second prediction result; A fusion unit is used to perform feature fusion on the first prediction result and the second prediction result based on the first weight and the second weight, and determine the recognition result of the speech to be recognized based on the obtained fusion feature.

10. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 8 by running the program in the memory.

11. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, cause the processor to implement the method as claimed in any one of claims 1 to 8.

Images