Method, device, equipment and storage medium for speech recognition

By combining audio and text feature information and utilizing the association between the first and second languages ​​to determine the confidence score of the speech recognition model, the problem of insufficient speech recognition accuracy in specific domains is solved, achieving high accuracy and cost-effectiveness in specific domains.

CN115472165BActive Publication Date: 2024-11-26FACE CUTE CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210803531.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-07
Publication Date
2024-11-26
Estimated Expiration
2042-07-07

AI Technical Summary

Technical Problem

Existing speech recognition models have insufficient accuracy in specific domains, and training multiple domain models is costly. General external language models do not cover general texts well, which affects recognition accuracy.

Method used

The intermediate confidence score is determined by combining audio and text features, and the confidence scores of candidate recognition results are determined by using first language association and second language association respectively. Different language associations are integrated to improve recognition accuracy.

Benefits of technology

Significantly improve recognition accuracy in specific domains while maintaining recognition accuracy for general scenarios and reducing training costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115472165B_ABST
    Figure CN115472165B_ABST
Patent Text Reader

Abstract

According to an embodiment of the present disclosure, a method, apparatus, device and storage medium for speech recognition are provided. The method includes determining an intermediate confidence score of a candidate recognition result of a speech to be recognized based on audio feature information and text feature information corresponding to the speech to be recognized, and determining a first language confidence score of the candidate recognition result based on the text feature information using a first language association relationship. The method also includes determining a second language confidence score of the candidate recognition result based on the text feature information using a second language association relationship. The method also includes determining a target confidence score of the candidate recognition result based on a comparison result of the first language confidence score and the second language confidence score and the intermediate confidence score. In this way, the accuracy of speech recognition for specific and general fields can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for speech recognition. Background Art

[0002] Automatic Speech Recognition (ASR) refers to the conversion of speech signals into machine-recognizable inputs, such as natural language text. With the development of artificial intelligence technology, machine learning models have been studied and developed to achieve speech recognition tasks, allowing speech recognition technology to be gradually applied to daily life. In some specific application scenarios, such as medical care and driving, there are often higher requirements for speech recognition accuracy. Therefore, domain-based speech recognition optimization has become a very important optimization direction. Summary of the invention

[0003] In a first aspect of the present disclosure, a speech recognition method is provided. The method includes: determining an intermediate confidence score of a candidate recognition result of a speech to be recognized based on audio feature information and text feature information corresponding to the speech to be recognized; determining a first language confidence score of the candidate recognition result based on the text feature information using a first language association relationship; determining a second language confidence score of the candidate recognition result based on the text feature information using a second language association relationship; and determining a target confidence score of the candidate recognition result based on a comparison result of the first language confidence score and the second language confidence score and the intermediate confidence score.

[0004] In a second aspect of the present disclosure, a device for speech recognition is provided. The device includes: an intermediate score determination module, configured to determine an intermediate confidence score of a candidate recognition result of a speech to be recognized based on audio feature information and text feature information corresponding to the speech to be recognized; a first language score determination module, configured to determine a first language confidence score of the candidate recognition result based on text feature information using a first language association relationship; a second language score determination module, configured to determine a second language confidence score of the candidate recognition result based on text feature information using a second language association relationship; and a target score determination module, configured to determine a target confidence score of the candidate recognition result based on a comparison result of the first language confidence score and the second language confidence score and the intermediate confidence score.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes: at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0010] Figure 2 A block diagram of a speech recognition architecture according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A flowchart showing a process of speech recognition according to some embodiments of the present disclosure is shown;

[0012] Figure 4A and Figure 4B A flowchart showing a model training process according to some embodiments of the present disclosure;

[0013] Figure 5 A flowchart showing a speech recognition process according to some embodiments of the present disclosure is shown;

[0014] Figure 6 A block diagram showing an apparatus for speech recognition according to some embodiments of the present disclosure; and

[0015] Figure 7 A block diagram of an electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0018] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0019] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0020] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0021] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0023] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0024] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

[0025] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association from input to output (also called the mapping of input to output) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. In the application stage, the model can be used to process the actual input based on the parameter values ​​obtained from the training to determine the corresponding output.

[0026] Figure 1 A schematic diagram of an example environment 100 is shown in which embodiments of the present disclosure can be implemented. Figure 1 In the environment 100, it is desired to train and use such a speech recognition model to implement speech recognition tasks.

[0027] like Figure 1 As shown, the environment 100 includes a model training system 110 and a model application system 120. Figure 1 In an example embodiment of the present invention, the model training system 110 is configured to train the speech recognition model 105 using training data. The training data may include a plurality of speech samples 112-1, 112-2, ... 112-N and corresponding text samples 114-1, 114-2, ... 114-N, where N is an integer greater than or equal to 1. For ease of discussion, the speech samples are collectively or individually referred to as speech samples 112, and the text samples are collectively or individually referred to as text samples 114. The speech sample 112 and the corresponding text sample 114 may form a speech-text sample pair, where the text sample 114 may indicate the speech recognition result of the speech sample 112.

[0028] Before training, the parameter values ​​of the speech recognition model 105 can be initialized, or the parameter values ​​pre-trained can be obtained through a pre-training process. Through the training process, the parameter values ​​of the speech recognition model 105 are updated and adjusted. After the training is completed, the speech recognition model 105 has the trained parameter values. Based on such parameter values, the speech recognition model 105 can realize the speech recognition result of determining the text type from the speech.

[0029] exist Figure 1 In the example, the model application system 120 receives an input speech to be recognized 132. The model application system 120 may be configured to use the trained speech recognition model 105 to perform speech recognition on the speech to be recognized 132 to obtain a speech recognition result 134. The speech recognition result 134 may include a text sequence.

[0030] exist Figure 1 In the present invention, the model training system 110 and the model application system 120 can be any system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. The terminal device can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Servers include but are not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0031] It should be understood that Figure 1 The components and arrangements in the illustrated environment are merely examples, and a computing system suitable for implementing the example embodiments described in the present disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown as separate, the model training system 110 and the model application system 120 may be integrated into the same system or device. The embodiments of the present disclosure are not limited in this respect.

[0032] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0033] As mentioned above, in some fields, there are often higher requirements for speech recognition accuracy. If the general speech recognition model is trained with training data from general scenarios, the model may not be able to accurately recognize speech in specific scenarios. For example, it may recognize professional vocabulary (such as medical professional vocabulary or driving commands, etc.) as general vocabulary with similar pronunciation.

[0034] In order to improve the accuracy of speech recognition in a specific field, one possible solution is to use the training data of a specific field to train the speech recognition model, so that the speech recognition model has the ability to accurately recognize the speech in the specific field. However, the difficulty lies in how to obtain enough speech-text sample pairs in a specific field. In addition, since the training overhead (computing resource overhead and time cost) of the speech recognition model is often relatively large, training multiple speech recognition models for different fields will greatly increase the cost.

[0035] The language model can characterize language associations from a text perspective. Although it may not be explicitly included in the speech recognition model, the speech recognition model can also be considered to have a part that can characterize language associations. However, the speech recognition model trained based on general text samples is not good at representing the language associations of texts in specific fields. Currently, it is also proposed to use lightweight text samples in specific fields to train an external language model (Language Model, LM), and use the language model specific to the field to improve the accurate recognition of the general speech recognition model in the specific field.

[0036] However, regarding how the domain-specific external language model affects the recognition results of the speech recognition model, some solutions propose to directly add the confidence score of the candidate recognition result calculated by the external language model to the confidence score calculated by the speech recognition result, so as to measure whether the candidate recognition result can be used as the final speech recognition result. Some other solutions propose to estimate the score belonging to the internal language model from the confidence score calculated by the speech recognition model, and then replace the score of the internal language model with the confidence score of the external domain-specific language model to obtain the final confidence score of the candidate recognition result.

[0037] However, through analysis, research and experiments, we found that since external language models are generated based on text samples in a specific field, they often do not cover text outside the field or general text collections. If we do not distinguish, the score of the external language model will directly affect the speech recognition results, which will seriously affect the recognition accuracy of the speech recognition model for general speech.

[0038] In an example embodiment of the present disclosure, an improved speech recognition scheme is provided. According to the scheme, an intermediate confidence score of a candidate recognition result of a speech to be recognized is determined based on audio feature information and text feature information corresponding to the speech to be recognized, and a first language confidence score of the candidate recognition result is also determined using a first language association relationship, and the language confidence score can be determined independently of the audio feature information. A second language association relationship is used to determine a second language confidence score of the candidate recognition result based on text feature information. Based on the comparison result of the first language confidence score and the second language confidence score and the intermediate confidence score, a target confidence score of the candidate recognition result is determined.

[0039] According to the scheme, the language confidence scores determined by different language associations are comprehensively considered, and different language associations are selectively enabled to intervene in the speech recognition results, thereby improving the speech recognition accuracy. In some embodiments, if one of the language associations is determined for a specific field, a significant increase in the recognition accuracy in the specific field can be achieved, while not affecting the recognition accuracy of general scenarios.

[0040] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0041] Figure 2 2 shows a block diagram of a speech recognition architecture 200 according to some embodiments of the present disclosure. Figure 2 As described, the speech recognition architecture 200 includes a speech recognition model 210 and a domain-specific language model 220. In some embodiments, the speech recognition model 210 and the domain-specific language model 220 may be configured to implement end-to-end speech recognition.

[0042] The speech recognition model 210 can characterize the association between speech and text, and is used to realize the conversion from speech to text. The model input of the speech recognition model 210 is the speech to be recognized 205, and the model output is the confidence score of the candidate recognition result (for example, including text). The domain-specific language model 220 can be used to realize the prediction of text sequence. In the field of machine learning, the language model can be used to characterize the language association relationship, so that it can be used to predict the probability of the generation of character (word) sequence and determine whether a language sequence is a normal sentence. In the task of text generation, the language association relationship or language model can also be defined as: given a text unit or multiple text units, the next linguistically meaningful text unit can be predicted through the language association relationship or language model. In some embodiments of the present disclosure, the domain-specific language model 220 can perform a text generation task, and its model input is one or more text units, and the model output is the confidence score of the predicted next text unit.

[0043] In addition, the speech recognition architecture 200 further includes a fusion module 230 , which is configured to control the influence of the output of the domain-specific language model 220 on the speech recognition result, and determine the speech recognition result 235 corresponding to the speech to be recognized 205 .

[0044] Figure 3 FIG. 3 shows a speech recognition process 300 according to some embodiments of the present disclosure. In some embodiments, as will be mentioned below, the process 300 may utilize Figure 2 The speech recognition architecture 200 is implemented, and the process 300 can be implemented in Figure 1 The model application system 120. In other embodiments, the process 300 can also be implemented using other speech recognition structures.

[0045] In block 310, the model application system 120 determines the intermediate confidence score of the candidate recognition result of the speech to be recognized based on the audio feature information and text feature information corresponding to the speech to be recognized. That is, the confidence score of the candidate recognition result, that is, whether it can be used as the recognition result of the speech to be recognized, can be determined from various information available for the speech to be recognized, including acoustic and text feature information. In some embodiments, the candidate recognition result may include text.

[0046] In some embodiments, the association between speech and text can be utilized to determine the intermediate confidence score of the candidate recognition result based on audio feature information and text feature information. In some embodiments, the speech recognition model 210 can be utilized to determine the intermediate confidence score of the candidate recognition result based on audio feature information and text feature information, wherein the speech recognition model 210 can characterize the association between speech and text. The speech recognition model 210 can be constructed based on any model structure suitable for implementing speech recognition. As an example, the speech recognition model 210 can be based on a connectionist temporal classification (CTC) model structure, an attention-based model structure, a recursive neural network (RNN) model structure, and the like.

[0047] In the model application stage, the speech recognition model 210 is a trained model. In some embodiments, the speech recognition model 210 can be trained as a general speech recognition model, so that the speech recognition in general scenarios can have a higher accuracy. Figure 4A FIG. 4 is a flowchart showing a model training process 400 for a speech recognition model 210 according to some embodiments of the present disclosure. The process 400 may be implemented, for example, in Figure 1 The model training system is implemented at 110.

[0048] In block 410, the model training system 110 obtains speech-text sample pairs, each of which may include speech and corresponding text. The speech may be used as a model input, and the text may be used as a reference value of the model output, which is used to determine the current error of the model so as to adjust the model parameter value. The speech-text sample pairs may be speech-text sample pairs that can be obtained from any appropriate data source.

[0049] In box 420, the model training system 110 uses the obtained speech-text sample pairs to train the speech recognition model 210. In some embodiments, the speech-text sample pairs are related to general scenarios and do not need to be targeted at specific fields. Using such training data, the trained speech recognition model 210 can have a higher accuracy for speech in general scenarios. The model training system 110 can use any appropriate model training algorithm to train the speech recognition model 210, and the embodiments of the present disclosure are not limited in this respect.

[0050] In box 430 , after the training is completed, the model training system 110 obtains the trained speech recognition model 210 to provide it to the model application system 120 for use.

[0051] Speech recognition can be considered as a sequence-to-sequence conversion process, that is, a conversion process of a speech sequence to a text sequence. The speech sequence corresponds to the speech to be recognized, which may include a plurality of speech units divided according to any appropriate division granularity. When performing speech recognition, the speech recognition model 210 may sequentially recognize each speech unit, extract audio feature information of the speech unit, also referred to as audio feature information, and determine the corresponding text unit.

[0052] Return to reference Figure 3 In some embodiments, the speech to be recognized 205 can be input into the speech recognition model 210 after some preprocessing. In some embodiments, the speech recognition model 210 can perform speech recognition iteratively. In the speech recognition process, the candidate recognition results may include one or more text units predicted for the speech to be recognized. In this article, the text unit is the basic component unit of recognition, which can be, for example, a phoneme, a character, a word, etc.

[0053] In some embodiments, during the iteration process, one or more previously recognized text units can be used as prior information as model input to further determine the next text unit, which can affect the recognition result. That is, the probability of recognizing the next text unit from the speech is determined under the condition that the previous one or more text units are known. In some embodiments, each step of the iteration can take into account the text units recognized in the previous step or steps. Figure 5 Such an iterative process is discussed in more detail.

[0054] The speech recognition model 210 can extract audio feature information of one or more speech units of the speech to be recognized through feature extraction, and extract text feature information (also called text feature information) of the candidate recognition results. Audio feature information is used to characterize feature information in speech from an acoustic perspective to provide useful information for speech recognition. Text feature information is used to characterize feature information of the candidate recognition results from a linguistic perspective.

[0055] In some embodiments, the speech recognition model 210 constructed based on various model structures can be considered to include a feature extraction part or encoder, and a predictor or decoder. The feature extraction part or encoder is used to perform feature extraction to determine audio feature information and text feature information. The predictor or decoder is configured to process the extracted text feature information to obtain a model output.

[0056] The speech recognition model 210 determines the confidence score of the current candidate recognition result based on the audio feature information and the text feature information (in the text, for the convenience of discussion, it is called the "intermediate confidence score"). The intermediate confidence score is a comprehensive measure of the confidence determination of the speech recognition model 210 regarding the candidate recognition result from the two aspects of audio and text features, that is, it determines whether the candidate recognition result is the correct recognition result or a partially correct recognition result of the speech to be recognized (when the end of the speech sequence is not recognized). If the intermediate confidence score is higher, it means that the probability that the speech recognition model 210 determines that the candidate recognition result corresponds to the speech to be recognized is higher. Conversely, if the intermediate confidence score is higher, it means that the probability that the speech recognition model 210 determines that the candidate recognition result corresponds to the speech to be recognized is lower.

[0057] In block 320, the model application system 120 uses the first language association relationship to determine the first language confidence score of the candidate recognition result based on the text feature information. The language association relationship can be used to measure the language confidence score of the candidate recognition result from a text perspective, such as whether the language can accurately reflect the information contained in the speech to be recognized.

[0058] In some embodiments, the first language association relationship may be an association relationship determined based on a text sample in a general scenario. The text sample used to determine the language association relationship is also referred to as a dictionary or a thesaurus. In some embodiments, the first language association relationship may be determined using at least a text sample in a speech-text sample pair used to train the speech recognition model 210. In some embodiments, the first language confidence score may be determined using a portion of the speech recognition model 210 that can characterize the language association relationship.

[0059] For the speech recognition model 210, since the model also takes into account the text information of the candidate recognition results, it can generally be understood that it has an internal language model for accurately generating a text sequence corresponding to the speech from a language perspective. Depending on the structure of the speech recognition model 210, such an internal language model may be explicitly configured, or may exist implicitly. In the embodiments of the present disclosure, it is expected that the domain-specific language model 220 outside the speech recognition model 210 can be used to promote accurate speech recognition in a specific domain. Therefore, in some embodiments, it is proposed to determine the language confidence score given by the speech recognition model 210 regarding the current candidate recognition result. Specifically, the language confidence score is determined based on the text feature information of the candidate recognition result, rather than based on the audio feature information. The first language confidence score is a comprehensive measure of the confidence determination of the speech recognition model 210 regarding the candidate recognition result from the aspect of text features, that is, to determine whether the candidate recognition result is a correct recognition result or a partially correct recognition result (when the end of the speech sequence is not recognized) for the speech to be recognized.

[0060] In some embodiments, the text feature information and the audio feature information can be considered to be generated in the feature extraction part or the decoder part of the speech recognition model 210, so the feature extraction part or the decoder part of the speech recognition model 210 can characterize the first language association relationship. In some embodiments, the first language confidence score can be generated by excluding the audio feature information and inputting the text feature information into the subsequent predictor part or the decoder part. If the first language confidence score is higher, it means that the probability that the speech recognition model 210 determines that the candidate recognition result corresponds to the speech to be recognized from the text perspective is higher. Conversely, if the first language confidence score is higher, it means that the probability that the speech recognition model 210 determines that the candidate recognition result corresponds to the speech to be recognized from the text perspective is lower.

[0061] At block 330 , the model application system 120 utilizes the second language association relationship to determine a second language confidence score for the candidate recognition result based on the text feature information.

[0062] In some embodiments, the second language association relationship may indicate a language association relationship in a specific target domain. The second language association relationship is different from the first language association relationship and may be determined based on a text sample different from the first language association relationship. The second language association relationship may be determined based on a text sample in a specific target domain. In some embodiments, the second language association relationship may be characterized by a language model, such as a domain-specific language model 220. The domain-specific language model 220 is a language model independent of a speech recognition model. The domain-specific language model 220 may be constructed based on a language model structure suitable for implementation. As an example, the domain-specific language model 220 may be based on a statistical language model, such as an n-gram language model, or a neural network language model, such as an RNN language model, or the like.

[0063] In the model application phase, the domain specific language model 220 is a trained model. Figure 4B FIG. 4 is a flowchart showing a model training process 402 for a domain-specific language model 220 according to some embodiments of the present disclosure. The process 402 may be implemented, for example, in Figure 1 The model training system is implemented at 110.

[0064] In box 450, the model training system 110 obtains text samples related to the target domain. In box 440, the model training system 110 uses the obtained text samples to train the domain-specific language model 220. Based on the text samples related to the target domain, the domain-specific language model 220 can be trained to perform more accurate text sequence generation for the target domain. The target domain can be set according to specific application needs, such as medical, driving, etc. Text samples in a specific domain are easier to obtain than using audio and text sample pairs in a specific domain. In addition, compared to training a domain-specific speech recognition model, the training overhead and training difficulty of a domain-specific language model are smaller.

[0065] Since only the text in the domain needs to be used to determine the language association relationship, such as training a domain-specific language model, the embodiments of the present disclosure can quickly optimize new domains with only a small amount of resources, and have strong practicality and scalability. In some applications, multiple domain-specific language models related to different domains can be trained as needed, and these domain-specific language models can share a common speech recognition model.

[0066] In block 460, after the training is completed, the model training system 110 obtains the trained domain-specific language model 220 to provide it to the model application system 120 for use. The domain-specific language model 220 trained by process 402 can represent language association relationships in a specific target domain.

[0067] Continue to return to reference Figure 3 In some embodiments, compared with the internal language model in the speech recognition model 210, the domain-specific language model 220 determines the second language confidence score of the current candidate recognition result based on the same text feature information. In this way, the domain-specific language model 220 measures whether the current candidate recognition result is the correct recognition result or a partially correct recognition result (when the end of the speech sequence is not recognized) of the speech to be recognized based on the knowledge in the specific domain. If the second language confidence score is higher, it means that the domain-specific language model 220 determines that the probability that the candidate recognition result corresponds to the speech to be recognized is higher. Conversely, if the second language confidence score is higher, it means that the domain-specific language model 220 determines that the probability that the candidate recognition result corresponds to the speech to be recognized is lower.

[0068] In some embodiments, the model application system 120 may also determine whether the speech to be recognized 205 is related to the target domain targeted by the domain-specific language model 220. For example, whether the speech to be recognized 205 is related to the target domain may be determined by explicit indication or by other auxiliary information. If it is determined that the speech to be recognized 205 is related to the target domain, the domain-specific language model 220 is used to determine the second language confidence score. In some embodiments, if there are multiple domain-specific language models, the domain related to the speech to be recognized 205 may be determined, and a language model of the corresponding domain may be selected from the multiple domain-specific language models to determine the second language confidence score of the candidate recognition result.

[0069] In block 340, the model application system 120 determines a target confidence score for the candidate recognition result based on the comparison result of the first language confidence score with the second language confidence score and the intermediate confidence score. Specifically, the model application system 120 can use the fusion module 230 in the speech recognition architecture 200 to determine the target confidence score for the candidate recognition result. The target confidence score for the candidate recognition result can be used to further determine the final speech recognition result 235 for the language to be recognized 205.

[0070] In an embodiment of the present disclosure, a first language confidence score determined by using a first language association relationship under a general domain is compared with a second language confidence model determined by using a second language association relationship under a specific domain, and the comparison result will intervene in the final confidence score of the candidate recognition result. Specifically, by comparison, it can be determined whether the second language association relationship (e.g., a domain-specific language model) has a higher language confidence than the first language association relationship (e.g., a speech recognition model) with respect to the candidate recognition result. Furthermore, a higher language confidence score can be used to determine the final confidence score of the current candidate recognition result and affect the final speech recognition result of the speech to be recognized.

[0071] According to an embodiment of the present disclosure, by fusing the language confidence scores of candidate recognition results based on different language association relationships, it is possible to fully consider the recognition optimization of specific fields and accurate recognition of general fields, which can improve the recognition accuracy of specific field speech and general speech in specific scenarios.

[0072] Specifically, in some embodiments, if the comparison result indicates that the second language association has a higher language confidence than the first language association for the candidate recognition result, the first language confidence score is removed from the intermediate confidence score to obtain a partial confidence score. The partial confidence score refers to the portion of the score from which the influence of the language or text features in the speech recognition model 210 is removed. Then, the target confidence score is determined by combining the partial confidence score and the second language confidence score. In some embodiments, if the comparison result indicates that the second language association has a lower language confidence than the first language association for the candidate recognition result, the intermediate confidence score is determined as the target confidence score for the candidate recognition result.

[0073] In some embodiments, the first language confidence score and the second language confidence score may be weighted, and then the weighted results may be compared and a target confidence score may be determined, which will be described in detail below.

[0074] For better understanding, the t-th step of speech recognition in the speech recognition performed iteratively in sequence will be discussed in detail below. Figure 5 A flow chart of a speech recognition process 500 according to some embodiments of the present disclosure is shown. For ease of discussion, the speech recognition model 210 and the domain-specific language model 220 are used to represent different language association relationships.

[0075] In the process 500, the target recognition result 502 of the t-1 step can be used as a model input of the speech recognition model 210 and the domain-specific language model 220. The target recognition result 502 of the t-1 step can include, for example, the t-1th text unit predicted for the speech to be recognized 205. In some embodiments, the t-2th text unit or more previously predicted text units can also be used as the target recognition result 502 of the t-1 step and input to the speech recognition model 210 and the domain-specific language model 220.

[0076] The speech recognition model 210 determines the intermediate confidence score 512 (expressed as AMScore) of the candidate recognition result of the t-th step according to the normal speech recognition process, which takes into account the audio feature information of the speech to be recognized 205 and the text feature information of the candidate recognition result of the t-th step (and the target recognition result 502 of the t-1-th step). The candidate recognition result of the t-th step may, for example, include the t-th text unit predicted for the speech to be recognized 205.

[0077] In addition, by constraining the input of the predictor or decoder of the speech recognition model 210, the language confidence score 514 (expressed as InterLMScore) of the candidate recognition result of the tth step is determined based on the text feature information without considering the audio feature information, which is the so-called "first language confidence score" mentioned above.

[0078] The domain-specific language model 220 can determine the language confidence score 522 (expressed as ExtLMScore) of the candidate recognition result in the t-th step, which is the so-called "second language confidence score" mentioned above, under the condition of the target recognition result 502 in the t-1th step.

[0079] At block 530 , a score comparison is performed on the language confidence score 514 InterLMScore and the language confidence score 522 ExtLMScore , such as at the fusion module 230 of the speech recognition architecture 200 .

[0080] In some embodiments, the language confidence score 514InterLMScore and the language confidence score 522ExtLMScore can be directly compared. If the language confidence score 522ExtLMScore exceeds the language confidence score 514InterLMScore, the fusion module 230 can determine that the domain-specific language model 220 has a higher language confidence than the speech recognition model 210 for the candidate recognition result of the t-th step. If the language confidence score 522ExtLMScore does not exceed (e.g., is less than or equal to) the language confidence score 514InterLMScore, the fusion module 230 can determine that the domain-specific language model 220 has a lower language confidence than the speech recognition model 210 for the candidate recognition result.

[0081] In some embodiments, the language confidence score 514InterLMScore and the language confidence score 522ExtLMScore may be weighted using a first weight (expressed as λ1) and a second weight (expressed as λ2) to obtain a first weighted confidence score (expressed as λ1). 1* The first weighted confidence score and the second weighted confidence score are then compared to determine the comparison result. By weighting, the difference in scoring of candidate recognition results by the domain-specific language model 220 and the internal language model in the speech recognition model 210 can be eliminated or reduced. For example, if the internal language model in the speech recognition model 210 tends to give a higher confidence score to most candidate recognition results, this will affect the effect of determining the target confidence score.

[0082] In some embodiments, the weights λ1 and λ2 used may be determined together with the determination process of the second language association relationship and the first language association relationship, for example, by the training process of the domain-specific language model 220 and the speech recognition model 210. In some embodiments, if the domain-specific language model 220 and the speech recognition model 210 are to be used in combination, the two models may be jointly trained using additional training data to adjust at least the weights λ1 and λ2.

[0083] In some embodiments, if λ2*ExtLMScore exceeds λ 1* InterLMScore, the fusion module 230 can determine that the domain-specific language model 220 has a higher language confidence than the speech recognition model 210 about the candidate recognition result. 1* InterLMScore, the fusion module 230 may determine that the domain specific language model 220 has a lower language confidence than the speech recognition model 210 regarding the candidate recognition result.

[0084] In some embodiments, if it is determined that the domain-specific language model 220 has a higher language confidence than the speech recognition model 210 regarding the candidate recognition result, for example, λ2*ExtLMScore exceeds λ 1* InterLMScore, in box 540, the fusion module 230 determines the target confidence score of the tth step as P=AMScore+λ2*ExtLMScore-λ1*InterLMScore.

[0085] If it is determined that the domain-specific language model 220 has a lower language confidence than the speech recognition model 210 regarding the candidate recognition result, at block 550 , the fusion module 230 determines a target confidence score for the t-th candidate recognition result as P=AMScore.

[0086] In some embodiments, the target confidence score at step t may be determined as follows:

[0087] P=AMScore+max(0,λ2*ExtLMScore-λ1*InterLMScore) (1)

[0088] After determining the target confidence score of the candidate recognition result of step t, the target recognition result of step t can be determined based on the target confidence score in box 560. For example, the target confidence scores of multiple candidate recognition results can be determined in step t, and the candidate recognition result with a higher target confidence score is used as the target recognition result of step t. Based on the target recognition result of step t, the speech recognition result 235 corresponding to the speech to be recognized 205 can be determined.

[0089] It should be understood that Figure 5 A specific example of speech recognition is shown. According to the configured model results, the inputs of the speech recognition model 210 and the domain-specific model 220 may be different in each iteration, and more other information may be considered, which may be set according to the specific application. The embodiments of the present disclosure are not limited in this respect.

[0090] Figure 6 A schematic structural block diagram of an apparatus 600 for speech recognition according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in the model application system 120. Each module / component in the apparatus 600 may be implemented by hardware, software, firmware or any combination thereof.

[0091] As shown in the figure, the device 600 includes an intermediate score determination module 610, which is configured to determine the intermediate confidence score of the candidate recognition result based on the audio feature information and text feature information corresponding to the speech to be recognized. The device 600 also includes a first language score determination module 620, which is configured to use the first language association relationship to determine the first language confidence score of the candidate recognition result based on the text feature information, and a second language score determination module 630, which is configured to use the second language association relationship to determine the second language confidence score of the candidate recognition result based on the text feature information. The device 600 also includes a target score determination module 640, which is configured to determine the target confidence score of the candidate recognition result based on the comparison result of the first language confidence score and the second language confidence score and the intermediate confidence score.

[0092] In some embodiments, the intermediate score determination module 610 is configured to utilize a speech recognition model to determine an intermediate confidence score of a candidate recognition result based on audio feature information and text feature information.

[0093] In some embodiments, the first language score determination module 620 is configured to determine the first language confidence score of the candidate recognition result based on the text feature information by utilizing the portion of the speech recognition model that represents the first language association relationship.

[0094] In some embodiments, the second language score determination module 630 is configured to determine the second language confidence score of the candidate recognition result based on the text feature information using a domain specific language model, the domain specific language model representing the second language association relationship.

[0095] In some embodiments, the target score determination module 640 includes: a first determination module configured to remove the first language confidence score from the intermediate confidence score to obtain a partial confidence score if the comparison result indicates that the second language association relationship has a higher language confidence than the first language association relationship with respect to the candidate recognition result, and determine the target confidence score by combining the partial confidence score and the second language confidence score; and a second determination module configured to determine the intermediate confidence score as the target confidence score if the comparison result indicates that the second language association relationship has a lower language confidence than the first language association relationship with respect to the candidate recognition result.

[0096] In some embodiments, the device 600 also includes: a weighting module, configured to use a first weight and a second weight to weight the first language confidence score and the second language confidence score respectively to obtain a first weighted confidence score and a second weighted confidence score; and a comparison result determination module, configured to determine the comparison result by comparing the first weighted confidence score and the second weighted confidence score.

[0097] In some embodiments, the comparison result determination module includes: a first result indication determination module, configured to determine that the comparison result indicates that the second language association relationship has a higher language confidence with respect to the candidate recognition result than the first language association relationship if the second weighted confidence score exceeds the first weighted confidence score; and a second result indication determination module, configured to determine that the comparison result indicates that the second language association relationship has a lower language confidence with respect to the candidate recognition result than the first language association relationship if the second weighted confidence score does not exceed the first weighted confidence score.

[0098] In some embodiments, the first weight and the second weight are determined together with the determination of the second language association and the first language association.

[0099] In some embodiments, the speech recognition model is trained based on speech and text sample pairs related to general scenarios. In some embodiments, the domain-specific language model is trained based on text samples related to the target domain.

[0100] In some embodiments, the second language association relationship indicates a language association relationship in the target domain. In some embodiments, the second language score determination module 630 includes: a domain determination module configured to determine whether the speech to be recognized is related to the target domain; and a domain-based language score determination module configured to determine the second language confidence score using the second language association relationship if it is determined that the speech to be recognized is related to the target domain.

[0101] In some embodiments, the apparatus 600 further includes: a recognition result determination module configured to determine a speech recognition result of the speech to be recognized based on a target confidence score of the candidate recognition result.

[0102] Figure 7 1 shows a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 7 The electronic device 700 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown can be used to implement Figure 1 The electronic device 700 may include or be implemented as a model application system 120 and / or a model training system 110. Figure 6 Device 600.

[0103] like Figure 7 As shown, the electronic device 700 is in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capabilities of the electronic device 700.

[0104] The electronic device 700 typically includes a plurality of computer storage media. Such media may be any available media accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the electronic device 700.

[0105] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0106] The communication unit 740 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 700 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0107] The input device 750 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 760 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 700, or communicate with any device that allows the electronic device 700 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0108] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0109] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0110] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0111] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0112] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0113] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A speech recognition method, comprising: Determining an intermediate confidence score of a candidate recognition result of the speech to be recognized based on audio feature information and text feature information corresponding to the speech to be recognized; Determining a first language confidence score of the candidate recognition result based on the text feature information using a first language association relationship, wherein the first language association relationship is a language association relationship determined based on a text sample in a general scenario; Determining a second language confidence score of the candidate recognition result based on the text feature information using a second language association relationship, wherein the second language association relationship indicates a language association relationship in a target domain; as well as Determining a target confidence score of the candidate recognition result based on a comparison result of the first language confidence score and the second language confidence score and the intermediate confidence score, comprising: If the comparison result indicates that the second language association has a higher language confidence with respect to the candidate recognition result than the first language association, removing the first language confidence score from the intermediate confidence score to obtain a partial confidence score, and determining the target confidence score by combining the partial confidence score and the second language confidence score; as well as If the comparison result indicates that the second language association relationship has a lower language confidence with respect to the candidate recognition result than the first language association relationship, the intermediate confidence score is determined as the target confidence score.

2. The method of claim 1, wherein determining the intermediate confidence score of the candidate recognition result comprises: Using a speech recognition model, an intermediate confidence score of the candidate recognition result is determined based on the audio feature information and the text feature information.

3. The method of claim 2, wherein determining the first language confidence score of the candidate recognition result comprises: The first language confidence score of the candidate recognition result is determined based on the text feature information using the portion of the speech recognition model that represents the first language association relationship.

4. The method according to claim 1, wherein determining the second language confidence score of the candidate recognition result comprises: A second language confidence score of the candidate recognition result is determined based on the text feature information using a domain specific language model, the domain specific language model characterizing the second language association relationship.

5. The method according to claim 1, further comprising: Using a first weight and a second weight to weight the first language confidence score and the second language confidence score respectively to obtain a first weighted confidence score and a second weighted confidence score; as well as The comparison result is determined by comparing the first weighted confidence score and the second weighted confidence score.

6. The method of claim 5, wherein determining the comparison result comprises: If the second weighted confidence score exceeds the first weighted confidence score, determining that the comparison result indicates that the second language association has a higher language confidence with respect to the candidate recognition result than the first language association; as well as If the second weighted confidence score does not exceed the first weighted confidence score, determining that the comparison result indicates that the second language association has a lower language confidence with respect to the candidate recognition result than the first language association. 7 . The method of claim 5 , wherein the first weight and the second weight are determined together with a determination process of the second language association relationship and the first language association relationship.

8. The method of claim 2, wherein the speech recognition model is trained based on speech and text sample pairs associated with common scenarios.

9. The method of claim 4, wherein the domain-specific language model is trained based on text samples related to the target domain.

10. The method of claim 1, wherein determining the second language confidence score comprises: Determining whether the speech to be recognized is related to the target domain; as well as If it is determined that the speech to be recognized is related to the target domain, the second language association relationship is used to determine the second language confidence score.

11. The method according to claim 1, further comprising: Based on the target confidence score of the candidate recognition result, a speech recognition result of the speech to be recognized is determined.

12. A device for speech recognition, comprising: An intermediate score determination module, configured to determine an intermediate confidence score of a candidate recognition result of the speech to be recognized based on audio feature information and text feature information corresponding to the speech to be recognized; A first language score determination module is configured to determine a first language confidence score of the candidate recognition result based on the text feature information using a first language association relationship, wherein the first language association relationship is a language association relationship determined based on a text sample in a general scenario; A second language score determination module configured to determine a second language confidence score of the candidate recognition result based on the text feature information using a second language association relationship, wherein the second language association relationship indicates a language association relationship in a target domain; as well as a target score determination module configured to determine a target confidence score of the candidate recognition result based on a comparison result of the first language confidence score and the second language confidence score and the intermediate confidence score; The target score determination module comprises: a first determination module configured to, if the comparison result indicates that the second language association relationship has a higher language confidence with respect to the candidate recognition result than the first language association relationship, remove the first language confidence score from the intermediate confidence score to obtain a partial confidence score, and determine a target confidence score by combining the partial confidence score and the second language confidence score; as well as The second determination module is configured to determine the intermediate confidence score as the target confidence score if the comparison result indicates that the second language association relationship has a lower language confidence with respect to the candidate recognition result than the first language association relationship.

13. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the method according to any one of claims 1 to 11 when executed by the at least one processing unit.

14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 11 when executed by a processor.