Speech recognition method, electronic device, program product, and storage medium

Abstract

The embodiments of the present disclosure disclose a speech recognition method, electronic device, program product and storage medium. The method decodes the speech to be recognized to obtain a first decoding result, including multiple first word sequences and the start time and end time, acoustic probability and language probability of each word in the multiple first word sequences; calculates the confidence of each word in the multiple first word sequences based on the acoustic probability and language probability of each word in the multiple first word sequences; obtains a second decoding result based on the words in the multiple first word sequences whose confidence is greater than a first preset threshold, including multiple second word sequences and the start time and end time, acoustic probability and language probability of each word in the multiple second word sequences; obtains a speech recognition result based on the multiple second word sequences and the start time and end time, acoustic probability and language probability of each word therein, which can improve the accuracy of the speech recognition result, thereby improving the application effect based on speech recognition.

Description

Technical Field

[0001] The present disclosure relates to a speech recognition method and apparatus, an electronic device, a program product, and a storage medium. Background Art

[0002] Speech recognition is a technology that uses speech as its research object. Through speech signal processing and pattern recognition, machines can automatically recognize and understand human spoken language, converting speech into text. With the development of the mobile internet, speech recognition is becoming increasingly important and serves as the foundation for many other applications. For example, speech recognition technology enables applications such as voice dialing and voice navigation. The more accurate the speech recognition results, the better the performance of speech-based applications.

[0003] When existing speech recognition systems return the text sentence of the entire speech recognition result, they will return the confidence level of the text sentence. The receiving end will judge the credibility of the text sentence based on whether the confidence level of the text sentence is greater than a preset threshold, and thus decide the next action.

[0004] In the process of realizing the present invention, the inventor discovered through research that since the existing speech recognition system returns the confidence of the entire text sentence, and the entire text sentence may contain inaccurately recognized words, when the receiving end judges that the credibility of the text sentence is higher based on the confidence of the entire text sentence and proceeds to the next action, the presence of inaccurately recognized words in the entire text sentence may lead to erroneous actions, thereby reducing the application effect based on speech recognition. Summary of the Invention

[0005] The embodiments of the present disclosure provide a speech recognition method and apparatus, an electronic device, a program product, and a storage medium to improve the accuracy of speech recognition results, at least to a certain extent solve the problem of erroneous actions caused by inaccurately recognized words in the entire text sentence, and improve the application effect based on speech recognition.

[0006] According to one aspect of an embodiment of the present disclosure, there is provided a speech recognition method, comprising:

[0007] Decoding the speech to be recognized to obtain a first decoding result, wherein the first decoding result includes: a plurality of first word sequences and a start time and an end time, an acoustic probability, and a language probability of each word in the plurality of first word sequences;

[0008] Calculating the confidence of each word in the plurality of first word sequences based on the acoustic probability and the language probability of each word in the plurality of first word sequences respectively;

[0009] Obtaining a second decoding result based on the words in the plurality of first word sequences whose confidences are greater than a first preset threshold, the second decoding result including: a plurality of second word sequences and a start time and an end time, an acoustic probability, and a language probability of each word in the plurality of second word sequences;

[0010] Based on the multiple second word sequences and the start time and end time, acoustic probability and language probability of each word in the multiple second word sequences, a speech recognition result is obtained, and the language recognition result includes: a first sentence and the confidence of the first sentence, so as to perform corresponding actions based on the speech recognition result.

[0011] Optionally, in any embodiment of the present disclosure, calculating the confidence of each word in the multiple first word sequences based on the acoustic probability and the language probability of each word in the multiple first word sequences respectively includes:

[0012] Taking each word in each of the multiple first word sequences as a current word, and calculating a forward probability and a backward probability of the current word based on the acoustic probability and the language probability of the current word;

[0013] Calculating the posterior probability of the current word in the first word sequence based on the forward probability, backward probability and language probability of the current word;

[0014] The posterior probabilities of the current word in the multiple first word sequences are superimposed to obtain the confidence of the current word.

[0015] Optionally, in any embodiment of the present disclosure, obtaining a speech recognition result based on the plurality of second word sequences and the start time and end time, acoustic probability, and language probability of each word in the plurality of second word sequences includes:

[0016] Determining a second word sequence with the highest comprehensive score among the plurality of second word sequences based on the start time and end time, acoustic probability, and language probability of each word in the plurality of second word sequences;

[0017] Obtaining a first sentence based on the second word sequence with the highest comprehensive score;

[0018] The confidence of the first sentence is obtained based on the confidence of each word in the first sentence.

[0019] Optionally, in any embodiment of the present disclosure, before decoding the speech to be recognized, the method further includes:

[0020] Using the endpoint detection model, perform voice endpoint detection on the audio signal collected by the audio acquisition module to obtain the starting point and end point of at least one voice activity segment;

[0021] Based on the starting point and end point of the at least one voice activity segment, the at least one voice activity segment is cut out from the audio signal, so that each voice activity segment in the at least one voice activity segment is used as the voice to be recognized, and the operation of decoding the voice to be recognized is performed to obtain a first decoding result.

[0022] Optionally, in any embodiment of the present disclosure, after calculating the confidence of each word in the multiple first word sequences based on the acoustic probability and the language probability of each word in the multiple first word sequences, the method further includes:

[0023] In response to the confidence of each word in the multiple first word sequences being greater than a second preset threshold and less than a third preset threshold, the speech to be recognized is added to a noise set as a noise sample for training the endpoint detection model, and / or as foreground noise for training the acoustic model; wherein, the noise set includes at least one noise sample, and the second preset threshold is less than the third preset threshold.

[0024] Optionally, in any embodiment of the present disclosure, the method further includes:

[0025] The endpoint detection model is trained using noise samples in the noise set.

[0026] Optionally, in any embodiment of the present disclosure, after calculating the confidence of each word in the multiple first word sequences based on the acoustic probability and the language probability of each word in the multiple first word sequences, the method further includes:

[0027] In response to the confidence of each word in the multiple first word sequences being greater than the third preset threshold and less than the first preset threshold, the speech to be recognized is added to the background voice set as a background voice sample to be used as background voice noise for training the acoustic model; wherein, the background voice set includes at least one background voice sample, and the third preset threshold is less than the first preset threshold.

[0028] Optionally, in any embodiment of the present disclosure, after calculating the confidence of each word in the multiple first word sequences based on the acoustic probability and the language probability of each word in the multiple first word sequences, the method further includes:

[0029] In response to the confidence of each word in the plurality of first word sequences being greater than a fourth preset threshold, obtaining a second sentence based on the plurality of first word sequences and the confidence of each word in the plurality of first word sequences; wherein the fourth preset threshold is not less than the first preset threshold;

[0030] The second sentence is used as the annotation information of the speech to be recognized, and the speech to be recognized and the annotation information of the speech to be recognized are added to a speech set as a speech sample for training an acoustic model and a language model; wherein the speech set includes at least one speech sample.

[0031] Optionally, in any embodiment of the present disclosure, adding the speech to be recognized and the annotation information of the speech to be recognized as a speech sample to a speech set includes:

[0032] Determining attribute information of the second sentence, the attribute information including any one or more of the following: field, application scenario, and geographical area;

[0033] The speech to be recognized and the annotation information of the speech to be recognized are taken as a speech sample, and added to the speech set corresponding to the attribute information, so as to be used for training the acoustic model and language model corresponding to the attribute information.

[0034] Optionally, in any embodiment of the present disclosure, the method further includes:

[0035] Constructing a training data set based on noise samples in a noise set, background vocal samples in a background vocal set, and speech samples in a speech set, wherein the training data set includes at least one noisy signal, and the noisy signal includes any one or more of the following: a noisy signal generated by mixing the speech sample and the noise sample as foreground noise, a noisy signal generated by mixing the speech sample and the background vocal sample as background vocal noise, and a noisy signal generated by mixing the speech sample, the noise sample as foreground noise, and the background vocal sample as background vocal noise, the noisy signal being annotated with annotation information of the speech sample used to generate the noisy signal;

[0036] The training data set is used to train an acoustic model and / or a language model for decoding the speech to be recognized.

[0037] According to another aspect of the embodiments of the present disclosure, there is provided a speech recognition device, comprising:

[0038] A speech recognition model is configured to decode the speech to be recognized to obtain a first decoding result, wherein the first decoding result includes: a plurality of first word sequences and a start time and an end time, an acoustic probability, and a language probability of each word in the plurality of first word sequences;

[0039] a confidence calculation module, configured to calculate the confidence of each word in the plurality of first word sequences based on the acoustic probability and the language probability of each word in the plurality of first word sequences;

[0040] an acquisition module, configured to obtain a second decoding result based on words in the plurality of first word sequences whose confidence is greater than a first preset threshold, the second decoding result comprising: a plurality of second word sequences and a start time and an end time, an acoustic probability, and a language probability of each word in the plurality of second word sequences;

[0041] A result determination module is used to obtain a speech recognition result based on the multiple second word sequences and the start time and end time, acoustic probability and language probability of each word in the multiple second word sequences, and the language recognition result includes: a first sentence and the confidence of the first sentence, so as to perform corresponding actions based on the speech recognition result.

[0042] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, including:

[0043] memory for storing computer programs;

[0044] The processor is configured to execute the computer program stored in the memory, and when the computer program is executed, the speech recognition method described in any one of the above embodiments of the present disclosure is implemented.

[0045] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the speech recognition method described in any of the above embodiments of the present disclosure is implemented.

[0046] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program / instruction, which implements the speech recognition method described in any of the above embodiments of the present disclosure when executed by a processor.

[0047] Based on the speech recognition method and device, electronic device, program product and storage medium provided by the above-mentioned embodiments of the present disclosure, after decoding the speech to be recognized to obtain a first decoding result, the confidence of each word is calculated based on the acoustic probability and language probability of each word in the multiple first word sequences, and then the first decoding result is filtered based on a first preset threshold to obtain a second decoding result. Furthermore, based on the multiple second word sequences in the second decoding result and the start time and end time of each word therein, as well as the acoustic probability and language probability, a speech recognition result is obtained, so as to perform corresponding actions based on the speech recognition result. Therefore, after decoding the speech to be recognized to obtain a decoding result, the embodiment of the present disclosure can eliminate words with low confidence in the decoding result and determine the speech recognition result only based on words with high confidence, thereby improving the accuracy of the speech recognition result. Therefore, when performing corresponding actions based on the speech recognition result, it avoids erroneous actions caused by inaccurately recognized words in the obtained sentence, which helps to improve the application effect based on speech recognition.

[0048] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0050] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0051] Figure 1 The flowchart of one embodiment of the speech recognition method disclosed herein.

[0052] Figure 2 This is a flowchart of another embodiment of the speech recognition method disclosed herein.

[0053] Figure 3 The flowchart of another embodiment of the speech recognition method disclosed herein.

[0054] Figure 4 The flowchart of another embodiment of the speech recognition method disclosed herein.

[0055] Figure 5 The following is a flowchart of another embodiment of the speech recognition method disclosed herein.

[0056] Figure 6 The figure is a structural diagram of an embodiment of the speech recognition device disclosed in the present invention.

[0057] Figure 7 FIG. 4 is a structural diagram of another embodiment of the speech recognition device disclosed in the present invention.

[0058] Figure 8 The figure is a schematic structural diagram of an application embodiment of the electronic device disclosed herein. DETAILED DESCRIPTION

[0059] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure.

[0060] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate a necessary logical order between them.

[0061] It should also be understood that in the embodiments of the present disclosure, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0062] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0063] In addition, the term "and / or" in this disclosure is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the related objects are in an "or" relationship.

[0064] It should also be understood that the description of the various embodiments in this disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0065] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0066] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0067] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0068] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0069] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate in conjunction with numerous other general-purpose or specialized computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with terminal devices, computer systems, servers, and other electronic devices include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, among others.

[0070] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media, including storage devices.

[0071] Speech recognition is a technology that uses speech as its research object and enables machines to automatically recognize and understand human spoken language through speech signal processing and pattern recognition. It allows computers to convert speech into text and obtain corresponding words or character sequences for a given waveform sequence.

[0072] Figure 1 FIG. 1 is a flow chart of an embodiment of the speech recognition method disclosed herein. Figure 1 As shown, the speech recognition method of this embodiment includes:

[0073] 102. Decode the speech to be recognized to obtain a first decoding result.

[0074] The first decoding result is a word graph, which includes: multiple first word sequences corresponding to multiple paths in the word graph, each first word sequence includes at least one word, and the start time and end time, acoustic probability and language probability of each word in the at least one word.

[0075] Optionally, operation 102 may be implemented by a speech recognition model (including an acoustic model and a language model).

[0076] Among them, the acoustic probability is used to represent the probability of pronunciation and phonemes of a certain segment of speech in the speech to be recognized, which can be obtained through the acoustic model.

[0077] The acoustic model outputs an acoustic recognition result, which includes multiple paths, each path including at least one phoneme and the acoustic probability of each phoneme in the at least one phoneme, and the at least one phoneme obtains each word included in a first word sequence based on the acquisition time sequence of the speech to be recognized. The acoustic model may include, but is not limited to: Gaussian Mixture-Model Hidden Markov Model (GMM-HMM), Recurrent Neural Networks (RNN), Feedforward Sequential Memory Networks (FSMN), etc., and the embodiments of the present disclosure are not limited to this.

[0078] Acoustic Probability After obtaining the acoustic recognition result, the acoustic recognition result can be input into a language model to obtain the language probability of each phoneme in the acoustic recognition result to a word (including a character).

[0079] The language model may include, but is not limited to, a rule-based language model, a statistical language model, or a neural network language model (NNLM), etc., which is not limited in the embodiments of the present disclosure.

[0080] The speech to be recognized in the embodiments of the present disclosure may be the original audio signal collected by an audio acquisition module (such as a microphone), or the audio signal after the original audio signal has been processed by the front-end signal; in addition, the speech to be recognized may be an audio signal collected in real time by an application (APP) with a voice interaction function, or a historical audio signal stored in the APP, and the embodiments of the present disclosure are not limited to this. Among them, the front-end signal processing may include, but is not limited to: Voice Activity Detection (VAD), noise reduction, acoustic echo cancellation (AEC), dereverberation processing, sound source localization, beam forming (BF), etc.

[0081] 104 , calculating the confidence of each word in the plurality of first word sequences based on the acoustic probability and the language probability of each word in the plurality of first word sequences.

[0082] The confidence level of a word indicates the credibility and reliability of the word as a result of speech recognition. The confidence level ranges from 0 to 1. A larger value indicates a higher credibility and reliability of the word as a result of speech recognition.

[0083] 106. Obtain a second decoding result based on the words in the plurality of first word sequences whose confidences are greater than a first preset threshold.

[0084] The second decoding result is the word graph obtained after filtering out words whose confidence is not greater than the first preset threshold and their corresponding start time and end time, acoustic probability and language probability from the above word graph, which includes: multiple second word sequences, the start time and end time, acoustic probability and language probability of each word in the multiple second word sequences.

[0085] The specific value of the first preset threshold can be set based on various factors such as different fields, application scenarios, geographical regions, and the accuracy requirements for speech recognition results, and can be updated as needed. For example, in a voice dialing application, the value of the first preset threshold can be set to 0.5, in a voice navigation application, the value of the first preset threshold can be set to 0.4, and so on. The embodiments of the present disclosure do not limit the specific value of the first preset threshold or the factors that influence the value.

[0086] 108. Based on the plurality of second word sequences and the start time and end time, acoustic probability and language probability of each word in the plurality of second word sequences, a speech recognition result is obtained, so as to perform corresponding actions based on the speech recognition result.

[0087] The language recognition result includes: a first sentence and a confidence level of the first sentence.

[0088] Among them, performing corresponding actions based on the voice recognition results refers to executing the work corresponding to the voice recognition results in application scenarios based on voice recognition (such as implementing voice dialing, voice navigation, voice song requesting, voice wake-up and other applications). For example, in the voice navigation application scenario, based on the language recognition result "navigate to Building A", the navigation action of navigating to Building A is executed.

[0089] This enables applications such as voice dialing and voice navigation. The more accurate the voice recognition results, the better the performance of applications based on voice recognition.

[0090] Based on the speech recognition method provided by the above-mentioned embodiment of the present disclosure, after decoding the speech to be recognized and obtaining a first decoding result, the confidence of each word is calculated based on the acoustic probability and language probability of each word in the multiple first word sequences, and then the first decoding result is filtered based on a first preset threshold to obtain a second decoding result. Furthermore, based on the multiple second word sequences in the second decoding result and the start time and end time of each word therein, as well as the acoustic probability and language probability, a speech recognition result is obtained so as to perform corresponding actions based on the speech recognition result. Therefore, after decoding the speech to be recognized and obtaining the decoding result, the embodiment of the present disclosure can eliminate the words with lower confidence in the decoding result and determine the speech recognition result only based on the words with higher confidence, thereby improving the accuracy of the speech recognition result. Therefore, when performing corresponding actions based on the speech recognition result, it avoids erroneous actions caused by the presence of inaccurately recognized words in the obtained sentence, which helps to improve the application effect based on speech recognition.

[0091] Figure 2 FIG. 1 is a flow chart of another embodiment of the speech recognition method disclosed herein. Figure 2 As shown, in Figure 1 Based on the embodiment shown, 104 in this embodiment may include:

[0092] 1042. Take each word in each of the multiple first word sequences as the current word, and calculate the forward probability and backward probability of the current word based on the acoustic probability and language probability of the current word and based on a preset calculation method, such as a forward and backward algorithm.

[0093] 1044. Based on the forward probability, backward probability, and language probability of the current word and a preset calculation method, calculate the posterior probability of the current word in the first word sequence.

[0094] 1046 , superimpose the posterior probabilities of the current word in the multiple first word sequences to obtain the confidence of the current word, that is, the confidence of the current word in the entire word graph.

[0095] Based on this embodiment, the confidence of the current word in the entire word graph can be objectively and accurately determined through the acoustic probability and language probability of each word, so as to objectively and accurately determine the credibility and reliability of the word as a speech recognition result, and thus determine whether to remove the word from the word graph, which helps to improve the accuracy of the speech recognition results.

[0096] Optionally, in some implementations of any embodiment of the present disclosure, in 108, a second word sequence with the highest comprehensive score in the above-mentioned multiple second word sequences can be determined based on the start time and end time, acoustic probability and language probability of each word in the above-mentioned multiple second word sequences, wherein the comprehensive score is determined based on the acoustic probability and language probability. Specifically, the score of each word can be determined based on the acoustic probability and language probability of the word, and the word with the highest score is selected from the above-mentioned multiple second word sequences for the same start time and end time. The second word sequence with the highest comprehensive score is constructed based on the words with the highest scores corresponding to each start time and end time, that is, the second word sequence corresponding to the optimal path in the word graph; then, based on the second word sequence with the highest comprehensive score, the first sentence is obtained, and then, based on the confidence of each word in the first sentence, the confidence of the first sentence is obtained.

[0097] In a specific implementation, the first sentence can be directly composed of the second word sequence with the highest comprehensive score, and the confidence of the first sentence can be obtained based on the confidence of each word in the first sentence.

[0098] For example, in some examples, when obtaining the confidence of the first sentence based on the confidence of each word in the first sentence, the average value of the confidence of each word in the first sentence, that is, the ratio between the sum of the confidence of each word in the first sentence and the number of words in the first sentence, can be used as the confidence of the first sentence. Alternatively, in other examples, the median of the confidence of each word in the first sentence can be sorted in order from large to small or from small to large to obtain a confidence sequence, and a confidence in the middle of the confidence sequence or the average of the two confidences in the middle of the confidence sequence is selected as the confidence of the first sentence. The embodiments of the present disclosure do not limit the specific method of obtaining the confidence of the first sentence.

[0099] Based on this embodiment, the confidence of the first sentence can be determined based on the confidence of each word in the first sentence, which improves the accuracy and precision of the sentence confidence, thereby helping to determine the reliability of the speech recognition result.

[0100] Generally speaking, a speech recognition system mainly consists of a front-end signal processing module, an acoustic model, and a language model. The front-end processing module is mainly used to perform operations such as speech endpoint detection, noise reduction, feature extraction, acoustic echo cancellation, dereverberation processing, sound source localization, and beamforming. The acoustic model and language model belong to the back-end processing. The acoustic model is mainly used to construct the probabilistic mapping relationship between input speech and output acoustic units (phonemes), while the language model is mainly used to construct the probabilistic collocation relationship between acoustic units (phonemes) and words, and between different words, making the recognized sentences more fluent and natural.

[0101] Optionally, before step 102 in any of the above embodiments of the present disclosure, the following steps may also be included:

[0102] Using the endpoint detection model, voice endpoint detection (VAD) is performed on the audio signal collected by the audio acquisition module to obtain the start and end points of at least one voice activity segment;

[0103] Based on the starting point and the end point of the at least one speech activity segment, at least one speech activity segment is extracted from the audio signal so as to perform the operations of the above embodiment using each speech activity segment in the at least one speech activity segment as the speech to be recognized.

[0104] Among them, VAD, also known as voice activity detection or voice boundary detection, refers to detecting the presence of voice in an audio signal in a noisy environment and accurately detecting the starting position of a voice segment in the audio signal. It is usually used in voice processing systems such as voice coding and voice enhancement, and plays a role in reducing the voice coding rate, saving communication bandwidth, reducing the energy consumption of mobile devices, and improving the recognition rate. The starting point of VAD is from silence to speech, and the ending point of VAD is from speech to silence. The judgment of the ending point of VAD requires a period of silence. The speech obtained by the front-end signal processing of the original audio signal includes the speech from the starting point to the end point of VAD. Therefore, the speech to be recognized in the embodiment of the present disclosure may also include a period of silence after the speech segment.

[0105] Based on this embodiment, effective speech activity segments can be accurately extracted from the audio signal for separate speech recognition, which is beneficial to avoid identifying non-speech frames collected by the audio acquisition module as speech frames in the speech activity segment, thereby helping to improve the accuracy of the speech recognition results and the speech recognition effect.

[0106] In another embodiment of the speech recognition method disclosed herein, after step 104, the method may further include: in response to the confidence level of each word in the plurality of first word sequences being greater than a second preset threshold and less than a third preset threshold, deeming the speech to be recognized as pure noise that does not contain a human voice, adding the speech to be recognized as a noise sample to a noise set for training an endpoint detection model and / or as foreground noise for training an acoustic model. The noise set includes at least one noise sample, and the second preset threshold is less than the third preset threshold.

[0107] The second preset threshold can be set to a lower value, and the third preset threshold can be set to a value higher than the second preset threshold and lower than the first preset threshold. For example, the second preset threshold can be set to 0, 0.05, etc., and the third preset threshold can be set to 0.25, 0.3, etc., so as to select pure noise that does not include foreground and background human voices. In actual applications, the specific values of the first preset threshold, the second preset threshold, and the third preset threshold can be determined based on various factors such as the specific application and environment, and can be adjusted as needed. The embodiments of the present disclosure do not limit the specific values of the first preset threshold, the second preset threshold, and the third preset threshold.

[0108] like Figure 3 As shown in FIG, it is a flow chart of another embodiment of the speech recognition method disclosed in the present invention. Figure 3 As shown, in some of these implementations, noise samples can be obtained as follows:

[0109] 202. Obtain words with confidence greater than a second preset threshold from the multiple first word sequences, obtain multiple third word sequences and decoding information of each word in the multiple third word sequences, wherein the decoding information includes the start time and end time, acoustic probability and language probability of the word.

[0110] Optionally, the words and their decoding information in the above-mentioned multiple first word sequences whose confidence is not greater than the second preset threshold can also be eliminated, and multiple third word sequences and the decoding information of each word in the multiple third word sequences can be obtained from the first decoding results.

[0111] 204. Obtain words whose confidence is not greater than a third preset threshold value from the above-mentioned multiple third word sequences, obtain multiple fourth word sequences and decoding information of each word in the multiple fourth word sequences, wherein the decoding information includes the start time and end time, acoustic probability and language probability of the word.

[0112] Optionally, the words and their decoding information in the above-mentioned multiple third word sequences whose confidence is greater than the third preset threshold can also be eliminated, that is, the multiple fourth word sequences and the decoding information of each word in the multiple third word sequences can be obtained from the multiple third word sequences and the decoding information of each word in the multiple third word sequences.

[0113] 206 , determining a fourth word sequence with the highest comprehensive score among the plurality of fourth word sequences based on the start time and end time, acoustic probability, and language probability of each word in the plurality of fourth word sequences.

[0114] 208. Based on the fourth word sequence with the highest comprehensive score, a third sentence is obtained.

[0115] 210, identifying whether the third sentence includes the word.

[0116] If the third sentence does not include the word, operation 212 is performed. If the second sentence includes the word, the subsequent process of this embodiment is not performed and the third sentence can be directly discarded.

[0117] 212 , adding the speech to be recognized as a noise sample to the noise set for training the above-mentioned endpoint detection model, and / or as foreground noise for training the acoustic model.

[0118] The noise set includes at least one noise sample.

[0119] Or, in the above Figure 3 In the illustrated embodiment, 204 may be executed first to eliminate words whose confidence is greater than a third preset threshold value from the multiple first word sequences, and then 202 may be executed to eliminate words whose confidence is not greater than a second preset threshold value, to obtain multiple fourth word sequences and decoding information of each word in the multiple fourth word sequences, and then operations 206-212 may be executed.

[0120] Alternatively, in other implementations, noise samples can also be obtained in the following manner: after removing words from multiple first word sequences whose confidence is greater than a third preset threshold and whose confidence is not greater than a second preset threshold, identify whether the multiple first word sequences obtained thereby still include words; if not, add the speech to be recognized as a noise sample to the noise set for training the above-mentioned endpoint detection model, and / or as foreground noise for training the acoustic model.

[0121] Alternatively, the embodiment of the present disclosure can also confirm that the speech to be recognized is pure noise that does not contain human voice based on the third preset threshold and the second preset threshold in any other way, and add the speech to be recognized as a noise sample to the noise set. The embodiment of the present disclosure does not limit the specific method of confirming that the speech to be recognized is pure noise that does not contain human voice.

[0122] Based on this embodiment, real and effective online audio noise data can be collected, and the collection of noise samples is achieved through the online speech recognition process. By obtaining online noise samples and using them to train the endpoint detection model, the performance of the endpoint detection model in the online speech recognition scenario can be improved, thereby helping to improve the accuracy of the final speech recognition results. In addition, the collection of noise samples is achieved through the online speech recognition process. By obtaining online noise samples, the collection of noise samples is achieved, solving the problems of difficulty and high cost in obtaining and expanding the noise set. When the noise samples in the noise set are used as foreground noise for training the acoustic model, the performance of the acoustic model can be improved, thereby helping to improve the accuracy of the final speech recognition results.

[0123] Optionally, in Figure 3The illustrated embodiment may further include, after 208 of any of the above embodiments of the present disclosure: training an endpoint detection model using noise samples in a noise set.

[0124] For example, in some examples, the endpoint detection model can be trained using noise samples in the noise set as follows:

[0125] Using the endpoint detection model, speech endpoint detection is performed on at least one noise sample in the noise set to obtain detection information of the speech activity segment in each noise sample in the at least one noise sample, and the detection information of the speech activity segment includes the starting point and end point of the speech activity segment; based on the difference between the detection information of the speech activity segment in each noise sample in the at least one noise sample and the annotation information of the at least one noise sample, the endpoint detection model is trained.

[0126] The above-mentioned operation of training the endpoint detection model using noise samples in the noise set is an iterative operation. By iteratively executing the above-mentioned operation of training the endpoint detection model using noise samples in the noise set, the training of the endpoint detection model is realized, that is, the parameters of the endpoint detection model are optimized and adjusted until the first preset training completion condition is met to obtain a trained endpoint detection model. The preset training completion condition may, for example, include but is not limited to any one or more of the following: the number of iterative trainings of the endpoint detection model (that is, the number of iterative executions of the above-mentioned operation of training the endpoint detection model using noise samples in the noise set) reaches a preset number of times (for example, 200 times), the difference between the detection information of the speech activity segment in each noise sample in the above-mentioned at least one noise sample and the annotation information of the at least one noise sample is less than or equal to a preset threshold, etc. The embodiments of the present disclosure do not impose any restrictions on this.

[0127] Based on this embodiment, the endpoint detection model is trained or iteratively trained using noise samples in the noise set, which is beneficial to improving the performance of the endpoint detection model in the online speech recognition scenario, thereby helping to improve the accuracy of the final speech recognition results.

[0128] In another embodiment of the speech recognition method disclosed herein, after step 104, the method may further include: in response to the confidence level of each word in the plurality of first word sequences being greater than a third preset threshold and less than the first preset threshold, deeming the speech to be recognized as noise containing background human voices, adding the speech to be recognized as a background human voice sample to a background human voice set to serve as background human voice noise for training the acoustic model. The background human voice set includes at least one background human voice sample, and the third preset threshold is less than the first preset threshold.

[0129] The third preset threshold can be set to a value higher than the second preset threshold and lower than the first preset threshold. For example, the third preset threshold can be set to 0.25, 0.3, etc., and the first preset threshold can be set to 0.4, 0.5, etc., so as to select noise that does not include foreground human voices but includes background human voices. In actual applications, the specific values of the first and third preset thresholds can be determined based on various factors such as the specific application and environment, and can be adjusted as needed. The embodiments of this disclosure do not limit the specific values of the first and third preset thresholds.

[0130] Figure 4 FIG. 1 is a flow chart of another embodiment of the speech recognition method disclosed herein. Figure 4 As shown, in some of these implementations, background vocal samples can be obtained in the following ways:

[0131] 302. Obtain words with confidence greater than a third preset threshold from the multiple first word sequences, obtain multiple fifth word sequences and decoding information of each word in the multiple fifth word sequences, wherein the decoding information includes the start time and end time, acoustic probability, and language probability of the word.

[0132] Optionally, words in the above-mentioned multiple first word sequences whose confidence is not greater than the third preset threshold can also be eliminated, that is, multiple fifth word sequences and decoding information of each word in the multiple fifth word sequences can be obtained from the first decoding results.

[0133] 304. Obtain words whose confidence is not greater than a first preset threshold value from the above-mentioned multiple fifth word sequences, obtain multiple sixth word sequences and decoding information of each word in the multiple sixth word sequences, wherein the decoding information includes the start time and end time, acoustic probability and language probability of the word.

[0134] Optionally, the words and their decoding information in the above-mentioned multiple fifth word sequences whose confidence is greater than the first preset threshold can also be eliminated, that is, the multiple sixth word sequences and the decoding information of each word in the multiple fifth word sequences can be obtained from the multiple fifth word sequences and the decoding information of each word in the multiple fifth word sequences.

[0135] 306 , determining a sixth word sequence with the highest comprehensive score among the plurality of sixth word sequences based on the start time and end time, acoustic probability, and language probability of each word in the plurality of sixth word sequences.

[0136] 308 , based on the sixth word sequence with the highest comprehensive score, a fourth sentence is obtained.

[0137] 310, identifying whether the fourth sentence includes the word.

[0138] If the fourth sentence does not include the word, operation 312 is performed. If the fourth sentence includes the word, the subsequent process of this embodiment is not performed and the fourth sentence can be directly discarded.

[0139] 312 , the speech to be recognized is added as a background human voice sample to a background human voice set, so as to be used as background human voice noise for training an acoustic model.

[0140] The background vocal set includes at least one background vocal sample.

[0141] Or, in the above Figure 4 In the illustrated embodiment, 304 may be executed first to eliminate words whose confidence is greater than a first preset threshold value from the multiple first word sequences, and then 302 may be executed to eliminate words whose confidence is not greater than a third preset threshold value, to obtain multiple sixth word sequences and decoding information of each word in the multiple sixth word sequences, and then operations 306-312 may be executed.

[0142] Alternatively, in other implementations, noise samples can also be obtained in the following manner: after removing words whose confidence is not greater than a third preset threshold and words whose confidence is greater than the first preset threshold from multiple first word sequences, it is identified whether the multiple first word sequences obtained thereby still include words; if not, the speech to be recognized is added to the background voice set as a background voice sample, and used as background voice noise for training the acoustic model.

[0143] Alternatively, the embodiment of the present disclosure can also confirm that the speech to be recognized is noise containing background human voices based on the third preset threshold and the first preset threshold in any other way, and add the speech to be recognized as background human voice noise to the background human voice set. The embodiment of the present disclosure does not limit the specific method of confirming that the speech to be recognized is background human voice noise.

[0144] Based on this embodiment, the collection of background voice samples is realized through the online speech recognition process, and real and effective online audio background voice data is collected. By obtaining online background voice samples, the collection of background voice samples is realized, which solves the problems of difficulty in obtaining background voice sets and high cost of obtaining labeled data. When used as background voice noise for training acoustic models, the performance of the acoustic model can be improved, thereby helping to improve the accuracy of the final speech recognition results.

[0145] Figure 5 FIG. 1 is a flow chart of another embodiment of the speech recognition method disclosed herein. Figure 5 As shown, in Figures 1 to 4 Based on the embodiment shown, this embodiment may further include, after step 104:

[0146] 402. In response to the confidence of each word in the plurality of first word sequences being greater than a fourth preset threshold, a second sentence is obtained based on the plurality of first word sequences and the confidence of each word in the plurality of first word sequences.

[0147] The fourth preset threshold is not less than the first preset threshold. For example, the fourth preset threshold may be set to 0.9, 0.95, or the like, and may be determined based on the performance of the acoustic model and language model to be trained. The higher the fourth preset threshold, the better the performance improvement effect on the acoustic model and language model when used to train the acoustic model and language model.

[0148] For example, in some implementations, the first word sequence with the highest comprehensive score in the above-mentioned multiple first word sequences can be determined based on the start time and end time of each word in the above-mentioned multiple first word sequences, as well as the acoustic probability and language probability. The comprehensive score is determined based on the acoustic probability and language probability. Specifically, the score of each word can be determined based on the acoustic probability and language probability of the word. From the above-mentioned multiple first word sequences, the word with the highest score is selected for the same start time and end time, and the first word sequence with the highest comprehensive score is constructed based on the words with the highest scores corresponding to each start time and end time, that is, the first word sequence corresponding to the optimal path in the word graph; then, based on the first word sequence with the highest comprehensive score, the second sentence is obtained.

[0149] If the confidences of all words in the second sentence are not greater than the fourth preset threshold, operation 406 is performed. Subsequent operations of this embodiment are not performed, and the second sentence can be selectively discarded.

[0150] 404 , using the second sentence as the annotation information of the speech to be recognized, and adding the speech to be recognized and the annotation information of the speech to be recognized as a speech sample to a speech set for training an acoustic model and / or a language model.

[0151] The speech set includes at least one speech sample.

[0152] In the embodiments of the present disclosure, when training an acoustic model and / or a language model based on speech samples in a sample speech set, the acoustic model and the language model can be trained separately, or the acoustic model and the language model can be trained end-to-end in a unified manner.

[0153] Based on the embodiments of the present disclosure, high-quality speech samples can be collected during the online speech recognition process to enrich the speech samples in the speech set, and a large amount of high-quality speech and corresponding annotation information can be obtained at a low cost. When used to train the acoustic model and / or language model, the recognition performance of the acoustic model and / or language model can be effectively improved, thereby solving the problem that the speech set sample data set is too small, the cost of obtaining annotation data is too high, and the recognition performance of the acoustic model and / or language model is difficult to significantly improve as a result.

[0154] Optionally, in some of these implementations, Figure 5 In the illustrated embodiment, attribute information of the second sentence can also be determined. This attribute information may include, but is not limited to, any one or more of the following: fields, application scenarios, and geographic regions. Fields include, for example, music, poetry, and technology; application scenarios include, for example, voice dialing, voice navigation, voice song requesting, and voice wake-up; and the geographic region refers to the geographical region to which the speech to be recognized belongs, such as Guangzhou, Henan, and Northeast China. Then, in step 404, the speech to be recognized and its annotated information can be used as a speech sample and added to the speech set corresponding to the attribute information to train the acoustic model and language model corresponding to the attribute information. For example, in some optional examples, the domain of the second sentence can be determined based on the inclusion of words from various preset domains in the second sentence. For example, if the second sentence includes the words "song," "songs," song titles, or singer names from the music domain, the domain of the second sentence is determined to be the music domain; if the second sentence includes the words "poetry," "poetic lyrics," poem titles, or poem authors from the poetry domain, the domain of the second sentence is determined to be the poetry domain.

[0155] Alternatively, in other optional examples, a pre-trained first classification model can be used to classify the domain to which the second sentence belongs, and the domain to which the second sentence belongs can be determined based on the classification result. The first classification model can be implemented using a neural network. The neural network can be trained using corpora and domain annotation information from different domains to obtain the first classification model.

[0156] For example, in some optional examples, a specific application (APP) may call a voice acquisition module to collect audio signals to obtain the aforementioned speech to be recognized, and provide the speech to be recognized to the speech recognition system for speech recognition. When the APP provides the speech to be recognized to the speech recognition system, it may carry the identification information of the APP. The identification information of the APP may be, for example, the name of the APP, the application field, etc. Based on the correspondence between the identification information of the APP and the application scenario, the application scenario corresponding to the identification information of the APP can be determined as the application scenario of the second sentence.

[0157] Alternatively, in some optional examples, a pre-trained second classification model can be used to classify the application scenario of the second sentence, and the application scenario of the second sentence can be determined based on the classification results. The second classification model can be implemented using a neural network. The neural network can be trained using corpus of different application scenarios and application scenario annotation information to obtain the second classification model.

[0158] For example, in some optional examples, a specific application (APP) can call a voice acquisition module to collect audio signals to obtain the above-mentioned voice to be recognized, and provide the voice to be recognized to the voice recognition system for voice recognition. The user can set the geographical area corresponding to his or her voice when registering or using the APP. When the user sends the voice to be recognized to the APP, the APP can obtain and carry the geographical area set by the user when providing the voice to be recognized to the voice recognition system, and can directly use the geographical area carried by the voice to be recognized as the geographical area of the second sentence. If the user has not set the geographical area corresponding to his or her voice, the APP can obtain the current geographical area as the geographical area of the second sentence through the positioning module (such as GPS) on the terminal where the user is located.

[0159] Based on the embodiments of the present disclosure, during the online speech recognition process, high-quality speech samples can be collected based on fields, application scenarios, and geographical areas to enrich the speech samples in the speech set. A large amount of speech and corresponding annotation information can be obtained at low cost for different fields, application scenarios, and geographical area dimensions. When used to train acoustic models and / or language models, the recognition performance of the trained acoustic models and / or language models can be effectively improved, thereby solving the problem that there are too few speech set sample data sets in some fields, application scenarios, and geographical areas, and the resulting problem that the recognition performance of the corresponding acoustic models and language models is difficult to significantly improve. In the case of scarce speech set resources, the recognition of the acoustic model and / or language model can be improved. The inventors of the present disclosure found through experimental research that, under the same other conditions, when the fourth preset threshold is set to 95%, the accuracy of the speech recognition results of the trained acoustic model and / or language model can reach more than 98%.

[0160] Optionally, in some of the implementations, it may also include: constructing a training data set based on noise samples in the noise set, background vocal samples in the background vocal set, and speech samples in the speech set, the training data set including at least one noisy signal, the noisy signal including any one or more of the following: a noisy signal generated by mixing a speech sample and a noise sample serving as foreground noise, a noisy signal generated by mixing a speech sample and a background vocal sample serving as background vocal noise, a noisy signal generated by mixing a speech sample, a noise sample serving as foreground noise, and a background vocal sample serving as background vocal noise, wherein the noisy signal is annotated with annotation information of the speech sample used to generate the noisy signal; and then, using the training data set, training an acoustic model and / or language model for decoding the speech to be recognized, i.e., adjusting the network parameters of the acoustic model and / or language model.

[0161] In which, when using a noisy signal to train an acoustic model, the first difference between the phonemes output by the acoustic model and the phonemes corresponding to the annotation information of the noisy signal can be compared, and the acoustic model can be iteratively trained based on the first difference until a first preset training completion condition is met, for example, the first difference is less than a preset first difference threshold, and / or the number of iterative training of the acoustic model reaches a first preset number, and so on.

[0162] When using a noisy signal to train a language model, a second difference between the word sequence corresponding to the word output by the language model and the annotation information of the noisy signal can be compared, and the language model can be iteratively trained based on the second difference until a second preset training completion condition is met, for example, the second difference is less than a preset second difference threshold, and / or the number of iterative training of the language model reaches a second preset number, and so on.

[0163] When using a noisy signal to simultaneously train an acoustic model and a language model, the first difference between the phonemes output by the acoustic model and the phonemes corresponding to the annotation information of the noisy signal can be compared, and the second difference between the word sequence corresponding to the word output by the language model and the annotation information of the noisy signal can be compared. The language model is iteratively trained based on the first difference and the second difference until a third preset training completion condition is met. The third preset training completion condition may, for example, include but is not limited to any one or more of the following: the first difference is less than a preset first difference threshold, the second difference is less than a preset second difference threshold, the mean between the first difference and the second difference is less than a preset third difference threshold, the number of iterative training of the acoustic model and the language model reaches a third preset number of times, and so on.

[0164] Speech recognition systems generally consist of two phases: training and recognition. Training refers to the training of the acoustic model and / or language model, and is typically performed offline. Recognition, which converts user speech into text, is typically performed online. Training the acoustic and language models requires a large amount of labeled audio data as training samples. Acquiring this labeled audio data is primarily a manual process, which is time-consuming, labor-intensive, and costly.

[0165] Based on the embodiments of the present disclosure, during the online speech recognition process, noise samples can be collected to expand the noise samples in the noise set, background voice samples can be collected to expand the background voice samples in the background voice set, and speech samples can be collected to expand the speech samples in the speech set. Then, the noise samples in the noise set are used as foreground noise, and the background voice samples in the background voice set are used as background voice noise. Based on the foreground noise and / or background voice noise, a training data set is constructed with the speech samples in the speech set, so that a real, rich and reliable training data set can be expanded for training acoustic models and / or language models, which can effectively improve the recognition performance of the trained acoustic models and / or language models in various environments and improve the application effect based on speech recognition.

[0166] When the speech set is divided based on attribute information, it can be used to train the acoustic model and / or language model corresponding to each attribute information, thereby improving the recognition performance of the acoustic model and / or language model corresponding to each attribute information in various environments, and improving the performance of the trained acoustic model and / or language model.

[0167] Any of the speech recognition methods provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server. Alternatively, any of the speech recognition methods provided in the embodiments of the present disclosure can be executed by a processor, such as a processor that executes any of the speech recognition methods mentioned in the embodiments of the present disclosure by invoking corresponding instructions stored in a memory. This will not be further described below.

[0168] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0169] Figure 6 FIG. 1 is a structural diagram of an embodiment of a speech recognition device disclosed herein. The speech recognition device of this embodiment can be used to implement the above-mentioned speech recognition method embodiments disclosed herein. Figure 6As shown, the speech recognition device of this embodiment includes: a speech recognition model 502, a confidence calculation module 504, an acquisition module 506 and a result determination module 508. Among them:

[0170] Speech recognition model 502 is configured to decode the speech to be recognized and obtain a first decoding result. The first decoding result includes: multiple first word sequences, as well as the start and end times, acoustic probabilities, and language probabilities of each word in the multiple first word sequences. Speech recognition model 502 may include an acoustic model and a language model.

[0171] The confidence calculation module 504 is configured to calculate the confidence of each word in the plurality of first word sequences based on the acoustic probability and the language probability of each word in the plurality of first word sequences.

[0172] The acquisition module 506 is used to obtain a second decoding result based on the words in the multiple first word sequences whose confidence is greater than the first preset threshold. The second decoding result includes: multiple second word sequences, and the start time and end time, acoustic probability and language probability of each word in the multiple second word sequences.

[0173] The result determination module 508 is used to obtain a speech recognition result based on the multiple second word sequences and the start time and end time, acoustic probability and language probability of each word in the multiple second word sequences. The language recognition result includes: a first sentence and the confidence of the first sentence, so as to perform corresponding actions based on the speech recognition result.

[0174] Based on the speech recognition device provided by the above-mentioned embodiment of the present disclosure, after decoding the speech to be recognized and obtaining a first decoding result, the confidence of each word is calculated based on the acoustic probability and language probability of each word in the multiple first word sequences, and then the first decoding result is filtered based on a first preset threshold to obtain a second decoding result. Furthermore, based on the multiple second word sequences in the second decoding result and the start time and end time of each word therein, as well as the acoustic probability and language probability, a speech recognition result is obtained, so as to perform corresponding actions based on the speech recognition result. Therefore, after decoding the speech to be recognized and obtaining the decoding result, the embodiment of the present disclosure can eliminate the words with low confidence in the decoding result and determine the speech recognition result only based on the words with high confidence, thereby improving the accuracy of the speech recognition result. Therefore, when performing corresponding actions based on the speech recognition result, it avoids erroneous actions caused by the presence of inaccurately recognized words in the obtained sentence, which helps to improve the application effect based on speech recognition.

[0175] Optionally, in some implementations, the confidence calculation module 504 may include: a first calculation unit, used to take each word in each first word sequence in the multiple first word sequences as the current word, and calculate the forward probability and backward probability of the current word based on the acoustic probability and language probability of the current word; a second calculation unit, used to calculate the posterior probability of the current word in the first word sequence in which it is located based on the forward probability, backward probability and language probability of the current word; and a superposition unit, used to superimpose the posterior probabilities of the current word in the multiple first word sequences to obtain the confidence of the current word.

[0176] Optionally, in some implementations, the result determination module 508 may include: a first determination unit, used to determine the second word sequence with the highest comprehensive score among the multiple second word sequences based on the start time and end time, acoustic probability and language probability of each word in the multiple second word sequences; a second determination unit, used to obtain the first sentence based on the second word sequence with the highest comprehensive score; and a third determination unit, used to obtain the confidence of the first sentence based on the confidence of each word in the first sentence.

[0177] Figure 7 FIG. 1 is a structural diagram of another embodiment of the speech recognition device disclosed in the present invention. Figure 7 As shown, based on the above embodiment, the speech recognition device of this embodiment may further include: an endpoint detection model 510, which is used to perform speech endpoint detection on the audio signal collected by the audio collection module to obtain the starting point and end point of at least one speech activity segment; and based on the starting point and end point of the at least one speech activity segment, cut out the at least one speech activity segment from the audio signal, so that each speech activity segment in the at least one speech activity segment is used as the speech to be recognized, and the speech recognition model 502 decodes the speech to be recognized.

[0178] Alternatively, see Figure 7 On the basis of the above embodiment, the speech recognition device of this embodiment may further include: a first collection module 512, for adding the speech to be recognized as a noise sample to a noise set in response to the confidence of each word in the multiple first word sequences being greater than the second preset threshold and less than the third preset threshold, for training the endpoint detection model, and / or as foreground noise for training the acoustic model; wherein the noise set includes at least one noise sample, and the second preset threshold is less than the third preset threshold.

[0179] Alternatively, see Figure 7 On the basis of the above embodiment, the speech recognition device of this embodiment may further include: a first training module 514, configured to train the above endpoint detection model 510 using noise samples in the noise set.

[0180] Alternatively, see Figure 7 Based on the above embodiment, the speech recognition device of this embodiment may further include: a second collection module 516, which is used to add the speech to be recognized as a background human voice sample to the background human voice set in response to the confidence of each word in the multiple first word sequences being greater than the third preset threshold and less than the first preset threshold, so as to be used as background human voice noise for training the acoustic model; wherein the background human voice set includes at least one background human voice sample, and the third preset threshold is less than the first preset threshold.

[0181] Alternatively, see Figure 7 Based on the above embodiment, the speech recognition device of this embodiment may further include: a sentence determination module 518 and a second collection module 520. The sentence determination module 518 is configured to, in response to the confidence of each word in the plurality of first word sequences being greater than a fourth preset threshold, obtain a second sentence based on the plurality of first word sequences and the confidence of each word in the plurality of first word sequences, where the fourth preset threshold is not less than the first preset threshold.

[0182] The third collection module 520 is used to use the second sentence as the annotation information of the speech to be recognized, and add the speech to be recognized and the annotation information of the speech to be recognized as a speech sample to a speech set for training the acoustic model and the language model, wherein the speech set includes at least one speech sample.

[0183] Optionally, in some implementations, the third collection module 522 is specifically used to: determine attribute information of the second sentence, the attribute information including any one or more of the following: field, application scenario, geographical area; take the speech to be recognized and the annotation information of the speech to be recognized as a speech sample, and add it to the speech set corresponding to the attribute information for training the acoustic model and language model corresponding to the attribute information.

[0184] Alternatively, see Figure 7 Based on the above embodiment, the speech recognition device of this embodiment may further include: a construction module 524 and a second training module 526.

[0185] Among them: construction module 524 is used to construct a training data set based on noise samples in the noise set, background vocal samples in the background vocal set and speech samples in the speech set, the training data set includes at least one noisy signal, and the noisy signal includes any one or more of the following: a noisy signal generated by mixing a speech sample and a noise sample serving as foreground noise, a noisy signal generated by mixing a speech sample and the background vocal sample serving as background vocal noise, and a noisy signal generated by mixing a speech sample, a noise sample serving as foreground noise and a background vocal sample serving as background vocal noise, wherein the noisy signal is annotated with annotation information of the speech sample used to generate the noisy signal.

[0186] The second training module 526 uses the training data set to train the acoustic model and / or language model used to implement the decoding of the speech to be recognized.

[0187] In addition, an embodiment of the present disclosure further provides an electronic device, including:

[0188] memory for storing computer programs;

[0189] The processor is configured to execute the computer program stored in the memory, and when the computer program is executed, the speech recognition method described in any one of the above embodiments of the present disclosure is implemented.

[0190] Figure 8 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. Figure 8 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be either or both of the first device and the second device, or a standalone device independent of them, and the standalone device may communicate with the first device and the second device to receive collected input signals from them.

[0191] like Figure 8 As shown, the electronic device includes one or more processors and memory.

[0192] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0193] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the speech recognition method of each embodiment of the present disclosure described above and / or other desired functions.

[0194] In one example, the electronic device may further include an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0195] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0196] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0197] Of course, to simplify, Figure 8 Only some of the components related to the present disclosure in the electronic device are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0198] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to perform the steps of the speech recognition method according to various embodiments of the present disclosure described in the above part of this specification.

[0199] The computer program product may be written in any combination of one or more programming languages to implement the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0200] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the speech recognition method according to various embodiments of the present disclosure described in the above part of this specification.

[0201] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0202] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0203] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0204] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For system embodiments, since they largely correspond to method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.

[0205] The block diagrams of the devices, devices, equipment, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0206] The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0207] It should also be noted that in the apparatus, device, and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0208] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0209] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A speech recognition method, characterized in that: include: Decoding the speech to be recognized to obtain a first decoding result, wherein the first decoding result includes: a plurality of first word sequences and a start time and an end time, an acoustic probability, and a language probability of each word in the plurality of first word sequences; Calculating the confidence of each word in the plurality of first word sequences based on the acoustic probability and the language probability of each word in the plurality of first word sequences respectively; Obtaining a second decoding result based on the words in the plurality of first word sequences whose confidences are greater than a first preset threshold, the second decoding result including: a plurality of second word sequences and a start time and an end time, an acoustic probability, and a language probability of each word in the plurality of second word sequences; Obtaining a speech recognition result based on the plurality of second word sequences and the start time and end time, acoustic probability, and language probability of each word in the plurality of second word sequences, the language recognition result including: a first sentence and a confidence score of the first sentence, so as to perform a corresponding action based on the speech recognition result; The step of calculating the confidence of each word in the plurality of first word sequences based on the acoustic probability and the language probability of each word in the plurality of first word sequences comprises: Taking each word in each of the multiple first word sequences as a current word, and calculating a forward probability and a backward probability of the current word based on the acoustic probability and the language probability of the current word; calculating a posterior probability of the current word in the first word sequence in which it is located based on the forward probability, the backward probability, and the language probability of the current word; and superimposing the posterior probabilities of the current word in the multiple first word sequences to obtain a confidence score for the current word; The obtaining of speech recognition results based on the plurality of second word sequences and the start time and end time, acoustic probability, and language probability of each word in the plurality of second word sequences includes: Determining a second word sequence with a highest comprehensive score among the multiple second word sequences based on the start time and end time, acoustic probability, and language probability of each word in the multiple second word sequences, wherein the second word sequence with the highest comprehensive score is composed of the words with the highest scores corresponding to the same start time and end time among the multiple second word sequences; Obtaining a first sentence based on the second word sequence with the highest comprehensive score; The confidence of the first sentence is obtained based on the confidence of each word in the first sentence.

2. The method according to claim 1, characterized in that Before decoding the speech to be recognized, the method further includes: Using the endpoint detection model, perform voice endpoint detection on the audio signal collected by the audio acquisition module to obtain the starting point and end point of at least one voice activity segment; Based on the starting point and end point of the at least one voice activity segment, the at least one voice activity segment is cut out from the audio signal, so that each voice activity segment in the at least one voice activity segment is used as the voice to be recognized, and the operation of decoding the voice to be recognized is performed to obtain a first decoding result.

3. The method according to claim 2, characterized in that After calculating the confidence of each word in the plurality of first word sequences based on the acoustic probability and the language probability of each word in the plurality of first word sequences, the method further includes: In response to the confidence of each word in the multiple first word sequences being greater than a second preset threshold and less than a third preset threshold, the speech to be recognized is added to a noise set as a noise sample for training the endpoint detection model, and / or as foreground noise for training the acoustic model; wherein, the noise set includes at least one noise sample, and the second preset threshold is less than the third preset threshold.

4. The method according to claim 3, characterized in that Also includes: The endpoint detection model is trained using noise samples in the noise set.

5. The method according to claim 3, characterized in that After calculating the confidence of each word in the plurality of first word sequences based on the acoustic probability and the language probability of each word in the plurality of first word sequences, the method further includes: In response to the confidence of each word in the multiple first word sequences being greater than the third preset threshold and less than the first preset threshold, the speech to be recognized is added to the background voice set as a background voice sample to be used as background voice noise for training the acoustic model; wherein, the background voice set includes at least one background voice sample, and the third preset threshold is less than the first preset threshold.

6. The method according to claim 2, characterized in that After calculating the confidence of each word in the plurality of first word sequences based on the acoustic probability and the language probability of each word in the plurality of first word sequences, the method further includes: In response to the confidence of each word in the plurality of first word sequences being greater than a fourth preset threshold, obtaining a second sentence based on the plurality of first word sequences and the confidence of each word in the plurality of first word sequences; wherein the fourth preset threshold is not less than the first preset threshold; The second sentence is used as the annotation information of the speech to be recognized, and the speech to be recognized and the annotation information of the speech to be recognized are added to a speech set as a speech sample for training an acoustic model and a language model; wherein the speech set includes at least one speech sample.

7. The method according to claim 6, characterized in that The step of adding the to-be-recognized speech and the annotation information of the to-be-recognized speech as a speech sample to a speech set includes: Determining attribute information of the second sentence, the attribute information including any one or more of the following: field, application scenario, and geographical area; The speech to be recognized and the annotation information of the speech to be recognized are taken as a speech sample, and added to the speech set corresponding to the attribute information, so as to be used for training the acoustic model and language model corresponding to the attribute information.

8. The method according to any one of claims 2 to 7, characterized in that: Also includes: Constructing a training data set based on noise samples in a noise set, background vocal samples in a background vocal set, and speech samples in a speech set, wherein the training data set includes at least one noisy signal, and the noisy signal includes any one or more of the following: a noisy signal generated by mixing the speech sample and the noise sample as foreground noise, a noisy signal generated by mixing the speech sample and the background vocal sample as background vocal noise, and a noisy signal generated by mixing the speech sample, the noise sample as foreground noise, and the background vocal sample as background vocal noise, the noisy signal being annotated with annotation information of the speech sample used to generate the noisy signal; The training data set is used to train an acoustic model and / or a language model for decoding the speech to be recognized.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to execute a computer program stored in the memory, and when the computer program is executed, implement the method described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images