Speech recognition method and apparatus, electronic device, and storage medium

Abstract

The application provides a speech recognition method and device, electronic equipment and storage medium, and relates to the technical field of speech recognition. The method comprises the following steps: inputting collected to-be-recognized speech into an acoustic model to obtain a first text output by the acoustic model; when it is detected that the first text comprises an entity word marker, inputting an entity word part in the first text into an entity language model to obtain a first confidence of the entity word part output by the entity language model, and inputting the first text into a general language model to obtain a second confidence of the first text output by the general language model; the entity word part comprises the entity word marker and a target entity word; and the target entity word is stimulated based on the first confidence and the second confidence to obtain a speech recognition result of the to-be-recognized speech. The technical scheme provided by the application can avoid the false triggering of the entity word in the speech recognition process.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and more particularly to a speech recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] Automatic Speech Recognition (ASR) is a technology that converts speech signals into text sequences, and it is widely used in scenarios such as voice assistants, voice search, and voice input. ASR models rely heavily on the quality and quantity of training data. Due to significant differences between data from different domains, an ASR model trained on sample data from one domain may have weak recognition capabilities in a new domain. Therefore, it is crucial to quickly adapt a trained ASR model to new domains.

[0003] Entity words are proper nouns in speech that have specific meanings, such as place names, company names, song titles, singer names, actor names, etc. They are key factors in the differences between different fields, and optimizing entity words can improve the effectiveness of a field.

[0004] In related technologies, domain entity word optimization can be achieved through either synthetic training data or shallow fusion of language models. Synthetic training data optimization optimizes entity words by adjusting the proportion of entity words mixed into the training corpus through high-level resampling. This can affect the true distribution of general words, causing general-scene speech to be stimulated to generate entity words, resulting in false entity word triggering. Shallow fusion of language models stimulates all words in all corpora, causing corpora that do not contain entity words to be stimulated as well, leading to false entity word triggering in general-scene speech. Summary of the Invention

[0005] This invention provides a speech recognition method, apparatus, electronic device, and storage medium to solve the problem of false triggering of entity words in the speech recognition process in the prior art.

[0006] This invention provides a speech recognition method, comprising:

[0007] The speech to be recognized is collected and input into an acoustic model to obtain the first text output by the acoustic model;

[0008] If entity word markers are detected in the first text, the entity word portion of the first text is input into an entity language model to obtain a first confidence score of the entity word portion output by the entity language model, and the first text is input into a general language model to obtain a second confidence score of the first text output by the general language model; the entity word portion includes the entity word markers and the target entity word;

[0009] The target entity words are stimulated based on the first confidence level and the second confidence level to obtain the speech recognition result of the speech to be recognized.

[0010] According to a speech recognition method provided by the present invention, the step of stimulating the target entity word based on the first confidence level and the second confidence level to obtain the speech recognition result of the speech to be recognized includes:

[0011] Based on the first confidence level and the second confidence level, the excitation coefficient is determined;

[0012] The target entity word is excited based on the excitation coefficient to obtain the speech recognition result of the speech to be recognized.

[0013] According to a speech recognition method provided by the present invention, the entity language model is trained in the following manner:

[0014] Obtain entity word samples, wherein the entity word samples include entity words and entity word start markers and entity word end markers that enclose the entity words;

[0015] Based on the entity word samples, the initial entity language model is trained to obtain the entity language model.

[0016] According to a speech recognition method provided by the present invention, the target entity word and the entity words in the entity word sample belong to the same domain.

[0017] According to a speech recognition method provided by the present invention, the acoustic model is trained in the following manner:

[0018] Obtain a first sample pair and a second sample pair. The first sample pair includes a first speech sample and an entity text sample corresponding to the first speech sample. The entity text sample includes entity words, non-entity words, and entity word start markers and entity word end markers that enclose the entity words. The second sample pair includes a second speech sample and a general text sample corresponding to the second speech sample.

[0019] Based on the first sample pair and the second sample pair, the initial acoustic model is trained to obtain the acoustic model.

[0020] According to a speech recognition method provided by the present invention, obtaining the first sample pair includes:

[0021] Obtain a sentence template, wherein the sentence template includes at least one slot;

[0022] By filling each slot in the sentence template with entity words, a text sample is obtained.

[0023] The text sample is converted from text to speech to obtain the first speech sample;

[0024] Add entity word start markers and entity word end markers to the text sample to wrap and fill the entity words, and you get the entity text sample;

[0025] The first speech sample and the entity text sample are identified as the first sample pair.

[0026] According to a speech recognition method provided by the present invention, the general language model is trained in the following manner:

[0027] Obtain the parallel text of the general text sample;

[0028] Based on the parallel texts, the initial general language model is trained to obtain the general language model.

[0029] The present invention also provides a voice recognition device, comprising:

[0030] The first speech processing module is used to collect the speech to be recognized and input the speech to be recognized into the acoustic model to obtain the first text output by the acoustic model;

[0031] The second speech processing module is configured to, upon detecting that the first text includes entity word markers, input the entity word portion of the first text into an entity language model to obtain a first confidence score of the entity word portion output by the entity language model, and input the first text into a general language model to obtain a second confidence score of the first text output by the general language model; the entity word portion includes the entity word markers and target entity words;

[0032] The activation module is used to activate the target entity word based on the first confidence level and the second confidence level to obtain the speech recognition result of the speech to be recognized.

[0033] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the speech recognition method as described above.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method as described above.

[0035] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the speech recognition method as described above.

[0036] The speech recognition method, apparatus, electronic device, and storage medium provided by this invention utilize an acoustic model to perform speech recognition on the speech to be recognized, obtaining a first text. When entity word markers are detected in the first text, the entity word portion of the first text is input into an entity language model to obtain a first confidence level of the entity word portion output by the entity language model. The first text is then input into a general language model to obtain a second confidence level of the first text output by the general language model. The entity word portion includes entity word markers and target entity words. In this way, entity words in the speech to be recognized can be marked by the acoustic model, and the entity language model can only recognize the entity word portion marked by the entity word markers. Then, based on the first and second confidence levels, the target entity word is stimulated. This allows the language model to shallowly fuse only stimulate the target entity word in the entity word portion without affecting the general effect of non-entity word portions, avoiding false triggering of entity words during speech recognition and improving the accuracy of the speech recognition results. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the speech recognition method provided in an embodiment of the present invention;

[0039] Figure 2 This is a flowchart of the training method for the entity language model in an embodiment of the present invention;

[0040] Figure 3 This is a flowchart of the training method for the acoustic model in an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of the structure of the speech recognition device provided in an embodiment of the present invention;

[0042] Figure 5 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without any creative efforts shall fall within the scope of protection of the present invention.

[0044] It should be noted that the serial numbers assigned to the objects described in the present invention itself, such as "first", "second", etc., are only used to distinguish the objects described and do not have any sequential or technical meaning.

[0045] The ASR model is an acoustic model. The accuracy of its speech recognition depends on the quality and quantity of the training data, and there are significant differences between the data in different fields, such as the news field, the music field, the medical field, the audiobook field, etc., where the sentence patterns and common words are different.

[0046] Entity words refer to the proper nouns contained in a piece of speech and have specific meanings, such as place names, company names, song names, singer names, actor names, and medical test names, etc., which are the key factors for the differences between different fields. For example, song names and singer names that appear in the music field may be relatively rare in the real estate field. If the ASR model trained with real estate corpus is applied to the speech recognition in the music field, it is very likely that words such as song names and singer names will not be accurately recognized.

[0047] Optimizing entity words can improve the recognition effect in a specific field, but it will cause the general effect to decline. For example, for the speech "帮我拿kuai di" in a general context, if it is in the real estate field, "kuai di" here should be recognized as "块地", but in the general context, "kuai di" in "帮我拿kuai di" should be recognized as "快递". During the optimization of entity words, it is easy to cause such pronunciations that exist in both entity corpus and general corpus to be mis-triggered as entity words in the general context.

[0048] One of the challenges of entity-based domain adaptation is that domain data is usually presented in text form. Therefore, entity training data can be constructed by synthesizing audio corpora, allowing the model to adapt to the entity domain at the data level. Specifically, given the sentence structure and slots of the target domain, domain text can be synthesized based on the sentence structure and slots. For example, given the sentence structure: "Play..."<artist_name> Singing<song_name> ",Will<artist_name> Change it to the singer's name "AA"<song_name> By replacing the song title with "abc", a synthesized corpus of "play AA singing abc" can be obtained. Then, text-to-speech technology can be used to convert this corpus into the corresponding audio. This method can yield a large amount of training data for the target domain. Adding this training data to the ASR model training can, to a certain extent, adapt it to the target domain.

[0049] To achieve good results using synthetic training data, the target domain training data needs to be highly resampled and then fused with general context data before training the ASR model. However, high resampling can affect the true distribution of the general context data, causing domain entity words to be activated even in the transcribed text of speech in the general context, leading to false triggering of entity words. Conversely, insufficient mixed corpus can result in under-activation of entity words in the entity corpus. Therefore, it is not easy to find a mixing ratio that balances the performance of both the entity corpus and the general corpus by adjusting the proportion of entity corpus mixed into the training corpus.

[0050] In addition, shallow fusion (SF) of language models (LM) can be used for domain entity optimization. This involves training a domain language model, such as the Chinese language model N-Ggram or the Weighted Finite State Transducer (WFST), using domain corpus. During the beam search process of the speech recognition model, the path score is incentivized with a certain incentive coefficient λ. The specific formula can be expressed as follows:

[0051]

[0052] Where x represents the speech to be recognized, y * Let y represent the speech recognition result, where y represents the text sequence output by the ASR model after performing speech recognition on the speech x to be recognized, and logp(y|x) represents the probability of the ASR model outputting y. LM (y) represents the probability predicted by the speech model for the text sequence y.

[0053] Shallow fusion of language models can stimulate domain-specific sentences using domain-specific speech models. However, according to formula (1), it will stimulate all tokens in all corpora, causing corpora that do not contain entity words to be stimulated as well, which in turn leads to false triggering of entity words in the general context. Here, a token is a basic unit in the text, such as a word, word segment, or phrase.

[0054] Based on this, the embodiments of the present invention use entity word tagging to enable the end-to-end acoustic model (ASR model) to output entity word tags for the synthesized corpus. Then, the entity language model and the general language model are used to stimulate the entity word part marked by the entity word tag based on the recognition result of the acoustic model. By tagging entity words, the shallow fusion of the language model only stimulates the entity words in the entity word part, without affecting the general effect of the non-entity word part.

[0055] The following is combined Figures 1-3 The speech recognition method of the present invention is described below. This speech recognition method can be applied to electronic devices such as terminal devices or servers. Terminal devices may include mobile phones, computers, in-vehicle devices, tablet computers, wearable devices, smart home devices, smart robots, etc.; servers may include standalone servers, cluster servers, or cloud servers, etc. This speech recognition method can also be applied to speech recognition devices installed in electronic devices such as terminal devices or servers, which can be implemented through software, hardware, or a combination of both. The following example illustrates this speech recognition method using an electronic device as the executing entity.

[0056] Figure 1 An exemplary flowchart of the speech recognition method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the speech recognition method may include the following steps 110 to 130.

[0057] Step 110: Collect the speech to be recognized and input it into the acoustic model to obtain the first text output by the acoustic model.

[0058] Electronic devices may include voice acquisition devices, such as microphones. The electronic devices can acquire the voice to be recognized through the voice acquisition device, and then input the voice to be recognized into an acoustic model. The acoustic model performs voice recognition on the voice to be recognized and obtains the first text output by the acoustic model.

[0059] The acoustic model is trained on an initial acoustic model based on a first sample pair and a second sample pair. The first sample pair includes a first speech sample and a corresponding entity text sample. The entity text sample includes entity words, non-entity words, and entity word start and end markers that enclose the entity words. In other words, the entity text sample can be a sentence containing both entity words and non-entity words, with the entity words in the sentence enclosed before and after entity word markers. The second sample pair includes a second speech sample and a corresponding general text sample.

[0060] Based on this, when the speech to be recognized is speech in the context of the target domain, such as speech in the context of the music domain, the acoustic model can convert the speech to be recognized into a first text containing entity word markers, and the part marked by the entity word markers in the first text is an entity word. The entity words in the first text can be located through the entity word markers.

[0061] For example, the acoustic model can be an end-to-end ASR model, such as a Recurrent Neural Network Transducer (RNNT) model or a Connectionist Temporal Classification (CTC) model based on neural networks.

[0062] Step 120: If entity word markers are detected in the first text, input the entity word part of the first text into the entity language model to obtain the first confidence of the entity word part output by the entity language model, and input the first text into the general language model to obtain the second confidence of the first text output by the general language model.

[0063] The entity word part includes entity word markers and target entity words.

[0064] Entity word markers can mark entity words in the first text. These markers can include an entity word start marker and an entity word end marker. The text content enclosed between the entity word start marker and the entity word end marker can be identified as the target entity word. Based on this, the position of the target entity word can be determined according to the entity word start marker and the entity word end marker. For example, the entity word start marker could be... <entity>The entity terminator can be <\entity>.

[0065] When an electronic device detects that the first text output by the acoustic model includes entity word markers, it can determine the entity word portion based on these markers. The entity word portion is a continuous sequence of at least one character or word. Then, the entity word portion is input into an entity language model, which predicts the probability of the corresponding sequence occurring, thus obtaining a first confidence level. Simultaneously, the first text is input into a general language model, which predicts the probability of the first text sequence occurring, thus obtaining a second confidence level.

[0066] The entity language model can be obtained by training the initial entity language model based on entity word samples. The entity word samples include entity words and entity word markers that mark the entity words. For example, the entity word can be wrapped by entity word start markers and entity word end markers.

[0067] A general language model can be obtained by training an initial general language model based on general text samples. These general text samples can be general text samples in the second sample pair when training the acoustic model, or they can be parallel transcripts of general text samples in the second sample pair, i.e., parallel texts of general text samples in the second sample pair.

[0068] For example, entity language models and general language models can employ statistical language models or neural network language models (NNLMs). Statistical language models can include N-Ggram models, such as 4-gram language models. Neural network language models can include, but are not limited to, Long Short-Term Memory (LSTM) models, feedforward neural network language models (NNLMs), or recurrent neural network based language models (RNNLMs).

[0069] Step 130: Activate the target entity words based on the first confidence level and the second confidence level to obtain the speech recognition result of the speech to be recognized.

[0070] After obtaining the first confidence score of the entity word portion output by the entity language model and the second confidence score of the first text output by the general language model, the electronic device can determine the activation coefficient based on the first and second confidence scores. This activation coefficient is then used to activate the target entity words in the first text, resulting in the speech recognition result of the speech to be recognized. In this way, during the shallow fusion of language models, activation is only applied to the entity word portion output by the acoustic model, without affecting the general performance of non-entity word portions, thus avoiding false triggering of entity words.

[0071] The speech recognition method provided in this invention utilizes an acoustic model to perform speech recognition on the speech to be recognized, obtaining a first text. When entity word markers are detected in the first text, the entity word portion of the first text is input into an entity language model to obtain a first confidence score of the entity word portion output by the entity language model. The first text is then input into a general language model to obtain a second confidence score of the first text output by the general language model. The entity word portion includes entity word markers and target entity words. In this way, entity words in the speech to be recognized can be marked by the acoustic model, and the entity language model can only recognize the entity word portion marked by the entity word markers. Then, based on the first and second confidence scores, the target entity word is stimulated. This allows the language model to shallowly fuse only stimulate the target entity word in the entity word portion without affecting the general effect of non-entity word portions, avoiding false triggering of entity words during speech recognition and improving the accuracy of the speech recognition results. The speech recognition speed is fast and the recognition rate is high.

[0072] based on Figure 1 In one example embodiment, the speech recognition method of the corresponding embodiment, which excites the target entity word based on a first confidence level and a second confidence level to obtain the speech recognition result of the speech to be recognized, may include: determining an excitation coefficient based on the first confidence level and the second confidence level; and exciting the target entity word based on the excitation coefficient to obtain the speech recognition result of the speech to be recognized.

[0073] For example, the difference or ratio between the first confidence level and the second confidence level can be determined as the excitation coefficient.

[0074] For example, when performing speech recognition on the speech to be recognized, the acoustic model can convert the speech into a text sequence, where the first text is the sequence of text output by the acoustic model. As the acoustic model continuously outputs the text sequence, the electronic device can detect this sequence. If no entity word marker is detected, the acoustic model's recognition result is used as the recognition result for the currently input speech. If an entity word marker is detected, starting from the detection of the entity word marker, the sequence output by the acoustic model is input into the entity language model, and all sequences output by the acoustic model up to the current time point are input into the general language model. The current output sequence of the acoustic model is then stimulated based on the first confidence level output by the entity language model and the second confidence level output by the general language model, until an entity word marker is detected. The acoustic model's recognition result is then used as the recognition result for the currently input speech.

[0075] For example, the speech recognition process can be formalized as the following formulas (2) to (4):

[0076]

[0077]

[0078]

[0079] Where, q ne (y|x) represents the speech recognition result of the speech to be recognized; t b This indicates the start time of the entity word marker, that is, the time when the entity word start marker is detected; t e This indicates the moment when the entity word ending marker ends, i.e., the moment when the entity word ending marker is detected; y <t This represents all outputs of the acoustic model up to the current time t; This represents all outputs of the acoustic model up to the entity word start marker time; p e2e (y t |x,y <t ) and p e2e (y t |y <t (x) represents the output of the acoustic model at the current time t; This represents the output of the acoustic model when entity word start markers are detected. This represents the probability value output by the entity language model. Indicates from t b The output of the acoustic model at time t-1; p ID (y t |y ＜t ) represents the probability value output by the general language model.

[0080] Based on this, it can be understood that the sentences output by the acoustic model include two categories: those containing entity words and those not containing entity words. For the case containing entity words, after the speech to be recognized is input into the acoustic model, during the decoding process, if the acoustic model does not decode the entity word start marker, it will use the principle represented by formula (2) to decode and output; after the acoustic model decodes the entity word start marker, for the part enclosed by the entity word start marker and the entity word end marker, the electronic device will use the principle represented by formula (3) to call the entity language model and the general language model to decode and output; after the acoustic model decodes the entity word end marker, the electronic device will use the principle represented by formula (4) to decode and output until the acoustic model decodes the entity word start marker again.

[0081] For example, in the music field, song titles and artist names are entity words. Suppose the collected speech to be recognized is "Play AB sings abc", where "AB" is the artist name and "abc" is the song title. Inputting this speech into an acoustic model for speech recognition yields the sentence "Play..." containing the entity words. <entity>AB<\entity> sang <entity>"abc<\entity>", where the entity words "AB" and "abc" are both marked with entity word start markers. <entity>Enclosed in the entity word ending marker `<\entity>`. During decoding, for the acoustic model, each word of the sentence containing entity words can be decoded sequentially. When the acoustic model decodes " <entity>"At that time, the electronic device can call the entity language model and the general language model. For the "A" decoded by the acoustic model, according to the principle of the above formula (3), it will input "A" into the entity language model. <entity>"Obtain the first confidence level, and simultaneously input 'play' into the general language model." <entity>"Obtain the second confidence level. Based on the first confidence level, the second confidence level, and the confidence level of "A" decoded by the acoustic model, the probability of decoding "A" is determined together. For "B" decoded by the acoustic model, according to the principle of the above formula (3), it will be input into the entity language model." <entity>A” obtains the first confidence score, and at the same time inputs “play” into the general language model. <entity>The second confidence level of "A" is obtained. Based on the first confidence level, the second confidence level, and the confidence level of "B" decoded by the acoustic model, the probability of decoding "B" is determined together. This process continues until the acoustic model decodes "entity". For the subsequent "a", the confidence level of "a" decoded by the acoustic model can be determined as the probability of decoding "a" according to the principle of the above formula (4). In this way, the language model can be used to stimulate only the entity word part without affecting the non-entity word part, thus avoiding the false triggering of entity words in the general context.

[0082] For example, suppose the collected speech to be recognized is "happy is the most important thing" in a general context, which does not contain entity words. If the speech to be recognized is input into an acoustic model for speech recognition, the acoustic model can decode each character of the sentence "happy is the most important thing" in turn, which does not contain entity words. The decoding result of the acoustic model can be used as the language recognition result of the speech to be recognized.

[0083] In one example embodiment of the speech recognition method based on the above embodiments, Figure 2 An exemplary flowchart of the training method for the entity language model in an embodiment of the present invention is shown, with reference to... Figure 2 As shown, the entity language model can be trained based on the following steps 210 to 220.

[0084] Step 210: Obtain entity word samples. Entity word samples include entity words and entity word start markers and entity word end markers that wrap the entity words.

[0085] Assume the entity word start marker is <entity>If the entity word ending marker is <\entity>, then the entity word samples used to train the entity language model are... <entity>And entities enclosed in `<entity>`. For example, in the music field, singer names, song titles, and song genres are proper nouns and can be considered entity words in the music field. Taking singer names and song titles as examples, we can construct the following two types of entity word samples: <entity>Singer's name and "entity" <entity>The song title is "<entity>". You can replace the artist name and the song title with different names to get entity word samples.

[0086] Step 220: Based on entity word samples, train the initial entity language model to obtain the entity language model.

[0087] After obtaining entity word samples, the initial entity language model can be trained using these samples to obtain the final entity language model. This initial entity language model can be a statistical language model or a neural network language model.

[0088] Understandably, entity word samples determine the domains that entity language models can adapt to. For example, entity word samples can be entity word samples from at least one different domain. In this way, entity language models can adapt to entity words from multiple different domains to improve the accuracy of entity word recognition in these domains.

[0089] Based on this, when performing shallow fusion of language models using the entity language model and the general language model, the target entity words stimulated by the first confidence score output by the entity language model and the second confidence score output by the general language model belong to the same domain as the entity words in the entity word samples. That is, when performing shallow fusion of language models using the entity language model and the general language model, positive stimulation can be applied to entity words in the first text output by the acoustic model that belong to the same domain as the entity words in the entity word samples, thereby improving the accuracy of domain-specific entity word recognition.

[0090] In one example embodiment of the speech recognition method based on the above embodiments, Figure 3 An exemplary flowchart of the acoustic model training method in an embodiment of the present invention is shown, with reference to... Figure 3 As shown, the acoustic model can be trained based on the following steps 310 to 320.

[0091] Step 310: Obtain the first sample pair and the second sample pair.

[0092] The first sample pair includes a first speech sample and a corresponding entity text sample. The entity text sample includes entity words, non-entity words, and entity word start markers and entity word end markers that enclose the entity words. The second sample pair includes a second speech sample and a corresponding general text sample, where the second speech sample is speech text in a general context.

[0093] The entity text sample can be a text sentence containing entity words, where each entity word is enclosed in an entity word start marker and an entity word end marker. For example, in the music field, entity words could include singer names and song titles, and the sentence structure of the entity text sample could be: play. <entity>AA<\entity> sang <entity>abc<entity>, where AA represents the singer's name and abc represents the song title. A large number of texts with this sentence structure can be constructed as entity text samples.

[0094] Specifically, based on the domain-specific sentence structure, slot filling can be performed using entity word start markers. <entity>The entity words enclosed by the entity word terminator <\entity> are filled into the slots of the sentence to obtain entity text samples.

[0095] The first speech sample can be generated using text-to-speech (TTS). Specifically, based on domain-specific sentence structures, entity words are filled into the slots of the sentence structure using entity word slot filling, resulting in text corpus without entity word markers. Then, TTS is used to synthesize audio from this text corpus to obtain the first speech sample. In essence, removing entity word markers from the entity text sample yields the text corpus without entity word markers.

[0096] Specifically, obtaining the first sample pair may include: obtaining a sentence template, which includes at least one slot; filling each slot of the sentence template with entity words to obtain a text sample; converting the text sample to speech to obtain a first speech sample; adding entity word start markers and entity word end markers to the text sample to enclose the filled entity words to obtain an entity text sample; and determining the first speech sample and the entity text sample as the first sample pair.

[0097] For example, in the music field, entity words include song titles and artist names. Let's assume the field's sentence structure is: Play.<artist_name> Singing<song_name> ,in,<artist_name> It's a slot for the singer's name.<song_name> For the song title slot, inserting the singer's name into the singer's name slot and the song title into the song title slot yields a text sample without entity word markers. For example, if the inserted song title is "abc" and the inserted singer's name is "AA", the resulting text sample without entity word markers would be "Play AA singing abc". This text sample can then be converted to speech to obtain the corresponding first speech sample. Alternatively, both the singer's name and song title can be marked with entity word start markers. <entity>Wrapping the entity terminator `<\entity>` in the corresponding slot yields an entity text sample with the entity terminator, which is "play". <entity>AA<\entity> sang <entity>"abc<\entity>". This yields the first sample pair.

[0098] Step 320: Based on the first sample pair and the second sample pair, train the initial acoustic model to obtain the acoustic model.

[0099] After obtaining the first and second sample pairs, the first and second sample pairs are mixed, and the initial acoustic model is trained using these sample pairs to obtain the acoustic model. The initial acoustic model can be an end-to-end ASR model, such as an RNNT model or a CTC model, but is not limited to these.

[0100] The dictionary used to train the acoustic model needs to contain tokens for entity word start and end markers, such as " <entity>The two tokens are " and "<entity>".

[0101] Based on the methods of the above embodiments, in one example embodiment, the general language model can be trained using parallel transcript corpora used to train the acoustic model. Specifically, the general language model can be trained as follows: obtaining parallel texts of general text samples; and training the initial general language model based on the parallel texts to obtain the general language model.

[0102] The initial general language model can be a statistical language model or a neural network language model.

[0103] The speech recognition method provided in this invention employs a method of training an acoustic model by mixing low-multiple resampled synthetic entity training data (first sample pair) into a general training corpus (second sample pair). Furthermore, entity words in the synthetic entity training corpus are wrapped with entity word markers, ensuring that the language model only positively excites the portion wrapped with entity word markers during excitation. This simultaneously avoids the entity word mis-triggering problem caused by synthetic training data and the entity word mis-triggering problem caused by shallow fusion of language models, thus preventing entity word mis-triggering during speech recognition.

[0104] The speech recognition method provided in this invention uses the method of marking entity words in a sentence, which can perform targeted optimization only on the entity word part of a sentence, avoiding the impact on other parts of the sentence and sentences that do not contain the entity to be optimized, and can avoid the impact on the general effect during the entity optimization process.

[0105] The speech recognition device provided by the present invention is described below. The speech recognition device described below and the speech recognition method described above can be referred to in correspondence.

[0106] Figure 4 An exemplary schematic diagram of the speech recognition device provided in an embodiment of the present invention is shown, with reference to... Figure 4 As shown, the voice recognition device may include:

[0107] The first speech processing module 410 is used to acquire the speech to be recognized and input the speech to be recognized into an acoustic model to obtain the first text output by the acoustic model; the second speech processing module 420 is used to input the entity word part of the first text into an entity language model when it is detected that the first text includes entity word markers to obtain the first confidence of the entity word part output by the entity language model, and input the first text into a general language model to obtain the second confidence of the first text output by the general language model, wherein the entity word part includes entity word markers and target entity words; the excitation module 430 is used to excite the target entity words based on the first confidence and the second confidence to obtain the speech recognition result of the speech to be recognized.

[0108] For example, the activation module 430 is specifically used to: determine the activation coefficient based on the first confidence level and the second confidence level; and activate the target entity word based on the activation coefficient to obtain the speech recognition result of the speech to be recognized.

[0109] For example, the speech recognition device further includes: a first training module for acquiring entity word samples, the entity word samples including entity words and entity word start markers and entity word end markers that enclose the entity words; and training an initial entity language model based on the entity word samples to obtain an entity language model.

[0110] For example, the target entity word and the entity words in the entity word sample belong to the same domain.

[0111] For example, the speech recognition device further includes: a second training module, used to acquire a first sample pair and a second sample pair, the first sample pair including a first speech sample and an entity text sample corresponding to the first speech sample, the entity text sample including entity words, non-entity words, and entity word start markers and entity word end markers that wrap the entity words; the second sample pair including a second speech sample and a general text sample corresponding to the second speech sample; and training an initial acoustic model based on the first sample pair and the second sample pair to obtain an acoustic model.

[0112] For example, the second training module can be specifically used to: obtain a sentence template, which includes at least one slot; fill each slot of the sentence template with entity words to obtain a text sample; convert the text sample to speech to obtain a first speech sample; add entity word start markers and entity word end markers to the text sample to wrap the filled entity words to obtain an entity text sample; and determine the first speech sample and the entity text sample as a first sample pair.

[0113] For example, the speech recognition device also includes: a third training module for acquiring parallel texts of general text samples; and training an initial general language model based on the parallel texts to obtain a general language model.

[0114] Figure 5 An example is a schematic diagram of the structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call logical instructions in the memory 530 to execute the speech recognition method provided in any of the above method embodiments.

[0115] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the speech recognition method provided in any of the above method embodiments.

[0117] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the speech recognition method provided in any of the above method embodiments.

[0118] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0119] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / entity> < / entity> < / entity> < / entity> < / entity> < / entity> < / entity> < / entity> < / entity> < / entity> < / entity> < / entity> < / entity> < / entity> < / entity> < / entity> < / entity> < / entity> < / entity> < / entity>

Claims

1. A speech recognition method, characterized in that, include: The speech to be recognized is collected and input into an acoustic model to obtain the first text output by the acoustic model; If entity word markers are detected in the first text, the entity word portion of the first text is input into an entity language model to obtain a first confidence score of the entity word portion output by the entity language model, and the first text is input into a general language model to obtain a second confidence score of the first text output by the general language model; the entity word portion includes the entity word markers and the target entity word; The target entity word is stimulated based on the first confidence level and the second confidence level to obtain the speech recognition result of the speech to be recognized; The step of stimulating the target entity word based on the first confidence level and the second confidence level to obtain the speech recognition result of the speech to be recognized includes: Based on the first confidence level and the second confidence level, the excitation coefficient is determined; The target entity word is excited based on the excitation coefficient to obtain the speech recognition result of the speech to be recognized.

2. The speech recognition method according to claim 1, characterized in that, The entity language model was trained in the following manner: Obtain entity word samples, wherein the entity word samples include entity words and entity word start markers and entity word end markers that enclose the entity words; Based on the entity word samples, the initial entity language model is trained to obtain the entity language model.

3. The speech recognition method according to claim 2, characterized in that, The target entity word and the entity words in the entity word sample belong to the same domain.

4. The speech recognition method according to claim 1, characterized in that, The acoustic model was trained in the following manner: Obtain a first sample pair and a second sample pair. The first sample pair includes a first speech sample and an entity text sample corresponding to the first speech sample. The entity text sample includes entity words, non-entity words, and entity word start markers and entity word end markers that enclose the entity words. The second sample pair includes a second speech sample and a general text sample corresponding to the second speech sample. Based on the first sample pair and the second sample pair, the initial acoustic model is trained to obtain the acoustic model.

5. The speech recognition method according to claim 4, characterized in that, The process of obtaining the first sample pair includes: Obtain a sentence template, wherein the sentence template includes at least one slot; By filling each slot in the sentence template with entity words, a text sample is obtained. The text sample is converted from text to speech to obtain the first speech sample; Add entity word start markers and entity word end markers to the text sample to wrap and fill the entity words, and you get the entity text sample; The first speech sample and the entity text sample are identified as the first sample pair.

6. The speech recognition method according to claim 4, characterized in that, The general language model was trained in the following manner: Obtain the parallel text of the general text sample; Based on the parallel texts, the initial general language model is trained to obtain the general language model.

7. A voice recognition device, characterized in that, include: The first speech processing module is used to collect the speech to be recognized and input the speech to be recognized into the acoustic model to obtain the first text output by the acoustic model; The second speech processing module is configured to, upon detecting that the first text includes entity word markers, input the entity word portion of the first text into an entity language model to obtain a first confidence score of the entity word portion output by the entity language model, and input the first text into a general language model to obtain a second confidence score of the first text output by the general language model; the entity word portion includes the entity word markers and target entity words; The activation module is used to activate the target entity word based on the first confidence level and the second confidence level to obtain the speech recognition result of the speech to be recognized; The step of stimulating the target entity word based on the first confidence level and the second confidence level to obtain the speech recognition result of the speech to be recognized includes: Based on the first confidence level and the second confidence level, the excitation coefficient is determined; The target entity word is excited based on the excitation coefficient to obtain the speech recognition result of the speech to be recognized.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the speech recognition method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech recognition method as described in any one of claims 1 to 6.

Images