Speech recognition device, speech recognition method, and program

Abstract

To provide a speech recognition device, a speech recognition method and a program capable of improving a recognition rate of speech recognition of an E2E model even if some terms are not registered.SOLUTION: A speech recognition device comprises: an acquisition part which acquires a speech signal; a speech feature quantity calculation part which calculates the speech feature quantity of the acquired speech signal; a first speech recognition part which performs speech recognition using a learnt first E2E model based upon the speech feature quantity, gives a first tag to a vocabulary part of a specific class of a text as a recognition result, and outputs the resulting part; a second speech recognition part which performs speech recognition using a learnt second E2E model based upon the speech feature quantity, gives a second tag to a vocabulary part of the specific class of a phoneme as a recognition result, and outputs the resulting part; a phoneme substitution part which substitutes the phoneme with the second tag for the vocabulary with the first tag of the text recognized by the first speech recognition part; and an output part which converts the phoneme with the second tag substituted by the phoneme substitution part into a text and outputs the text.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a speech recognition device, a speech recognition method, and a program. [Background technology]

[0002] In speech recognition technology, the End-to-End (E2E) model has been attracting attention as an alternative to the conventional DNN-HMM (Deep Neural Network-Hidden Markov model) model. Deep learning using the E2E model replaces machine learning systems that require multiple stages of processing from the time input data is given to the time the results are output with a single large neural network with multiple layers and modules that perform various processes.

[0003] The DNN-HMM model has the problem of accumulating errors because it processes by cascading an acoustic model and a language model. On the other hand, the E2E model outputs text directly from speech features, which is globally optimized and has been reported to improve recognition rates (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2021-501376 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the E2E model of the technology described in Patent Document 1 does not use a dictionary, so in order to recognize words that occur infrequently, such as people's names, the entire model needs to be retrained, and it is not possible to easily register people's names, terms, etc.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a speech recognition device, a speech recognition method, and a program that can improve the recognition rate even when there are terms that are not registered in E2E model speech recognition. [Means for solving the problem]

[0007] (1) In order to achieve the above object, a speech recognition device according to one embodiment of the present invention includes: an acquisition unit that acquires a speech signal; a speech feature calculation unit that calculates speech features of the acquired speech signal; a first speech recognition unit that performs speech recognition using a trained first end-to-end model based on the speech features, and outputs the recognition result text by attaching a first tag to a vocabulary part of a specific class in the recognition result text; a second speech recognition unit that performs speech recognition using a trained second end-to-end model based on the speech features, and outputs the recognition result text by attaching a second tag to a vocabulary part of a specific class in the recognition result phonemes; a phoneme substitution unit that replaces the vocabulary with the first tag in the text recognized by the first speech recognition unit with phonemes with the second tag; and an output unit that converts the phonemes with the second tag replaced by the phoneme substitution unit into text and outputs the text.

[0008] (2) Furthermore, a speech recognition device according to one aspect of the present invention is the speech recognition device described in (1), in which the output unit converts the second tagged phonemes replaced by the phoneme replacement unit into text with the highest similarity from among the language models in which text and phonemes are associated.

[0009] (3) Furthermore, in a speech recognition device according to one aspect of the present invention, the output unit converts and outputs the second-tagged phoneme replaced by the phoneme replacement unit when the similarity of the text with the highest similarity among the language models in which text and phonemes are associated is greater than a threshold value, and outputs the second-tagged phoneme replaced by the phoneme replacement unit as the first-tagged text recognized by the first speech recognition unit when the similarity of the text with the highest similarity among the language models stored in the language model storage unit is equal to or less than a threshold value.

[0010] (4) Furthermore, a speech recognition device according to one aspect of the present invention is a speech recognition device described in any one of (1) to (3), in which the first end-to-end model is trained for each utterance unit using a speech signal and text data, and the second end-to-end model is trained for each utterance unit using a speech signal and phoneme data.

[0011] (5) Furthermore, in one aspect of the present invention, the speech recognition device is a speech recognition device described in any one of (1) to (4), wherein, when there are multiple vocabulary parts of the specific class with the first tag in the text output by the first speech recognition unit, the phoneme replacement unit replaces the first vocabulary part of the specific class with the first tag with a phoneme with the second tag.

[0012] (6) Furthermore, a speech recognition device according to one aspect of the present invention is a speech recognition device described in any one of (1) to (5), wherein the vocabulary portion of the specific class is at least one proper noun among a person's name, a department name, a product name, a model name, a part name, and a place name.

[0013] (7) To achieve the above object, one aspect of the present invention provides a speech recognition method including: an acquisition unit acquiring a speech signal; a speech feature calculation unit calculating speech features of the acquired speech signal; a first speech recognition unit performing speech recognition using a trained first end-to-end model based on the speech features, and outputting the recognition result text with a first tag attached to a vocabulary part of a specific class; a second speech recognition unit performing speech recognition using a trained second end-to-end model based on the speech features, and outputting the recognition result text with a second tag attached to a vocabulary part of a specific class of phonemes; a phoneme substitution unit replacing the vocabulary with the first tag in the text recognized by the first speech recognition unit with the phonemes with the second tag; and an output unit converting the phonemes with the second tag replaced by the phoneme substitution unit into text and outputting the text.

[0014] (8) To achieve the above object, one aspect of the present invention provides a program that causes a computer to acquire a speech signal, calculate speech features of the acquired speech signal, perform speech recognition using a trained first end-to-end model based on the speech features, and output the recognition result text by attaching a first tag to a vocabulary part of a specific class, perform speech recognition using a trained second end-to-end model based on the speech features, and output the recognition result text by attaching a second tag to a vocabulary part of a specific class of phonemes, replace the first-tagged vocabulary of the text recognized using the first end-to-end model with the second-tagged phonemes, and convert the replaced second-tagged phonemes into text and output it. [Effects of the Invention]

[0015] According to (1) to (8), the recognition rate can be improved even if there are terms that are not registered in the E2E model speech recognition. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating an example of the configuration of a voice recognition device according to an embodiment; [Figure 2] FIG. 2 is a diagram illustrating an example of a voice recognition process according to an embodiment. [Figure 3] 10 is a flowchart of a speech recognition process according to an embodiment. [Figure 4] FIG. 1 is a diagram illustrating an example of speech recognition using a DNN-HMM model in a conventional method. [Figure 5] FIG. 1 is a diagram illustrating an example of voice recognition in the case of E2E in a conventional method. [Figure 6] FIG. 10 is a diagram showing evaluation conditions. [Figure 7] FIG. 10 is a diagram showing an example of an evaluation result. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component can be recognized. In all the drawings for explaining the embodiments, the same reference numerals are used for components having the same functions, and repeated explanations will be omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0018] [Example of voice recognition device configuration] 1 is a diagram illustrating an example of the configuration of a speech recognition device according to this embodiment. As illustrated in FIG. 1, the speech recognition device 1 includes an acquisition unit 101, a sound source localization unit 102, a sound source separation unit 103, an acoustic feature extraction unit 104, a model 105 (first end-to-end model), a text output unit 106 (first speech recognition unit), a model 107 (second end-to-end model), a phoneme typewriter output unit 108 (second speech recognition unit), a phoneme conversion unit 109, a language model storage unit 110, a similarity calculation unit 111 (output unit), a replacement unit 112 (output unit), and an output unit 113. Note that the speech recognition device 1 may not include a sound source localization unit or a sound source separation unit when the acquired speech signal contains speech from a single speaker.

[0019] The acquisition unit 101 acquires an audio signal. The audio signal may be collected by a microphone array, for example.

[0020] The sound source localization unit 102 determines the direction of each sound source for each frame of a predetermined length (for example, 20 ms) based on the M-channel audio signals assigned with microphone array numbers output by the acquisition unit 101 (sound source localization). In sound source localization, the sound source localization unit 102 calculates a spatial spectrum indicating the power for each direction using, for example, the MUSIC (Multiple Signal Classification) method. The sound source localization unit 102 determines the sound source direction for each sound source based on the spatial spectrum. Note that the sound source localization unit 102 may calculate sound source localization using another method, for example, the weighted delay and sum beam forming (WDS-BF) method, instead of the MUSIC method.

[0021] The sound source separation unit 103 acquires sound source direction information assigned with a sound source ID (identification information) output by the sound source localization unit 102 and the M-channel audio signals output by the acquisition unit 101. The sound source separation unit 103 separates the M-channel audio signals into sound source-specific audio signals (separated audio), which are audio signals indicating components of each sound source, based on the sound source directions indicated by the sound source direction information. When separating into the sound source-specific audio signals, the sound source separation unit 103 uses, for example, the GHDSS (Geometric-constrained High-order Decorrelation-based Source Separation) method. The sound source separation unit 103 obtains the spectrum of the separated audio signals. Note that the speech recognition device 1 may also include a speech period detection unit. When including the speech period detection unit, the speech period detection unit may detect a speech period (a start time and an end time of an utterance) based on the sound source direction information output by the sound source localization unit 102 and the M-channel audio signals output by the acquisition unit 101. The sound source separation unit 103 may also separate the sound sources using the speech intervals (start times and end times of speech) detected by the speech interval detection unit. Note that the sound source separation unit 103 may also regard an interval in which the magnitude of the audio signal is equal to or greater than a predetermined value as a speech interval.

[0022] The acoustic feature extraction unit 104 calculates acoustic features for each sound source from the spectrum output by the sound source separation unit 103. The acoustic feature extraction unit 104 calculates acoustic features, for example, by calculating a static Mel-Scale Log Spectrum (MSLS), a delta MSLS, and one delta power every predetermined time (for example, 10 ms). Note that the MSLS is obtained by using spectral features as features of the audio signal and performing an inverse discrete cosine transform on MFCCs (Mel Frequency Cepstrum Coefficients).

[0023] The model 105 is a learning model for speech recognition, such as the open source ESPnet model (https: / / kan-bayashi.github.io / asj-espnet2-tutorial / ). The model 105 is trained using a dataset of pairs of speech signals and text, in which class words to be customized in advance are tagged (for example, proper nouns such as person names, department names, product names, model names, part names, and place names).

[0024] The text output unit 106 inputs speech features into the model 105, and outputs text with class tags (first tags).

[0025] The model 107 is a learning model for speech recognition, such as the open source ESPnet model. The model 107 creates learning data of phoneme strings from the pronunciation of the text and performs model learning.

[0026] The phoneme typewriter output unit 108 inputs speech features into the model 107, and outputs a phoneme typewriter with class tags (second tags).

[0027] The phoneme conversion unit 109 converts the notation in the first tag <> of the text with the class tag into a phoneme string of the second tag corresponding to the notation in the first tag <>.

[0028] The language model storage unit 110 stores texts and phonemes in a class in association with each other. For example, the language model storage unit 110 stores the texts and phonemes in association with each other. <tanaka>and associate the text <Hatanaka> with the phoneme <hatanaka>associate and memorize.

[0029] The similarity calculation unit 111 compares each of the phonemes stored in the language model storage unit 110 one by one, and calculates, for example, the ratio of the number of matching phonemes included in the phoneme string to the total number of phonemes as the similarity (0 to 1).

[0030] The replacement unit 112 determines whether or not to replace the phoneme string based on the calculated similarity. For example, the replacement unit 112 converts the name notation to one with the largest similarity r that exceeds a threshold. If the similarity r is equal to or less than the threshold, the replacement unit 112 does not perform replacement and instead uses text output with class tags. The threshold is, for example, 0.8. The threshold is an example and may be set according to the purpose, class words, etc.

[0031] The output unit 113 outputs the text of the speech recognition result to an external device (a display device, a printing device, a computer, a mobile terminal, etc.). Note that the output unit 113 may have the functions of the similarity calculation unit 111 and the replacement unit 112, for example.

[0032] The speech recognition device 1 performs learning and speech recognition for each utterance unit (for example, one utterance period).

[0033] [Example of voice recognition processing] Next, an example of the voice recognition process of this embodiment will be described. Fig. 2 is a diagram for explaining an example of the voice recognition process according to this embodiment, and Fig. 3 is a flowchart of the voice recognition process according to this embodiment.

[0034] (Step S1) The acquisition unit 101 acquires an audio signal.

[0035] (Step S2) The acoustic feature extraction unit 104 calculates acoustic features of the acquired speech signal and outputs them to the text output unit 106 and the phoneme typewriter output unit 108.

[0036] (Step S3) The text output unit 106 outputs text with a class tag by inputting speech features into the model 105. For example, if the speech recognition result of the model 105 is "Tanaka should do this," the text output unit 106 assigns a class tag to "Tanaka," which has been recognized as a person's name, and outputs "Tanaka should do this" (g1).

[0037] (Step S4) The phoneme typewriter output unit 108 outputs a phoneme typewriter with class tags by inputting speech features into the model 107. For example, if the phoneme recognition result in the model 107 is "korewahatanakanitanomu (please ask me if I am in Hatanaka)", the phoneme typewriter output unit 108 adds a class tag to the phoneme hatanaka in the person's name to output "korewa <hatanaka>Output "nitanomu" (g2).

[0038] (Step S5) The phoneme conversion unit 109 converts the text with class tags into a phoneme string corresponding to the notation in <>. For example, if the text output is "I ask <Tanaka> to do this" and the phoneme typewriter output is "korewa" <hatanaka>In the case of "nitanomu," the phoneme conversion unit 109 converts "this is <hatanaka>Replace with "I ask you to" (g3).

[0039] (Step S6) The similarity calculation unit 111 compares each of the phonemes stored in the language model storage unit 110 one by one, and calculates, for example, the ratio of the number of matching phonemes included in the phoneme sequence to the total number of phonemes as the similarity r (0 to 1).

[0040] (Step S7) The replacement unit 112 determines whether the similarity r is greater than a threshold value. If the similarity r is greater than the threshold value (Step S7; YES), the replacement unit 112 proceeds to processing in Step S8. If the similarity r is equal to or less than the threshold value (Step S7; NO), the replacement unit 112 proceeds to processing in Step S9.

[0041] (Step S8) If the similarity r is greater than a threshold, the replacement unit 112 replaces the phoneme with a class word stored in the language model storage unit 110, and outputs the replaced phoneme from the output unit 113. For example, if the similarity r is 0.9, the replacement unit 112 replaces the phoneme with a person's name stored in the language model storage unit 110, and outputs "I'll leave this to Hatanaka" (g4) from the output unit 113.

[0042] (Step S9) If the similarity r is equal to or less than the threshold, the replacement unit 112 outputs the phoneme from the output unit 113 without replacing it with a class word stored in the language model storage unit 110. For example, if the similarity r is 0.7, the replacement unit 112 outputs "I'll leave this to Tanaka" (g5) from the output unit 113 without replacing it with a person's name stored in the language model storage unit 110.

[0043] In FIG. 3, for example, the processes of steps S3 and S4 may be performed in parallel or may be time-shared.

[0044] In the above-described processing, the replacement unit 112 may select the class word (vocabulary) with the highest similarity without comparing the similarity with a threshold value, and output it to the output unit 113.

[0045] As described above, in this embodiment, a tagged training dataset is used to train an E2E model so that vocabulary parts of a specific class can be identified from acoustic features. In this embodiment, vocabulary parts of a specific class are output as phonemes, and text corresponding to the phonemes is estimated using a dictionary. In this embodiment, the estimated text is inserted into the speech recognition results of the E2E model.

[0046] As a result, according to this embodiment, it is possible to improve the recognition rate even if there are terms that are not registered in the E2E model of speech recognition.

[0047] If one utterance contains multiple class words, the phoneme conversion unit 109 converts them into phonemes, for example, from left to right. For example, if the text output is "<Sato><Hatanaka>" and the phoneme typewriter output is " <hatanaka>", the phoneme conversion unit 109 converts the text <Sato> into phonemes <hatanaka>and does not convert <Hatanaka>.

[0048] Depending on the similarity value, the output unit 113 may present both the text output result and the text written in kanji based on the phoneme output. In such a case, the user (utterance) may select the appropriate one, and the speech recognition device 1 may store the selected result in the language model storage unit 110 in association with it. Alternatively, depending on the similarity value, the output unit 113 may present the class words recognized by the phonemes in, for example, "katakana" or "hiragana." The output unit 113 may also present the calculated similarity.

[0049] [Model in conventional methods] Here, a model in the conventional method will be described. Figure 4 is a diagram illustrating an example of speech recognition using a DNN-HMM model in a conventional method. The example in Figure 4 uses a DNN-based acoustic model, a pronunciation dictionary, and an N-gram language model. Therefore, if you want to register a new term, you can customize it relatively easily by registering it in the pronunciation dictionary or retraining the text part of the N-gram language model.

[0050] Figure 5 is a diagram illustrating an example of speech recognition in the case of E2E using a conventional method. As shown in Figure 5, when E2E is used, the model does not have the concept of a dictionary. Therefore, when registering a new term, it is necessary to retrain the entire model using many pairs of speech signals containing the new term and text in which the new term is used, which makes customization difficult.

[0051] In contrast to this, in this embodiment, the recognition results using the E2E model are output with class tags attached, as described above. Note that in conventional E2E output, class tags were included, so it was not possible to determine, for example, which people's names and proper nouns were in the output text.

[0052] For this reason, in this embodiment, class words (personal names or proper nouns depending on the application) are learned in advance, and tags are attached to the class words before output. Furthermore, in this embodiment, two E2E models are used, with one model attaching tags to class words in the text and the other model attaching tags to class words in phonemes that indicate pronunciation.

[0053] In this embodiment, in these two recognition results, the class words in the text are replaced with phoneme strings.

[0054] Furthermore, in this embodiment, in order to estimate what kind of kanji notation a class word recognized as a phoneme has, a pre-trained intra-class language model is referenced to extract the class word with the kanji notation that is most similar to the phoneme (reading).

[0055] In this embodiment, for example, the one with the highest similarity is extracted, and if the similarity is greater than a threshold, it is replaced and output, and if it is equal to or less than the threshold, the result is output as text without replacement. Note that, according to this embodiment, by comparing with the threshold, if a phoneme is not correctly recognized or if a class word to be recognized is not registered in the language model, the similarity will be a small value, and this type of error output can be prevented.

[0056] As a result, according to this embodiment, even class words that have not been learned by the E2E model can be properly recognized.

[0057] [Evaluation results] Next, the results of evaluating the speech recognition method of the present embodiment will be described. Figure 6 shows the evaluation conditions. As shown in Figure 6, the total number of utterances used in the evaluation was 1097, the number of person name locations was 68 (32 of which were unlearned), the number of person names was 33 (15 of which were unlearned), and the threshold was 0.0 to 1.0. The number of people registered in the language model within each class was 15, 33, and 133, respectively. In the evaluation, person names were used as class words.

[0058] FIG. 7 shows an example of the evaluation results. In FIG. 7, Total CER is the overall error rate (%), Class CER is the error rate (%) for the class words to be customized, and Out-of-Class CER is the error rate (%) for the rest. DNN-HMM is a conventional method that can be customized and is a reference value. E2E baseline is a speech recognition result using a conventional method that cannot be customized and is a comparative example.

[0059] As shown in Figure 7, in the case of 33 people, which is an example where the exact number of personal names is registered in the language model within the class, the error rate is improved compared to other registered numbers. Furthermore, in this case, even when the threshold is changed from 0 to 1.0, the error rate is low at any threshold.

[0060] In contrast, in the case of an example where there are insufficient names registered in the language model within a class (15 people missing, 18 people missing), the error rate is higher than when there are 33 people registered, but by setting the threshold to around 0.8, the error rate was improved compared to conventional methods.

[0061] In addition, in the case of 133 people, which is an example where there are too many personal names registered in the language model within the class, there is little change in the error rate even when the threshold is changed.

[0062] In this way, in the evaluation, the error rate of the customization target could be improved by setting the threshold for similarity to around 0.8, regardless of whether the number of personal names registered in the language model within the class was appropriate, insufficient, or too many.

[0063] Note that a program for realizing all or part of the functions of the speech recognition device 1 of the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to perform all or part of the processing performed by the speech recognition device 1. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0064] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0065] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0066] 1...Speech recognition device, 101...Acquisition unit, 102...Sound source localization unit, 103...Sound source separation unit, 104...Acoustic feature extraction unit, 105...Model, 106...Text output unit, 107...Model, 108...Phoneme typewriter output unit, 109...Phoneme conversion unit, 110...Language model storage unit, 111...Similarity calculation unit, 112...Replacement unit, 113...Output unit< / hatanaka> < / hatanaka> < / hatanaka> < / hatanaka> < / hatanaka> < / hatanaka> < / tanaka>

Claims

1. an acquisition unit that acquires an audio signal; a speech feature calculation unit that calculates speech features of the acquired speech signal; a first speech recognition unit that performs speech recognition using a trained first end-to-end model based on the speech features, and outputs a recognition result text by attaching a first tag to a vocabulary part of a specific class; a second speech recognition unit that performs speech recognition using a trained second end-to-end model based on the speech features, and that attaches a second tag to a vocabulary part of a specific class among phonemes that are the recognition result, and outputs the result; a phoneme replacement unit that replaces the first tagged vocabulary of the text with the second tagged phoneme when the similarity of the text recognized by the first speech recognition unit is greater than a threshold; an output unit that converts the second-tagged phonemes replaced by the phoneme replacement unit into text and outputs the text; A speech recognition device comprising:

2. the output unit converts the phonemes with the second tags replaced by the phoneme replacement unit into text with the highest similarity from a language model in which text and phonemes are associated with each other; 2. The speech recognition device according to claim 1.

3. An acquisition unit that acquires an audio signal; a speech feature calculation unit that calculates speech features of the acquired speech signal; a first speech recognition unit that performs speech recognition using a trained first end-to-end model based on the speech features, and outputs a recognition result text by attaching a first tag to a vocabulary part of a specific class; a second speech recognition unit that performs speech recognition using a trained second end-to-end model based on the speech features, and that attaches a second tag to a vocabulary part of a specific class among phonemes that are the recognition result, and outputs the result; a phoneme replacement unit that replaces the first tagged vocabulary in the text recognized by the first speech recognition unit with the second tagged phonemes; an output unit that converts the second-tagged phonemes replaced by the phoneme replacement unit into text and outputs the text; The output unit converting and outputting the phonemes with the second tags replaced by the phoneme replacement unit when the similarity of the text with the highest similarity among the language models in which the text and the phonemes are associated is greater than a threshold value; outputting the second-tagged phonemes replaced by the phoneme replacement unit as the first-tagged text recognized by the first speech recognition unit when the similarity of the text with the highest similarity among the language models stored in the language model storage unit is equal to or less than a threshold value; Voice recognition device.

4. the first end-to-end model is trained for each utterance unit using a speech signal and text data; The second end-to-end model is trained for each utterance unit using a speech signal and phoneme data.

3. The speech recognition device according to claim 1.

5. the phoneme substitution unit, when there are a plurality of vocabulary parts of the specific class tagged with the first tag in the text output by the first speech recognition unit, replaces the first vocabulary part of the specific class tagged with the first tag with the phoneme tagged with the second tag; 3. The speech recognition device according to claim 1.

6. the vocabulary portion of the specific class is at least one proper noun among a person's name, a department name, a product name, a model name, a part name, and a place name; 3. The speech recognition device according to claim 1.

7. The acquisition unit acquires the audio signal, a speech feature calculation unit that calculates speech features of the acquired speech signal; a first speech recognition unit performs speech recognition using a trained first end-to-end model based on the speech features, and outputs the recognition result text by attaching a first tag to a vocabulary part of a specific class; a second speech recognition unit performs speech recognition using a trained second end-to-end model based on the speech features, and outputs the recognition result by attaching a second tag to a vocabulary part of a specific class among the phonemes; a phoneme substitution unit, when a similarity of the text recognized by the first speech recognition unit is greater than a threshold, replacing the vocabulary with the first tag of the text with the phoneme with the second tag; an output unit converting the second tagged phonemes replaced by the phoneme replacement unit into text and outputting the text; Speech recognition methods.

8. On the computer, Acquire an audio signal, Calculating speech features of the acquired speech signal; performing speech recognition using a trained first end-to-end model based on the speech features, and attaching a first tag to a vocabulary part of a specific class in text that is a recognition result, and outputting the text; performing speech recognition using a trained second end-to-end model based on the speech features, and attaching a second tag to a vocabulary part of a specific class among the phonemes resulting from the recognition, and outputting the result; replacing the first tagged vocabulary of the text with the second tagged phonemes when a similarity of the text recognized using the first end-to-end model is greater than a threshold; converting the replaced phonemes with the second tags into text and outputting the text; program.

Images