Learning device generating model for detecting error of voice recognition, voice recognition device and program

Abstract

To improve voice recognition accuracy in a caption broadcast service.SOLUTION: A division part 10 of a learning device 1 divides text data for pre-learning into sentence units, and a tokenization part 11 generates a sentence token string. A deletion error processing part 12 deletes tokens from the sentence token string with a probability of 15% at random, and generates the sentence token string (deletion) and a binary data string. A learning part 14 pre-learns a model using the sentence token string (deletion) and the binary data string, performs fine tuning using a voice recognition result etc., and generates a deletion error detection model. An insertion error processing part 13 inserts the token into the sentence token string at random at the probability of 15% and generates the sentence token string (insertion) and the binary data string. A learning part 15 pre-learns the model using the sentence token string (insertion) and the binary data string, performs fine tuning using the voice recognition result etc., and generates an insertion error detection model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a learning device, a voice recognition device, and a program for generating a model for detecting errors in voice recognition, and more particularly to a technique for detecting errors in the recognition results obtained by voice recognition technology in closed caption broadcasting services. [Background technology]

[0002] Conventionally, there is a closed caption broadcasting service that displays the audio of a broadcast program on the screen as subtitles. As a technique for producing subtitles for live programs in this closed caption broadcasting service, a method is known in which the audio data is converted into text using a voice recognition technique to produce the subtitles, and the produced subtitles are then manually corrected (for example, see Non-Patent Document 1).

[0003] Furthermore, a technique is known for improving the recognition result when text is estimated from speech data using an end-to-end speech recognition model.

[0004] For example, a method using a replacement token detection task has been disclosed in order to efficiently pre-train a pre-training model for a natural language processing task (see, for example, Non-Patent Document 2). Also, a method using ELECTRA, a language model pre-trained with a replacement token detection task, has been disclosed in order to improve the accuracy of speech recognition (see, for example, Non-Patent Document 3). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Tomoyasu Komori, "Latest Trends in Automatic Subtitling for Live Broadcast Programs," [online], NHK Technical Research and Development, Summer 2020 Issue Commentary 02, NHK Broadcasting Technology Research Laboratories, [Retrieved January 13, 2022], Internet<URL:https: / / www.nhk.or.jp / strl / publica / rd / 182 / 3.html> [Non-Patent Document 2] Kevin Clark et al.,“ELECTRA:PRE-TRAINING TEXT ENCODERS AS DISCRIMINATORS RATHER THAN GENERATORS”,arXiv:2003.10555v1 [cs.CL] 23 Mar 2020 ICLR2020 [Non-Patent Document 3] Hayato Futami et al., “ASR RESCORING AND CONFIDENCE ESTIMATION WITH ELECTRA”,arXiv:2110.01857v1 [cs.CL] 5 Oct 2021 ASRU2021 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the method of Non-Patent Document 2 mentioned above, error detection for the output result of speech recognition only deals with substitution errors in which some words etc. are replaced with other words etc. in the output result.

[0007] For this reason, when the method described in the aforementioned Non-Patent Document 2 is applied to a subtitle broadcasting service, although it is possible to correct substitution errors in the output results of speech recognition, there is a problem in that it does not sufficiently contribute to improving the accuracy of speech recognition.

[0008] In general, when speech recognition technology is used for live broadcasting programs, the most common type of error is deletion errors, where some words are deleted from the speech recognition output, and insertion errors, where words unrelated to the original speech data are inserted, may also occur. For this reason, in order to improve the accuracy of speech recognition in closed caption broadcasting services, a method capable of dealing with not only substitution errors but also deletion errors and insertion errors has been desired.

[0009] In addition, when manually correcting the output result of speech recognition, the confirmation work requires a lot of time and effort. Therefore, in order to improve the efficiency of the subtitle correction work, a method is desired that indicates the location where an error occurs in the output result and also indicates the type of error (substitution error, deletion error, and insertion error).

[0010] Therefore, the present invention has been made to solve the above-mentioned problems, and its object is to provide a learning device and a voice recognition device that can improve the accuracy of voice recognition in subtitle broadcasting services. [Means for solving the problem]

[0011] In order to solve the above problem, the learning device of claim 1 is a learning device that generates an error detection model for detecting errors in speech recognition, the learning device comprising: a tokenization unit that tokenizes sentence data for pre-learning and generates a token sequence of a sentence; a deletion error processing unit that generates a token sequence of a sentence by randomly deleting tokens from the token sequence of the sentence generated by the tokenization unit, and generates a token sequence of a sentence including the deletion error, and generates a binary data sequence corresponding to the token sequence of the sentence including the deletion error, which indicates the presence or absence of the deletion error; a first learning unit that uses the binary data sequence generated by the deletion error processing unit as correct answer data and the token sequence of the sentence including the deletion error generated by the deletion error processing unit and the correct answer data to learn a relationship between the token sequence of a sentence and a data sequence corresponding to the token sequence of the sentence that indicates a probability of a deletion error for each token; and a second learning unit that learns a relationship between a token sequence of a sentence and a data sequence indicating a probability of an insertion error for each token, corresponding to the token sequence of the sentence, using the token sequence of the sentence including the insertion error generated by the insertion error processing unit and the correct answer data, with the binary data sequence generated by the insertion error processing unit as correct answer data, wherein the first learning unit further acquires a token sequence of a sentence including a deletion error generated by a predetermined speech recognition process, and uses a binary data sequence indicating the position of the presence or absence of the deletion error, corresponding to the token sequence of the sentence including the deletion error, as correct answer data, to perform fine tuning using the token sequence of the sentence including the deletion error and the correct answer data, thereby generating a deletion error detection model as a trained model, and the second learning unit further acquires a token sequence of a sentence including an insertion error generated by a predetermined speech recognition process, and uses a binary data sequence indicating the position of the presence or absence of the insertion error, corresponding to the token sequence of the sentence including the insertion error, as correct answer data,The method is characterized in that fine tuning is performed using the token sequence of the sentence including the insertion error and the correct answer data, and an insertion error detection model is generated as a trained model.

[0012] Furthermore, the voice recognition device of claim 2 receives voice data and generates text corresponding to the voice data by voice recognition processing, estimates a plurality of candidate sentence data from the voice data using a predetermined voice recognition model, and calculates a score S ASR The candidate sentence data and the score S ASR and for each of the N pieces of candidate sentence data output by the speech recognition unit, tokenizing the candidate sentence data to generate a token sequence of the candidate sentence, and estimating a data sequence indicating a probability of an error for each token corresponding to the token sequence of the candidate sentence using a predetermined substitution error detection model, and the deletion error detection model and the insertion error detection model generated by the learning device according to claim 1, thereby obtaining a data sequence for each model, and calculating a score S based on the data sequence for each model. disc and a score S corresponding to the candidate sentence data output by the speech recognition unit is calculated. ASR The score S disc an error detection unit that obtains a score S by correcting the score S by the above-mentioned error detection unit, selects one candidate sentence data from the N candidate sentence data as a recognition result based on the score S for each of the N candidate sentence data, and outputs the recognition result as text corresponding to the speech data.

[0013] The speech recognition device of claim 3 is the speech recognition device of claim 2, further comprising a recognition result correction unit, wherein when the error detection unit selects the recognition result based on the score S for each of the N candidate sentence data, the error detection unit further selects the data sequence for each model corresponding to the one candidate sentence data as a data sequence for each model of the recognition result, and further selects the score S corresponding to the one candidate sentence data as a data sequence for each model of the recognition result. ASRThe score S of the recognition result ASR , and the probability of error for each token included in the data string for each model of the recognition result and the score S of the recognition result are selected as ASR and based on the reliability scores of the replacement error, the deletion error, and the insertion error for each token in the recognition result, calculates the positions in the recognition result where each of the replacement error, the deletion error, and the insertion error has occurred, and generates an error-attached recognition result indicating the position and type of the error, and the recognition result correction unit displays the error-attached recognition result generated by the error detection unit on a screen, corrects the error-attached recognition result in accordance with a user operation, and outputs the corrected recognition result sentence as text corresponding to the speech data.

[0014] Further, the speech recognition device of claim 4 is the speech recognition device of claim 2, wherein the error detection unit includes a tokenization unit that tokenizes each of the N candidate sentence data and generates a token sequence of the candidate sentence, and a tokenization unit that uses the replacement error detection model to estimate the data sequence as a replacement error data sequence from the token sequence of the candidate sentence generated by the tokenization unit, uses the deletion error detection model to estimate the data sequence as a deletion error data sequence from the token sequence of the candidate sentence, and uses the insertion error detection model to estimate the data sequence as an insertion error data sequence from the token sequence of the candidate sentence, and calculates a score S of the candidate sentence data based on a probability of error for each of the tokens included in the replacement error data sequence, the deletion error data sequence, and the insertion error data sequence. disc and an error detection processing unit for obtaining the score S of the candidate sentence data. ASR The score S obtained by the error detection processing unit discand a recognition result selection unit that selects one candidate sentence data from the N candidate sentence data as the recognition result based on the score S for each of the N candidate sentence data obtained by the score correction unit, and outputs the recognition result as text corresponding to the speech data.

[0015] Further, the speech recognition device of claim 5 is the speech recognition device of claim 3, wherein the error detection unit includes a tokenization unit that tokenizes each of the N candidate sentence data and generates a token sequence of the candidate sentence, and a tokenization unit that uses the replacement error detection model to estimate the data sequence as a replacement error data sequence from the token sequence of the candidate sentence generated by the tokenization unit, uses the deletion error detection model to estimate the data sequence as a deletion error data sequence from the token sequence of the candidate sentence, and uses the insertion error detection model to estimate the data sequence as an insertion error data sequence from the token sequence of the candidate sentence, and calculates a score S of the candidate sentence data based on a probability of error for each of the tokens included in the replacement error data sequence, the deletion error data sequence, and the insertion error data sequence. disc and an error detection processing unit for obtaining the score S of the candidate sentence data. ASR The score S obtained by the error detection processing unit disc a score correction unit that calculates a score S of the candidate sentence data by correcting the score by the score correction unit, and a score correction unit that selects one candidate sentence data from the N candidate sentence data as the recognition result based on the score S for each of the N candidate sentence data calculated by the score correction unit, and selects the replacement error data sequence, the deletion error data sequence, and the insertion error data sequence corresponding to the one candidate sentence data, as well as the score S ASR The replacement error data string of the recognition result, the deletion error data string of the recognition result, the insertion error data string of the recognition result, and the score S of the recognition result ASRa recognition result selection unit for selecting a token as a replacement error data sequence of the recognition result selected by the recognition result selection unit, and a score S of the recognition result. ASR Based on the above, the probability of occurrence of a substitution error for each token in the recognition result is calculated as a substitution error reliability score, and the probability of an error for each token included in the deletion error data string of the recognition result and the score S of the recognition result are calculated as a substitution error reliability score. ASR Based on the above, the probability of occurrence of the deletion error for each token in the recognition result is calculated as a deletion error reliability score, and the probability of an error for each token included in the insertion error data string of the recognition result and the score S of the recognition result are calculated as a deletion error reliability score. ASR a confidence score calculation unit that calculates a probability of occurrence of the insertion error for each token in the recognition result as a confidence score of an insertion error based on a threshold value of the confidence score of the substitution error for each token in the recognition result calculated by the confidence score calculation unit, generates a binary data sequence indicating a position of the presence or absence of the substitution error corresponding to the token sequence of the recognition result by threshold processing of the confidence score of the deletion error for each token in the recognition result, and generates a binary data sequence indicating the position of the presence or absence of the deletion error corresponding to the token sequence of the recognition result by threshold processing of the confidence score of the deletion error for each token in the recognition result, and generates a binary data sequence indicating the position of the presence or absence of the insertion error corresponding to the token sequence of the recognition result by threshold processing of the confidence score of the insertion error for each token in the recognition result, and an error processing unit that generates the recognition result with errors based on the binary data sequences generated by the threshold value determination unit.

[0016] The program of claim 6 further provides a program for a computer constituting a learning device for generating an error detection model for detecting errors in speech recognition, the program comprising: a tokenization unit that tokenizes sentence data for pre-learning and generates a token sequence of a sentence; a deletion error processing unit that generates a token sequence of a sentence by randomly deleting tokens from the token sequence of the sentence generated by the tokenization unit, and generates a token sequence of a sentence including the deletion error, and generates a binary data sequence corresponding to the token sequence of the sentence including the deletion error, which indicates the presence or absence of the deletion error; a first learning unit that uses the binary data sequence generated by the deletion error processing unit as correct answer data and learns a relationship between the token sequence of a sentence and a data sequence corresponding to the token sequence of the sentence, which indicates a probability of a deletion error for each token, by using the token sequence of the sentence including the deletion error generated by the deletion error processing unit and the correct answer data; and a second learning unit that learns a relationship between a token sequence of a sentence and a data sequence indicating a probability of an insertion error for each token, corresponding to the token sequence of the sentence, using the token sequence of the sentence including the insertion error generated by the insertion error processing unit and the correct answer data, with the binary data sequence generated by the insertion error processing unit as correct answer data; the first learning unit further acquires a token sequence of a sentence including a deletion error generated by a predetermined speech recognition process, and performs fine tuning using the token sequence of the sentence including the deletion error and the correct answer data, using the binary data sequence indicating the position of the presence or absence of the deletion error, corresponding to the token sequence of the sentence including the deletion error, to generate a deletion error detection model as a trained model; the second learning unit further acquires a token sequence of a sentence including an insertion error generated by a predetermined speech recognition process, and uses the binary data sequence indicating the position of the presence or absence of the insertion error, corresponding to the token sequence of the sentence including the insertion error, as correct answer data;The method is characterized in that fine tuning is performed using the token sequence of the sentence including the insertion error and the correct answer data, and an insertion error detection model is generated as a trained model.

[0017] The program of claim 7 is a program for causing a computer constituting a speech recognition device which receives speech data and generates text corresponding to the speech data by speech recognition processing, to estimate a plurality of candidate sentence data from the speech data using a predetermined speech recognition model, and to generate a score S ASR The candidate sentence data and the score S ASR and for each of the N pieces of candidate sentence data output by the speech recognition unit, tokenizing the candidate sentence data to generate a token sequence of the candidate sentence, and estimating a data sequence indicating a probability of an error for each token corresponding to the token sequence of the candidate sentence using a predetermined substitution error detection model, and the deletion error detection model and the insertion error detection model generated by the learning device according to claim 1, thereby obtaining a data sequence for each model, and calculating a score S based on the data sequence for each model. disc and a score S corresponding to the candidate sentence data output by the speech recognition unit is calculated. ASR The score S disc and based on the score S for each of the N pieces of candidate sentence data, selects one piece of candidate sentence data as a recognition result from the N pieces of candidate sentence data, and outputs the recognition result as text corresponding to the speech data, thereby functioning as an error detection unit. Effect of the Invention

[0018] As described above, according to the present invention, it is possible to improve the accuracy of speech recognition in closed caption broadcasting services. [Brief description of the drawings]

[0019] [Figure 1]1 is a block diagram showing an example of the configuration of a learning device according to an embodiment of the present invention. [Diagram 2] 13 is a flowchart showing an example of processing by the learning device. [Diagram 3] 11 is a diagram illustrating an example of generating a sentence token string and a binary data string from sentence data. FIG. [Figure 4] 11A and 11B are diagrams illustrating an example of processing by a learning unit; [Diagram 5] 1 is a block diagram showing an example of the configuration of a voice recognition device according to an embodiment of the present invention; [Figure 6] FIG. 2 is a diagram illustrating an outline of a process example of a voice recognition device. [Figure 7] 4 is a flowchart showing an example of processing performed by a voice recognition device. [Figure 8] 4 is a block diagram showing a configuration example of an error detection unit; [Figure 9] 4 is a block diagram showing a configuration example of an error detection processing unit. [Figure 10] 11 is a block diagram showing an example of the configuration of a recognition result error processing unit. FIG. [Figure 11] 1A is a diagram showing an example of a correct recognition result, and FIG. 1B is a diagram showing an example of a recognition result with an error. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The present invention is characterized in that, in order to improve the output result of an end-to-end speech recognition model, a deletion error detection model is generated by training using a token sequence of a sentence after randomly deleting tokens, and an insertion error detection model is generated by training using a token sequence of a sentence after randomly inserting tokens.

[0021] The present invention is also characterized in that, using a conventional substitution error detection model, and a deletion error detection model and an insertion error detection model generated by the above-mentioned learning, the probability of an error occurring for each token (error probability) is calculated for each of N candidate sentences estimated by speech recognition, and one candidate sentence is selected by correcting the scores of the N candidate sentences. N is an integer of 2 or more.

[0022] As a result, the score of the candidate sentence estimated by speech recognition is corrected in terms of substitution errors, deletion errors, and insertion errors for each token, so that the selected candidate sentence becomes a highly accurate text, thereby making it possible to improve the speech recognition accuracy.

[0023] Here, examples of substitution errors, deletion errors, and insertion errors will be described. For example, assume that the correct speech recognition output is "aiueo". In this case, if the speech recognition erroneously recognizes "u" as "u", the output result "aiueo" will contain a substitution error.

[0024] If the voice recognition does not recognize "u", the output result "aiueo" will contain a deletion error. If the voice recognition recognizes "ka" between "u" and "e", the output result "aiu kaeo" will contain an insertion error.

[0025] [Learning device] First, a learning device according to an embodiment of the present invention will be described below. Fig. 1 is a block diagram showing an example of the configuration of a learning device according to an embodiment of the present invention, and Fig. 2 is a flowchart showing an example of processing by the learning device.

[0026] This learning device 1 is a device that performs pre-learning of a model using pre-learning text data, and fine-tunes the model using speech recognition results for fine-tuning, etc., to generate a deletion error detection model and an insertion error detection model as trained models.

[0027] The learning device 1 includes a division unit 10, a tokenization unit 11, a deletion error processing unit 12, an insertion error processing unit 13, learning units 14 and 15, a data generation unit 16, and a storage unit 17.

[0028] The division unit 10 receives pre-learning text data (step S201), divides the pre-learning text data into sentence units, and generates data for each sentence (sentence data) (step S202). Then, the division unit 10 outputs the sentence data to the tokenization unit 11.

[0029] The tokenizer 11 receives the sentence data from the splitter 10, tokenizes the sentence data, and generates a token string of the sentence (step S203). The tokenizer 11 then outputs the token string of the sentence to the deletion error processor 12 and the insertion error processor 13.

[0030] 3 is a diagram for explaining an example of generating a sentence token string and a binary data string from sentence data. For example, if the sentence data is "he has a pen", the sentence data "he has a pen" is tokenized by the process of step S203 of the tokenizer 11, and a sentence token string "he / has / a / pen" is generated. Here, the notation in which each word constituting the sentence data is divided by the symbol " / " is regarded as the sentence token string.

[0031] 1 and 2, the deletion error processing unit 12 inputs the token string of the sentence from the tokenization unit 11. Then, the deletion error processing unit 12 generates a deletion error by randomly deleting tokens from the token string of the sentence with a probability of 15%, and generates a token string of the sentence including a deletion error (hereinafter referred to as a token string of the sentence (deleted)) (step S204).

[0032] The deletion error processing unit 12 generates a binary data string (correct answer data) consisting of binary data for each token indicating the position of the presence or absence of a deletion error, corresponding to the token string (deletion) of the sentence (step S205). Specifically, the deletion error processing unit 12 generates a binary data string in which the points where a deletion error has occurred are represented as "1" and the points where a deletion error has not occurred are represented as "0", corresponding to the token string (deletion) of the sentence. This binary data string is used as correct answer data in the learning unit 14, which will be described later.

[0033] This generates a binary data string of one-hot vectors that sets the locations where deletion errors occur to "1" as the correct answer data, and generates a dataset consisting of the sentence token string (deletion) and the binary data string.

[0034] The deletion error processing unit 12 outputs the token string (deletion) of the sentence and the binary data string to the learning unit 14.

[0035] 3, in the above example, when the token string of the sentence is "he / has / a / pen", for example, "a" is deleted by randomly generating a deletion error in the token string of the sentence "he / has / a / pen" by the process of step S204 of the deletion error processing unit 12. Then, the token string of the sentence (deletion) "he / has / pen" is generated.

[0036] Furthermore, the process of step S205 of the deletion error processing unit 12 generates a binary data string "0,0,1" consisting of binary data indicating the position of the presence or absence of a deletion error, corresponding to the token string (deletion) "he / has / pen" of the sentence. Note that in order to handle the occurrence of multiple deletion errors, a "1" indicating the presence of a deletion error is set before the token corresponding to the deletion error.

[0037] 1 and 2, the learning unit 14 inputs the token sequence (deletion) of the sentence and the binary data sequence from the deletion error processing unit 12. Then, using the token sequence (deletion) of the sentence and the correct answer data which is a binary data sequence, the learning unit 14 performs pre-learning of a model in which the token sequence of the sentence is input data and a data sequence indicating the probability of a deletion error for each token corresponding to the token sequence of the sentence is output data (step S206).

[0038] Fig. 4 is a diagram for explaining an example of processing by the learning unit 14. As shown in Fig. 4, the learning unit 14 includes a model 20 and an update unit 21. A token string (deletion) of a sentence is input to the model 20, and a data string indicating the probability of a deletion error for each token is output from the model 20. Then, the update unit 21 updates the parameters of the model 20 so that correct answer data, which is a binary data string, matches the data string indicating the probability of a deletion error for each token.

[0039] 4, both of the structures of model 20 for detecting deletion errors and model 22 for detecting insertion errors use ELECTRA, a neural network similar to that described in the above-mentioned Non-Patent Document 2. This neural network is a model in which the final layer of BERT is changed to a sigmoid function.

[0040] In addition, in the pre-training and fine-tuning of the models 20 and 22 described later, a binary cross-entropy error is used as a loss function for binary classification of whether or not there is an error. Learning is performed by the backpropagation method so that the value of the loss function becomes small.

[0041] 1 and 2, the insertion error processor 13 inputs the token string of the sentence from the tokenizer 11. Then, the insertion error processor 13 generates an insertion error by randomly inserting a token into the token string of the sentence with a probability of 15%, and generates a token string of the sentence including an insertion error (hereinafter referred to as a token string of the sentence (insertion)) (step S207).

[0042] The insertion error processor 13 generates a binary data string (correct answer data) consisting of binary data for each token indicating the presence or absence of an insertion error, corresponding to the token string (insertion) of the sentence (step S208). Specifically, the insertion error processor 13 generates a binary data string in which the location where an insertion error occurred is represented as "1" and the location where no insertion error occurred is represented as "0", corresponding to the token string (insertion) of the sentence. This binary data string is used as correct answer data in the learning unit 15, which will be described later.

[0043] This generates a binary data string of a one-hot vector with a value of "1" at the location where the insertion error occurred as the correct answer data, and generates a dataset consisting of the sentence token string (insertion) and the binary data string.

[0044] The insertion error processor 13 outputs the token sequence (insertion) of the sentence and the binary data sequence to the learning unit 15.

[0045] 3, in the above example, when the token string of the sentence is "he / has / a / pen", an insertion error is randomly generated in the token string of the sentence "he / has / a / pen" by the process of step S207 of the insertion error processing unit 13, so that, for example, "up" is inserted between "a" and "pen". Then, the token string (insertion) of the sentence "he / has / a / up / pen" is generated.

[0046] In addition, by the processing of step S208 in the insertion error processing unit 13, a binary data string "0,0,0,1,0" consisting of binary data indicating the presence or absence of an insertion error is generated corresponding to the token string (insertion) "he / has / a / up / pen" of the sentence.

[0047] 1 and 2, the learning unit 15 inputs the token sequence (insertion) of the sentence and the binary data sequence from the insertion error processing unit 13. Then, using the token sequence (insertion) of the sentence and the correct answer data which is a binary data sequence, the learning unit 15 performs pre-learning of a model in which the token sequence of the sentence is input data and a data sequence indicating the probability of an insertion error for each token corresponding to the token sequence of the sentence is output data (step S209).

[0048] 4, the learning unit 15 includes a model 22 and an update unit 23. A token string (insertion) of a sentence is input to the model 22, and a data string indicating the probability of an insertion error for each token is output from the model 22. Then, the update unit 23 updates the parameters of the model 22 so that correct answer data, which is a binary data string, matches the data string indicating the probability of an insertion error for each token.

[0049] Returning to FIG. 1 and FIG. 2, after the pre-learning in steps S206 and S209 is completed, the data generating unit 16 receives the fine-tuning speech recognition result and the like.

[0050] The fine-tuning speech recognition results, etc., include the actual speech recognition results estimated using a pre-trained model, for example, as well as information on the location where an error occurred and the type of error.

[0051] The data generating unit 16 generates a token string (deletion) of a sentence reflecting the actual speech recognition result, etc., and a corresponding binary data string, based on the speech recognition result for fine tuning, etc., and also generates a token string (insertion) of a sentence and a corresponding binary data string (step S210). This binary data string is used as the correct answer data in the learning units 14 and 15 described later.

[0052] The data generation unit 16 outputs the token string (deletion) and the binary data string of the sentence to the learning unit 14, and outputs the token string (insertion) and the binary data string of the sentence to the learning unit 15.

[0053] The learning unit 14 inputs the token sequence (deletion) of the sentence and the binary data sequence from the data generating unit 16. Then, the learning unit 14 performs fine tuning of the model pre-trained in step S206 using the token sequence (deletion) of the sentence and the binary data sequence (step S211). The fine tuning process is as shown in FIG.

[0054] The learning unit 14 generates a deletion error detection model as a trained model through fine tuning in step S211, and stores this in the storage unit 17 (step S212).

[0055] The learning unit 15 inputs the token sequence (insertion) and the binary data sequence of the sentence from the data generating unit 16. Then, the learning unit 15 performs fine tuning of the model pre-trained in step S209 using the token sequence (insertion) and the binary data sequence of the sentence (step S213). The fine tuning process is as shown in FIG.

[0056] The learning unit 15 generates an insertion error detection model as a trained model through the fine tuning in step S213, and stores this in the storage unit 17 (step S214).

[0057] As described above, according to the learning device 1 of the embodiment of the present invention, the division unit 10 divides the pre-learning text data into sentence units and generates sentence data for each sentence, and the tokenization unit 11 tokenizes the sentence data and generates a token string for the sentence.

[0058] The deletion error processing unit 12 generates a token string (deletion) of the sentence by randomly deleting tokens from the token string of the sentence with a probability of 15%, and generates a binary data string (correct answer data).

[0059] The learning unit 14 uses the sentence token sequence (deletion) and the correct answer data which is a binary data sequence to perform pre-learning of a model in which the sentence token sequence is input data and a data sequence indicating the probability of a deletion error for each token corresponding to the sentence token sequence is output data. After the pre-learning, the learning unit 14 performs fine tuning using the sentence token sequence (deletion) generated from the speech recognition result for fine tuning and the correct answer data which is a binary data sequence, thereby generating a deletion error detection model as a trained model.

[0060] The insertion error processor 13 generates a token string (insertion) for the sentence by randomly inserting a token into the token string of the sentence with a probability of 15%, and generates a binary data string (correct answer data).

[0061] The learning unit 15 uses the sentence token sequence (insertion) and the correct answer data, which is a binary data sequence, to perform pre-learning of a model in which the sentence token sequence is input data and a data sequence indicating the probability of an insertion error for each token corresponding to the sentence token sequence is output data. After the pre-learning, the learning unit 15 performs fine tuning using the sentence token sequence (insertion) generated from the speech recognition result for fine tuning and the like and the correct answer data, which is a binary data sequence, to generate an insertion error detection model as a trained model.

[0062] As a result, the speech recognition device 2 described later can use a conventional substitution error detection model, as well as a deletion error detection model and an insertion error detection model generated by the learning device 1 to determine the probability of an error occurring for each token for each of the N candidate sentences estimated by speech recognition, and can select one candidate sentence by correcting the scores of the N candidate sentences.

[0063] As a result, the score of the candidate sentence estimated by speech recognition is corrected in terms of substitution errors, deletion errors, and insertion errors for each token, so that the selected candidate sentence becomes a highly accurate text. In addition to reducing substitution errors that may occur due to speech recognition, deletion errors and insertion errors can also be reduced. This makes it possible to improve the speech recognition accuracy.

[0064] [Voice recognition device] Next, a description will be given of a voice recognition device according to an embodiment of the present invention. Fig. 5 is a block diagram showing an example of the configuration of a voice recognition device according to an embodiment of the present invention, and Fig. 6 is a diagram for explaining an outline of an example of processing performed by the voice recognition device.

[0065] This speech recognition device 2 uses the conventional substitution error detection model generated in accordance with Non-Patent Document 2, as well as the deletion error detection model and insertion error detection model generated by the learning device 1, to determine the probability of an error occurring for each token in N candidate sentences estimated by speech recognition, correct the scores of the N candidate sentences to select one candidate sentence, calculate a confidence score for each token by error type for the recognition result of the one candidate sentence, and generate an error-labeled recognition result that clearly indicates the location and type of the error.

[0066] The speech recognition device 2 includes a speech recognition unit 30, memory units 31 and 34, an error detection unit 32, and a recognition result correction unit 33. The memory unit 31 stores a preset conventional substitution error detection model, as well as a deletion error detection model and an insertion error detection model generated by the learning device 1.

[0067] The speech recognition device 2 receives speech data in the speech recognition unit 30, performs speech data recognition processing using a speech recognition model (step S601), and generates N pieces of candidate sentence data and a score S ASR This candidate sentence data may contain speech recognition errors.

[0068] The score S of this candidate sentence for speech recognitionASR A higher value indicates a lower possibility that a speech recognition error has occurred, and a lower value indicates a higher possibility that a speech recognition error has occurred.

[0069] The speech recognition device 2 performs an error detection process using a substitution error detection model (step S602-1), an error detection process using a deletion error detection model (step S602-2), and an error detection process using an insertion error detection model (step S602-3) in the error detection unit 32, thereby obtaining a score S that reflects the substitution errors, deletion errors, and insertion errors for each of the N candidate sentence data. disc Request.

[0070] Then, the speech recognition device 2 calculates a score S for each of the N pieces of candidate sentence data in the error detection unit 32. ASR Score S disc Based on the score S, one candidate sentence data is selected as the recognition result, and an error-added recognition result generation process (step S603) is performed, such as calculating a reliability score for each token of the recognition result, to generate an error-added recognition result.

[0071] In the speech recognition device 2, the recognition result correction unit 33 performs a correction process (step S604) on the recognition result with errors in accordance with a user operation, and generates and outputs text corresponding to the speech data.

[0072] Furthermore, the speech recognition device 2 stores the recognition result with errors for the portion corrected by the user operation in the recognition result correction unit 33, and the position and type of the error, as a DB (database) in the storage unit 34. This DB is used as data for speech recognition results for fine tuning in the learning device 1 shown in Fig. 1, and can generate an error detection model that continuously improves the error detection accuracy.

[0073] 7 is a flowchart showing an example of processing by the voice recognition device 2. Hereinafter, an example of processing by the voice recognition device 2 will be described with reference to FIGS.

[0074] <Voice recognition unit 30> The speech recognition unit 30 inputs speech data for which text of the speech recognition result is to be generated (step S701). Then, the speech recognition unit 30 estimates candidate sentence data using a preset speech recognition model and obtains a score S ASR The N candidate sentences and the N scores S ASR The speech recognition unit 30 identifies N pieces of candidate sentence data and N scores S ASR to the error detection unit 32.

[0075] <Error detection unit 32> The error detection unit 32 receives N candidate sentence data and N scores S from the speech recognition unit 30. ASR and reads out a replacement error detection model, a deletion error detection model and an insertion error detection model from the storage unit 31. Then, the error detection unit 32 generates an error-added recognition result (data in which the position and type of an error have been added to the candidate sentence data) by the processes of steps S703 to S708 described later, and outputs the error-added recognition result to the recognition result correction unit 33. The error detection unit 32 will be described in detail below with reference to Figs. 8 to 10.

[0076] 8 is a block diagram showing an example of the configuration of the error detection unit 32. The error detection unit 32 includes a tokenization unit 40, an error detection processing unit 41, a score correction unit 42, a recognition result selection unit 43, and a recognition result error processing unit 44.

[0077] (Tokenization unit 40) The tokenization unit 40 receives the N pieces of candidate sentence data from the speech recognition unit 30, tokenizes each of the N pieces of candidate sentence data, and generates a token sequence of the candidate sentence. The tokenization unit 40 then outputs the token sequences of the N candidate sentences to the error detection processing unit 41 and the recognition result selection unit 43.

[0078] (Error detection processing unit 41) The error detection processing unit 41 receives the token strings of the N candidate sentences from the tokenization unit 40, and reads out the substitution error detection model, the deletion error detection model, and the insertion error detection model from the storage unit 31.

[0079] The error detection processing unit 41 uses the substitution error detection model, the deletion error detection model, and the insertion error detection model to detect a data string D j (Step S703). j is the error probability for each of the multiple tokens that make up the candidate sentence data.

[0080] Here, j indicates the type of each model (each error) of the substitution error detection model, the deletion error detection model, and the insertion error detection model, and j = 1, 2, 3. D1 is a substitution error data string estimated using the substitution error detection model, D2 is a deletion error data string estimated using the deletion error detection model, and D3 is an insertion error data string estimated using the insertion error detection model.

[0081] The error detection processing unit 41 detects a data string D j Based on this, a score S is calculated by integrating substitution, deletion and insertion errors. disc is calculated (step S704).

[0082] The error detection processing unit 41 calculates N scores S disc to the score correction unit 42, and a data string D for each model in each of the token strings of the N candidate sentences is output. j to the recognition result selection unit 43.

[0083] 9 is a block diagram showing an example of the configuration of the error detection processing unit 41. This error detection processing unit 41 includes a substitution error detection unit 50, a deletion error detection unit 51, an insertion error detection unit 52, and a scoring unit 53.

[0084] The substitution error detection unit 50 inputs the token strings of the N candidate sentences from the tokenization unit 40, and reads out the substitution error detection model from the storage unit 31. Then, the substitution error detection unit 50 estimates the data string D1 of the substitution error for each of the token strings of the N candidate sentences using the substitution error detection model.

[0085] The substitution error detection unit 50 outputs the data string D1 containing the N substitution errors to the scoring unit 53 and the recognition result selection unit 43.

[0086] The deletion error detection unit 51 inputs the token strings of the N candidate sentences from the tokenization unit 40, and reads out the deletion error detection model from the storage unit 31. Then, the deletion error detection unit 51 estimates the data string D2 of the deletion error for each of the token strings of the N candidate sentences, using the deletion error detection model.

[0087] The deletion error detection unit 51 outputs the data string D2 containing the N deletion errors to the scoring unit 53 and the recognition result selection unit 43.

[0088] The insertion error detection unit 52 inputs the token strings of the N candidate sentences from the tokenization unit 40, and reads out the insertion error detection model from the storage unit 31. Then, the insertion error detection unit 52 estimates the data string D3 of the insertion error for each of the token strings of the N candidate sentences, using the insertion error detection model.

[0089] The insertion error detection unit 52 outputs a data string D3 containing N insertion errors to the scoring unit 53 and the recognition result selection unit 43.

[0090] The scoring unit 53 inputs a data string D1 containing N substitution errors from the substitution error detection unit 50, a data string D2 containing N deletion errors from the deletion error detection unit 51, and further inputs a data string D3 containing N insertion errors from the insertion error detection unit 52.

[0091] The scoring unit 53 calculates a score S, which is an integration of substitution errors, deletion errors, and insertion errors, for each of the token strings of the N candidate sentences, using the following formula: disc Then, the scoring unit 53 calculates the N scores S disc to the score correction unit 42.

number

[0092] i indicates the i-th position of the token in the candidate sentence data, and j indicates the type of each model (each error) of the substitution error detection model, the deletion error detection model, and the insertion error detection model, as described above. j denotes the preset hyperparameters in the jth model, and D j (i) Let y denote the output (error probability) of the jth model for the token in the i-th position, y denotes the token sequence of the candidate sentence, and L denotes the number of tokens in the candidate sentence.

[0093] This score S disc is the absolute value |S disc The larger the |, the more likely it is that an error has occurred for that token, and the absolute value |S disc The smaller | is and the closer it is to 0, the less likely it is that an error has occurred for that token.

[0094] (Score correction unit 42) Returning to FIG. 8, the score correction unit 42 receives the N scores S disc The speech recognition unit 30 receives N scores S ASR Enter.

[0095] The score correction unit 42 calculates the speech recognition score S estimated in step S702 for each of the token sequences of the N candidate sentences. ASR The score S calculated in step S704 disc The score S is corrected by the following formula (step S705):ASR , score S disc and the length of the token sequence of the candidate sentence. The score correction unit 42 outputs the N scores S to the recognition result selection unit 43.

number

[0096] β is a preset hyperparameter, and β|y| is a correction term for the length of the token sequence of the candidate sentence, which takes a larger value the longer the token sequence of the candidate sentence is, and takes a smaller value the shorter it is. In other words, the longer the token sequence of the candidate sentence is, the higher the score S will be, and the shorter the token sequence of the candidate sentence is, the lower the score S will be.

[0097] The higher the score S, the lower the possibility that an error has occurred, and the lower the score (closer to 0), the more likely it is that an error has occurred.

[0098] As a result, for each of the N candidate sentence data, a speech recognition score S ASR Score S disc etc., to obtain a score S that takes into account not only speech recognition errors, but also token substitution errors, deletion errors, and insertion errors, as well as the length of the candidate sentence data.

[0099] (Recognition result selection unit 43) The recognition result selection unit 43 receives N scores S from the score correction unit 42, token sequences of the N candidate sentences from the tokenization unit 40, and a data sequence D for each model in each of the token sequences of the N candidate sentences from the error detection processing unit 41. j The speech recognition unit 30 outputs N scores S ASR Enter the following respectively.

[0100] The recognition result selection unit 43 identifies the highest score S among the N scores S, and selects one candidate sentence data corresponding to the highest score S among the N candidate sentence data. In other words, the recognition result selection unit 43 selects the token sequences of the N candidate sentences, the data sequence D for each model for each of the N candidate sentence data, j , and N scores S ASR Among them, the token sequence of one candidate sentence corresponding to the highest score S, and the data sequence D for each model in the token sequence of one candidate sentence j , and one score S ASR (Step S706).

[0101] The recognition result selection unit 43 selects a token sequence of one candidate sentence as a recognition result, and selects a data sequence D for each model in the token sequence of one candidate sentence. j Let D1, D2, and D3 be the data sequences of the recognition results, and one score S ASR The recognition result score S ASR to the recognition result error processing unit 44.

[0102] (Recognition result error processing unit 44) The recognition result error processing unit 44 receives the recognition result, the data strings D1, D2, and D3 of the recognition result, and the score S of the recognition result from the recognition result selection unit 43. ASR Then, the recognition result error processing unit 44 inputs the data strings D1, D2, and D3 of the recognition result and the score S of the recognition result. ASR Based on this, a reliability score for each token according to the error type is calculated (step S707).

[0103] Here, when the recognition result error processing unit 44 calculates the reliability score for each token by error type, the data strings D1, D2, and D3 of the recognition result (data string D j (i) ) is used as a confidence level indicating that no j-type error (substitution error, deletion error, insertion error) has occurred for the i-th token of the recognition result.

[0104] The recognition result error processing unit 44 generates an error-added recognition result indicating the position and type of the error by threshold judgment of the confidence score for each token by error type in the recognition result (step S708). Then, the recognition result error processing unit 44 outputs the error-added recognition result to the recognition result correction unit 33.

[0105] 10 is a block diagram showing an example of the configuration of the recognition result error processing unit 44. This recognition result error processing unit 44 includes a data string processing unit 60, a reliability score calculation unit 61, a threshold determination unit 62, and an error processing unit 63.

[0106] The data string processing unit 60 receives the data strings D1, D2, and D3 of the recognition results from the recognition result selection unit 43. The data string processing unit 60 then calculates (1-D1), (1-D2), and (1-D3), and outputs (1-D1), (1-D2), and (1-D3) to a confidence score calculation unit 61.

[0107] The reliability score calculation unit 61 receives (1-D1), (1-D2), and (1-D3) from the data string processing unit 60, and also receives the recognition result score S from the recognition result selection unit 43. ASR Then, the reliability score calculation unit 61 calculates the score S of the recognition result based on (1-D1), (1-D2), and (1-D3). ASR The weighted sum of these is calculated and used as the confidence score for each token for each error type in the recognition result.

[0108] As a result, in the recognition result, a confidence score for each token reflecting a substitution error, a confidence score for each token reflecting a deletion error, and a confidence score for each token reflecting an insertion error are obtained.

[0109] A higher confidence score for each token indicates a lower possibility that an error has occurred for that token (higher confidence), and a lower confidence score indicates a higher possibility that an error has occurred for that token (lower confidence).

[0110] The confidence score calculation unit 61 outputs the confidence score for each token for each error type in the recognition result to the threshold determination unit 62 .

[0111] The threshold determination unit 62 inputs the confidence score for each token for each error type in the recognition result from the confidence score calculation unit 61. Then, the threshold determination unit 62 performs threshold determination for the confidence score for each token for each error type (for each of substitution error, deletion error, and insertion error), and generates a binary data string consisting of binary data for each token indicating the presence or absence of an error or a position.

[0112] Specifically, the threshold determination unit 62 compares the reliability score with a preset threshold for each token, and when it is determined that the reliability score exceeds the threshold, it determines that the token has not had an error (is reliable). On the other hand, when it is determined that the reliability score is equal to or less than the threshold, it determines that the token has had an error (is not reliable).

[0113] The threshold determination unit 62 generates a binary data string in which the points where it is determined that an error has occurred are set to "1" and the points where it is determined that no error has occurred are set to "0."

[0114] This generates a binary data sequence of one-hot vectors that sets the value of "1" where a substitution error occurs, a binary data sequence of one-hot vectors that sets the value of "1" where a deletion error occurs, and a binary data sequence of one-hot vectors that sets the value of "1" where an insertion error occurs.

[0115] The threshold determination unit 62 outputs a binary data string for each error type in the recognition result to an error processing unit 63.

[0116] The error processing unit 63 receives binary data strings for each type of error in the recognition result from the threshold determination unit 62, and also receives the recognition result (the token string of the selected candidate sentence) from the recognition result selection unit 43.

[0117] The error processing unit 63 generates an error-added recognition result that indicates the position and type of the error for the candidate sentence data by adding a color, symbol, or the like indicating the occurrence of an error as a correction candidate portion to the portion of the token of the recognition result that corresponds to the portion of "1" (the portion where the error occurred) included in the binary data string for each error type. Then, the error processing unit 63 outputs the error-added recognition result to the recognition result correction unit 33.

[0118] <Recognition result correction unit 33> 5 and 7, the recognition result correction unit 33 receives the recognition result with errors from the error processing unit 63 of the recognition result error processing unit 44 provided in the error detection unit 32, and displays the recognition result with errors on a screen. Then, the recognition result correction unit 33 determines whether or not a correction has been input, according to an operation by the user who has confirmed the recognition result with errors displayed on the screen (step S709).

[0119] If the recognition result correction unit 33 determines in step S709 that correction input has been made (step S709: Y), it performs correction processing on the candidate sentence data of the recognition result with errors in accordance with user operation, and outputs the corrected recognition result sentence as text corresponding to the voice data input by the voice recognition device 2 (step S710).

[0120] As a result, tokens with substitution errors, tokens with deletion errors, and tokens with insertion errors contained in the recognition result with errors are corrected, and a sentence with the recognition result from which substitution errors etc. have been removed is obtained.

[0121] Then, the recognition result correction unit 33 stores the error-attached recognition results of the portions where the deletion errors and deletion error corrections have been performed by the user operation, and the positions and types of the errors as a DB in the storage unit 34 (step S711), and the process proceeds to step S712.

[0122] This DB is used as data of speech recognition results for fine tuning by the learning device 1 shown in Fig. 1. As a result, a highly accurate deletion error detection model and insertion error detection model are generated.

[0123] Fig. 11(a) is a diagram showing an example of a correct recognition result, and Fig. 11(b) is a diagram showing an example of a recognition result with errors. In Fig. 11(b), the square frame indicated by e1 indicates that a substitution error has occurred, the square frame and "^" indicated by e2 indicate that a deletion error has occurred, and the square frame and "-" indicated by e3 indicate that an insertion error has occurred.

[0124] As shown in Figure 11(a), an example of a correct recognition result corresponding to speech data is, "The large number of vehicles that had been stranded on a section of the outbound lanes of the Shin-Meishin Expressway heading from Mie Prefecture to Shiga Prefecture was cleared at around 8:00 a.m. on the 26th, more than a day after the road closure."

[0125] It is assumed that the speech recognition unit 30 and the error detection unit 32 process the speech data to generate a recognition result with errors as shown in FIG. 11(b).

[0126] The recognition result correction unit 33 then displays the recognition result with errors as shown in Figure 11(b) on the screen, and allows the user to recognize the location and type of the error by using a square frame, "^", and "-" symbols as shown in e1, e2, and e3.

[0127] The recognition result correction unit 33, in accordance with the user's operation, corrects "superstition", where a substitution error indicated by e1 has occurred, to the correct "Meishin", and corrects the "^" part, where a deletion error indicated by e2 has occurred, to "wa," so that the sentence becomes correct. Also, in accordance with the user's operation, the recognition result correction unit 33 deletes the "-" part ("i"), where an insertion error indicated by e3 has occurred, thereby correcting the insertion error.

[0128] Returning to FIG. 5 and FIG. 7, the recognition result correction unit 33 outputs the sentence of the corrected recognition result (for example, the example in FIG. 11(a)) as text corresponding to the speech data.

[0129] On the other hand, if the recognition result correction unit 33 determines in step S709 that there is no correction input (step S709: N), the recognition result correction unit 33 proceeds to step S712. In this case, the recognition result correction unit 33 outputs the sentence of the recognition result with errors as text, assuming that the recognition result with errors generated in step S708 is error-free.

[0130] The voice recognition device 2 determines whether or not processing of all voice data has been completed (step S712), and if it determines that processing of all voice data has not been completed (step S712: N), it proceeds to step S701 and processes the next voice data.

[0131] On the other hand, if the voice recognition device 2 determines that the processing for all the voice data has been completed (step S712: Y), it ends the processing.

[0132] As described above, according to the speech recognition device 2 of the embodiment of the present invention, the speech recognition unit 30 inputs speech data, estimates candidate sentence data using a speech recognition model, and calculates a score S ASR The N candidate sentences and the N scores S ASR Output.

[0133] The error detection unit 32 detects the data string D using the conventional substitution error detection model, as well as the deletion error detection model and the insertion error detection model generated by the learning device 1 for each of the N candidate sentence data. j and estimate the score S by integrating the substitution error etc. disc Calculate the speech recognition score S ASR Score S disc Then, the score S after correction is calculated.

[0134] The error detection unit 32 identifies the highest score S among the N scores S, selects one candidate sentence data corresponding to the highest score S as a recognition result, and outputs the recognition result data strings D1, D2, and D3 and the recognition result score S. ASRThe error detection unit 32 then generates an error-marked recognition result by performing a threshold determination on the reliability scores.

[0135] The recognition result correction unit 33 corrects the recognition result with errors in accordance with user operation, outputs the corrected recognition result sentence as text corresponding to the voice data, and stores the recognition result with errors for the corrected part, as well as data on the position and type of the errors, in the memory unit 34 as a DB.

[0136] This results in a score S of the candidate sentence data estimated by speech recognition. ASR is the score S estimated in terms of substitution, deletion and insertion errors for each token. disc etc., and the recognition result of one candidate sentence data is selected using score S, so that the recognition result becomes a highly accurate text.

[0137] That is, in addition to reducing substitution errors that may occur in speech recognition, deletion errors and insertion errors can also be reduced, thereby improving the accuracy of speech recognition.

[0138] Furthermore, by presenting the user with the recognition result with errors, which indicates the location and type of the errors, the user can recognize the location and type of the errors, which makes it easier to correct the errors. This makes it possible to improve the efficiency of the subtitle correction work.

[0139] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept thereof.

[0140] In the above embodiment, the recognition result correction unit 33 of the speech recognition device 2 performs a correction process on the candidate sentence data of the recognition result with an error in accordance with a user operation, and outputs the sentence of the recognition result after the correction as a text corresponding to the speech data input by the speech recognition device 2.

[0141] In response to this, the recognition result selection unit 43 provided in the error detection unit 32 of the speech recognition device 2 may be configured to, when identifying the highest score S among the N scores S, output one candidate sentence data corresponding to the highest score S as text corresponding to the speech data input by the speech recognition device 2.

[0142] A normal computer can be used as the hardware configuration of the learning device 1 and the voice recognition device 2 according to the embodiment of the present invention. The learning device 1 and the voice recognition device 2 are configured by a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, an interface, etc.

[0143] Each function of the splitting unit 10, tokenization unit 11, deletion error processing unit 12, insertion error processing unit 13, learning units 14 and 15, data generation unit 16 and memory unit 17 provided in the learning device 1 is realized by having a CPU execute a program that describes these functions.

[0144] Furthermore, the voice recognition unit 30, the storage units 31, 34, the error detection unit 32 and the recognition result correction unit 33 provided in the voice recognition device 2 are also realized by causing the CPU to execute a program describing these functions.

[0145] These programs are stored in the storage medium and are read and executed by the CPU. These programs can also be distributed by storing them in a storage medium such as a magnetic disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), or a semiconductor memory, and can also be transmitted and received via a network. [Explanation of symbols]

[0146] 1 Learning device 2. Voice recognition device 10 divisions 11,40 Tokenization section 12 Deletion error handling section 13 Insertion error processing section 14,15 Learning Department 16 Data Generation Section 17,31,34 Storage section 20,22 Model 21,23 Update section 30 Voice Recognition Unit 32 Error detection section 33 Recognition result correction section 41 Error detection processing unit 42 Score Correction Section 43 Recognition result selection unit 44 Recognition result error processing section 50 Substitution error detection unit 51 Deletion error detection unit 52 Insertion Error Detection Unit 53 Scoring section 60 Data Column Processing Unit 61 Reliability score calculation unit 62 Threshold judgment unit 63 Error processing section S ASR ,S disc ,Score D j ,D1,D2,D3 data string

Claims

1. 1. A learning device for generating an error detection model for detecting errors in speech recognition, comprising: a tokenizer that tokenizes sentence data for pre-training and generates a token sequence of the sentence; a deletion error processing unit that generates a deletion error by randomly deleting tokens from the token sequence of the sentence generated by the tokenization unit, generates a token sequence of the sentence including the deletion error, and generates a binary data sequence that indicates the presence or absence of the deletion error corresponding to the token sequence of the sentence including the deletion error; a first learning unit that uses the binary data sequence generated by the deletion error processing unit as correct answer data and a token sequence of a sentence including the deletion error generated by the deletion error processing unit and the correct answer data to learn a relationship between the token sequence of the sentence and a data sequence indicating a probability of a deletion error for each token corresponding to the token sequence of the sentence; an insertion error processing unit that generates an insertion error by randomly inserting a token into the token sequence of the sentence generated by the tokenization unit, generates a token sequence of the sentence including the insertion error, and generates a binary data sequence that indicates the presence or absence of the insertion error corresponding to the token sequence of the sentence including the insertion error; a second learning unit that uses the binary data sequence generated by the insertion error processing unit as correct answer data and a token sequence of a sentence including the insertion error generated by the insertion error processing unit and the correct answer data to learn a relationship between the token sequence of the sentence and a data sequence indicating a probability of an insertion error for each token corresponding to the token sequence of the sentence, The first learning unit further includes: obtaining a token sequence of a sentence including a deletion error generated by a predetermined speech recognition process, and using a binary data sequence indicating the presence or absence of the deletion error and corresponding to the token sequence of the sentence including the deletion error as correct answer data, performing fine tuning using the token sequence of the sentence including the deletion error and the correct answer data, thereby generating a deletion error detection model as a trained model; The second learning unit further includes: A learning device characterized by acquiring a token sequence of a sentence including an insertion error generated by a predetermined speech recognition process, using a binary data sequence indicating the presence or absence of the insertion error and the position thereof corresponding to the token sequence of the sentence including the insertion error as correct answer data, performing fine tuning using the token sequence of the sentence including the insertion error and the correct answer data, and generating an insertion error detection model as a trained model.

2. A voice recognition device receives voice data and generates text corresponding to the voice data through voice recognition processing, A plurality of candidate sentence data are estimated from the speech data using a predetermined speech recognition model, and a score S ASR N pieces (N is an integer of 2 or more) of candidate sentence data in descending order of the score S ASR A speech recognition unit that outputs For each of the N pieces of candidate sentence data output by the speech recognition unit, tokenize the candidate sentence data to generate a token string of the candidate sentence; A data sequence indicating a probability of an error for each token corresponding to the token sequence of the candidate sentence is estimated from the token sequence of the candidate sentence by using a preset substitution error detection model, and the deletion error detection model and the insertion error detection model generated by the learning device according to claim 1, thereby obtaining a data sequence for each model, and calculating a score S based on the data sequence for each model. disc Seeking The score S corresponding to the candidate sentence data output by the speech recognition unit ASR The score S disc The score S is calculated by correcting the score by an error detection unit that selects one candidate sentence data from the N candidate sentence data as a recognition result based on the score S for each of the N candidate sentence data, and outputs the recognition result as a text corresponding to the speech data; A voice recognition device comprising:

3. 3. The speech recognition device according to claim 2, Further, a recognition result correction unit is provided, The error detection unit When selecting the recognition result based on the score S for each of the N candidate sentence data, a data sequence for each model corresponding to the one candidate sentence data is further selected as a data sequence for each model of the recognition result, and the score S corresponding to the one candidate sentence data is ASR The score S of the recognition result ASR Select as the probability of an error for each token included in the data string for each model of the recognition result, and the score S of the recognition result ASR Calculating the probability of occurrence of each of substitution errors, deletion errors, and insertion errors for each token in the recognition result as a confidence score based on the determining positions in the recognition result where the substitution errors, the deletion errors, and the insertion errors have occurred based on the reliability scores of the substitution errors, the deletion errors, and the insertion errors for each token in the recognition result, and generating an error-labeled recognition result indicating the positions and types of the errors; The recognition result correction unit a recognition result including errors generated by the error detection unit is displayed on a screen, the recognition result including errors is corrected in accordance with user operation, and the corrected recognition result sentence is output as text corresponding to the speech data.

4. 3. The speech recognition device according to claim 2, The error detection unit a tokenizer for tokenizing each of the N pieces of candidate sentence data and generating a token string of the candidate sentence; Using the replacement error detection model, the data sequence is estimated as a replacement error data sequence from the token sequence of the candidate sentence generated by the tokenization unit, using the deletion error detection model, the data sequence is estimated as a deletion error data sequence from the token sequence of the candidate sentence, and using the insertion error detection model, the data sequence is estimated as an insertion error data sequence from the token sequence of the candidate sentence, and a score S of the candidate sentence data is calculated based on the probability of errors for each of the tokens included in the replacement error data sequence, the deletion error data sequence, and the insertion error data sequence. disc An error detection processing unit for determining The score S of the candidate sentence data ASR The score S obtained by the error detection processing unit disc a score correction unit that calculates a score S of the candidate sentence data by correcting the score S by the above-mentioned a recognition result selection unit that selects one candidate sentence data from the N candidate sentence data as the recognition result based on the score S for each of the N candidate sentence data obtained by the score correction unit, and outputs the recognition result as a text corresponding to the speech data; A voice recognition device comprising:

5. 4. The speech recognition device according to claim 3, The error detection unit a tokenizer for tokenizing each of the N pieces of candidate sentence data and generating a token string of the candidate sentence; Using the replacement error detection model, the data sequence is estimated as a replacement error data sequence from the token sequence of the candidate sentence generated by the tokenization unit, using the deletion error detection model, the data sequence is estimated as a deletion error data sequence from the token sequence of the candidate sentence, and using the insertion error detection model, the data sequence is estimated as an insertion error data sequence from the token sequence of the candidate sentence, and a score S of the candidate sentence data is calculated based on the probability of errors for each of the tokens included in the replacement error data sequence, the deletion error data sequence, and the insertion error data sequence. disc An error detection processing unit for determining The score S of the candidate sentence data ASR The score S obtained by the error detection processing unit disc a score correction unit that calculates a score S of the candidate sentence data by correcting the score S by the above-mentioned Based on the score S for each of the N pieces of candidate sentence data obtained by the score correction unit, one piece of candidate sentence data is selected from the N pieces of candidate sentence data as the recognition result, and the replacement error data sequence, the deletion error data sequence, and the insertion error data sequence corresponding to the one piece of candidate sentence data, as well as the score S ASR The replacement error data string of the recognition result, the deletion error data string of the recognition result, the insertion error data string of the recognition result, and the score S of the recognition result are ASR a recognition result selection unit for selecting The probability of an error for each token included in the replacement error data sequence of the recognition result selected by the recognition result selection unit and the score S of the recognition result ASR The probability of occurrence of a substitution error for each token in the recognition result is calculated as a reliability score of the substitution error based on the above, and the probability of an error for each token included in the deletion error data string of the recognition result and the score S of the recognition result are calculated as a reliability score of the substitution error. ASR The probability of the deletion error occurring for each token in the recognition result is calculated as a reliability score of the deletion error based on the above, and the probability of the error for each token included in the insertion error data string of the recognition result and the score S of the recognition result are calculated as a reliability score of the deletion error. ASR a confidence score calculation unit that calculates a probability of occurrence of the insertion error for each token in the recognition result based on the a threshold determination unit that generates a binary data sequence indicating the position of the presence or absence of the substitution error corresponding to the token sequence of the recognition result by threshold processing on the reliability score of the substitution error for each token in the recognition result calculated by the reliability score calculation unit, generates a binary data sequence indicating the position of the presence or absence of the deletion error corresponding to the token sequence of the recognition result by threshold processing on the reliability score of the deletion error for each token in the recognition result, and generates a binary data sequence indicating the position of the presence or absence of the insertion error corresponding to the token sequence of the recognition result by threshold processing on the reliability score of the insertion error for each token in the recognition result; an error processing unit that generates the recognition result with errors based on each of the binary data strings generated by the threshold determination unit; A voice recognition device comprising:

6. A computer constituting a learning device for generating an error detection model for detecting errors in speech recognition, A tokenizer that tokenizes the pre-training sentence data and generates a token sequence of the sentence; a deletion error processing unit that generates a deletion error by randomly deleting tokens from the token sequence of the sentence generated by the tokenization unit, generates a token sequence of the sentence including the deletion error, and generates a binary data sequence indicating the presence or absence of the deletion error corresponding to the token sequence of the sentence including the deletion error; a first learning unit that uses the binary data sequence generated by the deletion error processing unit as correct answer data, and learns a relationship between a token sequence of a sentence and a data sequence indicating a probability of a deletion error for each token corresponding to the token sequence of the sentence, by using the token sequence of the sentence including the deletion error generated by the deletion error processing unit and the correct answer data; an insertion error processing unit that generates an insertion error by randomly inserting a token into the token sequence of the sentence generated by the tokenization unit, generates a token sequence of the sentence including the insertion error, and generates a binary data sequence that indicates the presence or absence of the insertion error corresponding to the token sequence of the sentence including the insertion error; and a second learning unit that functions as a second learning unit that learns a relationship between a token sequence of a sentence and a data sequence indicating a probability of an insertion error for each token, the data sequence corresponding to the token sequence of the sentence, using the binary data sequence generated by the insertion error processing unit as correct answer data and the token sequence of the sentence including the insertion error generated by the insertion error processing unit and the correct answer data; The first learning unit further includes: obtaining a token sequence of a sentence including a deletion error generated by a predetermined speech recognition process, and using a binary data sequence indicating the presence or absence of the deletion error and corresponding to the token sequence of the sentence including the deletion error as correct answer data, performing fine tuning using the token sequence of the sentence including the deletion error and the correct answer data, thereby generating a deletion error detection model as a trained model; The second learning unit further includes: A program comprising: acquiring a token sequence of a sentence including an insertion error generated by a predetermined speech recognition process; using a binary data sequence indicating the presence or absence of the insertion error and the position corresponding to the token sequence of the sentence including the insertion error as correct answer data; performing fine tuning using the token sequence of the sentence including the insertion error and the correct answer data; and generating an insertion error detection model as a trained model.

7. A computer constituting a voice recognition device which receives voice data and generates text corresponding to the voice data through voice recognition processing, A plurality of candidate sentence data are estimated from the speech data using a predetermined speech recognition model, and a score S ASR N pieces (N is an integer of 2 or more) of candidate sentence data in descending order of the score S ASR A speech recognition unit that outputs For each of the N pieces of candidate sentence data output by the speech recognition unit, tokenize the candidate sentence data to generate a token string of the candidate sentence; A data sequence indicating a probability of an error for each token corresponding to the token sequence of the candidate sentence is estimated from the token sequence of the candidate sentence by using a preset substitution error detection model, and the deletion error detection model and the insertion error detection model generated by the learning device according to claim 1, thereby obtaining a data sequence for each model, and calculating a score S based on the data sequence for each model. disc Seeking The score S corresponding to the candidate sentence data output by the speech recognition unit ASR The score S disc The score S is calculated by correcting the score by A program for functioning as an error detection unit that selects one candidate sentence data from the N candidate sentence data as a recognition result based on the score S for each of the N candidate sentence data, and outputs the recognition result as text corresponding to the speech data.

Images