Speech recognition device, speech recognition method, and speech recognition program

Abstract

To improve voice recognition performance.SOLUTION: A voice recognition device 10 comprises a recognition unit 20B, and an output control unit 20D. The recognition unit 20B generates a voice recognition result from voice data, by using topic data 31 in which a topic appearance probability for each lapse time of each of a plurality of topics changing according to the lapse time is registered in advance, a recognition model 32 which is associated with each of the plurality of topics, and includes an adaptation language model 32C for outputting the voice recognition result from the voice data, and the lapse time. The output control unit 20D outputs the voice recognition result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] An embodiment of the present invention relates to a speech recognition device, a speech recognition method, and a speech recognition program. [Background technology]

[0002] Subtitle generation technology that utilizes speech recognition technology is known. Subtitle generation technology is used, for example, in the field of subtitling for television programs and the like. However, speech recognition can sometimes result in erroneous recognition due to background sounds, slip-ups, speech containing technical terms, and the like. For this reason, users are encouraged to correct the speech recognition results and output the corrected speech recognition results (see, for example, Non-Patent Document 1). Various techniques have been considered for improving speech recognition performance. For example, a speech recognition device has been disclosed that estimates a topic from speech recognition results and uses the estimated topic to adjust the language score used for speech recognition (see, for example, Patent Document 1).

[0003] Patent Document 1 discloses a technology for extracting text corresponding to a speech recognition result from topic information such as a TV program script or progress schedule, and obtaining text related to the extracted text from external data on a website. Patent Document 1 also discloses a technology for adjusting a language score used in speech recognition using text obtained from external data.

[0004] However, conventional technology adjusts language scores by obtaining text related to speech recognition results from external data on websites, making it difficult to apply to real-time subtitling for news programs and the like, and in some cases it has not been possible to improve speech recognition performance. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 5723711 [Non-patent literature]

[0006] [Non-Patent Document 1] NHK Science and Technology Research Institute, "NHK Science and Technology Research Institute R&D 2020 Summer Issue Commentary 02 Latest Trends in Automatic Subtitling for Live Broadcast Programs," [online], Special Issue on Image and Audio Processing Technology for Program Production Support, Summer 2020 Issue No. 182, [Retrieved April 28, 2022], Internet<https: / / www.nhk.or.jp / strl / publica / rd / 182 / 3.html> Summary of the Invention [Problem to be solved by the invention]

[0007] An object of the present invention is to provide a speech recognition device, a speech recognition method, and a speech recognition program that can improve speech recognition performance. [Means for solving the problem]

[0008] A speech recognition device according to an embodiment includes a recognition unit and an output control unit. The recognition unit generates a speech recognition result from speech data using topic data in which a topic appearance probability for each elapsed time for each of a plurality of topics that changes over time is registered in advance, a recognition model that is linked to each of the plurality of topics and includes an adapted language model for outputting a speech recognition result from speech data, and the elapsed time. The output control unit outputs the speech recognition result. The output control unit outputs the speech recognition result and the topic used for speech recognition of the speech recognition result to a display unit, and outputs the topic whose topic appearance probability is equal to or greater than a threshold to the display unit in a different display format from the topic whose topic appearance probability is less than the threshold. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a functional block diagram of a voice recognition device. [Figure 2A] Schematic diagram of the data structure of topic data. [Figure 2B] Schematic diagram of the data structure of topic data. [Figure 3A] FIG. [Figure 3B] Schematic diagram of a correction converter. [Figure 4A]FIG. 10 is a schematic diagram of the data structure of revision history data. [Figure 4B] FIG. 10 is a schematic diagram of the data structure of revision history data. [Figure 5] Schematic diagram of a display screen. [Figure 6] FIG. [Figure 7] 10 is a flowchart showing the flow of information processing. [Figure 8] FIG. 2 is a diagram showing an example of a hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0010] The speech recognition device, speech recognition method, and speech recognition program will be described in detail below with reference to the accompanying drawings.

[0011] FIG. 1 is a functional block diagram of an example of a voice recognition device 10 according to this embodiment.

[0012] The voice recognition device 10 is an information processing device that executes voice recognition processing on input voice data and outputs the voice recognition result.

[0013] The speech recognition device 10 includes a control unit 20, a storage unit 30, a UI (user interface) unit 40, and a communication unit 50. The control unit 20, the storage unit 30, the UI unit 40, and the communication unit 50 are connected to each other so as to be able to communicate with each other via a bus or the like.

[0014] The storage unit 30 stores various types of information. In this embodiment, the storage unit 30 stores topic data 31, a recognition model 32, a correction model 33, and correction history data 34. Details of these data and models will be described later.

[0015] The UI unit 40 includes a display unit 40A and an input unit 40B. The display unit 40A displays various types of information. The display unit 40A is, for example, a display. The input unit 40B accepts operation instructions from a user. The input unit 40B is, for example, a pointing device such as a mouse or a touchpad, a keyboard, etc. The UI unit 40 may be a touch panel in which the display unit 40A and the input unit 40B are integrated. The UI unit 40 may also be configured to further include an output function using a speaker, a projection device, etc.

[0016] The communication unit 50 is a communication interface for communicating with an external information processing device of the speech recognition device 10. For example, the communication unit 50 communicates with an external information processing device or electronic device via a wired network such as Ethernet (registered trademark), a wireless network such as Wi-Fi (Wireless Fidelity) or Bluetooth (registered trademark), or the like.

[0017] The control unit 20 executes information processing in the speech recognition device 10. The control unit 20 includes an acquisition unit 20A, a recognition unit 20B, a correction unit 20C, an output control unit 20D, a memory control unit 20E, an update unit 20F, and a topic data update unit 20G.

[0018] The acquisition unit 20A, recognition unit 20B, correction unit 20C, output control unit 20D, storage control unit 20E, update unit 20F, and topic data update unit 20G are realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), i.e., by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0019] In addition, the information stored in the storage unit 30 and at least a part of the above-mentioned units included in the control unit 20 may be configured to be mounted on an external information processing device communicably connected to the voice recognition device 10.

[0020] The acquisition unit 20A acquires voice data. The voice data is data of voice. The voice may be, for example, a voice spoken by a user, a voice output from various devices via a speaker, etc. In this embodiment, a description will be given assuming that the voice data is data of voice spoken by a user.

[0021] For example, the acquisition unit 20A acquires audio data of continuous audio in time series by sequentially digitizing real-time audio collected by a microphone (not shown) connected to the control unit 20. The acquisition unit 20A may acquire audio data by reading audio data from an audio file that has been recorded in advance and stored in the storage unit 30 or the like.

[0022] The recognition unit 20B generates a speech recognition result from the speech data acquired by the acquisition unit 20A, using the topic data 31, the recognition model 32, and the elapsed time.

[0023] The topic data 31 is data in which the topic appearance probability for each elapsed time for each of a plurality of topics that change over time is registered in advance.

[0024] A topic is information that indicates the subject of the content represented by the voice data.

[0025] The elapsed time is information that indicates the elapsed time from a predetermined timing. For example, the elapsed time may be the elapsed time from the timing when speech starts in the voice data acquired by the acquisition unit 20A, the elapsed time from the timing when voice recognition by the control unit 20 starts, the elapsed time from the first topic in the topic data 31, etc.

[0026] In this embodiment, an example will be described in which the elapsed time is the time elapsed from the start of speech recognition of the speech data acquired by the acquisition unit 20A. If the start points of the elapsed time used for the data to be processed are different, the control unit 20 adjusts the elapsed time using information related to the timestamp included in each data so that the start points match, and then executes each process described below. In this embodiment, the description will continue assuming that the start points of the elapsed time used for each data are the same.

[0027] FIG. 2A is a schematic diagram showing an example of the data structure of topic data 31A. Topic data 31A is an example of topic data 31 when the audio data is Japanese audio data. As shown in FIG. 2A, topic data 31A is a database that associates, for each elapsed time, topics that appear in the audio data at that elapsed time with the topic appearance probability of that topic at that elapsed time. FIG. 2A shows an example in which topics and topic appearance probabilities are linked for each elapsed time of a predetermined period indicated in minutes.

[0028] FIG. 2B is a schematic diagram showing an example of the data structure of topic data 31B. Topic data 31B is an example of topic data 31 when the audio data is Japanese audio data. In FIG. 2B, the horizontal axis represents elapsed time, and the vertical axis represents the topic appearance probability. As shown in FIG. 2B, topic data 31A may be data that graphically represents, for each elapsed time, the topics that appear in the audio data at that elapsed time and the topic appearance probability of that topic at that elapsed time. FIG. 2B shows an example in which topics and topic appearance probabilities are linked for each elapsed time of a predetermined period indicated in minutes.

[0029] 2A and 2B show examples of topics such as "general," "sports," and "weather." Figures 2A and 2B show an example in which the topics with the highest topic appearance probability change over time in the order "general," "sports," and "weather."

[0030] The data format of the topic data 31 is not limited to a database or a graph format. For example, the topic data 31 may be data that represents the topic appearance probability of a topic corresponding to elapsed time according to a commonly used probability distribution such as a normal distribution.

[0031] 2A and 2B show an example in which the elapsed time of each topic shown in topic data 31 is measured in minutes. However, the elapsed time is not limited to minutes. For example, the elapsed time of each topic shown in topic data 31 may be measured in a period longer than the broadcast time of a typical television program. Specifically, the elapsed time may be measured in weeks, months, seasons, event periods, etc.

[0032] For example, during the Olympic Games, there is a high possibility that the topic "Olympics" will be featured in news programs and the like. For this reason, the control unit 20 stores in advance in the storage unit 30 topic data 31, which defines, for example, the duration of the Olympic Games, which is an example of an event period, as the elapsed time, the topic "Olympics" as the topic with the highest topic appearance probability corresponding to the elapsed time, and the topic "sports" as the topic with the next highest topic appearance probability. The control unit 20 may also generate in advance a general-purpose language model 32B and an adapted language model 32C for the Olympics and register them as the recognition model 32. Details of the general-purpose language model 32B and the adapted language model 32C will be described later.

[0033] The topic data 31 may be information generated in advance at least before the recognition process by the recognition unit 20B, and may be generated either before or after the user speaks.

[0034] When the topic data 31 is information generated in advance before a user speaks, the topic data 31 may be generated in advance using, for example, information on a program progress schedule sheet or information on lecture notes. A program progress schedule sheet is information used in the production of a broadcast program that describes the planned progress of the program being produced. Lecture notes are notes that describe the speech content to be used in a lecture.

[0035] When the topic data 31 is information generated after a user speaks, the topic data 31 is, for example, information on a lecture record compiled after a lecture or information on minutes of a meeting.

[0036] The topic data 31 may be generated in advance by the control unit 20, or may be generated in advance by external information processing.

[0037] Returning to Figure 1, we continue the explanation.

[0038] The recognition model 32 is a pre-trained model used for speech recognition. The recognition model 32 includes an acoustic model 32A, a general-purpose language model 32B, and an adapted language model 32C.

[0039] The acoustic model 32A is a machine learning model that receives speech data as input and outputs the phoneme occurrence probability of each phoneme included in the speech data for each elapsed time. The acoustic model 32A may be a model that has been trained in advance using a known method.

[0040] The general-purpose language model 32B is a machine learning model that receives phonemes and phoneme occurrence probabilities for each elapsed time included in the speech data as input and outputs a general-purpose candidate word string for each elapsed time. The general-purpose language model 32B may be a model that has been trained in advance by a known method.

[0041] The adapted language model 32C is a machine learning model for outputting a speech recognition result from speech data. The adapted language model 32C is linked to each of a plurality of topics and pre-stored in the storage unit 30. That is, the recognition model 32 includes a plurality of adapted language models 32C, and the plurality of adapted language models 32C are linked to different topics.

[0042] In detail, the adapted language model 32C is a machine learning model that receives a phoneme string represented by a general candidate word string as input and outputs a candidate word string. The adapted language model 32C may be a model that has been trained in advance by a known method.

[0043] As described above, the recognition unit 20B generates a speech recognition result from the speech data acquired by the acquisition unit 20A, using the topic data 31, the recognition model 32, and the elapsed time.

[0044] In detail, for each elapsed time of the speech data acquired by the acquisition unit 20A, the recognition unit 20B generates a speech recognition result using an adapted language model 32C linked to the topic with the highest topic appearance probability corresponding to the elapsed time.

[0045] The recognition unit 20B inputs the speech data acquired by the acquisition unit 20A to the acoustic model 32A. Through this processing, the recognition unit 20B derives the phoneme occurrence probability for each phoneme included in the speech data for each elapsed time as an output from the acoustic model 32A.

[0046] The recognition unit 20B then arranges pairs of the derived phonemes and their phoneme occurrence probabilities in time series along with the speech data, and inputs these pairs to the general-purpose language model 32B. Through these processes, the recognition unit 20B derives a general-purpose candidate word string for each elapsed time as an output from the general-purpose language model 32B.

[0047] Furthermore, the recognition unit 20B inputs the phoneme string represented by the general candidate word string corresponding to the elapsed time into each of the adapted language models 32C corresponding to each of the multiple topics, thereby deriving multiple candidate word strings.

[0048] Then, for each of the multiple candidate word strings derived for each elapsed time and topic, the recognition unit 20B calculates a candidate word string score by adding the product of the language score obtained by the adapted language model 32C corresponding to each topic and the corresponding topic occurrence probability, the acoustic score obtained by the acoustic model 32A, and the general-purpose language score obtained by the general-purpose language model 32B.

[0049] That is, the recognition unit 20B calculates the candidate word string score for each of the plurality of candidate word strings for each elapsed time and topic using the following formulas (1) and (2).

[0050]

number

[0051] In formula (1), n ​​represents the ID (identification information) of the topic, and t represents the elapsed time. n,t represents the candidate word sequence score for a candidate word sequence of elapsed time t and topic n. t represents the acoustic score at time t. t represents the general language score at elapsed time t. λ represents a constant. n,t represents the topic appearance probability of topic n over time t. n,t represents the adaptive linguistic score for elapsed time t and topic n.

[0052] The recognition unit 20B calculates a score Score for each topic in the topic data 31 using the above formula (1). n,t Alternatively, to reduce the amount of calculation, the score may be calculated only for a plurality of topics whose topic appearance probability corresponding to the elapsed time t in the topic data 31 is equal to or greater than a threshold. For example, this threshold is 0.5, but is not limited to this value.

[0053] Specifically, assume a situation in which "sports" and "Olympics" are identified as topics with topic appearance probabilities equal to or greater than a threshold corresponding to a certain elapsed time t in the topic data 31. Also assume a situation in which the topic appearance probability of the topic "sports" is the highest, and the topic appearance probability of the topic "Olympics" is the next highest. In this case, the recognition unit 20B calculates the topic appearance probability Topic n,t Then, the topic appearance probability of the topic "sports" and the topic appearance probability of the topic "Olympics" can be determined.

[0054] The recognition unit 20B adopts the candidate word string with the highest candidate word string score for each elapsed time from among the plurality of candidate word strings derived for each elapsed time and topic as the speech recognition result corresponding to each elapsed time.

[0055] Through these processes, the recognition unit 20B generates, for each elapsed time of the speech data, a speech recognition result using the adapted language model 32C linked to the topic with the highest probability of topic occurrence corresponding to the elapsed time, as the speech recognition result corresponding to that elapsed time of the speech data.

[0056] Next, the correction unit 20C will be described.

[0057] The correction unit 20C generates a corrected speech recognition result by correcting the speech recognition result using a correction model 33 for correcting the speech recognition result. The corrected speech recognition result is a result of correcting at least some of the words that make up the speech recognition result to a correct recognition result.

[0058] The correction model 33 is a machine learning model used to correct the speech recognition result. The correction model 33 includes a correction base model 33A and a correction converter 33B.

[0059] The correction base model 33A is a machine learning model for acquiring first feature quantities of a speech recognition result from the speech recognition result. The correction base model 33A is trained in advance using a pair of a speech recognition result and a correct answer text. The correct answer text represents a correct speech recognition result. The first feature quantities are tensors output from an intermediate layer of the correction base model 33A. The first feature quantities are sometimes referred to as intermediate representations or latent representations.

[0060] The correction converter 33B is a machine learning model for outputting a corrected speech recognition result from the first feature amount.

[0061] FIG. 3A is an explanatory diagram of an example of the correction process by the correction unit 20C.

[0062] The correction unit 20C acquires a first feature of the speech recognition result by using the speech recognition result received from the recognition unit 20B and the correction base model 33A. The correction unit 20C inputs the speech recognition result to the correction base model 33A, and acquires the first feature as an output from the intermediate layer of the correction base model 33A.

[0063] The correction unit 20C inputs the acquired first feature amount to the correction converter 33B to generate a corrected speech recognition result.

[0064] 3B is a schematic diagram showing an example of the configuration of the correction converter 33B. For example, the correction converter 33B is composed of nodes representing each of a plurality of words and arcs, which are paths representing the connection relationships between the plurality of words. In FIG. 3B, the words, which are nodes, are represented by circles, and the paths, which are arcs, are represented by arrows.

[0065] The correction unit 20C searches for a path representing the concatenation of the multiple words defined by the correction converter 33B using the similarity between the input first feature and the second feature, which is a latent expression of each of the multiple words defined by the correction converter 33B. Cosine similarity may be used as this similarity. In this search, the correction unit 20C excludes words whose similarity is equal to or less than a threshold from the path search candidates. The correction unit 20C then searches for the shortest path and outputs a word string consisting of the words that make up the shortest path as the corrected speech recognition result.

[0066] Continuing the explanation, returning to Fig. 1, it is preferable that the correction unit 20C uses the elapsed time when calculating the similarity score, which is the similarity used in the search.

[0067] That is, the correction unit 20C generates a corrected speech recognition result in which the similarity to the first feature value increases as the second feature value of a word matches a corrected character string included in the corrected speech recognition result linked to a correction time in the correction history data 34 that is closer to the elapsed time of the word for which similarity is to be calculated. The corrected character string means a character string that is different from the corresponding speech recognition result among the character strings that make up the corrected speech recognition result. In other words, the corrected character string is a character string that is different between the corrected speech recognition result and the speech recognition result. Note that the character string may be one character or multiple characters.

[0068] The correction history data 34 is data for managing the history of corrections made by the user to the voice recognition result. The correction history data 34 is updated by the storage control unit 20E, which will be described later.

[0069] The correction history data 34 is data that links together a speech recognition result, a corrected speech recognition result, a topic used in the speech recognition of the speech recognition result, and a correction time, which is an elapsed time.

[0070] The corrected speech recognition result represents a speech recognition result corrected by the user for the speech recognition result output from the recognition unit 20B or the corrected speech recognition result corrected by the correction unit 20C for the speech recognition result. In this embodiment, a case will be described in which the corrected speech recognition result represents a speech recognition result corrected by the user for the corrected speech recognition result, as an example. The correction time represents the elapsed time corresponding to the corrected speech recognition result corrected by the user.

[0071] 4A is a schematic diagram showing an example of the data configuration of correction history data 34A. The correction history data 34A is an example of the correction history data 34 when the speech data is Japanese speech data. As shown in FIG. 4, the correction history data 34A registers the correction time, topic, speech recognition result, and corrected speech recognition result in association with each other.

[0072] Fig. 4B is a schematic diagram showing an example of the data configuration of correction history data 34B. The correction history data 34B is an example of the correction history data 34 when the speech data is English speech data. As shown in Fig. 4B, the correction history data 34B registers the time of correction, which is the time of correction, the topic, the speech recognition result, and the corrected speech recognition result in association with each other.

[0073] Continuing the explanation by returning to Fig. 1, when the elapsed time is used to calculate the similarity score, the correction unit 20C calculates the similarity score using the following formula (3).

[0074]

number

[0075] In formula (3), n represents the ID (identification information) of the topic. t represents the elapsed time. FixScore n,t represents the similarity score between the first feature of the speech recognition result for elapsed time t and topic n and the second feature, which is the latent expression of each of the multiple words defined in the corrective converter 33B. fix represents the corrected string included in the corrected speech recognition result. asr represents the speech recognition result or the corrected speech recognition result. asr The case where the corrected speech recognition result is shown will be explained as an example.

[0076] logP(W fix |W asr ) represents the uncorrected similarity score. n (t,W fix ) represents the weighting value using the correction time. n (t,W fix ) is a word that constitutes the corrected character string included in the corrected speech recognition result, among the multiple words defined in the correction converter 33B, and the closer the correction time (elapsed time) of the word is to the elapsed time t of the comparison word included in the speech recognition result, the higher its value.

[0077] In this way, the correction unit 20C may use the elapsed time to calculate the similarity score, which is the similarity used in the search, so that the closer the time is to the previous correction time, the higher the similarity score calculated.

[0078] By calculating a higher similarity score the closer the correction unit 20C is to the previous correction time, the correction unit 20C can generate a corrected speech recognition result that corrects misrecognitions due to speech recognition of currently spoken speech data in more real time and with higher accuracy.

[0079] Furthermore, the correction unit 20C may search for a path of the correction converter 33B using a corrected similarity obtained by adding a weight value corresponding to the topic used to derive the speech recognition result to the similarity score, which is the similarity used for the search.

[0080] For example, the correction history data 34 may be configured to further associate a weight value corresponding to each topic. The weight value corresponding to the topic may be registered in, for example, an operation instruction by a user. Then, the correction unit 20C calculates fn(t, W fix ) can be a weight value associated with the topic corresponding to the modification time, which is the elapsed time t. fix ) may be a multiplication or addition value of a weighting value using the modification time and a weighting value associated with the topic.

[0081] In this way, by using the correction similarity to which the weight value corresponding to the topic used by the correction unit 20C to derive the speech recognition result is added, it is possible to generate a corrected speech recognition result that is corrected in real time and with high accuracy for a speech recognition result that includes a corrected word or a speech recognition result that includes a word that was corrected at a more recent correction time.

[0082] For example, news that differs from the script may be urgently inserted and spoken. In such cases, specific words contained in the urgently inserted news content are often spoken multiple times. For example, if the urgently inserted news is an earthquake alert, it is expected that the location of the earthquake will be repeatedly spoken. Therefore, the correction unit 20C uses a corrected similarity to which a weight value corresponding to the topic used by the correction unit 20C to derive the speech recognition result can generate a corrected speech recognition result that is corrected in real time and with high accuracy.

[0083] The output control unit 20D outputs the speech recognition result by the recognition unit 20B. If the control unit 20 is configured to include a correction unit 20C, the output control unit 20D outputs the speech recognition result corrected by the correction unit 20C as the speech recognition result.

[0084] The output control unit 20D outputs the voice recognition result to the display unit 40A. The output control unit 20D may also output the voice recognition result to an external information processing device via the communication unit 50. The output control unit 20D may also output the voice recognition result to various output devices such as a speaker or a projector connected to the control unit 20 so as to be able to communicate with the control unit 20.

[0085] In this embodiment, the output control unit 20D outputs the speech recognition result to the display unit 40A. The output control unit 20D outputs the speech recognition result and the topic used for the speech recognition of the speech recognition result to the display unit 40A. The output control unit 20D may output topics whose topic appearance probability is equal to or greater than a threshold to the display unit 40A in a display format different from that of topics whose topic appearance probability is less than the threshold. The threshold may be determined in advance. Furthermore, the threshold may be changeable as appropriate in response to a user's operation instruction on the input unit 40B, etc. A different display format means different character color, character size, character format, etc.

[0086] 5 is a schematic diagram of an example of a display screen 60A. The display screen 60A is an example of the display screen 60 when the voice data is in Japanese. The display screen 60 is an example of a display screen for the voice recognition result by the output control unit 20D.

[0087] For example, the display screen 60A includes a recognition result display field 62A and a topic display field 62B. The recognition result display field 62A is a display field that displays, for each elapsed time, the topic with the highest topic appearance probability corresponding to the elapsed time in the topic data 31 in association with the speech recognition result. The topic display field 62B is a display field that displays, for each elapsed time, the topic with the highest topic appearance probability corresponding to each elapsed time.

[0088] Returning to Figure 1, we continue the explanation.

[0089] The storage control unit 20E acquires the speech recognition result corrected by the user for the speech recognition result, and stores it in the correction history data .

[0090] The user operates the input unit 40B while viewing the display screen 60 displayed on the display unit 40A to input a correction character string for the displayed speech recognition result, thereby inputting the corrected speech recognition result. The storage control unit 20E associates the corrected speech recognition result input by the user's operation instruction of the input unit 40B, the speech recognition result from which the corrected speech recognition result was corrected, the topic used for the speech recognition of the speech recognition result, and the correction time, which is the elapsed time, and stores them in the correction history data 34.

[0091] Therefore, for example, the correction history data 34A shown in Fig. 4A is stored in the storage unit 30. In the case of the correction history data 34A expressed in Japanese shown in Fig. 4A, a corrected speech recognition result in which the character string "eh" included in the speech recognition result has been corrected to the corrected character string "A" is registered. Also, in the case of the correction history data 34A, a corrected speech recognition result in which the character string "eh-oki" included in the speech recognition result has been corrected to the corrected character "oki" is registered.

[0092] Furthermore, the storage unit 30 stores, for example, correction history data 34B shown in Fig. 4B. In the case of the correction history data 34B expressed in English shown in Fig. 4B, a corrected speech recognition result in which a character string "blockade" included in the speech recognition result has been corrected to a corrected character string "Block A" is registered. In addition, in the case of the correction history data 34B, a corrected speech recognition result in which a character string "loke" included in the speech recognition result has been corrected to a corrected character string "Oki" is registered.

[0093] Returning to Figure 1, we continue the explanation.

[0094] The update unit 20F updates the corrected model 33 using the correction history data .

[0095] FIG. 6 is an explanatory diagram of an example of the update process by the update unit 20F.

[0096] The update unit 20F obtains a first feature by inputting the speech recognition result output by the output control unit 20D into the correction base model 33A. Then, the update unit 20F updates the correction converter 33B of the correction model 33 by training the correction converter 33B of the correction model 33 using, as training data, a pair of the obtained first feature and a corrected speech recognition result corresponding to the speech recognition result included in the correction history data 34.

[0097] As described above, the correction base model 33A is a machine learning model that has been trained in advance using a pair of a speech recognition result and a correct text. For example, the correction base model 33A can be created by using a well-known technique, BART (Bidirectional Auto-Regressive Transformer), to fine-tune a model that has been trained in advance using a large corpus using a pair of a speech recognition result and a correct text.

[0098] The update unit 20F then inputs the speech recognition result output by the output control unit 20D into the corrected base model 33A created in advance, thereby obtaining a first feature that is a latent expression.

[0099] The update unit 20F then extracts the obtained first feature amount and a corrected character string that is the difference between the speech recognition result and a corrected speech recognition result corresponding to the speech recognition result included in the correction history data 34. The update unit 20F then updates the correction converter 33B using the first feature amount, the corrected character string, and the correction location that is the position of the corrected character string in the corrected speech recognition result.

[0100] First, the update unit 20F initializes the correction converter 33B. The initialized correction converter 33B has a loop structure as shown in Fig. 3B. An arc that matches any input word and outputs the input word as is is added to the top node of the initialized correction converter 33B.

[0101] The update unit 20F obtains a first feature, which is a latent expression, by inputting the speech recognition result output by the output control unit 20D into a previously created corrected base model 33A. The update unit 20F also obtains a pair of the speech recognition result and a corrected speech recognition result corresponding to the speech recognition result. The update unit 20F then extracts a corrected character string, which is the difference between the speech recognition result and the corrected speech recognition result.

[0102] The update unit 20F then adds arcs and nodes to the correction converter 33B so that the first feature, which is a latent expression of the speech recognition result, is input and the corrected speech recognition result is output. In detail, the update unit 20F adds a node and an arc between the start node and the end node for each word, which is a plurality of nodes included in the correction converter 33B.

[0103] By these processes, the update unit 20F updates the correction converter 33B from the first feature so that the latest corrected speech recognition result for the speech recognition result of the first feature is output as the corrected speech recognition result. By these update processes, the update unit 20F updates the correction model 33.

[0104] Therefore, the correction model 33 is updated so as to output a corrected speech recognition result that is a correct speech recognition result that corresponds more closely to real-time speech.

[0105] Returning to Figure 1, we continue the explanation.

[0106] The topic data update unit 20G updates the topic data 31 using the revision history data .

[0107] For example, the topic data update unit 20G updates the topic data 31 based on the revision history data 34 so as to reduce the topic appearance probability of topics whose number of revisions per unit time is equal to or exceeds a threshold. The threshold number of revisions and the unit time may be determined in advance. The unit time is, for example, 10 minutes, but is not limited to this time. The threshold number of revisions and the unit time may be changed as appropriate by the user operating the input unit 40B, for example.

[0108] Specifically, the topic data update unit 20G counts the number of revisions per unit time for each topic based on the revision history data 34. Then, the topic data update unit 20G identifies from the revision history data 34 topics whose number of revisions per unit time is equal to or greater than a threshold value, or whose revision rate per unit time is equal to or greater than a threshold value.

[0109] The topic data update unit 20G then changes the topic appearance probability in the topic data 31 linked to the topic identified based on the revision history data 34 so as to lower the topic appearance probability. At this time, the topic data update unit 20G changes the topic appearance probability in the topic data 31 linked to the topic identified from the revision history data 34 and the elapsed time, which is the revision time corresponding to the topic. The topic data update unit 20G then allocates the probabilities evenly so that the topic appearance probabilities of other topics are increased by the amount of the reduced probability.

[0110] Through these processes, the topic data update unit 20G updates the topic data 31.

[0111] The topic data update unit 20G may also update the topic data 31 by the following process.

[0112] For example, the topic data update unit 20G counts the number of revisions per unit time for each topic based on the revision history data 34. Then, the topic data update unit 20G identifies topics whose number of revisions per unit time is equal to or exceeds a threshold value or whose rate of revisions per unit time is equal to or exceeds a threshold value from the revision history data 34. Furthermore, the topic data update unit 20G identifies, as similar topics, other topics associated with speech recognition results that match or are similar to the revised speech recognition results corresponding to the identified topics in the revision history data 34.

[0113] If a similar topic can be identified from the revision history data 34, the topic data update unit 20G identifies the elapsed time, which is the revision time corresponding to the similar topic in the revision history data 34. Then, the topic data update unit 20G reduces the topic appearance probability associated with the topic corresponding to the identified elapsed time among the topics associated with elapsed times within a period including the identified elapsed time in the topic data 31, and increases the topic appearance probability associated with a topic that matches the similar topic.

[0114] Through these processes, the topic data update unit 20G updates the topic data 31 so as to reduce the topic appearance probability of topics whose number of revisions per unit time is equal to or exceeds a threshold value.

[0115] Therefore, the topic data update unit 20G can update the topic data 31 so as to improve the speech recognition performance in the speech recognition device 10 that outputs speech recognition results for speech spoken in real time, even when the topic changes sequentially.

[0116] Next, an example of the flow of information processing executed by the voice recognition device 10 of this embodiment will be described.

[0117] FIG. 7 is a flowchart showing an example of the flow of information processing executed by the voice recognition device 10 of this embodiment.

[0118] The acquisition unit 20A acquires voice data (step S100).

[0119] The recognition unit 20B identifies, for each elapsed time of the speech data acquired in step S100, a topic appearance probability corresponding to the elapsed time from the topic data 31 (step S102). Then, for each elapsed time, the recognition unit 20B identifies an adapted language model 32C linked to the topic with the highest topic appearance probability identified in step S102 (step S104). Then, the recognition unit 20B generates a speech recognition result for the speech data acquired in step S100 using the identified adapted language model 32C for each elapsed time (step S106).

[0120] The correction unit 20C generates a corrected speech recognition result by correcting the speech recognition result generated in step S106 using the correction model 33 (step S108).

[0121] The output control unit 20D displays the corrected speech recognition result generated in step S108 as the speech recognition result on the display unit 40A (step S110). For example, the output control unit 20D displays a display screen 60A shown in FIG. 5 on the display unit 40A.

[0122] The storage control unit 20E acquires a speech recognition result corrected by the user for the speech recognition result displayed in step S110 (step S112), and stores the corrected speech recognition result acquired in step S112 in the correction history data 34 (step S114).

[0123] The update unit 20F updates the corrected model 33 using the correction history data 34 stored in step S114 (step S116).

[0124] The topic data update unit 20G uses the revision history data 34 stored in step S114 to update the topic data 31 (step S118), and then ends this routine.

[0125] As described above, the speech recognition device 10 of this embodiment includes a recognition unit 20B and an output control unit 20D. The recognition unit 20B generates a speech recognition result from speech data using topic data 31 in which the topic appearance probability for each elapsed time of a plurality of topics that changes with elapsed time is registered in advance, a recognition model 32 linked to each of the plurality of topics and including an adapted language model 32C for outputting a speech recognition result from speech data, and the elapsed time. The output control unit 20D outputs the speech recognition result.

[0126] Here, prior art discloses a technique for extracting text corresponding to speech recognition results from topic information such as television program scripts and progress schedules, and then obtaining text related to the extracted text from external data on a website. The prior art also discloses a technique for adjusting a language score used for speech recognition using text obtained from external data. However, because the prior art adjusts the language score by obtaining text related to the speech recognition results from external data on a website, it is difficult to apply the technique to real-time subtitling for programs such as news programs, and there are cases where it is not possible to improve speech recognition performance.

[0127] On the other hand, in the speech recognition device 10 of this embodiment, the recognition unit 20B generates a speech recognition result from speech data using the topic appearance probability for each elapsed time of multiple topics that changes with elapsed time, a recognition model 32 including an adapted language model 32C linked to each of the multiple topics, and the elapsed time.

[0128] In this way, the speech recognition device 10 of this embodiment generates a speech recognition result from speech data using the topic appearance probability of the topic according to the elapsed time of the speech data and the adapted language model 32C of the topic according to the elapsed time.

[0129] Therefore, even when the topic changes depending on the elapsed time of the speech data, the speech recognition device 10 of this embodiment can generate a speech recognition result that follows the change of topic.

[0130] Therefore, the voice recognition device 10 of this embodiment can improve the voice recognition performance.

[0131] Furthermore, the speech recognition device 10 of this embodiment can generate speech recognition results that follow topic changes even when the topic changes depending on the elapsed time of the speech data, and therefore can be suitably applied to a system that generates subtitles in real time.

[0132] For example, the speech recognition device 10 of this embodiment can be suitably applied to a system that provides speech recognition results such as subtitles in real time when live broadcasts such as television news programs are aired in real time.

[0133] Furthermore, in the speech recognition device 10 of this embodiment, the correction unit 20C generates a corrected speech recognition result by correcting the speech recognition result using the correction model 33 for correcting the speech recognition result.

[0134] Therefore, in addition to the above-mentioned effects, the speech recognition device 10 of this embodiment can immediately correct and output a corrected speech recognition result when the speech recognition result contains an error.

[0135] Furthermore, in the speech recognition device 10 of this embodiment, the update unit 20F updates the correction model 33 using correction history data 34 that links the speech recognition result, the corrected speech recognition result corrected by the user, and the correction time.

[0136] Therefore, the speech recognition device 10 of this embodiment can output a speech recognition result that immediately reflects corrections made by the user to the speech recognition result. Therefore, in addition to the above effects, the speech recognition device 10 of this embodiment can reduce the development costs, correction costs, and user correction burdens required for speech recognition such as subtitling.

[0137] (Hardware configuration) Next, the hardware configuration of the voice recognition device 10 of the above embodiment will be described.

[0138] FIG. 8 is a diagram showing an example of the hardware configuration of the voice recognition device 10 of the above embodiment.

[0139] The speech recognition device 10 of the above embodiment has a hardware configuration that utilizes a normal computer, in which a CPU 80, a ROM (Read Only Memory) 82, a RAM (Random Access Memory) 84, an HDD 86, an I / F unit 88, etc. are interconnected via a bus 90.

[0140] The CPU 80 is a calculation device that controls information processing executed by the speech recognition device 10 of the above embodiment. The RAM 84 stores data necessary for various processes by the CPU 80. The ROM 82 stores programs and the like that realize various processes by the CPU 80. The HDD 86 stores data. The I / F unit 88 is an interface for sending and receiving data to and from other devices.

[0141] The programs for executing the various processes executed by the speech recognition device 10 of the above embodiment are provided in advance in the ROM 82 or the like.

[0142] The program executed by the speech recognition device 10 of the above embodiment may be provided by being recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk) in a format that can be installed or executed by these devices.

[0143] The program executed by the speech recognition device 10 of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program for executing each of the above processes in the speech recognition device 10 of the above embodiment may be provided or distributed via a network such as the Internet.

[0144] The programs for executing the various processes executed by the speech recognition device 10 of the above embodiment are generated in such a way that the above-mentioned units are generated on the main storage device.

[0145] The various information stored in the HDD 86 may be stored in an external device, in which case the external device and the CPU 80 may be connected via a network or the like.

[0146] Although the embodiments of the present disclosure have been described above, the above embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the inventions and their equivalents as set forth in the claims. [Explanation of symbols]

[0147] 10 Voice recognition device 20B Recognition part 20C Correction Department 20D Output control section 20E Memory control unit 20F Update Department 20G Topical Data Update Department 31 Topic Data 32 Recognition Model 32C Adaptive Language Model 33 Correction Model 33A Correction Base Model 33B Correction Converter 34 Revision History Data 40A display

Claims

1. a recognition unit that generates a speech recognition result from the speech data using topic data in which a topic appearance probability for each elapsed time of a plurality of topics that changes with the elapsed time is registered in advance, a recognition model that is linked to each of the plurality of topics and includes an adaptive language model for outputting a speech recognition result from the speech data, and the elapsed time; an output control unit that outputs the speech recognition result; Equipped with The output control unit outputting the speech recognition result and the topic used for the speech recognition of the speech recognition result to a display unit; outputting the topic having the topic appearance probability equal to or greater than a threshold value to the display unit in a display format different from that of the topic having the topic appearance probability less than the threshold value; Voice recognition device.

2. The recognition unit generating the speech recognition result using the adapted language model associated with the topic having the highest topic appearance probability corresponding to the elapsed time for each elapsed time of the speech data; The speech recognition device according to claim 1 .

3. a correction unit that generates a corrected speech recognition result by correcting the speech recognition result using a correction model for correcting the speech recognition result; Equipped with The output control unit outputting the corrected speech recognition result as the speech recognition result; The speech recognition device according to claim 1 .

4. The correction model is a correction base model for acquiring a first feature of the speech recognition result from the speech recognition result; a correction converter that outputs the corrected speech recognition result from the first feature; Including, the first feature is a latent representation output from an intermediate layer of the correction base model; The correction unit using the speech recognition result and the correction base model to acquire the first feature of the speech recognition result, and inputting the acquired first feature to the correction converter to generate the corrected speech recognition result; 4. The speech recognition device according to claim 3.

5. The correction unit a path representing a concatenation relationship between the plurality of words defined by the correction converter is searched for using a similarity between the input first feature and a second feature which is a latent expression of each of the plurality of words defined by the correction converter, and an output for the path with the greatest similarity is output as the corrected speech recognition result; 5. The speech recognition device according to claim 4.

6. The correction unit generating the corrected speech recognition result such that the similarity between the second feature of the word and the first feature is increased as the second feature of the word matches the corrected character string included in the corrected speech recognition result associated with the closer modification time; 6. The speech recognition device according to claim 5.

7. The correction unit searching for the path using a modified similarity obtained by adding a weight value corresponding to the topic used in deriving the speech recognition result to the similarity; 6. The speech recognition device according to claim 5.

8. a storage control unit that acquires a speech recognition result corrected by a user for the speech recognition result, and stores the speech recognition result, the corrected speech recognition result, and the correction time, which is the elapsed time, in correction history data that are linked together; an update unit that updates the correction model using the correction history data; The speech recognition device according to claim 3 .

9. The update unit a first feature obtained by inputting the speech recognition result into a correction base model that has been trained in advance using a pair of the speech recognition result and a correct text; the corrected speech recognition result corresponding to the speech recognition result included in the correction history data; and The correction model is updated by training the correction model using the pairs of 9. The speech recognition device according to claim 8.

10. a topic data updating unit that updates the topic data using the revision history data; The speech recognition device according to claim 8, comprising:

11. The topic data update unit updating the topic data based on the revision history data so as to reduce the topic appearance probability of the topic whose number of revisions per unit time is equal to or exceeds a threshold value; The speech recognition device according to claim 10.

12. a recognition step of generating a speech recognition result from the speech data using topic data in which a topic appearance probability for each elapsed time of a plurality of topics that changes with the elapsed time is registered in advance, a recognition model that is linked to each of the plurality of topics and includes an adaptive language model for outputting a speech recognition result from the speech data, and the elapsed time; an output control step of outputting the speech recognition result; Including, The output control step includes: outputting the speech recognition result and the topic used for the speech recognition of the speech recognition result to a display unit; outputting the topic having the topic appearance probability equal to or greater than a threshold value to the display unit in a display format different from that of the topic having the topic appearance probability less than the threshold value; Speech recognition methods.

13. A speech recognition program to be executed by a computer, comprising: a recognition step of generating a speech recognition result from the speech data using topic data in which a topic appearance probability for each elapsed time of a plurality of topics that changes with the elapsed time is registered in advance, a recognition model that is linked to each of the plurality of topics and includes an adaptive language model for outputting a speech recognition result from the speech data, and the elapsed time; an output control step of outputting the speech recognition result; Including, The output control step includes: outputting the speech recognition result and the topic used for the speech recognition of the speech recognition result to a display unit; outputting the topic having the topic appearance probability equal to or greater than a threshold value to the display unit in a display format different from that of the topic having the topic appearance probability less than the threshold value; Speech recognition program.

Images