Program, information processing device, and information processing method

Abstract

This invention provides a program, an information processing device, and an information processing method that improve the recognition performance of speech containing many unique words and phrases. [Solution] This is a program that causes a computer to perform a segment detection step, a phrase detection step, a recognition step, a generation step, and a correction step. The segment detection step detects the utterance segments contained in the audio data. The phrase detection step detects one or more first phrases from the utterance segments, which are at least a part of one or more specified phrases, and outputs a detection result indicating the one or more detected first phrases. The recognition step recognizes the audio of the utterance segments and outputs a recognition result. The generation step generates correction information to correct the recognition result using the detection result. The correction step corrects the recognition result by inputting the correction information into a language model.

Description

Technical Field

[0001] Embodiments of the present invention relate to a program, an information processing apparatus, and an information processing method.

Background Art

[0002] A large vocabulary continuous speech recognition system (hereinafter, speech recognition system) that recognizes and converts speech spoken at meetings, lectures, etc. into text is known. Also, an end-to-end speech recognition method that directly converts speech into text has been developed and is being utilized in actual applications.

[0003] In the above speech recognition system and speech recognition method, there is a demand for more accurately recognizing speech containing user-specific phrases (such as proper nouns) such as personal names, organizations, and technical terms.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Non-Patent Document 2

[0006] The present invention aims to provide a program, an information processing device, and an information processing method that can improve the recognition performance of speech containing many unique words and phrases. [Means for solving the problem]

[0007] The program of this embodiment is a program that causes a computer to perform a segment detection step, a phrase detection step, a recognition step, a generation step, and a correction step. The segment detection step detects utterance segments contained in the audio data. The phrase detection step detects one or more first phrases from the utterance segment that are at least a part of one or more specified phrases, and outputs a detection result indicating the one or more detected first phrases. The recognition step recognizes the audio of the utterance segment and outputs a recognition result. The generation step generates correction information to correct the recognition result using the detection result. The correction step corrects the recognition result by inputting the correction information into a language model. [Brief explanation of the drawing]

[0008] [Figure 1] A block diagram of the configuration of the information processing device according to the first embodiment. [Figure 2] Flowchart of the recognition process in the first embodiment. [Figure 3] A diagram showing an example of a generated correction prompt. [Figure 4] A diagram showing an example of a generated correction prompt. [Figure 5] Block diagram of the information processing device of the second embodiment. [Figure 6] Flowchart of the recognition process in the second embodiment. [Figure 7] Hardware configuration diagram of an information processing device according to the first or second embodiment. [Modes for carrying out the invention]

[0009] A preferred embodiment of the information processing device according to this invention will be described in detail below with reference to the attached drawings.

[0010] As mentioned above, end-to-end methods are developing as a speech recognition technology. For example, OpenAI's Whisper achieves high performance in general speech recognition tasks by training end-to-end on a large-scale speech and text pair dataset (approximately 680,000 hours) publicly available on the internet.

[0011] In the following, we will mainly explain examples where the unit of processing is a word. For example, a term unique to a user is a proper noun. The unit of processing is not limited to a word; for example, it may be a phrase containing multiple words. In the following, proper nouns such as names of people, organizations, and technical terms that are specified by the user are called user words. A user word corresponds to one or more specified terms.

[0012] In end-to-end speech recognition systems, the following methods can be used to more accurately recognize speech containing proper nouns such as names of people, organizations, and technical terms. (M1) Additional audio data containing proper nouns and corresponding transcribed text are collected, or pseudo-data is generated using speech synthesis technology, and then the model is fine-tuned (domain adapted). (M2) Apply bias to the proper nouns to be recognized. Specifically, during the decoding of the recognition process, increase the score of word sequences or subword sequences corresponding to proper nouns by a predetermined fixed value so that those proper nouns are output in the recognition result (speech recognition result). Alternatively, if the surrounding context is to be considered, collect text containing proper nouns, create a second language model, and use the second language model during the decoding of the recognition process (Shallow-fusion type). (M3) A sequence of subwords encoded with proper nouns is added to a portion of the input of the end-to-end decoding unit. As a post - processing of speech recognition, the recognition result is corrected (compensated, rectified, calibrated) using generative AI such as large language models (LLMs). For example, the recognition result is corrected by an LLM using a prompt including a hypothesis of the recognition result.

[0013] In AI technology, an LLM trained with a vast amount of text data can be used to generate natural sentences. For example, by the user creating an appropriate prompt, the AI can generate a more accurate and useful response. A prompt is an instruction and question given by the user to the AI. For example, when correcting a text, the following types of prompts can be considered. "You are a professional editor. Please correct the grammar errors in the following text. # Input text: Enter the text to be corrected here"

[0014] Each of the above - mentioned methods has the following problems respectively. (M1) The workload of preparing a large - scale speech and text paired dataset for fine - tuning is extremely large, and it is practically difficult to prepare a large amount of training data in multiple domains. Also, to fine - tune the model, a computational load during training is required. (M2) It is known that when the number of proper nouns adding bias increases, the performance deteriorates significantly. Therefore, when actually used in a system, it is necessary to limit the number of registered words. Also, to train a second - language model, a computational load during training is required. (M3) There is a limit to the number of registrable sub - word sequences. (M4) There is a limit to the number of input characters of the LLM.

[0015] Another technique to improve speech recognition performance involves recognizing specific words (hot words) in the speech and then adjusting the speech recognition parameters based on the attribute information of those specific words. In this technique, the acoustic segment targeting the specific words and the acoustic segment used for speech recognition are separate. In other words, the speech recognition result does not include the specific words. Therefore, even with such a technique, it is not possible to improve speech recognition performance when there are many proper nouns (specific words).

[0016] (First Embodiment) The information processing device of the first embodiment generates a correction prompt using the speech recognition result within the utterance section and the result of detecting user words contained in the speech within the utterance section, and generates the final recognition result using the correction prompt. This makes it possible to improve the recognition performance of speech containing many proper nouns.

[0017] The information processing device 100 can be used, for example, in a system that recognizes and transcribes speech containing many technical terms spoken at meetings and lectures to create meeting minutes.

[0018] Figure 1 is a block diagram showing an example of the configuration of an information processing device 100 according to the first embodiment. As shown in Figure 1, the information processing device 100 includes a storage unit 120, an acquisition unit 101, a section detection unit 102, a word detection unit 103, a recognition unit 104, a generation unit 105, a modification unit 106, and an output control unit 111.

[0019] The memory unit 120 stores various types of information used in the information processing device. For example, the memory unit 120 stores the user dictionary 121 used by the word detection unit 103, the speech recognition model 122 used by the recognition unit 104, and the large-scale language model (LLM) 123.

[0020] User dictionary 121 is a dictionary that stores one or more user words specified by the user. Each user word includes, for example, the following elements. Note that a user word does not need to include all of the following elements; for example, it must include at least the spelling and pronunciation (reading). • Notation: Words to be detected • Pronunciation: Information about the pronunciation of user-defined words (e.g., phoneme sequence) • Part of speech: Information about the part of speech of the user's word (e.g., proper noun, person's name, organization name, etc.) • Domain: Domain information for the user's word (e.g., information systems, medical, etc.) • Priority: User word priority (e.g., low, medium, high)

[0021] Users can add new user words to the user dictionary 121 and edit user words already registered in the user dictionary 121. User words may also be registered by specifying only their spelling. In this case, for example, an estimation process may be performed to estimate the pronunciation from the spelling, and the estimated pronunciation may be registered in the user dictionary in association with the specified spelling.

[0022] Figure 1 shows an example of a configuration in which the LLM123 is located inside the information processing device 100, but the LLM123 may be located in an external device. For example, the information processing device 100 may be configured to utilize the LLM123 provided by a generation AI system connected via a network or the like.

[0023] The storage unit 120 can be composed of any commonly used storage medium, such as flash memory, memory card, RAM (Random Access Memory), HDD (Hard Disk Drive), and optical disc. Some or all of the data stored in the storage unit 120 (user dictionary 121, speech recognition model 122, and LLM 123) may be stored in physically different storage media, or in different storage areas of the same physically storage medium.

[0024] The acquisition unit 101 acquires various types of information used by the information processing device 100. For example, the acquisition unit 101 acquires audio data to be recognized. The method by which the acquisition unit 101 acquires information may be any method. For example, the acquisition unit 101 acquires audio data by collecting an audio signal via an audio input device such as a microphone and converting it to a digital format. The acquisition unit 101 may also acquire audio data by reading a pre-recorded digital audio file (e.g., WAV format, MP3 format, etc.). The acquisition unit 101 outputs the acquired audio data to the section detection unit 102.

[0025] The segment detection unit 102 detects speech segments contained in the audio data. Any conventional method can be used to detect speech segments from the audio data, but for example, the following method can be applied.

[0026] For example, the segment detection unit 102 analyzes the audio data received from the acquisition unit 101 and detects speech segments using voice activity detection (VAD) technology. A speech segment is a section of the audio signal that contains audio signals actually spoken by a person. The segment detection unit 102 identifies the start and end points of speech based on the energy level or frequency characteristics of the audio signal and detects the section from the start point to the end point as a speech segment. VAD technologies include energy-based detection technology and frequency-based detection technology.

[0027] In energy-based detection technology, the interval detection unit 102, for example, divides the audio signal into short time windows, calculates the energy of each window, and determines that the interval in which the energy exceeds a certain threshold is an utterance interval.

[0028] In frequency-based detection, the interval detection unit 102 calculates, for example, Mel-frequency cepstrum coefficients (MFCCs) as feature quantities of the speech signal, sets a threshold based on the energy of the MFCCs, and determines that intervals exceeding the threshold are speech intervals.

[0029] The segment detection unit 102 may detect speech segments by using machine learning techniques such as support vector machines (SVMs) and neural networks to determine whether each frame is speech based on the features of the MFCC.

[0030] The segment detection unit 102 outputs the audio data of the detected speech segment to the word detection unit 103 and the recognition unit 104.

[0031] The word detection unit 103 detects user words from the detected utterance segment. For example, the word detection unit 103 detects one or more user words WA (first phrases) from the utterance segment, which are at least a part of the user words stored in the user dictionary 121, and outputs a detection result (user word detection result) indicating the one or more detected user words WA (phrase detection unit).

[0032] The word detection unit 103 detects user words (WA) in the following sequence, for example, and outputs the detection result. (D1) Extraction of speech features: Extract features (e.g., MFCC) from the speech signal within the speech segment. (D2) Phoneme Recognition: Based on the extracted features and acoustic model, the speech signal is converted into a sequence of phonemes. (D3) Word Matching: The phoneme sequence is compared with the pronunciation information in the user dictionary 121 to detect matching user words (WA). Hidden Markov Models (HMMs) and other deep learning methods are used for word matching. (D4) Output of detection results: The detection results showing the detected user word WA are output to the generation unit 105.

[0033] The detection results will include, for example, the following elements. However, the detection results do not need to include all of the following elements; for example, they will include at least the user word and its position. • Detected user words: User words detected within the utterance segment • Occurrence location: The position (start and end point) where a user word is detected within an utterance. • Confidence level: Information (score) indicating the confidence level of the detection result.

[0034] The recognition unit 104 recognizes the speech in the utterance section and outputs the recognition result. For example, the recognition unit 104 recognizes the speech data within the utterance section and outputs text (string) representing the speech recognition result. Speech recognition by the recognition unit 104 can be implemented using any conventional recognition technology. For example, the recognition unit 104 can use a modular method in which an acoustic model and a language model (language model for speech recognition) are trained separately, or an end-to-end method.

[0035] In the modular approach, the recognition unit 104 outputs the recognition result text in the following sequence. (R1) Extraction of speech features: Features are extracted from the speech signal within the utterance section. MFCCs and spectral features are used as features. (R2) Application of Acoustic Models: Based on speech features, an acoustic model is used to convert the speech signal into a sequence of phonemes. An acoustic model is a model that has learned the correspondence between speech signals and phonemes (the smallest units of sound). Examples of acoustic models that can be used include Hidden Markov Models (HMMs), Deep Neural Networks (DNNs), and Recurrent Neural Networks (RNNs). (R3) Application of a language model: A language model is used to convert a sequence of phonemes into a sequence of words. The language model generates the most appropriate sequence of words based on word occurrence probabilities and contextual information. The language model can be, for example, an n-gram model, an RNN, or a transformer. (R4) Decoding: The output of the acoustic model and the output of the language model are combined to generate the final recognition result. For example, the Viterbi algorithm and beam search are used. (R5) Output of recognition results: The sequence of words (text) obtained as a result of decoding is output to the generation unit 105 as the recognition result.

[0036] In an end-to-end approach, the recognition unit 104 integrates the steps (R2-R4) of applying the acoustic model, applying the language model, and decoding, which were performed in a modular manner, and generates text (recognition result) directly from the speech signal using a single neural network (speech recognition model 122). For example, in an end-to-end approach, the recognition unit 104 uses a speech recognition model 122 based on deep learning techniques such as RNNs, convolutional neural networks (CNNs), and transformer models to output the recognition result text.

[0037] The recognition result may include, for example, the following elements. However, the recognition result does not necessarily need to include all of the following elements; for example, it may include at least the recognized text. • Recognition text: Speech recognition text within the utterance segment. • Location of each word: The position of each word within the utterance (start and end points). • Confidence level: Information (score) indicating the confidence level of the recognition result.

[0038] The recognition unit 104 may be configured to use multiple speech recognition models 122 and output multiple recognition results obtained by the multiple speech recognition models 122.

[0039] The generation unit 105 uses the recognition result and the detection result to generate a correction prompt (correction information) for correcting the recognition result. The generation unit 105 outputs the generated correction prompt to the correction unit 106. Details of the generation of the correction prompt will be described later.

[0040] The correction unit 106 corrects the recognition result using a correction prompt to obtain the final recognition result. For example, the correction unit 106 corrects the recognition result by inputting a correction prompt to the LLM 123. The correction unit 106 outputs the result corrected by the LLM 123 as the final recognition result.

[0041] Any method can be used to correct the recognition result using LLM123 and correction prompts. Below, we will describe an example of processing when a transformer model is used with LLM123. When a transformer model is used with LLM123, the corrected result is generated by correcting the recognition result in the following way. (A1) Input encoding: When a modification prompt (input sequence) is input to the transformer model, the input sequence first passes through the embedding layer and is converted into a numerical vector. (A2) Positional encoding: The transformer model performs positional encoding on the encoded data in order to preserve the positional information of the input sequence. (A3) Encoder processing: The encoder consists of multiple self-attention layers and a feedforward layer, and outputs a high-dimensional vector representation that includes contextual information of the input sequence. (A4) Processing by the decoder: The decoder also consists of multiple self-attention layers and feedforward layers, and uses the encoder output to predict the next word. (A5) Output generation: The output of the decoder's final layer is transformed into a probability distribution of the next word through the softmax function, the word with the highest probability is selected, and output as the next word.

[0042] The process from (A3) to (A5) is repeated until the entire sequence is generated.

[0043] The output control unit 111 controls the output of various types of information used by the information processing device 100. For example, the output control unit 111 outputs the corrected result (final recognition result) of the recognition result obtained by the correction unit 106. The method of outputting the information can be any method, but for example, it can be displayed on a display device or transmitted to an external device via a network.

[0044] At least a portion of each of the above parts (acquisition unit 101, interval detection unit 102, word detection unit 103, recognition unit 104, generation unit 105, correction unit 106, and output control unit 111) may be implemented by one or more processing units. Each of the above parts may be implemented by, for example, one or more processors. For example, each of the above parts may be implemented by having a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) execute a program, i.e., by software. Each of the above parts may be implemented by a processor such as a dedicated IC (Integrated Circuit), i.e., by hardware. Each of the above parts may be implemented by using both software and hardware. When multiple processors are used, each processor may implement one of the above parts, or two or more of the above parts.

[0045] The information processing device 100 may be composed of one physical device or multiple physical devices. For example, the information processing device 100 may be built on a cloud environment. Furthermore, each part of the information processing device 100 may be distributed and provided on multiple devices.

[0046] Next, the recognition process by the information processing device 100 of the first embodiment will be described. Figure 2 is a flowchart showing an example of the recognition process in the first embodiment.

[0047] The acquisition unit 101 acquires the audio data to be recognized (step S101). The section detection unit 102 detects speech sections from the audio data (step S102). The word detection unit 103 refers to the user dictionary 121 which stores user words and detects whether a user word from the user dictionary 121 is included in each speech section, and outputs a detection result including the detected user word WA and its position (step S103).

[0048] The recognition unit 104 recognizes the speech within each utterance segment and outputs the recognition result (step S104). The generation unit 105 generates a correction prompt using the user word detection result and the recognition result (step S105). The correction unit 106 inputs the correction prompt to the LLM 123 and corrects the recognition result (step S106). The output control unit 111 outputs the recognition result corrected by the LLM 123 as the final recognition result (step S107).

[0049] Next, we will explain the details of generating correction prompts. Figures 3 and 4 show examples of generated correction prompts. For the sake of explanation, the correction prompts are divided into correction prompt 300a in Figure 3 and correction prompt 300b in Figure 4. A correction prompt is information that includes, for example, correction prompt 300a followed by correction prompt 300b. The underlining within the correction prompts is added for the sake of explanation.

[0050] As shown in Figures 3 and 4, the correction prompt includes, for example, the following elements: • Meaning of terms: For example, meaning 301a for recognition result (speech recognition result), and meaning 301b for detection result (user word detection result). • Task description: For example, Task 302 demonstrates correcting recognition results using detection results. • Correction rules: For example, correction rule 303 when correcting recognition results using detection results. • Sample: For example, one or more samples including detection results, recognition results, and correction results 304 ·Recognition result 305 • Detection result: 306 ·Final result 307

[0051] Modification rule 303 consists of specific modification rules such as replacement, deletion, and addition. Figure 3 shows an example of modification rule 303 containing 14 rules.

[0052] Sample 304 represents a sample of the final result obtained according to modification rule 303. Figure 4 shows an example of Sample 304 containing four samples. Each sample includes the speech recognition result, the user word detection result, and the final result.

[0053] Recognition result 305 represents the speech recognition result by the recognition unit 104. In Figure 4, the string (text) of the recognition result output from the recognition unit 104 is inserted into <Recognition Result Text>.

[0054] The detection result 306 represents the detection result of the user word by the word detection unit 103. In Figure 4, the detection result of the user word output from the word detection unit 103 is inserted in <Detection Result>. Although not shown in Figure 4, the detection result 306 includes at least the user word and its location.

[0055] Final result 307 is included in the correction prompt to obtain the result of the correction by LLM123.

[0056] Next, we will provide further details on specific examples of information to include in a correction prompt.

[0057] For example, the generation unit 105 may generate a correction prompt that includes the domain of each of the one or more user word WAs indicated by the detection results of the word detection unit 103. As described above, the user dictionary 121 may include domain information of user words. Domain information is information that indicates the domain (category, context) to which the user word belongs. Domain information may indicate domains such as medical terms, legal terms, and technical terms.

[0058] For example, the generation unit 105 generates a correction prompt that includes a detection result, namely a user word WA, which is associated with a domain indicated by the domain information, as the user word detection result 306. The generation unit 105 also generates a correction prompt that includes a correction rule indicating that the recognition result should be corrected considering the domain, such as the ninth correction rule.

[0059] The generation unit 105 may generate a correction prompt that includes the pronunciation of one or more user word WAs indicated by the detection result of the word detection unit 103. For example, the generation unit 105 generates a correction prompt that includes a detection result, namely a user word detection result 306, which includes a user word WA whose pronunciation is associated with a pronunciation included in the user dictionary 121. The generation unit 105 also generates a correction prompt that includes a correction rule that indicates that the recognition result should be corrected considering the pronunciation, such as the 10th correction rule. This ensures that pronunciation information is considered when correcting the recognition result, enabling more accurate corrections.

[0060] The generation unit 105 may generate a correction prompt that includes the part of speech of one or more user word WAs indicated by the detection result of the word detection unit 103. For example, the generation unit 105 generates a correction prompt that includes a detection result, namely a user word detection result 306, which includes a user word WA associated with the part of speech contained in the user dictionary 121. The generation unit 105 also generates a correction prompt that includes a correction rule indicating that the recognition result should be corrected considering the part of speech, such as the 11th correction rule. This ensures that part of speech information is considered when correcting the recognition result, enabling more accurate corrections.

[0061] The generation unit 105 may add correction rules to the correction prompt that limit the correction range based on the location of the user word. For example, the generation unit 105 may generate a correction prompt that includes a correction rule that limits the correction range of the recognition result, such as the seventh correction rule. The correction range (character position) can be determined from the recognition result, the length of the utterance segment, and the location of the detection result. The length of the utterance segment is passed to the generation unit 105 from, for example, the word detection unit 103 or the recognition unit 104.

[0062] The correction range may be determined from the occurrence location of the recognition result and the occurrence location (start and end location) of the user word WA. In this case, for example, the recognition unit 104 outputs the occurrence location (start and end location) for each word or subword of the recognition result. By using a correction rule that limits the correction range, location information is taken into consideration when correcting the recognition result, enabling more accurate correction.

[0063] The generation unit 105 may add a rule to the correction prompt that limits the number of corrections to the recognition result. For example, the generation unit 105 may generate a correction prompt that includes a correction rule that limits the number of corrections to the recognition result so that the number of corrections is less than or equal to an upper limit, such as the 12th correction rule. This makes it possible to avoid, for example, the output of an incorrect correction result due to repeated corrections.

[0064] The generation unit 105 may generate a correction prompt that includes past correction results. For example, if the correction unit 106 corrects the recognition result, it stores the correction result in the storage unit 120. The generation unit 105 then generates a correction prompt that includes the correction result stored in the storage unit 120. The correction result may also be included in the correction prompt as a sample associated with the detection result and the recognition result.

[0065] Past correction results may be correction results for the previous utterance segment. This allows contextual information in consecutive utterance segments to be considered, resulting in consistent corrections. For example, if a specific user word WA that was used for correction in the previous utterance segment is detected again in the next utterance segment, the information of the correction result for that user word WA can be reflected in the correction prompt.

[0066] The generation unit 105 may generate correction prompts that include the priority of each of the one or more user word WAs indicated by the detection result. For example, the generation unit 105 may generate correction prompts that include correction rules that modify the result to prioritize and adopt user word WAs with higher priority, such as the 13th correction rule.

[0067] If user words with associated priorities are registered in the user dictionary, the generation unit 105 may include the priority associated with the user word WA in the user dictionary in the correction prompt.

[0068] The generation unit 105 may include a priority based on the confidence level of the user word WA detection by the word detection unit 103 in the correction prompt. In this case, when the word detection unit 103 detects a user word WA, it outputs a detection result that further includes the confidence level of one or more user word WAs. The priority based on confidence level is the confidence level itself, or a value obtained by an operation on the confidence level (such as multiplication by a coefficient).

[0069] The generation unit 105 may generate a correction prompt that includes the confidence level of the recognition result by the recognition unit 104. In this case, the recognition unit 104 outputs the recognition result word by word. For example, the recognition unit 104 outputs a recognition result that includes one or more words WB (second phrases) recognized from the speech, and the confidence level of each of the one or more words WB.

[0070] For example, the generation unit 105 generates a correction prompt that includes a correction rule, such as the 14th correction rule, which prioritizes correcting word WBs with a lower confidence level than other word WBs. Confidence level is an indicator of the recognition accuracy of each word, and it is useful for prioritizing the correction of words with low confidence levels during the correction process. This improves the accuracy of the recognition results.

[0071] In a configuration where the recognition unit 104 outputs multiple recognition results using multiple speech recognition models 122, the generation unit 105 may generate a correction prompt that includes multiple recognition results. For example, the generation unit 105 may generate a correction prompt that includes multiple recognition results from multiple speech recognition models 122, as well as an instruction to select the most likely correction result from among the correction results for each of the multiple recognition results. The generation unit 105 may also generate a correction prompt that includes multiple recognition results, as well as an instruction to output multiple correction results for each of the multiple recognition results.

[0072] Thus, in the first embodiment, a correction prompt is generated from the speech recognition result and the user word detection result for the speech segment, and the recognition result is corrected by inputting the correction prompt into the LLM. This makes it possible to perform speech recognition with high accuracy, for example, for continuous speech containing many technical terms such as those in a lecture, simply by creating a user dictionary 121 (user word list) in advance.

[0073] Furthermore, in this embodiment, it is not necessary to include all user words specified by the user in the correction prompt; only user words detected from the utterance segment are included in the correction prompt. This avoids the problem of limitations on the number of input characters for the LLM, as seen in the method described above (M4).

[0074] (Second embodiment) The information processing device of the second embodiment differs from the first embodiment in that it utilizes a speech recognition engine capable of registering user words. In the second embodiment, detected user words are registered in the speech recognition engine as words to be recognized.

[0075] Figure 5 is a block diagram showing an example of the configuration of the information processing device 100-2 according to the second embodiment. As shown in Figure 5, the information processing device 100-2 includes a storage unit 120, an acquisition unit 101, a section detection unit 102, a word detection unit 103-2, a recognition unit 104-2, a generation unit 105, a modification unit 106, and an output control unit 111.

[0076] In the second embodiment, the functions of the word detection unit 103-2 and the recognition unit 104-2 differ from those of the first embodiment. Other configurations and functions are the same as those in Figure 1, which is a block diagram of the information processing device 100 of the first embodiment, and are therefore given the same reference numerals, and their description is omitted here.

[0077] The word detection unit 103-2 differs from the word detection unit 103 of the first embodiment in that it further includes a function to output one or more user words (WA) indicated by the detection result to the recognition unit 104-2.

[0078] The recognition unit 104-2 is configured to utilize a speech recognition engine capable of registering detected user words. For example, the recognition unit 104-2 recognizes speech using one or more user words (WA) output from the word detection unit 103-2 as the target for recognition.

[0079] Next, the recognition process by the information processing device 100-2 of the second embodiment will be described with reference to Figure 6. Figure 6 is a flowchart showing an example of the recognition process in the second embodiment.

[0080] Steps S201 to S203 are the same as steps S101 to S103 in the information processing device 100 of the first embodiment, so their explanation will be omitted.

[0081] In this embodiment, the word detection unit 103-2 outputs the detection result, including the detected user word and its location, to the recognition unit 104-2 (step S204).

[0082] The recognition unit 104-2 registers only the user words output from the word detection unit 103 with the speech recognition engine to be added as recognition targets. Then, the recognition unit 104-2 recognizes the speech within each utterance segment and outputs the recognition result so that the added user words are also recognized (step S205). Since only the detected user words are added to the registration, the problem of limitations on the number of words that can be registered, as in the method described above (M3), does not occur.

[0083] Steps S206 to S208 are the same as steps S105 to S107 in the information processing device 100 of the first embodiment, so their explanation will be omitted.

[0084] Thus, in the second embodiment, the speech recognition performance can be further improved.

[0085] As explained above, according to the first and second embodiments, the recognition performance of speech containing many unique words can be improved.

[0086] Next, the hardware configuration of the information processing device according to the first or second embodiment will be described using Figure 7. Figure 7 is an explanatory diagram showing an example of the hardware configuration of the information processing device according to the first or second embodiment.

[0087] The information processing device in the first or second embodiment includes a control device such as a CPU (Central Processing Unit) 51, a storage device such as a ROM (Read Only Memory) 52 or RAM (Random Access Memory) 53, a communication interface 54 for communication via a network, and a bus 61 for connecting the various parts.

[0088] The program to be executed in the information processing device of the first or second embodiment is provided pre-installed in a ROM 52 or the like.

[0089] The program executed by the information processing device of the first or second embodiment may be configured to be provided as a computer program product by recording it in an installable or executable file format onto a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0090] Furthermore, the information processing device of the first or second embodiment may be configured to store the program on a computer connected to a network such as the Internet and provide it by allowing download via the network. Alternatively, the information processing device of the first or second embodiment may be configured to provide or distribute the program via a network such as the Internet.

[0091] A program executed in the information processing device of the first or second embodiment can cause a computer to function as a part of the information processing device described above. This computer can read a program from a computer-readable storage medium onto its main memory and execute it using the CPU 51.

[0092] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0093] 100,100-2 Information Processing Device 101 Acquisition Department 102 Section detection unit 103,103-2 Word detection unit 104,104-2 Recognition part 105 Generation part 106 Correction section 111 Output Control Unit 120 Storage section 121 User Dictionary 122 Speech Recognition Models 123 Large-Scale Language Models (LLMs)

Claims

1. to the computer A segment detection step for detecting speech segments contained in audio data, A phrase detection step that detects one or more first phrases from the utterance section, which are at least a part of one or more designated phrases, and outputs a detection result showing the one or more detected first phrases, A recognition step that recognizes the speech in the aforementioned speech segment and outputs the recognition result, A generation step of generating correction information for correcting the recognition result using the detection result, A correction step involves inputting the aforementioned correction information into a language model to correct the recognition result, A program to execute.

2. The generation step is, The meaning of the above detection result and the above recognition result, A task to demonstrate correcting the recognition result using the detection result, The correction rules for correcting the recognition result using the detection result, One or more samples including the detection result and the recognition result and the correction result, The above detection results and, The aforementioned recognition result and, The correction information including the above is generated The program according to claim 1.

3. The generation step is, The system generates the correction information, which includes correction rules that limit the range of correction of the recognition result, determined from the recognition result, the length of the utterance interval, and the location of the detection result. The program according to claim 1.

4. The generation step is, The system generates the corrected information which includes the pronunciation of one or more of the first words indicated by the detection result. The program according to claim 1.

5. The generation step is, The system generates the corrected information which includes one or more parts of speech of the first phrase indicated by the detection result. The program according to claim 1.

6. The generation step is, The correction information is generated, which includes the priority of each of the one or more first phrases indicated by the detection result. The program according to claim 1.

7. Each of the one or more specified terms is associated with the priority, The generation step generates the modification information, which includes the priority associated with each of the one or more first phrases indicated by the detection result. The program according to claim 6.

8. The word detection step outputs the detection result, which further includes the confidence level of each of the one or more first words detected. The generation step generates the correction information, which includes the priority based on the confidence level of each of the one or more first phrases indicated by the detection result. The program according to claim 6.

9. The generation step is, The correction information is generated which includes the domains of one or more first phrases indicated by the detection result. The program according to claim 1.

10. The recognition step outputs the recognition result which includes one or more second words recognized from the speech and the confidence level of each of the one or more second words. The generation step generates the modified information which includes modification rules that prioritize modifying the second phrase whose confidence level is lower than that of other second phrases. The program according to claim 1.

11. The correction step involves storing the corrected result, obtained by correcting the recognition result, in the storage unit. The generation step generates the modified information, which includes the modified result stored in the storage unit. The program according to claim 1.

12. The word detection step outputs one or more of the first words indicated by the detection result to the recognition step. The recognition step involves recognizing the speech using one or more of the outputted first phrases as the object of recognition. The program according to claim 1.

13. A segment detection unit that detects speech segments contained in audio data, A phrase detection unit detects one or more first phrases, which are at least a part of one or more designated phrases, from the utterance section, and outputs a detection result indicating the one or more detected first phrases. A recognition unit that recognizes the speech in the aforementioned speech segment and outputs the recognition result, A generation unit that generates correction information for correcting the recognition result using the detection result, A correction unit that corrects the recognition result by inputting the correction information into the language model, An information processing device equipped with the following features.

14. An information processing method performed by an information processing device, A segment detection step for detecting speech segments contained in audio data, A phrase detection step that detects one or more first phrases from the utterance section, which are at least a part of one or more designated phrases, and outputs a detection result showing the one or more detected first phrases, A recognition step that recognizes the speech in the aforementioned speech segment and outputs the recognition result, A generation step of generating correction information for correcting the recognition result using the detection result, A correction step involves inputting the aforementioned correction information into a language model to correct the recognition result, Information processing methods including

Images