Voice Recognition Device

Abstract

The purpose of the present invention is to provide a sound recognition device capable of ascertaining the mood of a conversation at, for example, a meeting. A sound recognition device 100 comprises: a sound acquisition unit 101 that functions as an audio information acquisition unit which acquires audio information; a sound recognition unit 102 that acquires a recognition result text of a sound of a user A (first audio information) out of the audio information; a nonverbal sound recognition unit 103 that determines that a sound of a user B (second audio information) with respect to (associated with) the sound of the user A (first audio information) is the occurrence of a nonverbal event (for example, a nod); and a result output unit 105 that outputs event tag information indicative of an event in association with the recognition result text.

Description

[Technical field]

[0001] The present invention relates to a voice recognition device that performs voice recognition. [Background technology]

[0002] Patent Document 1 describes a conference support system that distinguishes between speech segments and non-speech segments contained in voice data during a conference or the like, and recognizes text data indicating speech from the speech segments. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-45208 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in a meeting or the like, it is not possible to recognize laughter or the like, and it is not possible to provide the appropriate atmosphere. Therefore, it is difficult to analyze the quality of the meeting atmosphere, etc.

[0005] An object of the present invention is to provide a voice recognition device capable of grasping the atmosphere of a conversation in a meeting or the like. [Means for solving the problem]

[0006] The voice recognition device of the present invention includes a sound information acquisition unit that acquires sound information, and a sound information processing unit that performs voice recognition processing on first sound information among the sound information to obtain a recognition result, determines that second sound information related to the first sound information is the occurrence of a non-verbal event, and outputs information indicating the event in association with the recognition result. Effect of the Invention

[0007] According to the present invention, it is possible to obtain a speech recognition result that makes it easy to grasp the atmosphere of a conversation in a meeting or the like. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a functional configuration of a speech recognition device 100 according to the present disclosure. [Diagram 2] 10 is an explanatory diagram showing the reliability of a speech recognition result and the reliability of each recognition event; FIG. [Diagram 3] FIG. 13 is a diagram showing an example of a processing result for an utterance. [Figure 4] FIG. 13 is a diagram showing an outline of a determination process when a determination section is changed. [Diagram 5] 3 is a flowchart showing the operation of the voice recognition device 100. [Figure 6] FIG. 13 is a diagram illustrating a process when an utterance is in English. [Figure 7] FIG. 13 is a diagram showing an example of a result output in a process of adding event tag information. [Figure 8] 13 is a diagram showing an example of processing determination for user A and user B. [Figure 9] FIG. 13 is a diagram showing an example of a result output in another example. [Figure 10] FIG. 13 is a diagram showing an example of a result output in another example. [Figure 11] An example in which event tag information is associated with recognition result text will be described. [Figure 12] FIG. 13 is a block diagram showing a functional configuration of a voice recognition device 100a according to a modification of the present disclosure. [Figure 13] FIG. 13 is a diagram showing a specific example of correction. [Figure 14] FIG. 13 is a block diagram showing a functional configuration of a voice recognition device 100b according to another modified example. [Figure 15] 13 is a graph showing the frequency of event types by topic. [Figure 16] 1 is a diagram illustrating an example of a hardware configuration of a voice recognition device 100 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The embodiments of the present disclosure will be described with reference to the accompanying drawings. Whenever possible, the same parts are designated by the same reference numerals and redundant description will be omitted.

[0010] 1 is a block diagram showing the functional configuration of a speech recognition device 100 according to the present disclosure. As shown in the figure, the speech recognition device 100 includes a speech acquisition unit 101, a speech recognition unit 102, a non-language speech recognition unit 103 (non-language sound recognition unit), a score determination unit 104, and a result output unit 105. Each component will be described below.

[0011] The voice acquisition unit 101 is a part that acquires voice during a meeting, a lecture, or the like. For example, the voice acquisition unit 101 is a microphone. However, the voice acquisition unit 101 is not limited to this, and may be a part that acquires a voice signal transmitted by wire or wirelessly. The voice acquisition unit 101 detects a voice section from a voice waveform signal, and outputs each section to the voice recognition unit 102 and the non-language voice recognition unit 103.

[0012] The speech recognition unit 102 performs speech recognition processing on speech sounds or non-speech sounds in the speech section output from the speech acquisition unit 101 using a known language model and acoustic model, acquires recognition result text for each recognition unit, and derives the reading and reliability of the recognition result text. The speech recognition unit 102 outputs the recognition result text, its reading, and reliability for each recognition unit, such as an utterance unit, a sentence unit, a phrase unit, a word unit, a kana or phoneme unit, or a time unit.

[0013] This reliability is information indicating how much the recognition result in the case of speech recognition processing can be trusted, and is generally indicated between 0 and 1, but is not limited thereto, and may be expressed as an integer or between 0 and 100. It may also be a normalized numerical value. In the present disclosure, the reliability of the speech recognition result is calculated based on the reliability stored in the language model and the acoustic model, but is not limited thereto, and other known reliability deriving methods, such as end-to-end speech recognition, may be used.

[0014] The non-language speech recognition unit 103 is a part that performs recognition processing on non-language sounds using a known recognition model for non-language sounds for speech sounds or non-language sounds in the speech section output from the speech acquisition unit 101. This non-language speech recognition unit 103 generates a reliability for each event according to the type of non-language sound for each recognition time according to the recognition unit recognized by the speech recognition unit 102. For example, when the speech recognition unit 102 recognizes on a word-by-word basis, the speaking time for the recognized word becomes the recognition unit of the non-language speech recognition unit 103.

[0015] An event indicates a type of non-verbal sound, and is classified as a positive event or a negative event. For example, in the present disclosure, a non-verbal sound refers to a non-verbal sound based on voice, such as laughter, a back-channel sound, a nod sound (which can be distinguished as positive or negative), a sigh, a sneeze, a cough, and other sounds, such as keyboard sounds and music sounds, such as background music. For example, a positive event indicates that it is laughter, and a negative event indicates that it is not laughter. The non-verbal speech recognition unit 103 recognizes these non-verbal sounds and generates a confidence level for the positive event and the negative event.

[0016] The reliability for an event is information indicating how much trust can be placed in the recognition result in the case of non-verbal speech recognition processing. This reliability is generally shown between 0 and 1, but is not limited to this, and may be represented by an integer, or may be represented between 0 and 100. Also, it may be a normalized numerical value. This reliability is obtained based on the reliability stored in the recognition model for recognizing non-verbal speech, but is not limited to this, and other known methods for deriving reliability may also be used.

[0017] Figure 2 is an explanatory diagram showing the reliability of the speech recognition result text and the reliability for each event. As shown in the figure, assume there is an utterance content of "ahahaha" (spoken as "ahahaha"). This indicates laughter.

[0018] The speech recognition unit 102 attempts to recognize the speech as verbal speech, and thus recognizes the text data "あ母派" (converted to Japanese kanji) and the reading "アハハハ" (ahahaha). The speech recognition unit 102 derives the reliability for the recognition result text recognized using the acoustic model and the language model.

[0019] On the other hand, the non-verbal speech recognition unit 103 derives the reliability for each event. In Figure 2, the reliability corresponding to the presence or absence of laughter is derived. That is, the reliability that "ahahaha" (spoken as "ahahaha") is not laughter and the reliability that it is laughter are calculated. In Figure 2, the reliability indicating that there is no laughter is 0.3 and the reliability indicating that there is laughter is 0.7 is calculated.

[0020] In addition, the reliability of the text data as laughter may be calculated. Also, as non-verbal speech, the reliability of other types such as the presence or absence of coughing or the presence or absence of keyboard sounds may be calculated.

[0021] The non-verbal voice recognition unit 103 has recognition models for recognizing laughter, coughing, or keyboard sounds respectively, and can calculate the respective confidence levels based on these recognition models. Of course, it may also be equipped with recognition models for recognizing other sounds such as responses, nods, and sneezes. Additionally, it may be equipped with a recognition model that outputs the confidence levels of each event for multiple events.

[0022] The score determination unit 104 is a part that determines which of the recognition result text for the language voice recognized by the voice recognition unit 102 and the confidence level for the event recognized by the non-verbal voice recognition unit 103 is appropriate.

[0023] The detailed processing will be described with reference to FIG. 2. In FIG. 2, it is assumed that the voice “ahahaha” is input, and processing is performed in the voice recognition unit 102 and the non-verbal voice recognition unit 103 respectively. The score determination unit 104 compares the confidence level of the recognition result text “a mother school” (Japanese where “ahahaha” is recognized, here an example of incorrect conversion) of the voice recognition unit 102 with the confidence levels of the positive and negative events (no laughter / yes laughter) of the non-verbal voice recognition unit 103. In FIG. 2, the voice recognition unit 102 outputs a confidence level of the recognition result text: 0.3. Also, the non-verbal voice recognition unit 103 outputs a confidence level of the negative event (no laughter): 0.3 and a confidence level of the positive event (yes laughter): 0.7.

[0024] The score determination unit 104 determines the recognition result text or event with the highest confidence level. In FIG. 2, since the confidence level of the positive event (yes laughter): 0.7 is the highest confidence level, “ahahaha” is determined as an event indicating laughter.

[0025] As described above, the score determination unit 104 may determine the recognition result text or event based on the highest reliability, or may use values ​​adjusted by weighting for each reliability. For example, the score determination unit 104 may determine which event the utterance content is based on a value obtained by multiplying the reliability of a positive event or a negative event by a predetermined coefficient. More specifically, the score determination unit 104 may determine whether the utterance content is a non-speech sound by comparing a value obtained by multiplying the reliability of a positive event by 2 with the reliability of a negative event.

[0026] Moreover, the score determination unit 104 may determine whether a positive event in a non-speech sound is appropriate by multiplying the reliability of the positive event by 0.7, subtracting 0.1 from the reliability, and comparing the result with a threshold value. The coefficients and threshold values ​​may be fixed values ​​stored in advance in a memory or the like, or an input unit may be provided so that they can be input from the outside. Furthermore, the given coefficients and threshold values ​​may be varied by a predetermined formula. Moreover, the score determination unit 104 is not limited to comparing the reliability of the positive event and the negative event with two values, but may include other values, for example, may compare with three or more values. For example, laughter and coughing may be considered as positive events, no laughter and no coughing as negative events, and keyboard sounds may be considered as noise events, and the three reliability values ​​of the three events may be compared.

[0027] Such weighting adjustment is determined, for example, according to the attributes or type of the user who uses the speech recognition device 100 of the present disclosure, or the contents of the conference. For example, in the case of a conference content that is likely to cause laughter, laughter is easily recognized, but there are also conference contents in which this is not the case. In such cases, laughter is difficult to recognize as laughter. For such conferences or users, the adjustment can be made as described above to enable accurate recognition.

[0028] The result output unit 105 selects a highly reliable recognition result text based on the determination result by the score determination unit 104, or adds event tag information corresponding to the event and outputs it. When outputting, the result output unit 105 inputs information on the event determined by the score determination unit 104 and acquires event tag information corresponding to the event from a storage unit or the like. This storage unit or the like stores the event tag information in advance. The event tag information is, for example, information that graphically represents a non-verbal sound, and in the case of laughter, it is a mark that graphically represents laughter. The event tag information may be, for example, text information that is defined in advance. The output may be output to an external terminal, or may be displayed on a display screen.

[0029] Fig. 3 is a diagram showing an example of a processing result for a certain utterance. Fig. 3 shows, as an example of the processing result, a recognition result text, a recognition result morpheme, a recognition result text reliability, an event reliability (laugh), an event reliability (cough), an event reliability (keyboard sound), a judgment result, a result output, and a supplementary output. Also, the reliability and result output for each word are shown.

[0030] The following speech content is acquired, and the recognition result by the speech recognition unit 102 is obtained.

[0031] Speech: I'm, um, hahaha, kind of, (cough: Gogh), (keyboard sounds: clink clink), fine. The above utterance is in Japanese and is pronounced as watashiwa ano- hahaha nanka (gohho) kakakaka ii. In Japanese, the "wa" that comes after the subject is pronounced as wa.

[0032] Recognition result text: I'm a mother, something like Van Gogh, or something like that. The above is the recognition result text converted to Japanese. Here, coughing and keyboard sounds are mixed in. This recognition result text is segmented into romanized characters watashi, wa, ano-, hahaha, nanka, gohho, kakakaka, and ii, and converted to Japanese. Here, for the utterance "watashiwa," the recognition results "watashi" and "wa" are obtained. The reliability of the recognition result text is derived as 0.95 and 0.91, respectively. Meanwhile, the reliability of each event for "watashi" by the non-language speech recognition unit 103 is as follows:

[0033] Confidence of laughter positive event: 0.23 Confidence of laughter negation event: 0.90 Confidence for positive cough event: 0.21 Confidence for negative cough event: 0.70 Confidence of positive keyboard event: 0.15 Confidence of negative keyboard sound event: 0.75 The reliability of each flag for "wa" by the non-language speech recognition unit 103 is as follows:

[0034] Confidence of laughter positive event: 0.35 Confidence of laughter negation event: 0.85 Confidence for positive cough event: 0.05 Confidence for negative cough event: 0.81 Confidence of positive keyboard event: 0.12 Confidence of negative keyboard sound event: 0.85 The score determination unit 104 selects the most reliable recognition result texts "watashi" (I) and "wa" (Wa) based on these reliability values, and the result output unit 105 outputs "watashiwa" (I) (watashiwa).

[0035] On the other hand, for the utterance: "ano-" and the recognition result text: "ano-", the speech recognition unit 102 determines the speech recognition reliability to be 0.9 and recognizes it as a filler.

[0036] Furthermore, for the speech: "hahaha" (hahaha) and the recognition result text: "mother's voice" (text in which "hahaha" is recognized) (confidence level: 0.1), the non-language speech recognition unit 103 calculates a confidence level of a positive event (laughing voice included): 0.7. Since the confidence level of the positive event (laughing voice included) is high compared to the confidence level of the recognition result text and other positive events (coughing, keyboard sounds), the score determination unit 104 determines that the recognition result: "mother's voice" is laughter. The result output unit 105 outputs event tag information indicating that laughter has occurred as a supplementary output. Note that the result output unit 105 does not have to output the event tag information.

[0037] In the present disclosure, the score determination unit 104 determines that the recognition result text determined to be a filler based on the recognition result morpheme is a filler even if the recognition result text has a high reliability. Then, the result output unit 105 does not output the recognition result text as a result output. Note that, as necessary, event tag information indicating that it is a filler may be output as a supplementary output.

[0038] In the example of FIG. 3, the reliability is calculated for each word, and the judgment is made based on the reliability for each word, but the present invention is not limited to this. The score judgment unit 104 may change the recognition unit of the recognition result text output by the speech recognition unit 102 and the non-language speech recognition unit 103 to a judgment unit for score judgment and make a judgment. For example, in FIG. 3, the speech recognition unit 102 and the non-language speech recognition unit 103 calculate the reliability for each word, but the score judgment unit 104 may integrate the reliability for each phrase or sentence, and make a judgment based on the reliability integrated for each phrase or sentence. By changing the score judgment range in this way, for example, the position where the event tag information indicating the non-language sound is added can be changed, making the text easier to read. For example, when the score is judged for each sentence, the event tag information is added to the end of the sentence.

[0039] Fig. 4 is a diagram showing an outline of the processing contents when the score determination unit is changed to the sentence unit. For the sake of convenience, the description of the event reliability, etc. is simplified compared to Fig. 3. Assume that the following speech content is input and the recognition result text is obtained.

[0040] Utterance: I, um, (laughing: hahaha), kind of, (cough: gogh), (keyboard noises: cack-cack), good. The above utterance is in Japanese and is pronounced as watashiwa ano- hahaha nanka (gohho) kakakaka ii. In Japanese, the "wa" that comes after the subject is pronounced as wa. Recognition result text: I, um, Mother, something, Van Gogh, or something, good The above is the recognition result text converted to Japanese. Coughing and keyboard sounds are mixed in. This recognition result text, in romanized notation, shows that it has been segmented into watashi, wa, ano-, hahaha, nanka, gohho, kakakaka, and ii and converted to Japanese. Here, the score determination unit 104 performs score determination on a sentence-by-sentence basis. That is, the score determination unit 104 sums up the speech recognition reliability and the non-language speech recognition reliability in one sentence of the recognition result text. For example, in the example of FIG. 4, the score determination unit 104 calculates the total reliability of the recognition result text in that one sentence and the total reliability of the events (respective reliability of the positive event and the negative event). Based on this total, the score determination unit 104 determines whether the recognition result text is appropriate and whether an event exists. In the example of FIG. 4, since the total reliability of the recognition result text and the positive event is equal to or greater than a predetermined value, the score determination unit 104 determines that the recognition result text includes speech, laughter, coughing, or keyboard sounds.

[0041] Then, the score determination unit 104 outputs each recognition result text whose reliability is equal to or greater than a predetermined value, such as "I," "I," "um," "something," and "good," as well as outputting event information (laughing, coughing, keyboard sounds), to the result output unit 105. The result output unit 105 acquires event tag information from the event information and outputs it together with the recognition result text whose reliability is equal to or greater than a predetermined value.

[0042] This makes it possible to output event tag information on a sentence-by-sentence basis and add event tag information to the end of the recognition result text, resulting in easy-to-read sentences.

[0043] Next, an operation of the speech recognition device 100 of the present disclosure will be described. Fig. 5 is a flowchart showing the operation of the speech recognition device 100. The speech acquisition unit 101 acquires a speech waveform signal (S101), performs speech section detection from the speech waveform signal, and outputs the speech (or other sound) in the speech section as a speech signal to the speech recognition unit 102 and the non-language speech recognition unit 103 (S102).

[0044] The speech recognition unit 102 performs speech recognition processing on the speech signal and outputs the recognition result text, reading, and reliability (S103). In addition, the non-language speech recognition unit 103 performs non-language speech recognition processing on the speech signal and outputs the reliability of each event for each recognition target time (S104).

[0045] The score determination unit 104 determines the validity of the recognition result text or the event for each recognition target based on the reliability of the recognition result text by the speech recognition processing and the reliability of each event recognized by the non-language speech recognition (S105).

[0046] The result output unit 105 selects appropriate recognition result text from the recognition result text based on the determination result, or obtains and outputs event tag information (S106).

[0047] By performing such processing, speech and non-speech sounds can be recognized during a meeting, etc.

[0048] Next, application examples of other languages ​​will be described. The above speech examples are for Japanese, but similar processing is naturally possible for other languages. Figure 6 is a diagram showing processing when the speech is in English. Note that the description is simplified for convenience of explanation. In Figure 6, the following speech is made.

[0049] Utterance: I go (laughing: hahaha) to (cough: off coff cough) (keyboard noises: clatter) school. Recognition result text:I go ah the her head to Costoco caca grata. Here, the speech recognition reliability of the recognition result text "I", "go", "to", and "school" recognized by the speech recognition unit 102 is high.

[0050] On the other hand, the recognition result texts "the her head", "Costoco", and "caca gratta" recognized by the speech recognition unit 102 have low recognition result text reliability but high event reliability. For example, the recognition result text "the her head" has a high laughter event reliability. This is because the laughter "hahaha" is being recognized as a speech sound.

[0051] The same is true for the recognition result texts "costoco" and "caca grata", which are derived with low speech confidence as a result of attempting to recognize a cough or keyboard sounds, respectively, as speech sounds. On the other hand, since these utterances are coughs and keyboard sounds, it is necessary not to output the recognition result text for them.

[0052] In the example of FIG. 6, the following result is output based on the recognition result text confidence and event confidence:

[0053] Result output:I go to school. In this way, it is possible to prevent non-speech sounds from being output even for English speech recognition.

[0054] Next, a process of adding event tag information (marks such as images, symbols, or text) indicating what event the non-speech sound is related to when the speech recognition device 100 in the present disclosure recognizes the non-speech sound will be described. In the following, when there are expressions and processes unique to the Japanese language, they will be described with the Japanese notation.

[0055] 7A and 7B are diagrams showing a specific example, in which Fig. 7A shows the actual speech content, and Fig. 7B shows the result output based on the recognition result.

[0056] 7 shows a conversation between users A and B. To simplify the explanation, the content of the conversation is omitted and shown as "---". In FIG. 7(a), user A is laughing while talking, and user B is making interjections in response to the conversation. The speech recognition unit 102 and the non-language speech recognition unit 103 can recognize who is speaking based on the speech channel or sound source separation, etc. For example, written in English, it would be as follows: A: I am Japanese. (Laughs) I live in Tokyo. B: yeah (answer) yeah (answer)

[0057] In Fig. 7, the voice recognition unit 102 and the non-language voice recognition unit 103 distinguish between the speech (sound sources) of user A and user B, and perform voice recognition processing and non-language voice recognition processing, respectively. The voice recognition unit 102 and the non-language voice recognition unit 103 recognize the elapsed time of the speech, and perform voice recognition processing and non-language voice recognition processing on the speech of user A while recognizing the elapsed time of the speech. Similarly, the voice recognition unit 102 and the non-language voice recognition unit 103 perform voice recognition processing and non-language voice recognition processing on the speech of user B while recognizing the elapsed time of the speech. Therefore, the result output unit 105 can use these elapsed times to indicate that user B is making a backchannel response in response to the speech of user A.

[0058] 8 is a diagram showing an example of processing determination for user A and user B. As shown in the diagram, the voice recognition unit 102 and the non-language voice recognition unit 103 perform voice recognition processing and non-language voice recognition processing by distinguishing between user A and user B, respectively, based on a sound source separation technique.

[0059] As shown in FIG. 8, the speech recognition unit 102 and the non-language speech recognition unit 103 that recognize the speech of user A recognize the speech of user A, "It's." (pronounced "desu" and a verb in Japanese). This "it's." indicates the ending of a word in Japanese. In Japanese, the ending of a word becomes a verb, but in other languages, it does not necessarily become a verb. In addition, the speech recognition unit 102 and the non-language speech recognition unit 103 that recognize the speech of user B recognize the backchannel event of user B, "un." In addition, the speech recognition unit 102 and the non-language speech recognition unit 103 each grasp the passage of time. Although omitted in FIG. 8, the elapsed time is associated and managed for each recognition result text. In addition, when recognizing the ending of a word in an English conversation, it is determined from the entire conversation.

[0060] The result output unit 105 outputs "desu" (desu) of user A and the event tag information (backchannel mark) of user B in accordance with the determination process of the score determination unit 104. At that time, the result output unit 105 outputs the recognized recognition result text "desu" (desu) and the event tag information:backchannel mark at positions according to the time lapse of the utterance.

[0061] 7(b) is a diagram showing a specific example of the result output. As shown in the diagram, the result output unit 105 outputs an interjection mark in accordance with the elapsed time of the speech of the user B. As a result, the interjection mark corresponds to the position of the recognition result text of the user A's "Watashi wa ------ desu." (meaning "I am ------.").

[0062] In this way, by attaching event tag information according to the content and timing of speech of a plurality of users, the atmosphere of the conversation content can be accurately expressed.

[0063] 9 is a diagram showing an example of a result output in another example. As shown in the figure, FIG. 9(a) shows the contents of the utterance. The speech recognition unit 102 and the non-language speech recognition unit 103 recognize the contents of the utterance of the user A and the user B, respectively. The result output unit 105 outputs the recognition result text and the event tag information according to the determination result of the score determination unit 104.

[0064] In FIG. 9B, the result output unit 105 adds event tag information to the recognition result text for each sentence. That is, the event tag information is added for each user's utterance. In FIG. 9B, the result output unit 105 adds user B's event tag information: backchannel mark x1) to user A's recognition result text "----desu." (meaning I am ----.). Similarly, the result output unit 105 adds user A and B's event tag information: A's comedy mark x1, B's backchannel mark x1 to the recognition result text of user A's next utterance "----nanndonda yo." (meaning I think -----. Note that this is not limited to think, and varies depending on the conversation). As described above, the non-verbal speech recognition unit 103 recognizes the time when user B's backchannel occurs, and can insert it into user A's recognition result text according to that time.

[0065] 9(c), the result output unit 105 outputs the recognition result text and the event tag information "It's ----. What is ----? (meaning I am ----. I think -----.) Mr. A: 1 laughing mark, Mr. B: 2 interjection marks" for each paragraph (each user who spoke). Then, the result output unit 105 operates to output the event tag information to the end of the recognition result text of the speech of user A.

[0066] That is, the result output unit 105 adds event tag information of a mark indicating that user A laughed and a mark indicating that user B gave a backchannel to the recognition result text and outputs the added event tag information. At this time, the non-verbal speech recognition unit 103 recognizes that user A laughed once and that user B gave a backchannel twice, and the result output unit 105 adds a numerical value indicating the number of times to the mark. Note that, in this case, an utterance and a response thereto in a conversation between multiple users is regarded as one paragraph unit. A paragraph unit means a user unit that uttered the recognition result text, but may also be determined based on the interval between utterances.

[0067] In the example of FIG. 9(d), a similar process is performed, but the result output unit 105 adds a laughing mark to the recognition result text of user A, and adds the event tag information (interjection mark), which is the recognition result of user B, to the end of the recognition result text of user A.

[0068] That is, the result output unit 105 outputs the recognition result text and event tag information of user A based on the speaker including the recognition result text, i.e., the utterance of user A, and when user B's utterance is only an event, outputs the event tag information of user B to the end of the recognition result text (or event tag information) of user A. Note that when the recognition result of user B's utterance includes the recognition result text, the result output unit 105 may output the event tag information together with the recognition result text of user B.

[0069] Next, a result output example in another example will be shown. FIG. 10 is a diagram showing a result output example in another example. According to this diagram, the result output unit 105 can recognize a break in a topic based on the speech contents (or the recognition result text) of the user A and the user B. For example, when the result output unit 105 detects a character string indicating an intention to change the topic, such as "by the way," it performs a process of adding a mark to the end of the topic up to that point. In FIG. 10, the result output unit 105 determines that the point where the user B makes a backchannel response of "hmmm" is the end of one topic, and adds a laughing mark for the user A and a backchannel response mark for the user B to the end of the topic. In addition, a topic can be estimated or a scene can be divided using a topic estimation engine or a topic segmentation engine, which are well-known technologies.

[0070] This allows event tag information to be added to each topic. Note that topics can be used to distinguish between a time period during which an explanation is given and a time period during which questions and answers are answered in a meeting, for example.

[0071] In Fig. 10(b), a mark according to the number of times or a number indicating the number of times is added, but if an event occurs a certain number of times or more, an image mark emphasizing the frequency may be added. For example, if there are 10 or more laughs, a big smiley face mark may be added.

[0072] 7 and 9 show an example in which laughter occurs, but the voice recognition device 100 may prepare event tag information (image, mark, text) for each event and according to the occurrence frequency in a storage unit such as a memory (not shown) and change the event tag information according to the frequency. That is, a reference value for the occurrence frequency may be prepared and the event tag information may be changed according to the reference value.

[0073] For example, if laughter occurs two or more times, a mark indicating laughter is added, and if coughing occurs five or more times, a mark indicating coughing is added. Event tag information may also be generated according to the frequency, for example, if coughing occurs less than five times, a mark indicating coughing is added, and if coughing occurs five or more times, a mark indicating a worried face is added. In addition, in a meeting or the like, in a section with many laughter events (a predetermined frequency or more), an approval mark can be used instead of a laughing mark.

[0074] Regarding keyboard sounds, if keyboard sounds are generated a predetermined number of times or less, a mark indicating this may be added. If the volume exceeds a certain level, a mark indicating a stronger indication (for example, that the keystrokes are loud) may be added. Furthermore, if the voice recognition unit 102 performs voice recognition and the recognition result text is output more than a certain amount (if the reliability of the recognition result text is equal to or greater than a predetermined value), a mark indicating an even stronger warning message may be added.

[0075] These reference values ​​or the predetermined number of times may be a predetermined value, or may be determined based on the frequency of occurrence of an event in the entire conversation. For example, the frequency of occurrence of a certain event in all utterances may be averaged by time unit (or by paragraph unit or sentence unit, etc.) to set the reference value (or the predetermined number of times), and if the frequency of occurrence of the event is equal to or higher than this, event tag information may be added.

[0076] Moreover, instead of a mark, the event tag information may be an emoticon such as a smiley face or a video. Moreover, instead of adding event tag information, the color of the characters in the recognition result text of the immediately preceding utterance may be changed. For example, in the case of a smiley face, the recognition result text may be in a bright color (such as yellow) or the background may be in a bright color (the color of the blue sky). For example, in the case of a sigh, the recognition result text may be in a dark color (such as gray) or the background may be in a dark color (gray).

[0077] In the above, the contents of the event tag information are changed depending on the type of event, the score determined by the score determination unit 104, and the frequency of the event. However, when multiple events occur, the display format of the event tag information may be changed depending on the proportion of occurrence of each event relative to the total number of events.

[0078] Also, when a dark event such as a sigh occurs, a warning or advice may be displayed and a message may be output to tell the user to speak in a brighter voice.

[0079] In the present disclosure, the speech recognition device 100 may perform speech recognition processing and non-language speech recognition processing on one or more user conversations in real time, or may perform speech recognition processing and non-language speech recognition processing on recorded conversation data.

[0080] When performing real-time speech recognition processing and non-language speech recognition processing in a conference or the like, the speech recognition device 100 may output (display) only to a specific person (terminal), or output (display) so that everyone can see. Also, in a web conference or the like, if there is a display column associated with each speaker (terminal), it may be displayed in the column corresponding to that speaker. Also, it may be displayed only to user A, or may be displayed on the screen of other participants such as user C or D.

[0081] Next, an example in which event tag information and recognition result text are associated with each other and can be switched between display will be described. FIG. 11 is a diagram showing an example. FIG. 11(a) shows the actual speech content, and FIG. 11(b) shows an example of the result output. As shown in FIG. 11(b), the result output unit 105 outputs the recognition result text recognized by the speech recognition unit 102 and the event tag information recognized by the non-language speech recognition unit 103. At that time, the result output unit 105 links the event tag information (laughing mark L1 or backchannel marks L2, L3) with the recognition result text and manages the recognition result output.

[0082] When the result output unit 105 displays the recognition result text and event tag information on the display as the result output, if a user viewing the display selects the event tag information L (L1 to L3) (e.g., by clicking with a mouse), the result output unit 105 accepts the selection and outputs the recognition result text of the event tag information to the display, which then displays it (Figure 11(c)).

[0083] This process may be performed when the result of the determination by the score determination unit 104 satisfies a predetermined condition. That is, when a certain recognition result text and its corresponding event reliability do not reach a predetermined value, it may be difficult for the score determination unit 104 to accurately determine which one is valid. In that case, the result output unit 105 may acquire the recognition result text, the event tag information, and their respective reliability from the score determination unit 104, and enable switching between the display by a user operation.

[0084] When the result output unit 105 outputs the recognition result to another external terminal, the recognition result is output in association with the recognition result text. The external terminal displays the recognition result text and the event tag information, and when the event tag information L (L1 to L3) is selected (e.g., clicked with a mouse) by a user operation of the external terminal, the external terminal displays the recognition result text associated with the event tag information.

[0085] Next, a process of correcting a speech recognition result will be described. Fig. 12 is a block diagram showing a functional configuration of a speech recognition device 100a according to a modification of the present disclosure. As shown in the figure, the speech recognition device 100a includes a correction unit 106 in addition to the functional configuration of the speech recognition device 100 in Fig. 1.

[0086] Correction unit 106 is a part that corrects the recognition result text or event tag information of the result output that is output and displayed on the display from result output unit 105. This display displays the recognition result text and event tag information, and when correction unit 106 receives a correction portion (a part of the recognition result text or event tag information) indicated by a pointer or the like according to a user's operation, it displays one or more correction candidates corresponding to the correction portion on the display to the user. For example, the correction candidates are displayed in a pull-down menu.

[0087] The result output unit 105 associates the recognition result text and event tag information for the same utterance, and manages (stores) them as correction candidates. The recognition result text includes text converted to kanji, text in hiragana, text in katakana, and other converted symbols or text.

[0088] Then, the correction unit 106 switches to one of the correction candidates selected by the user, and the result output unit 105 outputs and displays the correction candidate on the display. The correction candidates are the recognition result text and events recognized by the speech recognition unit 102 and the non-language speech recognition unit 103. As described above, the correction candidates include text converted into kanji, text remaining in hiragana, text remaining in katakana, or other converted symbols or text.

[0089] FIG. 13 is a diagram showing a specific example of the correction. FIG. 13(a) shows the spoken content, and FIG. 13(b) is a diagram showing an example of a correction screen by the user. As shown in the figure, the correction unit 106 moves the pointer P according to the user's operation. When the correction part indicated by the pointer P is selected, the correction unit 106 displays a correction candidate B. When the user selects an arbitrary candidate from the correction candidates B, the result output unit 105 outputs the candidate to the display as a result output. In FIG. 13, the correction candidates B include "mother" and the text in which "hahaha" is recognized as it is can also be selected. In the present disclosure, "mother" is erroneously recognized text.

[0090] In the figure, the event tag information is described as the object to be corrected, but it goes without saying that the recognition result text may also be the object to be corrected.

[0091] Next, the counting of the occurrence frequency of events will be described. The voice recognition device 100a in the modified example of the present disclosure may count the frequency of events when performing voice recognition. By counting the occurrence frequency of events, the importance of the conversation, the quality of the conversation, and the like can be determined. FIG. 14 is a block diagram showing the functional configuration of a voice recognition device 100b in another modified example. In addition to the functional configuration of the voice recognition device 100, this voice recognition device 100b includes a counting unit 107.

[0092] The tallying unit 107 is a part that tallies the occurrence frequency of events recognized by the non-language speech recognition unit 103 and determined by the score determination unit 104. For example, the tallying unit 107 tallies the types of events, such as laughter, nodding, etc., determined by the score determination unit 104, and their occurrence frequencies. In this case, the tallying unit 107 may aggregate by time period, paragraph, topic, or speaker. Also, aggregation may be performed by the presence of a long pause, a change of paragraph, etc. This allows a user analyzing the speech recognition results to determine which utterances are important and their importance, based on the frequency of laughter or nodding for each classification, such as time period or topic.

[0093] In determining the topic division, the counting unit 107 determines a predetermined character string (for example, "by the way") obtained by the speech recognition unit 102 through speech recognition as the topic division, but of course the topic may be determined by other methods. Similarly, the speech recognition unit 102 and the non-language speech recognition unit 103 can distinguish speakers based on speakers distinguished by sound source separation or speech channel.

[0094] FIG. 15 is a graph showing the frequency of event types for each topic. To provide this graph, the speech recognition device 100b can provide information on the occurrence frequency of each event for each topic. Note that a table may be used instead of a graph. Also, instead of a topic, the occurrence frequency of each event for each speaker may be provided. This makes it possible to obtain an analysis result of whether each topic or speaker is a positive or negative topic or speaker.

[0095] Similarly, the time of occurrence of an event may be referenced and classified into time periods. A time period in which there was a lot of laughter or nodding may be regarded as a positive time period, and a time period in which there was little laughter or nodding or a lot of sighing may be regarded as a negative time period, and these may be regarded as the analysis results.

[0096] Next, the effects of the speech recognition device 100 of the present disclosure will be described.

[0097] The speech recognition device 100 of the present disclosure includes a speech acquisition unit 101 that functions as a sound information acquisition unit that acquires sound information, a speech recognition unit 102 that acquires recognition result text of user A's speech (first sound information) from the sound information, a non-verbal speech recognition unit 103 that determines that user B's speech (second sound information) (related to) user A's speech (first sound information) is the occurrence of a non-verbal event (e.g., a nod), and a result output unit 105 that associates event tag information indicating the event with the recognition result text and outputs it.

[0098] With this configuration, the occurrence of an event can be associated with the recognition result and presented to a user, etc. For example, event tag information (additional information) such as a mark indicating laughter, which is one of the non-speech sound recognition results, is added to the recognition result text, or the recognition result text is processed by coloring it, etc. Therefore, when the occurrence of an event is visually associated, a user who receives the event tag information can intuitively and sensually recognize the event. In the present disclosure, whether it is a user or a different user, adding event tag information to the recognition result text enables intuitive or sensually recognition of the event.

[0099] The additional information such as marks may be represented by colors in addition to symbols.

[0100] Furthermore, when the occurrence frequency of an event satisfies a predetermined condition, the result output unit 105 in the voice recognition device 100 outputs the event tag information in association with the recognition result text.

[0101] For example, if the frequency of laughter is more than a certain number of times, a mark indicating laughter is added to the recognition result text. This allows the user to intuitively grasp the atmosphere of the conversation. In other words, the atmosphere of a conversation is different if laughter occurs once or twice compared to if laughter occurs more than once.

[0102] Furthermore, the result output unit 105 associates event tag information (information indicating an event) of user B with the recognition result text for a group of sound information (user A's speech and user B's speech) that includes user A's speech and satisfies a predetermined condition. For example, the predetermined condition is that the group of sound information is divided into sentences, paragraphs, or topics. By adding event tag information to the end of a sentence, paragraph, or topic, it is possible to easily grasp the event in a group of conversation.

[0103] Furthermore, when an event such as that of user B occurs, the result output unit 105 adds event tag information to the end of the recognition result text of user A. That is, the event tag information is added to the end of the recognition result text in an utterance by a different speaker. This allows the user to intuitively grasp the atmosphere of the conversation.

[0104] The speech recognition device 100a further includes a correction unit 106 that corrects the recognition result text or information indicating an event. The speech recognition unit 102 performs speech recognition processing on the speeches (first sound information and the second sound information) of the user A and the user B to obtain the respective recognition result texts. The non-language speech recognition unit 103 performs non-language speech recognition processing on the speeches (first sound information and the second sound information) of the user A and the user B to determine the occurrence of each event. The correction unit 106 corrects the respective recognition result texts or event tag information of these pieces of sound information.

[0105] This allows the recognition result text and event tag information to be corrected, and a correct recognition result to be obtained.

[0106] In the voice recognition device 100, the voice recognition unit 102 and the non-language voice recognition unit 103 perform voice recognition processing and event occurrence determination for the sound information (first sound information, second sound information) of both users A and B. Then, the result output unit 105 acquires the recognition result text and event tag information according to the determination result of the score determination unit 104. Then, the result output unit 105 outputs, for example, the event tag information of user B's laughter together with the recognition result text of the laughter to a display. The operating user looking at this display can switch between the event tag information of user B and the recognition result text for display. That is, when the operating user performs a switching operation (for example, selection by a pointer), the result output unit 105 performs output control for switching between the display of the event tag information and the recognition result.

[0107] If the output destination is an external terminal rather than a display, the result output unit 105 outputs the event tag information and the recognition result text of user B to the external terminal. This output corresponds to output control that enables switching between the event tag information and the recognition result text. The external terminal displays the event tag information and enables switching between the display of the event tag information and the recognition result text according to the user operation of the external terminal.

[0108] In the above embodiment, the reliability was determined and the result was output in accordance with the determination within the speech recognition device 100, but these processes may be requested from an external terminal. That is, as a process for not outputting the result based on the speech recognition process, the speech recognition device 100 outputs the recognition result text from the speech recognition process and its reliability, as well as the recognition result (event, etc.) from the non-language speech recognition process and its reliability to the external terminal. The external terminal can obtain the recognition result text, etc. based on this information.

[0109] In the speech recognition device 100 of the present disclosure, the sound information processing unit further includes a score determination unit 104 that determines the recognition results of the speech recognition unit 102 and the non-language speech recognition unit 103, and a result output unit 105 that processes and outputs the recognition result text by the speech recognition unit 102 in accordance with the determination. That is, based on the determination result by the score determination unit 104, the result output unit 105 does not output the non-language sound part of the recognition result text, and outputs only the language part.

[0110] This makes it possible to obtain easy-to-read recognition result text because the recognition result text for non-speech sound parts is not output.

[0111] In the speech recognition device 100 of the present disclosure, the speech recognition unit 102 derives speech sound confidence for the sound information being speech, i.e., confidence for the recognition result text, and the non-language speech recognition unit 103 derives non-language sound confidence for the sound information being non-language, i.e., confidence for the event. Then, the score determination unit 104 determines the recognition results (recognition result text and each event) by the speech recognition unit 102 and the non-language speech recognition unit 103 based on these confidences (speech sound confidence and non-language sound confidence).

[0112] Here, the reliability of an event indicating the non-speech sound reliability indicates the reliability that the speech waveform signal (sound information) is a non-speech sound and the reliability that the speech waveform signal (sound information) is not a non-speech sound, i.e., indicates a positive event and a negative event, respectively.

[0113] Then, the score determination unit 104 performs a weighting process on at least one of the reliability of the positive event and the reliability of the negative event, and performs a determination process.

[0114] By carrying out such weighting processing, it is possible to make a determination according to the attributes or type of the user and the contents of the conference.

[0115] In the above, the non-verbal sounds include at least one of laughter, coughing, nodding, sneezing, and keyboard sounds, but are not limited to these.

[0116] In addition, in the present disclosure, the speech recognition unit 102 performs speech recognition in a predetermined speech recognition unit (sentence unit, phrase unit, word unit, etc.), and the non-language speech recognition unit 103 performs non-language speech recognition in time units according to the predetermined speech recognition unit.

[0117] This enables recognition of non-speech sounds in accordance with the units of speech recognition.

[0118] The score determination unit 104 may determine the recognition result in a determination unit different from the speech recognition unit. For example, speech recognition and non-language speech recognition may be performed in units of words, and the score may be determined in units of sentences.

[0119] The block diagrams used in the description of the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. The method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by wire, wireless, etc.). The functional block may be realized by combining the one device or the multiple devices with software.

[0120] Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocation, mapping, assignment, etc. For example, a functional block (component) that performs the function of transmission is called a transmitting unit or a transmitter. In either case, as described above, there is no particular limitation on the method of realization.

[0121] For example, the voice recognition devices 100, 100a, and 100b in an embodiment of the present disclosure may function as a computer that performs processing of the voice recognition method of the present disclosure. The voice recognition device 100 will be described below, but the same applies to the voice recognition devices 100a and 100b. FIG. 16 is a diagram showing an example of a hardware configuration of the voice recognition device 100 according to an embodiment of the present disclosure. The above-mentioned voice recognition device 100 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

[0122] In the following description, the term "apparatus" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the speech recognition apparatus 100 may be configured to include one or more of the devices shown in the drawings, or may be configured to exclude some of the devices.

[0123] Each function in the speech recognition device 100 is realized by loading a specific software (program) onto hardware such as a processor 1001 and a memory 1002, causing the processor 1001 to perform calculations, control communications via a communication device 1004, and control at least one of reading and writing of data in the memory 1002 and the storage 1003.

[0124] The processor 1001, for example, operates an operating system to control the entire computer. The processor 1001 may be configured with a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the above-mentioned speech recognition unit 102 and non-language speech recognition unit 103, etc. may be realized by the processor 1001.

[0125] Moreover, the processor 1001 reads out a program (program code), a software module, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes according to the programs. As the programs, a program that causes a computer to execute at least a part of the operations described in the above-mentioned embodiment is used. For example, the voice recognition unit 102 may be realized by a control program stored in the memory 1002 and running on the processor 1001, and other functional blocks may be similarly realized. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may be transmitted from a network via a telecommunication line.

[0126] The memory 1002 is a computer-readable recording medium, and may be composed of at least one of, for example, a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Random Access Memory (RAM), etc. The memory 1002 may be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store a program (program code), software modules, etc. that are executable to implement a speech recognition method according to an embodiment of the present disclosure.

[0127] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. Storage 1003 may be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including at least one of memory 1002 and storage 1003.

[0128] The communication device 1004 is hardware (transmission / reception device) for performing communication between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 1004 may be configured to include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc., in order to realize at least one of, for example, Frequency Division Duplex (FDD) and Time Division Duplex (TDD). For example, the above-mentioned voice acquisition unit 101, etc. may be realized by the communication device 1004. The voice acquisition unit 101 may be implemented as a transmission unit and a reception unit that are physically or logically separated.

[0129] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0130] In addition, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0131] Furthermore, the speech recognition device 100 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0132] The notification of information is not limited to the aspects / embodiments described in the present disclosure, and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI), Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB), System Information Block (SIB))), other signals, or a combination thereof. In addition, the RRC signaling may be called an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.

[0133] The order of the steps, sequences, flow charts, etc. of each aspect / embodiment described in this disclosure may be changed unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0134] The input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be transmitted to another device.

[0135] The determination may be based on a value represented by a single bit (0 or 1), a Boolean (true or false) value, or a numerical comparison (e.g., comparison with a predetermined value).

[0136] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

[0137] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented as modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0138] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0139] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.

[0140] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0141] In addition, terms explained in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings.

[0142] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment" can be interpreted as "assuming," "expecting," "considering," etc.

[0143] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0144] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0145] Any reference to an element using a designation such as "first", "second" etc. used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

[0146] When "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Further, when used in this disclosure, the term "or" is not intended to be an exclusive or.

[0147] In this disclosure, where articles have been added due to translation, such as, for example, a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0148] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different." [Explanation of symbols]

[0149] 100: speech recognition device, 101: speech acquisition unit, 102: speech recognition unit, 103: non-language speech recognition unit, 104: score determination unit, 105: result output unit, 106: correction unit, 107: aggregation unit.

Claims

1. a sound information acquisition unit that acquires sound information; a sound information processing unit that performs speech recognition processing on first sound information by one speaker among the sound information to obtain a recognition result, and determines that second sound information by a different speaker related to the first sound information is the occurrence of a non-verbal event, and outputs information indicating the event in association with the recognition result; a correction unit that corrects the recognition result or the information indicating the event; With The sound information processing unit includes: performing speech recognition processing on the first sound information and the second sound information to obtain respective recognition results, and performing non-language speech recognition processing on the first sound information and the second sound information to determine occurrence of each event; the correction unit corrects the first sound information and the second sound information by using the recognition results or the information indicating the event, Speech recognition device.

2. A sound information acquisition unit that acquires sound information; a sound information processing unit that performs speech recognition processing on first sound information by one speaker among the sound information to obtain a recognition result, and determines that second sound information by a different speaker related to the first sound information is the occurrence of a non-verbal event, and outputs information indicating the event in association with the recognition result; Equipped with The sound information processing unit includes: performing a speech recognition process on the second sound information to obtain a recognition result; outputting a recognition result of the second sound information together with information indicating the event; performing output control to enable switching between presentation of the information indicating the event and presentation of the recognition result of the second sound information; Speech recognition device.

3. The sound information processing unit includes: adding additional information indicating the event to the recognition result; 3. The speech recognition device according to claim 1.

4. The sound information processing unit includes: outputting the recognition result and information indicating the event for each speaker; When adding additional information indicating an event of the second sound information by the one speaker to the recognition result of the first sound information by the one speaker, the additional information is output in association with the recognition result.

4. The speech recognition device according to claim 3.

5. The sound information processing unit includes: processing the recognition result according to the event; The speech recognition device according to any one of claims 1 to 4.

6. When an occurrence frequency of the event satisfies a predetermined condition, the sound information processing unit outputs information indicating the event in association with the recognition result. The speech recognition device according to any one of claims 1 to 5.

7. The sound information processing unit includes: associating information indicating the event with a recognition result for a group of sound information that includes the first sound information and satisfies a predetermined condition; The speech recognition device according to any one of claims 1 to 6.

8. The predetermined condition is that the sound information group is divided into sentence units, paragraph units, or topic units.

8. The speech recognition device according to claim 7.

9. the sound information processing unit, when an event caused by a speaker different from the one speaker of the first sound information occurs, adds information indicating the event to the end of the recognition result for the sound information group.

9. A speech recognition device according to claim 7 or 8.

Images