Audio information processing device and audio information processing method

Abstract

To provide a speech information processing device and a speech information processing method which enable a user to complete texting of a desired composition without carrying out complex work when ongoing speech production for texting is interrupted.SOLUTION: A speech information processing device 1 sequentially texts user's utterances during a speech reception period for receiving uttered speeches to be texted from a user, and automatically outputs contents uttered thus far from the user during the speed reception period when it can be recognized that user's speech production has been interrupted. Thus, in a situation where it can be recognized that user's speech production has been interrupted, the user can recognize the contents of compositions uttered and texted thus far by himself.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a voice information processing device and a voice information processing method, and is particularly suitable for use in a voice information processing device and a voice information processing method that convert a user's spoken voice into text. [Background technology]

[0002] Conventionally, there exists a voice information processing device that inputs a user's spoken voice, converts the input voice into text, and transmits the text as a message in a chat application or as an e-mail. By using this type of device, a user can transmit a text of a desired content to a partner by speaking, without performing an operation using his / her hands.

[0003] Patent Document 1 describes a technique in which, when an interrupt occurs while inputting a telephone number in a telephone, data processed up to that point is temporarily saved in a non-volatile memory, and the data is restored after the interrupt process ends. Patent Document 2 describes a technique in which, when a signal loss occurs while recording a received signal in a recording and playback device in a digital broadcast receiving system, a missing information signal for the missing time is generated and recorded in the recording and playback device, and video or audio is output for the missing portion based on the generated and recorded missing information signal. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2007-151188 A [Patent Document 2] JP 2003-319280 A Summary of the Invention [Problem to be solved by the invention]

[0005] When a user is converting his / her own speech into text using the above-mentioned conventional speech information processing device, the user's speech may be interrupted in the middle of the speech due to some reason. Then, after the speech is interrupted, when the reason for the interruption is resolved and the user resumes the speech for text conversion, the user often does not remember the exact content of the sentence that he / she has spoken up to that point, and does not know exactly where and what content of the sentence he / she should speak. In such a case, the user must temporarily cancel the sentence that has been converted into text up to that point, and start speaking again from the beginning of the sentence, which is troublesome for the user.

[0006] The present invention has been made to solve such problems, and has an object to enable a user to complete the conversion of a desired sentence into text without performing complicated tasks when speech is interrupted midway through the conversion. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, in the present invention, during a voice reception period in which spoken voice to be converted into text is received from the user, the user's speech is sequentially converted into text, while when it can be determined that the user's speech has been interrupted, the content of the utterances already spoken by the user during the voice reception period is automatically output as audio. Effect of the Invention

[0008] According to the present invention configured as described above, when the user's speech is deemed to have been interrupted, the speech content that the user has spoken up to that point is automatically output as voice. Therefore, by listening to the output voice, the user can understand the content of the sentence that he / she has spoken up to that point, and recognize how far he / she has spoken and where he / she should speak from. This allows the user to resume speaking from the middle of the sentence without canceling the sentence that has already been converted into text. Therefore, according to the present invention, when the speech for conversion to text is interrupted in the middle of the speech, the user can complete the conversion of the desired sentence into text without performing complicated work. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of the configuration of a voice information processing device according to a first embodiment of the present invention. [Diagram 2] 4 is a flowchart showing an operation example of the voice information processing device according to the first embodiment of the present invention. [Diagram 3] FIG. 11 is a block diagram showing an example of a functional configuration of a voice information processing device according to a second embodiment of the present invention. [Figure 4] 10 is a flowchart showing an operation example of the voice information processing device according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] First Embodiment Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of the functional configuration of a voice information processing device 1. The voice information processing device 1 according to this embodiment is a device mounted on a vehicle. The voice information processing device 1 has a function of providing a user with an environment for performing text chat in which multiple people exchange text messages. In particular, the voice information processing device 1 according to this embodiment has a function of inputting a voice spoken by a passenger (hereinafter simply referred to as a "user") using the device during a voice reception period (described later), converting the sentence indicated by the input voice into text, and transmitting the textual sentence as a message. By using this function, the user can create and transmit a message to be sent to the other party in a text chat without inputting the message manually. Hereinafter, the vehicle on which the voice information processing device 1 is mounted is referred to as the "own vehicle".

[0011] As shown in Fig. 1, a microphone 2 and a speaker 3 are connected to the voice information processing device 1. The microphone 2 is provided at a position where it can pick up the speech of a user aboard the vehicle. The microphone 2 picks up the speech and outputs an audio signal of the picked up speech. The speaker 3 is provided inside the vehicle, inputs an audio signal, and emits a speech based on the input audio signal.

[0012] As shown in Fig. 1, the audio information processing device 1 includes, as its functional configuration, an audio output unit 10, an audio input unit 11, and an audio information processing unit 12. The above-mentioned functional blocks 10 to 12 can be configured by any of hardware, a DSP (Digital Signal Processor), and software. For example, when configured by software, the above-mentioned functional blocks 10 to 12 are actually configured by a computer's CPU, RAM, ROM, etc., and are realized by the operation of a program stored in a recording medium such as a RAM, ROM, hard disk, or semiconductor memory. Among the functional configuration, the audio output unit 10 inputs an audio signal, drives a speaker 3 based on the input audio signal, and causes the speaker 3 to emit audio based on the audio signal.

[0013] The operation of the voice information processing device 1 when the operation mode is chat mode and a sentence spoken by the user is converted into text and sent as a message will be described below. The chat mode is an operation mode in which the user can use the voice information processing device 1 to perform text chat with a desired person (which may be multiple people). The transition to the chat mode is performed by the user operating the operation means (which may be a touch panel) of the voice information processing device 1 or by a voice instruction. At that time, the settings required for sending a message, such as the designation of the person to perform the text chat with, are also appropriately performed.

[0014] When a user wishes to convert a desired sentence into text using the voice information processing device 1 and send it as a message, the user speaks a message start word consisting of a predetermined fixed sentence, then speaks the sentence that the user wishes to convert into text, and then speaks a message end word consisting of a predetermined fixed sentence. An example of the message start word is the sentence "message start", and an example of the message end word is the sentence "message end". In other words, in this embodiment, the period from the end of speaking the message start word to the start of speaking the message end word is the period during which the voice of the sentence to be converted into text is accepted. This period corresponds to the "voice acceptance period".

[0015] Furthermore, in this embodiment, if the user wishes to send a sentence that has been spoken within the voice reception period after uttering the message end word, the user utters a message transmission word. An example of the message transmission word is the sentence "send message." In response to the user uttering the message transmission word, the sentence that the user has spoken is transmitted to the other party.

[0016] While the operation mode is the chat mode, the voice input unit 11 inputs a voice signal output by the microphone 2, performs analog / digital conversion processing including sampling, quantization, and encoding on the voice signal, and other signal processing to generate voice data (hereinafter referred to as "input voice data"), and buffers the data in the buffer 13. The buffer 13 is a storage area formed in a work area such as a RAM. The input voice data is voice waveform data sampled at a predetermined sampling period (16 kHz as an example).

[0017] The voice information processing unit 12 constantly analyzes the input voice data buffered in the buffer 13, and monitors whether or not the voice waveform of the message start word has appeared in the input voice data. In this embodiment, the voice pattern of the message start word (= the pattern of the voice waveform when the message start word is spoken) is registered in advance. Multiple voice patterns may be registered. The voice information processing unit 12 constantly compares the voice waveform of the input voice data with the voice pattern related to the message start word, and calculates the similarity using a predetermined method, and when the similarity reaches a certain level or more, it determines that the waveform of the message start word has appeared in the input voice data. Hereinafter, the voice information processing unit 12 detecting that the waveform of the message start word has appeared in the input voice data will be expressed as "the voice information processing unit 12 detects the message start word" as appropriate.

[0018] When the message start word is detected, the voice information processing unit 12 performs voice recognition on the input voice data buffered in the buffer 13 as needed, converts the sentences recorded in the input voice data into text, and writes them as text in the text data stored in a storage unit (not shown). Hereinafter, this process will be referred to as "text conversion process." The input voice data is appropriately converted into text by performing morphological analysis, syntactic structure analysis, semantic structure analysis, etc. based on existing technology related to natural language processing. Artificial intelligence technology may be used as part of the technology. Also, the voice information processing unit 12 may be configured to perform the text conversion process in cooperation with an external device. For example, the text conversion process may be performed in cooperation with a cloud server that provides a service of converting voice data into text.

[0019] In parallel with the text conversion process, the voice information processor 12 constantly analyzes the input voice data buffered in the buffer 13, and monitors whether or not the voice waveform of the message end word has appeared in the input voice data. This monitoring is performed in the same manner as the monitoring of whether or not the voice waveform of the message start word has appeared, based on the voice pattern of the message end word registered in advance.

[0020] When it is detected that the voice waveform of the message end word appears in the input voice data, the voice information processing unit 12 ends the text conversion process. After that, the voice information processing unit 12 analyzes the input voice data buffered in the buffer 13 as needed, and monitors whether the voice waveform of the message sending word appears in the input voice data. This monitoring is performed in the same manner as the monitoring of whether the voice waveform of the message start word appears, based on the voice pattern of the message sending word registered in advance.

[0021] When it is detected that the voice waveform of a message transmission word appears in the input voice data, the voice information processing unit 12 transmits a message regarding the text described in the sentence data to a predetermined server via the network N in accordance with a protocol.

[0022] Furthermore, in parallel with the text conversion process, the voice information processing unit 12 analyzes the input voice data buffered in the buffer 13 at any time and monitors whether the voice waveform of the cancel word appears in the input voice data. The cancel word is, for example, the sentence "message canceled." The monitoring is performed in the same manner as the monitoring of whether the voice waveform of the message start word appears, based on the voice pattern of the cancel word registered in advance. When it detects that the voice waveform of the cancel word appears in the input voice data, the voice information processing unit 12 cancels the text conversion process and further deletes the text that has been written in the sentence data up to that point. After that, the voice information processing unit 12 starts monitoring again whether the voice waveform of the message start word appears in the input voice data.

[0023] Furthermore, the voice information processing unit 12 executes the following process during the execution of the text conversion process, that is, from when the message start word is detected until the text conversion process is ended or canceled. That is, it determines whether or not a period during which the user does not speak has continued for a predetermined time or more. A period during which the user does not speak for a predetermined time or more means the following. For example, suppose that the user utters the sentence "Hello." In this case, it means that after the utterance of "Hello" is completed, a predetermined time or more has passed without the user uttering anything.

[0024] The voice information processing unit 12 analyzes the input voice data, and when the sound pressure value of the voice waveform once exceeds a first threshold (a threshold for determining that speech has been made), then falls below a second threshold (a threshold for determining that speech has not been made; this may be the same value as the first threshold) and the state of being below the second threshold continues for a predetermined period of time or more, it determines that the period during which the user has not made speech has exceeded the predetermined period of time. However, any method of determination may be used.

[0025] When detecting that a period of time during which the user does not speak has continued for a predetermined time or longer, the voice information processing unit 12 executes the following process. That is, the voice information processing unit 12 causes the voice output unit 10 to output, by voice, the sentence indicated by the text described up to that point in the sentence data (=text already generated in the text conversion process). Hereinafter, the voice output by the voice output unit 10 in this manner is referred to as "already-converted voice", and the voice information processing unit 12 causing the voice output unit 10 to output the already-converted voice is simply expressed as "the voice information processing unit 12 outputs the already-converted voice". The already-converted voice corresponds to "voice corresponding to the content of the utterance already uttered by the user during the voice reception period" in the claims.

[0026] To describe in detail the processing of the voice information processing unit 12, the voice information processing unit 12 generates voice data for outputting the sentence indicated by the text described in the sentence data as voice. The voice data is appropriately generated by existing technology such as voice synthesis technology. The voice information processing unit 12 then outputs a voice signal based on the voice data to the voice output unit 10, thereby emitting a voice based on the voice data from the speaker 3.

[0027] Thereafter, the voice information processing unit 12 continues to execute the text conversion process, and when there is an utterance for conversion to text, converts the uttered sentence into text, while when there is an utterance by the user and a period of time during which the user does not make any utterance thereafter continues for a predetermined period of time or more, the voice information processing unit 12 again outputs the converted voice to text. The voice information processing unit 12 also executes detection of the message end word and the cancel word in parallel.

[0028] With the above configuration, the voice information processing device 1 operates, for example, in the following manner. For example, assume that the user is thinking of uttering a sentence, "I'm on my way by car now. I've just passed point A. My estimated arrival time is 13:00. The road is congested, so I may arrive late. I'll contact you again when I get closer." Then, assume that the user utters the message start word, and then stops uttering the sentence, "I'm on my way by car now. I've just passed point A," for some reason. One example of the reason is that the vehicle is approaching an intersection or has started parking, so it becomes necessary to concentrate on driving, or that the user has to pay a toll at a gate on the road. Note that the part uttered by the user is converted into text by the text conversion process of the voice information processing unit 12, and the text is written in the text data.

[0029] In this case, after the sentence "~We have just passed point A," if a predetermined time or more has passed without any speech, the speech information processing unit 12 of the speech information processing device 1 according to this embodiment automatically causes the speech output unit 10 to output speech related to the already generated text. In this example, the sentence "We are now on our way by car. We have just passed point A" is output by speech.

[0030] As a result of the above process, the following effects are achieved. That is, when the user resumes the speech for text conversion after interrupting the speech for text conversion, the user is required to newly speak from the sentence immediately after the part that was completed. However, it may happen that the user does not accurately remember the content of the sentence that was spoken so far and does not know exactly where to start and what content of the sentence to utter. In this example, the user should start speaking from "The estimated time of arrival is...", but it may happen that the user does not know exactly how far he or she has spoken or where to start speaking. In such a case, it is possible to cancel the voice input up to that point by speaking a cancel word and start the work of converting the desired sentence into text and sending it as a message from the beginning, but such work is cumbersome for the user.

[0031] On the other hand, the present embodiment provides the following effect. That is, if a period of time during which the user does not speak continues for a considerable period of time, it can be considered that the user's speech has been interrupted. This is because, usually, when a user speaks a series of sentences to be sent as a message in order to convert them into text, there is no case where the user does not speak for an unnecessarily long period of time during the speech.

[0032] According to the voice information processing device 1 of this embodiment, when the user's speech is interrupted, the text that the user has converted to text is automatically output as voice. Therefore, the user can understand the content of the text that the user has converted to text by listening to the output voice. This allows the user to resume speaking from the middle of the text without canceling the text that has already been converted to text. Therefore, according to this embodiment, when the speech is interrupted in the middle of the speech for conversion to text, the user can complete the conversion of the desired text to text without performing any cumbersome work.

[0033] In addition, after the user stops speaking for text conversion, the user may forget that he / she was speaking for text conversion. In such a case, according to the present embodiment, the sentence that has already been converted into text based on the user's speech is automatically output as a voice, which can make the user realize that he / she was in the middle of speaking (naturally, the content of the sentence that has already been spoken can also be recognized).

[0034] Next, a method of processing voice information by the voice information processing device 1 will be described with reference to a flowchart. The flowchart in Fig. 2 shows the operation of the voice information processing unit 12 when the chat mode is on. As shown in Fig. 2, the voice information processing unit 12 constantly analyzes input voice data buffered in the buffer 13, and monitors whether or not the voice waveform of the message start word has appeared in the input voice data (step SA1). If it has appeared (step SA1: YES), the voice information processing unit 12 starts a text conversion process (step SA2).

[0035] Next, the voice information processing unit 12 monitors whether or not the voice waveform of the message end word appears in the input voice data (step SA3), whether or not the voice waveform of the cancel word appears in the input voice data (step SA4), and whether or not the period during which no speech is made has exceeded a predetermined time (step SA5). If the voice waveform of the message end word appears in the input voice data (step SA3: YES), the voice information processing unit 12 ends the text conversion process (step SA6) and monitors whether or not the voice waveform of the message transmission word appears in the input voice data (step SA7). If the voice waveform of the message transmission word appears in the input voice data (step SA7: YES), the voice information processing unit 12 transmits a message for the text described in the sentence data (step SA8).

[0036] If the voice waveform of the cancel word appears in the input voice data (step SA4: YES), the voice information processing unit 12 cancels the text conversion process (step SA9) and returns the processing procedure to step SA1.

[0037] If the period of no speech has been longer than the predetermined time (step SA5: YES), the voice information processing unit 12 causes the voice output unit 10 to output by voice the sentence indicated by the text described in the text data up to that point (= the text already generated in the text conversion process) (step SA10). After that, the voice information processing unit 12 returns the processing procedure to step SA3.

[0038] <Modification of the first embodiment> In the first embodiment, when a period during the voice reception period during which no speech is made reaches a predetermined time or longer, the voice information processing unit 12 is configured to output a sentence indicated by text already generated up to that point as voice (already-converted voice) to the voice output unit 10. In this regard, the voice information processing unit 12 may be configured to output a voice based on the voice data stored in the buffer 13 (= a recorded voice of the user's speech) to the voice output unit 10 instead of the already-converted voice. In this configuration, the voice (= a recorded voice of the user's speech) output instead of the already-converted voice corresponds to "a voice corresponding to the content of the speech already uttered by the user during the voice reception period" in the claims.

[0039] In this case, for example, the voice information processing unit 12 extracts voice data corresponding to a portion already spoken by the user during the voice receiving period from the input voice data stored in the buffer 13, and outputs a voice signal based on the extracted voice data to the voice output unit 10. This modification is applicable to the second embodiment (including the modification of the second embodiment) described later.

[0040] <Second embodiment> Next, a second embodiment will be described. FIG. 3 is a block diagram showing an example of a functional configuration of a voice information processing device 1A according to this embodiment. In the following description of the second embodiment, the same elements as those in the first embodiment are given the same reference numerals, and detailed description thereof will be omitted. In this embodiment, for convenience of explanation, it is assumed that the user who uses the voice information processing device 1A is the driver. However, this is for convenience of explanation, and naturally, a passenger other than the driver may be the user who uses the voice information processing device 1A.

[0041] As is clear from a comparison between Fig. 1 and Fig. 3, the voice information processing device 1A according to this embodiment includes a voice information processing device 12A instead of the voice information processing device 12 according to the first embodiment. A camera 14 is connected to the voice information processing device 1A according to this embodiment. The camera 14 is provided at a position where it can capture an image of the upper body including the face of a user when the user is seated in the driver's seat. The camera 14 captures images at a predetermined cycle and outputs captured image data based on the captured image to the voice information processing device 12A.

[0042] The voice information processing unit 12 according to the first embodiment causes the voice output unit 10 to output voice related to sentences that have already been converted into text (already-converted voice) when a period during which the user does not speak continues for a predetermined time or longer during a voice reception period. On the other hand, the voice information processing unit 12A according to the present embodiment outputs already-converted voice when the user does not speak for a predetermined time or longer after the user's face moves in a manner that looks outside through a side window.

[0043] In more detail, when the voice information processing unit 12A detects a message start word, it recognizes an image of the upper half of a person's body (= an image of the upper half of the user's body) in the captured image data input from the camera 14 at a predetermined cycle using existing recognition technology, and continuously analyzes the image of the upper half of the body to monitor whether or not an action that would be taken when looking outside through a side window (turning the face toward the side window and looking outside) has been performed. This monitoring is performed based on existing action recognition technology. It goes without saying that this monitoring may also be performed using a model learned by deep learning or other machine learning techniques.

[0044] When the voice information processing unit 12A detects that the user has performed an action that would be taken when looking outside through a side window, if a period of time after the detection during which the user does not speak continues for a predetermined period of time or longer, the voice information processing unit 12A automatically outputs the converted voice.

[0045] According to this embodiment, the following effects are achieved. That is, when it is considered that the user's speech has been interrupted, the sentence that the user has already converted to text is automatically output as a voice, so that the same effects as those of the first embodiment can be obtained. Furthermore, when the user looks outside through the side window and no speech for conversion to text is made for a predetermined time or more, it can be strongly presumed that the driver has interrupted the speech for conversion to text due to looking at the scenery outside, compared to when the period of no speech simply continues for a predetermined time or more. Based on this, according to this embodiment, it is possible to output the already converted to text voice in a situation where it can be more strongly presumed that the speech has been interrupted, compared to the first embodiment.

[0046] Next, the operation of the voice information processing device 1A according to this embodiment will be described with reference to the flowchart of FIG. 4. In the flowchart of FIG. 4, the same processes as those in the flowchart of FIG. 2 are assigned the same step numbers, and their description will be omitted. As shown in FIG. 4, the voice information processing device 1A according to this embodiment executes a process in step SB1 that is different from the process in step SA5 of FIG. 2. That is, in step SB1, the voice information processing unit 12A monitors whether or not the period during which the driver's face moves in a manner that the driver is looking outside through the side window and no further speech is made has exceeded a predetermined time. Then, in step SB1, if the period during which the driver's face moves in a manner that the driver is looking outside through the side window and no further speech is made has exceeded a predetermined time (step SB1: YES), the processing procedure proceeds to step SA1. Note that, in the second embodiment, the voice information processing unit 12A may be configured to output a recorded voice of the user's speech instead of the already-texted voice, as described in the modified example of the first embodiment.

[0047] <First Modification of Second Embodiment> Next, a first modified example of the second embodiment will be described. In the second embodiment, the voice information processing unit 12A outputs a pre-text voice when the user (driver) does not speak for a predetermined time or more after the user's face moves in a manner that the user is looking outside through the side window. In this regard, the voice information processing unit 12A according to this modified example executes the following process. Note that this modified example is based on the premise that a car navigation device is installed in the vehicle. That is, the voice information processing unit 12A according to this modified example outputs a pre-text voice based on the input from the camera 14 when the user does not speak for a predetermined time or more after the user's face moves in a manner that the user is looking at the display screen of the car navigation device.

[0048] The operation of the voice information processing device 1A according to this modified example will be explained with reference to the flowchart in Figure 4. In step SB1, the voice information processing unit 12A monitors whether or not the user has spoken for a predetermined period of time or more after the user's face moves in a manner that indicates that the user is looking at the display screen of the car navigation device.

[0049] If the user does not make a speech for conversion to text for a predetermined time or more after looking at the display screen of the car navigation device, it can be strongly estimated that the driver stopped making a speech for conversion to text due to looking at the display screen. Based on this, according to this modification, it is possible to obtain the same effect as the second embodiment.

[0050] In the second embodiment and this modification, the voice information processing unit 12A monitors whether the user's face has moved in a predetermined manner based on the image captured by the camera 14, and when the user does not speak for a predetermined time or more after the face has moved in the predetermined manner, causes the voice output unit 10 to output a voice related to the already generated text. However, the example of the configuration is not limited to the illustrated one. As an example, the configuration may be such that the voice information processing unit 12A monitors whether the user's face has moved in a manner in which the user is looking at other passengers other than the user, and executes a corresponding process, or may monitor whether the user's face has moved in a manner in which the user is looking at a member provided in the vehicle, such as a rearview mirror or a side mirror, and executes a corresponding process. As described in the modification of the first embodiment, in the first modification of the second embodiment, the voice information processing unit 12A may output a recorded voice of the user's speech instead of the already converted voice.

[0051] <Second Modification of Second Embodiment> Next, a second modified example of the second embodiment will be described. The voice information processing unit 12A according to this modified example monitors whether the user's face has become an expression concentrating on driving based on the photographing result of the camera 14, and outputs the already-converted voice when the user does not speak for a predetermined time or more after the expression has become. The monitoring is performed based on existing facial expression recognition technology. Of course, the monitoring may be performed using a model learned by deep learning or other machine learning techniques. It is assumed that the driver concentrates on driving and this is reflected in the facial expression when the vehicle is about to enter an intersection, while entering the intersection, when parked in a parking lot, when driving on a congested road, etc.

[0052] The operation of the voice information processing device 1A according to this modified example will be described with reference to the flowchart in Figure 4. In step SB1, the voice information processing unit 12A monitors whether the user has not spoken for a predetermined period of time or more after the user's face has taken on an expression of concentration on driving.

[0053] If the user does not make a speech for conversion to text for a predetermined period of time after the user's facial expression shows that he or she is concentrating on driving, it can be strongly assumed that the driver has stopped making a speech for conversion to text due to the driver's concentration on driving. Therefore, according to the configuration of this modification, it is possible to obtain the same effect as the second embodiment.

[0054] In this modification, the voice information processing unit 12A monitors whether the user's face has a predetermined expression based on the image captured by the camera 14, and when the user does not speak for a predetermined time or more after the predetermined expression, causes the voice output unit 10 to output a voice related to the already generated text. However, the example of the configuration is not limited to the illustrated one. As an example, it may be configured to monitor whether the user's face has a surprised expression and execute a corresponding process. Also, in the second modification of the second embodiment, as described in the modification of the first embodiment, the voice information processing unit 12A may be configured to output a recorded voice of the user's speech instead of the already converted voice.

[0055] <Third Modification of Second Embodiment> Next, a third modified example of the second embodiment will be described. The audio information processing unit 12A according to this modified example monitors whether or not the user has started yawning based on the image capturing result of the camera 14. When the audio information processing unit 12A detects that the user has started yawning, it monitors whether or not the yawning has ended, and outputs the converted-to-text audio when it detects that the yawning has ended. The detection of the start / end of the yawning is performed based on existing image recognition technology. It goes without saying that the monitoring may be performed using a model learned by deep learning or other machine learning techniques.

[0056] The operation of the voice information processing device 1A according to this modified example will be described with reference to the flowchart in FIG. 4. In step SB1, after the user starts yawning, the voice information processing unit 12A monitors whether the yawning has ended.

[0057] While the user is yawning, it can be considered that the user has stopped speaking for text conversion due to the yawning. Based on this, according to the configuration of this modification, it is possible to obtain the same effect as the second embodiment. As described in the modification of the first embodiment, in the third modification of the second embodiment, the voice information processing unit 12A may be configured to output a recorded voice of the user's speech instead of the already-converted voice.

[0058] <Fourth Modification of the Second Embodiment> Next, a fourth modified example of the second embodiment will be described. The audio information processing unit 12A according to this modified example monitors whether or not the user has started a telephone call based on the image captured by the camera 14. When the audio information processing unit 12A detects that the user has started a telephone call, it monitors whether or not the call has ended, and outputs the already-converted voice when it detects that the call has ended. Furthermore, the audio information processing unit 12A stops converting audio data based on the voice input by the audio input unit 11 into text during the period during which the user is making a call (=the period from when the start of the call is detected to when the end of the call is detected). It is assumed that the user's telephone call is a call made by the user using his or her own mobile phone in the car.

[0059] The detection of the start / end of a call by a user is performed based on an existing image recognition technology. Of course, the monitoring may be performed using a model learned by deep learning or other machine learning techniques. In this modification, the voice information processing unit 12A detects the start / end of a call based on the image capture result of the camera 14, but the method of detection is not limited to the exemplified method. As an example, the voice information processing device 1A and a mobile phone are communicatively connected, and a predetermined signal is transmitted from the mobile phone to the voice information processing device 1A at the start and end of a call, and the voice information processing unit 12A may detect the start / end of a call based on the predetermined signal.

[0060] The operation of the voice information processing device 1A according to this modified example will be described with reference to the flowchart in FIG. 4. In step SB1, after a user starts a telephone call, the voice information processing unit 12A monitors whether the call has ended.

[0061] While the user is talking on the phone, it can be considered that the user has interrupted the speech for conversion to text due to the fact. Therefore, according to this modification, the same effect as in the second embodiment can be obtained. Furthermore, the speech input by the speech input unit 11 during a call is not speech for conversion to text, but speech for a telephone call, and should not be converted to text. And according to this modification, it is possible to prevent speech that should not be converted to text from being converted to text.

[0062] In the fourth modified example, the voice information processing unit 12A may execute the following process when outputting the already-converted voice to the voice output unit 10. That is, the voice information processing unit 12A may execute the following process to output, to the voice output unit 10, a sentence indicating that the voice input unit 11 has stopped converting the voice inputted into text during a user's call, together with the already-converted voice (voice corresponding to the spoken content). For example, the voice information processing unit 12A executes the following process. That is, the voice information processing unit 12A first outputs the already-converted voice. Next, the voice information processing unit 12A outputs a sentence with the content "Voice during a telephone call has not been converted into text. You can continue inputting." The voice data that is the source of the voice is prepared in advance. The process illustrated is merely an example, and for example, the voice information processing unit 12A may execute the following process to output, as a voice, a sentence indicating that the voice input unit 11 has stopped converting the voice inputted into text during a user's call, and then output the already-converted voice.

[0063] Also, in the fourth modification, the voice information processing unit 12A may be configured to output a recorded voice of the user's speech instead of the already-converted voice, as described in the modification of the first embodiment.

[0064] In the third and fourth modified examples, an example of a configuration was described in which the voice information processing unit 12A detects that the user is in a predetermined state in which he or she cannot make a speech for text conversion, and outputs the already-converted voice when the predetermined state is resolved after the user has entered the predetermined state. However, the example of the configuration is not limited to the illustrated one. As an example, the voice information processing unit 12A may be configured to detect that the user has started and finished eating, and output the already-converted voice when the user has finished eating.

[0065] Although the embodiments (including modifications) of the present invention have been described above, each of the above-mentioned embodiments merely shows an example of the realization of the present invention, and the technical scope of the present invention should not be interpreted as being limited thereby. In other words, the present invention can be embodied in various forms without departing from the gist or main characteristics thereof.

[0066] For example, in the first embodiment, the transmission of text by the voice information processing unit 12 is performed as the transmission of a message in a text chat, but the transmission of text is not limited to the manner exemplified in each embodiment. For example, the transmission of text may be performed by email. Furthermore, the transmission of text does not only mean the transmission to a specific person, but is a concept that broadly includes the transmission of text to an external device, such as the transmission of text to a server or a specific host device. For example, the transmission of text also includes the transmission of text according to a protocol to a message posting site or a bulletin board site. The above also applies to the second embodiment.

[0067] In the first embodiment, the voice information processing device 1 is provided in a vehicle. However, the voice information processing device 1 does not necessarily have to be a device provided in a vehicle. The same applies to the second embodiment. That is, the present invention is widely applicable to voice information processing devices that convert a user's utterance into text.

[0068] In the above embodiments, the voice reception period is started by the user uttering the message start word. In this regard, the voice reception period may be started when the user performs a predetermined operation on a touch screen or other input means, or when the user makes a predetermined gesture in a configuration capable of detecting gestures. The same applies to the message end word, message send word, and cancel word. [Explanation of symbols]

[0069] 1, 1A Voice information processing device 10 Audio output section 11 Audio input section 12, 12A Audio information processing section 14 Camera

Claims

1. A voice input unit for inputting voice; an audio output unit that outputs audio; a voice information processing unit that converts into text the voice input unit during a voice reception period during which a spoken voice to be converted into text is received from a user; The audio information processing unit includes: During the voice reception period, the voice output unit sequentially converts the user's utterance into text, while automatically outputting the contents of the utterance already uttered by the user during the voice reception period by voice when it can be determined that the user's utterance has been interrupted, A camera is connected to capture the user's face. The audio information processing unit monitors whether or not the user's face has moved in a predetermined manner based on the photographing result of the camera, and when the user does not speak for a predetermined period of time or more after the face has moved in the predetermined manner, causes the audio output unit to output a sound corresponding to the content of the utterance.

13. A voice information processing apparatus comprising:

2. Provided in the vehicle, 2. The voice information processing apparatus according to claim 1, wherein the predetermined manner is a manner in which the face of the user is looking outside through a side window.

3. A vehicle equipped with a car navigation device, 2. The voice information processing apparatus according to claim 1, wherein the predetermined manner is a manner in which the face of the user is looking at a display screen of a car navigation system.

4. A voice input unit for inputting voice; an audio output unit that outputs audio; a voice information processing unit that converts into text the voice input unit during a voice reception period during which a spoken voice to be converted into text is received from a user; The audio information processing unit includes: During the voice reception period, the voice output unit sequentially converts the user's utterance into text, while automatically outputting the contents of the utterance already uttered by the user during the voice reception period by voice when it can be determined that the user's utterance has been interrupted, A camera is connected to capture the user's face. The voice information processing unit monitors whether or not the user's face has a predetermined expression based on the photographing result of the camera, and when the user does not speak for a predetermined time or more after the predetermined expression has been observed, outputs a voice corresponding to the speech content to the voice output unit.

13. A voice information processing apparatus comprising:

5. Provided in the vehicle, The predetermined facial expression is a facial expression of the user concentrating on driving.

5. The audio information processing apparatus according to claim 4.

6. A voice input unit for inputting voice; an audio output unit that outputs audio; a voice information processing unit that converts into text the voice input unit during a voice reception period during which a spoken voice to be converted into text is received from a user; the voice information processing unit sequentially converts the user's speech into text during the voice reception period, while automatically outputting, to the voice output unit, speech content already uttered by the user during the voice reception period when it can be determined that the user's speech has been interrupted, the voice information processing unit detects that a user is in a predetermined state in which the user cannot make an utterance for conversion into text, and, after the user is in the predetermined state, causes the voice output unit to output a voice corresponding to the utterance content when the predetermined state is resolved; A camera is connected to capture the user's face. The audio information processing unit detects whether the user has started to yawn based on the image capturing result of the camera, and outputs audio corresponding to the yawning content to the audio output unit when the user has started to yawn and then finished yawning.

13. A voice information processing apparatus comprising:

7. The voice information processing unit outputs, when outputting the spoken content by voice to the voice output unit, a sentence indicated by a text that has already been generated.

7. The voice information processing device according to claim 1, wherein the voice information processing device further comprises: a first input section;

8. The voice information processing unit outputs a recorded voice of the user's speech when the voice output unit outputs the speech content by voice.

7. The voice information processing device according to claim 1, wherein the voice information processing device further comprises: a first input section;

9. A first step of converting a voice input by a voice input unit of the voice information processing device into text during a voice reception period during which a voice information processing unit of the voice information processing device receives an uttered voice to be converted into text from a user; a second step of the voice information processing unit of the voice information processing device sequentially converting the user's utterance into text during the voice receiving period, while automatically outputting the contents of the utterance already uttered by the user during the voice receiving period to the voice output unit of the voice information processing device by voice when the user's utterance can be considered to have been interrupted; A camera is connected to capture the user's face. In the second step, the audio information processing unit monitors whether or not the user's face has moved in a predetermined manner based on the photographing result of the camera, and when the user has not spoken for a predetermined period of time or more after the face has moved in the predetermined manner, causes the audio output unit to output a voice corresponding to the spoken content.

23. A method for processing audio information comprising the steps of:

10. A first step of converting a voice input by a voice input unit of a voice information processing device into text during a voice reception period during which a voice information processing unit of the voice information processing device receives a spoken voice to be converted into text from a user; a second step of the voice information processing unit of the voice information processing device sequentially converting the user's utterance into text during the voice receiving period, while automatically outputting the contents of the utterance already uttered by the user during the voice receiving period to the voice output unit of the voice information processing device by voice when the user's utterance can be considered to have been interrupted; A camera is connected to capture the user's face. In the second step, the voice information processing unit monitors whether or not the user's face has a predetermined expression based on the photographing result of the camera, and when the user does not speak for a predetermined time or more after the predetermined expression has been obtained, outputs a voice corresponding to the utterance content to the voice output unit.

23. A method for processing audio information comprising the steps of:

11. A first step of converting a voice input by a voice input unit of a voice information processing device into text during a voice reception period during which a voice information processing unit of the voice information processing device receives a spoken voice to be converted into text from a user; a second step of the voice information processing unit of the voice information processing device sequentially converting the user's utterance into text during the voice receiving period, while automatically outputting the contents of the utterance already uttered by the user during the voice receiving period to the voice output unit of the voice information processing device by voice when the user's utterance can be considered to have been interrupted; In the second step, the voice information processing unit detects that a user is in a predetermined state in which the user cannot make an utterance for converting the voice into text, and, after the user is in the predetermined state, causes the voice output unit to output a voice corresponding to the utterance content when the predetermined state is resolved; A camera is connected to capture the user's face. In the second step, the audio information processing unit detects whether the user has started yawning based on the image capturing result of the camera, and causes the audio output unit to output audio corresponding to the yawning content when the user has started yawning and then finished yawning.

23. A method for processing audio information comprising the steps of:

Images