Audio output device and audio output method

Abstract

To provide a voice output device and a voice output method for preventing a subject from missing a message.SOLUTION: A voice output device 1 includes a voice output control unit 12 that determines whether or not a start condition that is established when a subject can normally listen to a voice in a predetermined space is established when a message is received by a message reception part 11, causes a voice output unit to start voice output of the message when the start condition is established, and suspends voice output of the message by the voice output unit when not being established. Voice output is not performed immediately in response to reception of the message but voice output is performed only when the subject is in a state of being able to normally listen to the message, and in other cases, voice output of the message is suspended.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an audio output device and an audio output method, and is particularly suitable for use in an audio output device and an audio output method having a function of outputting a received message by voice. [Background technology]

[0002] Conventionally, there exists a voice output device that is installed in a space formed in a vehicle or a room in a house, has a function of receiving messages in a chat app or messages (emails) in an email system, and outputs the contents of the message as voice when the message is received. This type of voice output device usually outputs the message as voice as soon as the message is received so that the target person can immediately recognize the contents of the message in response to receiving the message. Patent Document 1 describes an in-vehicle information device that recognizes the contents of the voice uttered by a vehicle occupant and outputs the contents of the voice in a manner according to the environment around the vehicle or the operating state of the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2014 / 002128 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional voice output device described above, the message is output as soon as it is received. Therefore, even when the intended recipient is in a situation where he or she cannot hear the message or cannot concentrate on listening to the message, voice output is always performed in response to the reception of the message, which can cause the intended recipient to miss the message.

[0005] The present invention has been made to solve such problems, and has an object to prevent a target person from missing a message. [Means for solving the problem]

[0006] In order to solve the above problems, in the present invention, an audio output device is installed in a specified space and has an audio output unit that outputs audio. When a message is received, it is determined whether a start condition is met, which is met when the subject can normally hear the audio in the specified space, and if the start condition is met, the audio output unit is made to start audio output of the message, and if the start condition is not met, the audio output unit is made to suspend audio output of the message. In this case, to determine whether the start condition is met, the photographed image data generated by photographing the subject's face is analyzed using facial expression recognition technology. ,versus It is determined whether or not the condition that the subject is in a relaxed state is satisfied. Effect of the Invention

[0007] According to the present invention configured as described above, audio output is not performed immediately upon receiving a message, but is performed only when the target person is in a state in which he or she can normally hear the message, and audio output of the message is put on hold in other cases, thereby preventing the target person from missing out on the message. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of a functional configuration of a sound output device and a mobile terminal according to a first embodiment of the present invention. [Diagram 2] 4 is a flowchart showing an example of the operation of the audio output device according to the first embodiment of the present invention. [Diagram 3] 11 is a block diagram showing an example of the functional configuration of a sound output device and a mobile terminal according to a second embodiment of the present invention. FIG. [Figure 4] 10 is a flowchart showing an example of the operation of the audio output device according to the second embodiment of the present invention. [Diagram 5] FIG. 11 is a block diagram showing an example of the functional configuration of a sound output device and a mobile terminal according to a third embodiment of the present invention. [Figure 6] 13 is a flowchart showing an example of the operation of the audio output device according to the third embodiment of the present invention. [Figure 7] FIG. 13 is a block diagram showing an example of the functional configuration of a sound output device and a mobile terminal according to a fourth embodiment of the present invention. [Figure 8] 13 is a flowchart showing an example of the operation of the audio output device according to the fourth embodiment of the present invention. [Figure 9] FIG. 13 is a block diagram showing an example of the functional configuration of a sound output device and a mobile terminal according to a fifth embodiment of the present invention. [Figure 10] 13 is a flowchart showing an example of the operation of the audio output device according to the fifth embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] First Embodiment Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of the functional configuration of an audio output device 1 according to this embodiment, and a mobile terminal 2 connected to the audio output device 1. The audio output device 1 is a device installed in an interior space (corresponding to the "predetermined space" in the claims) formed inside a vehicle, and as an example, a so-called car navigation system can function as the audio output device 1. In this embodiment, the vehicle in which the audio output device 1 is installed is not an autonomous vehicle, but is basically an automobile that runs by being driven by a driver (however, this includes an automobile in which a certain level of autonomous driving is performed in a limited environment). Hereinafter, the vehicle in which the audio output device 1 is installed is referred to as the "own vehicle."

[0010] As shown in Fig. 1, the audio output device 1 is connected to an audio processing device 3 provided in the vehicle interior space. The audio processing device 3 includes a D / A converter, an amplifier, and a speaker provided in the vehicle, and performs D / A conversion on an input audio signal, amplifies the signal, and outputs the signal as audio from the speaker. The audio output device 1 is also connected to an in-vehicle camera 4. The in-vehicle camera 4 is an imaging device provided in the vehicle interior space, and captures an image of an area including the driver's seat at a predetermined period, and outputs captured image data based on the captured image data to an audio output control unit 12 (described later) of the audio output device 1.

[0011] The mobile terminal 2 is a mobile terminal brought into the vehicle by a passenger of the vehicle, and as an example, a so-called smartphone, a mobile phone other than a smartphone, or a tablet computer without a telephone function can function as the mobile terminal 2. In this embodiment, for the sake of simplicity, it is assumed that the mobile terminal 2 is a terminal owned by the driver. A text chat application (hereinafter referred to as a "chat app") that enables messages to be exchanged in chat format with other people is installed in the mobile terminal 2. The chat app has a function of providing a user interface related to text chat on the mobile terminal 2, as well as a function of transmitting text data for voice output to the voice output device 1 in response to reception of message data related to a message (this function will be described in detail later).

[0012] As shown in FIG. 1, the audio output device 1 includes, as its functional components, an audio output device side communication unit 10, a message receiving unit 11, an audio output control unit 12, and an audio output unit 13. The mobile terminal 2 includes, as its functional components, a mobile terminal side communication unit 20 and a chat application execution unit 21. The above-mentioned functional blocks 10-13, 20, and 21 can be configured by any of hardware, a DSP (Digital Signal Processor), and software. For example, when configured by software, the above-mentioned functional blocks 10-13, 20, and 21 are actually configured by including a CPU, RAM, ROM, etc. of a computer, and are realized by the operation of a program stored in a recording medium such as a RAM, ROM, hard disk, or semiconductor memory. The above also applies to other embodiments described later.

[0013] The audio output device side communication unit 10 of the audio output device 1 and the mobile terminal side communication unit 20 of the mobile terminal 2 communicate wirelessly according to a predetermined communication standard related to wireless communication. The communication standard related to wireless communication is, for example, Bluetooth (registered trademark) or a communication standard related to wireless LAN. Note that the audio output device 1 and the mobile terminal 2 may be connected by wire, and the audio output device side communication unit 10 and the mobile terminal side communication unit 20 may communicate by wire according to a predetermined communication standard related to wired communication. The chat application execution unit 21 of the mobile terminal 2 executes various processes by hardware such as a CPU reading and executing the chat application and a program associated with the chat application (including a part of a program of the OS and an API).

[0014] When the operation mode is the message notification mode, the audio output device 1 has a function of outputting audio information related to a message (hereinafter referred to as "message audio") in response to reception of message data related to a text chat message by the mobile terminal 2. The driver (target person) who is the owner of the mobile terminal 2 can confirm the contents of the message by listening to the message audio output by the audio output device 1. Note that the transition to the message notification mode is performed by an explicit instruction from the driver when both the audio output device 1 and the mobile terminal 2 are powered on. Below, the processing of the mobile terminal 2 and the audio output device 1 after the mobile terminal 2 receives message data when the operation mode is the message notification mode will be described in detail.

[0015] The mobile terminal 2 can access a network N including the Internet. The network N may be accessed by directly accessing a mobile communication network, or by indirectly accessing the mobile communication network using a tethering function of a portable router. When message data related to a text chat message is transmitted to the mobile terminal 2 from another terminal on which a chat application is installed, the chat application execution unit 21 of the mobile terminal 2 receives the message data. In response to receiving the message data, the chat application execution unit 21 executes processing based on a pre-set setting, such as sounding a ringtone to notify the user that a message has been received, or displaying necessary information on the display unit of the mobile terminal 2.

[0016] In addition to this process, the chat application execution unit 21 according to this embodiment executes a process of generating text data for voice output and transmitting it to the mobile terminal 2. The text data for voice output is data in which the supplementary information and the message body included in the message data are described in text. As will be apparent later, the information described as text in the text data for voice output is ultimately output as a message voice by the voice output device 1. The supplementary information is information for supplementing matters related to the message before outputting the message body as voice. In this embodiment, the supplementary information includes the name of the sender and the date and time of receipt of the message data.

[0017] For example, suppose that chat application execution unit 21 receives a message body saying "Hello" from a sender named "AA" at "April 1st, 13:30." In this case, the information described as text in the text data for voice output is "October 1st, 13:30. From AA. Hello." If the message body contains symbols, stamps, or pictograms, chat application execution unit 21 omits them or converts them into information expressed as text according to predetermined rules.

[0018] The message receiving unit 11 of the voice output device 1 receives the text data for voice output transmitted by the chat application executing unit 21, and stores it in the receiving buffer 22. The receiving buffer 22 is a storage area formed in a work area such as a RAM. The text data for voice output is an object that the message receiving unit 11 receives as a message, and corresponds to a "message" in the claims. As a result of the above processing being performed between the chat application executing unit 21 and the message receiving unit 11, the text data for voice output is stored in the receiving buffer 22 immediately in response to reception of the message data by the chat application executing unit 21. Hereinafter, the message receiving unit 11 receiving the text data for voice output and storing this text data for voice output in the receiving buffer 22 may be simply expressed as "the message receiving unit 11 receiving a message."

[0019] When the message receiving unit 11 stores text data for voice output in the receiving buffer 22 (=when the message receiving unit 11 receives a message), the voice output control unit 12 executes the following process. That is, by analyzing the captured image data input from the in-vehicle camera 4, the voice output control unit 12 determines whether or not a first start condition, that is, the driver is present in the vehicle interior, is satisfied.

[0020] Here, the driver does not always sit in the driver's seat in the vehicle interior space while the message notification mode is on, and may go outside the vehicle interior space. For example, the driver may go outside the vehicle interior space to put luggage in the trunk of the vehicle (or take luggage out of the trunk), or to go shopping for a short time. If the driver has gone outside the vehicle interior space and is not present in the vehicle interior space (i.e., the first start condition is not established), the driver cannot normally hear the sound output by the sound output device 1. On the other hand, if the driver is present in the vehicle interior space (i.e., the first start condition is established), the driver can normally hear the message sound output by the sound output device 1, since the driver is present within a range where the driver can hear the message sound output by the sound output device 1. Therefore, the first start condition can be said to be a condition that is established when the driver (target person) can normally hear the sound in the vehicle interior space.

[0021] The audio output control unit 12 judges whether or not the first start condition is satisfied by the following method. That is, when a person is seated in the driver's seat, it can be assumed that a driver is present in the vehicle interior space, and when a person is not seated in the driver's seat, it can be assumed that a driver is not present in the vehicle interior space. Based on this, the audio output control unit 12 specifies the area of ​​the driver's seat in the input photographed image data. The area of ​​the driver's seat in the photographed image data is determined in advance. Next, the audio output control unit 12 judges whether or not an image of a human face is included in the area of ​​the driver's seat by an existing face recognition technology, and if it is included, it judges that the first start condition is satisfied, and if it is not included, it judges that the first start condition is not satisfied.

[0022] When the first start condition is satisfied, the voice output control unit 12 controls the voice output unit 13 to start voice output of a message voice based on the text data for voice output stored in the reception buffer 22. Specifically, the voice output control unit 12 outputs a start notification signal to the voice output unit 13. When the start notification signal is input, the voice output unit 13 generates voice data for outputting information described as text in the text data for voice output as voice. As an example, the voice data is data of a voice waveform in which a voice is sampled at a predetermined sampling period. The voice data is appropriately generated by using a voice synthesis technique or other existing techniques. Next, the voice output unit 13 outputs a voice signal based on the voice data to the voice processing device 3, thereby causing the voice processing device 3 to output a voice based on the voice data (= message voice). As a result, when the first start condition is satisfied, voice output of the message voice is started immediately in response to reception of the message by the message receiving unit 11. Therefore, the driver can immediately recognize the contents of the message in response to reception of the message.

[0023] On the other hand, if the first start condition is not satisfied, the audio output control unit 12 suspends the audio output of the message audio by the audio output unit 13. Specifically, the audio output control unit 12 does not output a start notification signal to the audio output unit 13 at that time. The audio output control unit 12 continues to input the captured image data from the in-vehicle camera 4 and continues to analyze the captured image data to monitor whether the first start condition is satisfied. That is, the audio output control unit 12 monitors whether the driver is seated in the driver's seat. For example, the audio output control unit 12 continues to analyze the captured image data intermittently input, monitors whether a new image of a human face is included in the area of ​​the driver's seat, and determines that the first start condition is satisfied when such a state is reached.

[0024] When the first start condition is satisfied, the audio output control unit 12 controls the audio output unit 13 to start audio output of the text data for audio output stored in the receiving buffer 22. Specifically, the audio output control unit 12 outputs a start notification signal to the audio output unit 13.

[0025] As described above, if the first start condition is not satisfied at the time when the message receiving unit 11 receives the text data for voice output, the voice output control unit 12 suspends the voice output of the message voice by the voice output unit 13, and then starts the voice output of the message voice when the first start condition is satisfied. This prevents the message voice from being output when the driver is not in the vehicle interior, causing the driver to miss the message voice. In other words, according to this embodiment, voice output is not performed immediately in response to the reception of a message, but is performed only when the target person can normally hear the message, and voice output of the message is suspended otherwise, preventing the target person from missing the message.

[0026] Next, an operation example of the voice output device 1 according to this embodiment will be described using a flowchart. FIG. 2 is a flowchart showing an operation example of the voice output device 1 after receiving text data for voice output from the mobile terminal 2. As shown in FIG. 2, the message receiving unit 11 of the voice output device 1 receives the text data for voice output and stores it in the receiving buffer 22 (step SA1). When the text data for voice output is stored in the receiving buffer 22, the voice output control unit 12 determines whether or not the first start condition is satisfied based on the image data captured by the in-vehicle camera 4 (step SA2). As described above, the first start condition is a condition that a person is seated in the driver's seat.

[0027] If the first start condition is met (step SA2: YES), the audio output control unit 12 controls the audio output unit 13 to start audio output of the message audio (step SA3). If the first start condition is not met (step SA2: NO), the audio output control unit 12 suspends audio output of the message audio by the audio output unit 13 (step SA4). Next, the audio output control unit 12 monitors whether the first start condition is met (step SA5). If the first start condition is met (step SA5: YES), the audio output control unit 12 controls the audio output unit 13 to start audio output of the message audio (step SA6).

[0028] <First Modification of First Embodiment> Next, a first modified example of the first embodiment will be described. In the following description, the audio output control unit 12 (or a component corresponding to the audio output control unit 12 in an embodiment other than the first embodiment) causing the audio output unit 13 to start audio output of a message audio may be expressed simply as "the audio output control unit 12 starts audio output", and the audio output control unit 12 suspending audio output of a message audio by the audio output unit 13 may be expressed simply as "the audio output control unit 12 suspends audio output".

[0029] In the first embodiment, the target person who listens to the message voice is the driver. In this regard, in this modified example, the target person is a person other than the driver. Hereinafter, an example of the configuration and processing of the voice output device 1 according to this modified example will be briefly described. First, the shooting area of ​​the in-vehicle camera 4 is the entire in-vehicle space. In particular, the in-vehicle camera 4 is provided at a position where the face of a person sitting in any seat provided in the in-vehicle space can be shot from the front. A plurality of in-vehicle cameras 4 may be provided. In addition, image data in which an image of the face of the target person is recorded (hereinafter referred to as "face image data") is registered in advance in the voice output device 1. In addition, the target person is basically the owner of the mobile terminal 2.

[0030] Then, when the message receiving unit 11 receives the message, the audio output control unit 12 judges whether or not a first start condition that a target person exists in the vehicle interior space is established. Specifically, the audio output control unit 12 judges whether or not the captured image data input from the in-vehicle camera 4 includes an image of the face of the same person as the face indicated by the registered face image data. This judgment is performed based on an existing face recognition technology. As a simple example, the audio output control unit 12 calculates the degree of approximation by performing an existing multi-dimensional vector comparison process for each of the feature vectors of the face image indicated by the registered face image data and the feature vectors of the face image included in the captured image data, and judges that the captured image data includes an image of the face of the same person as the face indicated by the registered face image data when the captured image data includes an image of the face of the same person as the face indicated by the registered face image data. When the captured image data includes an image of the face of the same person as the face indicated by the registered face image data, the audio output control unit 12 judges that the first start condition is established, and when the captured image data does not include the image, the audio output control unit 12 judges that the first start condition is established.

[0031] The operation of the audio output device 1 after determining whether the first start condition is satisfied is the same as that of the first embodiment. According to the configuration of this modification, similarly to the first embodiment, audio output is not performed immediately in response to the reception of a message, but audio output is performed only when the target person is in a state where he or she can normally hear the message, and audio output of the message is suspended in other cases, so that it is possible to prevent the target person from missing the message.

[0032] <Other Modifications of the First Embodiment> Next, other modified examples of the first embodiment will be described. In the above first embodiment, the vehicle is not an autonomous vehicle, but the vehicle may be an autonomous vehicle. In the above first embodiment, the audio output control unit 12 determines whether the first start condition is satisfied by analyzing the image data captured by the in-vehicle camera 4. In this regard, the configuration may be such that the first start condition is satisfied by another method. As an example, the configuration may be such that a sensor (weight sensor, switch sensor, optical sensor, etc.) that detects whether or not the driver is seated in the driver's seat is provided in the driver's seat, and the audio output control unit 12 determines whether or not the first start condition is satisfied based on the detection value of the sensor.

[0033] <Second embodiment> Next, a second embodiment will be described. FIG. 3 is a block diagram showing an example of a functional configuration of a voice output device 1A according to this embodiment. In the description of the second embodiment, the same elements as those in the first embodiment are given the same reference numerals, and detailed description thereof will be omitted. As is clear from a comparison between FIG. 1 and FIG. 3, the voice output device 1A according to this embodiment includes a voice output control unit 12A instead of the voice output control unit 12 according to the first embodiment. Note that, in this embodiment, as in the first embodiment, the vehicle is not an autonomous vehicle, and the person who is to listen to the message voice is the driver.

[0034] The audio output control unit 12A executes the following process when the message receiving unit 11 receives a message. That is, the audio output control unit 12A judges whether or not the vehicle is parked. This judgment is performed, for example, by judging whether or not the parking brake is applied. If the vehicle is not parked, the audio output control unit 12A starts audio output without judging whether or not a second start condition described below is satisfied.

[0035] On the other hand, when the vehicle is parked, the audio output control unit 12A analyzes the captured image data input from the in-vehicle camera 4 to determine whether or not the second start condition, that is, the driver seated in the driver's seat, is not asleep, is satisfied. At the time when it is determined whether or not the second start condition is satisfied, the vehicle is parked and the driver may be asleep for a rest. If the driver is asleep (i.e., the first start condition is not satisfied), the driver cannot normally hear the audio output by the audio output device 1A. On the other hand, if the driver is not asleep (i.e., the first start condition is satisfied), the driver can normally hear the message audio in that the driver is awake and can hear and recognize the message audio output by the audio output device 1. Therefore, the second start condition can be said to be a condition that is satisfied when the driver (target person) can normally hear the audio in the vehicle interior space.

[0036] The audio output control unit 12A judges whether the second start condition is satisfied based on existing technology. To explain a simple example, the audio output control unit 12A identifies an image of a human face in the driver's seat area in the input photographed image data based on existing face recognition technology, and regards the identified face image as the driver's face image. Then, the audio output control unit 12A analyzes the identified face image and judges whether the driver is asleep. For example, the audio output control unit 12A continuously analyzes photographed image data for a certain period (e.g., 5 seconds) and judges whether the eyes have been closed for a certain period or more. Then, if the eyes have been closed for a certain period or more, the audio output control unit 12A judges that the driver is asleep and the second start condition is not satisfied, and otherwise judges that the second start condition is satisfied.

[0037] If the second start condition is met, the audio output control unit 12A starts audio output. On the other hand, if the second start condition is not met, the audio output control unit 12A suspends audio output. Then, the audio output control unit 12A continues to analyze the captured image data input from the in-vehicle camera 4, and monitors whether the second start condition is met. In other words, the audio output control unit 12A monitors whether the driver is no longer asleep (=awake). If the second start condition is met, the audio output control unit 12A starts audio output.

[0038] As described above, if the second start condition is not satisfied at the time when the message receiving unit 11 receives the message, the audio output control unit 12A holds back the audio output, and then starts the audio output when the second start condition is satisfied. This prevents the message audio from being output when the driver is asleep, causing the driver to miss the message audio. In other words, according to this embodiment, audio output is not performed immediately in response to the reception of the message, but is performed only when the target person is in a state where he or she can normally hear the message, and audio output of the message is held back otherwise, preventing the target person from missing the message.

[0039] Next, an operation example of the voice output device 1A according to the present embodiment will be described using a flowchart. FIG. 4 is a flowchart showing an operation example of the voice output device 1A after receiving text data for voice output from the mobile terminal 2. As shown in FIG. 4, the message receiving unit 11 of the voice output device 1A receives the text data for voice output and stores it in the receiving buffer 22 (step SB1). When the text data for voice output is stored in the receiving buffer 22, the voice output control unit 12A judges whether the vehicle is parked or not (step SB2). If the vehicle is not parked (step SB2: NO), the voice output control unit 12A controls the voice output unit 13 to start the voice output of the message voice (step SB3). After the process of step SB3, the flowchart ends. On the other hand, if the vehicle is parked (step SB2: YES), the voice output control unit 12A judges whether the second start condition is established or not based on the captured image data of the in-vehicle camera 4 (step SB4). As described above, the second start condition is a condition that the driver is not in a sleeping state.

[0040] If the second start condition is met (step SB4: YES), the audio output control unit 12A controls the audio output unit 13 to start audio output of the message voice (step SB5). If the second start condition is not met (step SB4: NO), the audio output control unit 12A suspends audio output of the message voice by the audio output unit 13 (step SB6). Next, the audio output control unit 12A monitors whether the second start condition is met (step SB7). If the second start condition is met (step SB7: YES), the audio output control unit 12A controls the audio output unit 13 to start audio output of the message voice (step SB8).

[0041] <First Modification of Second Embodiment> Next, a first modified example of the second embodiment will be described. In this modified example, the vehicle is an autonomous vehicle having a fully autonomous driving function. It is assumed that the physical, technical, and legal environment for fully autonomous driving has been established, and the vehicle can run in a fully autonomous driving mode on public roads. During the fully autonomous driving, the driver is allowed to sleep.

[0042] Now, when the message receiving unit 11 receives a message while the vehicle is traveling in a fully automated driving mode, the voice output control unit 12A according to this modification determines whether the second start condition is satisfied without determining whether the vehicle is parked. This is because, in this modification, the driver may be asleep not only when the vehicle is parked, but also when the vehicle is traveling in a fully automated driving mode. The operation of the voice output device 1A after the determination is the same as that of the second embodiment. According to this modification, it is possible to prevent the driver from missing the message voice due to the driver being asleep when the vehicle is traveling in a fully automated driving mode.

[0043] <Other Modifications of the Second Embodiment> Next, other modified examples of the second embodiment will be described. In the second embodiment, the audio output control unit 12A may be configured to immediately start audio output only when both the first start condition and the second start condition are satisfied when the message receiving unit 11 receives a message, and to suspend audio output in other cases. In the above second embodiment, the driver is the target person who listens to the message audio, but the target person is not limited to the driver. When a person other than the driver is the target person, the audio output control unit 12A can determine whether a target person other than the driver is in a sleeping state based on the captured image data of the in-vehicle camera 4 by using the technology of the first modified example of the first embodiment. In the above second embodiment, the audio output control unit 12A determines whether the second start condition is satisfied by analyzing the captured image data of the in-vehicle camera 4. In this regard, the second start condition may be determined by another method. As an example, the audio output control unit 12A may be configured to acquire the driver's biological information (pulse wave, brain wave, etc.), determine whether the driver is in a sleeping state based on the biological information, and determine whether the second start condition is satisfied / not satisfied based on the biological information.

[0044] <Third embodiment> Next, a third embodiment will be described. FIG. 5 is a block diagram showing an example of a functional configuration of a voice output device 1B according to this embodiment. In the description of the third embodiment, the same elements as those in the first embodiment are given the same reference numerals, and detailed description thereof will be omitted. In this embodiment, the vehicle is not an autonomous driving automobile. In this embodiment, the target person who listens to the message voice is the driver, and the subject who performs the hands-free call described later is also the driver. As is clear from a comparison between FIG. 1 and FIG. 5, the voice output device 1B according to this embodiment includes a voice output control unit 12B instead of the voice output control unit 12 according to the first embodiment. In addition, the voice output device 1B according to this embodiment includes a hands-free call execution unit 30. The hands-free call execution unit 30 is a functional block that cooperates with the mobile terminal 2 to realize a hands-free call. Equipment necessary for realizing a hands-free call (for example, a microphone that inputs a speech voice) is appropriately installed. The hands-free call execution unit 30 outputs a signal indicating that a hands-free call is being performed to the voice output control unit 12B while a hands-free call is being performed.

[0045] The audio output control unit 12B executes the following process when the message receiving unit 11 receives a message. That is, the audio output control unit 12B judges whether or not a third start condition that the driver is not on the phone is satisfied. In this embodiment, the driver may make a hands-free call while driving the vehicle (obviously, the driver may make a hands-free call when not driving), and may make a call using the driver's own mobile terminal 2 or other device having a phone function when the vehicle is stopped. In this embodiment, the mobile terminal 2 can be used as a phone while maintaining the operation mode in the message notification mode.

[0046] Here, when the driver is on the phone (i.e., when the third condition is not met), the driver cannot concentrate on listening to the voice output by the voice output device 1B. On the other hand, when the driver is not on the phone (i.e., when the third condition is met), the driver can normally listen to the message voice in that he or she can concentrate on listening to the message voice without being interrupted by answering the phone. Therefore, the third start condition can be said to be a condition that is met when the driver (target person) can normally listen to the voice in the vehicle interior space.

[0047] The audio output control unit 12B judges whether or not the third start condition is satisfied by the following method. That is, when the message receiving unit 11 receives a message and receives a signal indicating that a hands-free call is being made from the hands-free call executing unit 30, the audio output control unit 12B judges that the driver is on a hands-free call (=on a phone call) and the third start condition is not satisfied. Furthermore, the audio output control unit 12B analyzes the captured image data input from the in-car camera 4 and judges whether or not the driver is on a phone call. Here, when the driver is on a phone call, the driver usually performs actions specific to when the driver is on a phone call, such as an action of holding a mobile phone to the ear or an action of talking by moving the mouth while wearing a headset. Based on this, the audio output control unit 12B analyzes the captured image data by an image analysis technique using a pattern image corresponding to an action specific to when the driver is on a phone call, and judges whether or not the driver is on a phone call. When the audio output control unit 12B judges that the driver is on a phone call as a result of analyzing the captured image data, the audio output control unit 12B judges that the third start condition is not satisfied. When it is not determined that the third start condition is not satisfied from the above two viewpoints, the audio output control unit 12B determines that the third start condition is satisfied.

[0048] When the third start condition is satisfied, the audio output control unit 12B starts audio output. On the other hand, when the third start condition is not satisfied, the audio output control unit 12B suspends audio output and monitors whether the third start condition is satisfied. In other words, the audio output control unit 12B monitors whether the driver is no longer on the phone. When it is detected that the third start condition is satisfied, the audio output control unit 12B starts audio output.

[0049] As described above, if the third start condition is not satisfied at the time when the message receiving unit 11 receives the message, the audio output control unit 12B holds the audio output, and then starts the audio output when the third start condition is satisfied. This configuration prevents the message audio from being output when the driver is answering the phone, causing the driver to be unable to concentrate on listening to the message audio and miss the message audio. In other words, according to this embodiment, audio output is not performed immediately in response to the reception of a message, but is performed only when the target person is in a state where the message can be normally heard, and the audio output of the message is held in other cases, preventing the target person from missing the message.

[0050] Next, an operation example of the voice output device 1B according to this embodiment will be described with reference to a flowchart. FIG. 6 is a flowchart showing an operation example of the voice output device 1B after receiving text data for voice output from the mobile terminal 2. As shown in FIG. 6, the message receiving unit 11 of the voice output device 1B receives the text data for voice output and stores it in the receiving buffer 22 (step SC1). When the text data for voice output is stored in the receiving buffer 22, the voice output control unit 12B determines whether or not the third start condition is satisfied based on the input from the hands-free call executing unit 30 and the captured image data of the in-car camera 4 (step SC2). As described above, the third start condition is a condition that the driver is not on the phone.

[0051] If the third start condition is met (step SC2: YES), the audio output control unit 12B controls the audio output unit 13 to start audio output of the message audio (step SC3). If the third start condition is not met (step SC2: NO), the audio output control unit 12B suspends audio output of the message audio by the audio output unit 13 (step SC4). Next, the audio output control unit 12B monitors whether the third start condition is met (step SC5). If the third start condition is met (step SC5: YES), the audio output control unit 12B controls the audio output unit 13 to start audio output of the message audio (step SC6).

[0052] <Modification of the third embodiment> Next, a modified example of the third embodiment will be described. In the third embodiment, the vehicle is not an autonomous vehicle, but it may be an autonomous vehicle. In addition, in the third embodiment, the voice output control unit 12B may be configured to immediately start voice output only when the message receiving unit 11 receives a message and an arbitrary combination of the third start condition and the first and second start conditions (including a combination of one condition and a combination of two conditions) is established, and to suspend voice output in other cases. In addition, in the third embodiment, the driver is the target person who listens to the message voice, but the target person is not limited to the driver. When a person other than the driver is the target person, the voice output control unit 12B can determine whether a target person other than the driver is on the phone based on the captured image data of the in-vehicle camera 4 by using the technology of the first modified example of the first embodiment. In addition, the voice output control unit 12B may be configured to determine whether the driver is on the phone by a method other than the method described in the third embodiment.

[0053] <Fourth embodiment> Next, a fourth embodiment will be described. FIG. 7 is a block diagram showing an example of a functional configuration of a voice output device 1C according to this embodiment. In the description of the fourth embodiment, the same elements as those in the first embodiment are given the same reference numerals, and detailed description thereof will be omitted. In this embodiment, the target person who listens to the message voice is the driver, and the vehicle is not an autonomous driving automobile. As is clear from a comparison between FIG. 1 and FIG. 7, the voice output device 1C according to this embodiment includes a voice output control unit 12C instead of the voice output control unit 12 according to the first embodiment.

[0054] When the message receiving unit 11 receives a message, the audio output control unit 12C executes the following process. That is, the audio output control unit 12C judges whether or not a fourth start condition that the driver is not talking is satisfied. The driver talking means that a passenger other than the driver is in the vehicle and the driver is talking to the passenger. Here, when the driver is talking (that is, when the fourth start condition is not satisfied), the driver cannot concentrate on listening to the voice output by the audio output device 1C. On the other hand, when the driver is not talking (that is, when the fourth start condition is satisfied), the driver can normally listen to the message voice in that he or she can concentrate on listening to the message voice without being disturbed by the conversation. Therefore, the fourth start condition can be said to be a condition that is satisfied when the driver (target person) can normally listen to the voice in the vehicle interior space.

[0055] The audio output control unit 12C judges whether the fourth start condition is satisfied by the following method. That is, the audio output control unit 12C identifies the image of the driver's face from the captured image data input from the in-vehicle camera 4 by the method described in the first embodiment. Next, the audio output control unit 12C identifies the mouth area in the image of the face, and tracks and analyzes the mouth area for a certain period of time (for example, 5 seconds). Based on the result of the analysis of the mouth area, if the mouth has not moved continuously for a certain period of time or more (the mouth remains closed or open), the audio output control unit 12C judges that the driver is not talking and the fourth start condition is satisfied, and otherwise judges that the driver is talking and the fourth start condition is not satisfied.

[0056] When the fourth start condition is satisfied, the audio output control unit 12C starts audio output. On the other hand, when the fourth start condition is not satisfied, the audio output control unit 12C suspends audio output and monitors whether the fourth start condition is satisfied. That is, the audio output control unit 12C monitors whether the driver is not talking. For example, the audio output control unit 12C continuously analyzes the captured image data to continuously monitor the movement of the driver's mouth, and when a state in which the mouth does not move continuously for a certain period of time or more appears, it determines that the fourth start condition is satisfied. When it is detected that the fourth start condition is satisfied, the audio output control unit 12C starts audio output.

[0057] As described above, if the fourth start condition is not satisfied at the time when the message receiving unit 11 receives the message, the audio output control unit 12C holds back the audio output, and then starts audio output when the fourth start condition is satisfied. This configuration can prevent the message audio from being output while the driver is talking, causing the driver to be unable to concentrate on listening to the message audio and miss the message audio. In other words, according to this embodiment, audio output is not performed immediately in response to the reception of a message, but is performed only when the target person can normally hear the message, and audio output of the message is held back otherwise, preventing the target person from missing the message.

[0058] Next, an operation example of the voice output device 1C according to this embodiment will be described with reference to a flowchart. FIG. 8 is a flowchart showing an operation example of the voice output device 1C after receiving text data for voice output from the mobile terminal 2. As shown in FIG. 8, the message receiving unit 11 of the voice output device 1C receives the text data for voice output and stores it in the receiving buffer 22 (step SD1). When the text data for voice output is stored in the receiving buffer 22, the voice output control unit 12C determines whether or not the fourth start condition is satisfied based on the image data captured by the in-vehicle camera 4 (step SD2). As described above, the fourth start condition is a condition that the driver is not talking.

[0059] If the fourth start condition is met (step SD2: YES), the audio output control unit 12C controls the audio output unit 13 to start audio output of the message audio (step SD3). If the fourth start condition is not met (step SD2: NO), the audio output control unit 12C suspends audio output of the message audio by the audio output unit 13 (step SD4). Next, the audio output control unit 12C monitors whether or not the fourth start condition is met (step SD5). If the fourth start condition is met (step SD5: YES), the audio output control unit 12C controls the audio output unit 13 to start audio output of the message audio (step SD6).

[0060] <First Modification of Fourth Embodiment> Next, a first modified example of the fourth embodiment will be described. In the above fourth embodiment, the audio output control unit 12C judges whether or not the fourth start condition that the driver is not talking is satisfied. In this regard, the audio output control unit 12C according to this modified example judges whether or not the fourth start condition that the driver is not talking loudly is satisfied. In detail, a microphone that inputs the driver's speech is connected to the audio output device 1C, and the sound pressure level of the voice input to the microphone is input to the audio output control unit 12C from an audio processing circuit that processes the input from the microphone.

[0061] Then, when the message receiving unit 11 receives a message, the audio output control unit 12C judges whether or not a fourth start condition that the driver is not talking loudly is satisfied. Specifically, when the driver is talking (whether or not the driver is talking is judged by the method described in the above fourth embodiment) and the input sound pressure level is equal to or higher than a threshold, the audio output control unit 12C judges that the fourth start condition is not satisfied, and otherwise judges that the fourth start condition is satisfied. Note that when the driver is talking and the sound pressure level of the voice input to the microphone is equal to or higher than a threshold, it can be regarded as a state in which the driver is talking at a certain volume or higher.

[0062] Here, when the driver is speaking softly, the degree of concentration on the conversation is low compared to when the driver is speaking loudly, and the driver is in a state where the driver can hear the voice message output without missing anything, or is in a state where the driver is likely to hear the voice message without missing anything. Based on this, according to this modification, when the driver is speaking softly, the voice message is output immediately in response to the reception of the message, so that the driver can be prevented from feeling uncomfortable when the voice message is suspended even though the driver is in a state where the voice message can be heard.

[0063] <Other Modifications of the Fourth Embodiment> Next, a modified example of the fourth embodiment will be described. In the fourth embodiment, the audio output control unit 12C may be configured to immediately start audio output only when the fourth start condition and any combination of the first to third start conditions (including a combination of one condition, a combination of two conditions, and a combination of three conditions) are satisfied when the message receiving unit 11 receives a message, and to suspend audio output in other cases. In this case, the content of the fourth start condition may be the content of the fourth start condition of the first modified example of the fourth embodiment. In addition, in the above fourth embodiment, the vehicle is not an autonomous vehicle, but it may be an autonomous vehicle. In addition, in the above fourth embodiment, the driver is the target person who listens to the message voice, but the target person is not limited to the driver. In this case, the audio output control unit 12C can determine whether a target person other than the driver is talking based on the captured image data of the in-vehicle camera 4 by using the technology of the first modified example of the first embodiment. In addition, the method for determining whether the driver is talking is not limited to the method exemplified in the above fourth embodiment, and any method that applies existing technology can be adopted.

[0064] <Fifth embodiment> Next, the fifth embodiment will be described. FIG. 9 is a block diagram showing an example of a functional configuration of the audio output device 1D according to this embodiment. In this embodiment, the target person who listens to the message voice is the driver, and the vehicle is not an autonomous driving automobile. As is clear from a comparison between FIG. 1 and FIG. 9, the audio output device 1D according to this embodiment includes an audio output control unit 12D instead of the audio output control unit 12 according to the first embodiment. When the message receiving unit 11 receives a message, the audio output control unit 12D executes the following process. That is, the audio output control unit 12D determines whether or not a fifth start condition that the driver is in a relaxed state is satisfied. Here, if the driver is not in a relaxed state (i.e., if the fifth start condition is not satisfied), the driver may not be able to concentrate and listen to the voice output by the audio output device 1D. On the other hand, if the driver is in a relaxed state (i.e., if the fifth start condition is satisfied), the driver can normally listen to the message voice output by the audio output device 1D in that the driver can concentrate and listen to the message voice output by the audio output device 1D in a calm state. Therefore, the fifth start condition can be said to be a condition that is met when the driver (target person) can normally hear the sound in the vehicle interior space.

[0065] The audio output control unit 12D judges whether the fifth start condition is satisfied by the following method. That is, the audio output control unit 12D identifies an image of the driver's face from the captured image data input from the in-vehicle camera 4 by the method described in the first embodiment, and judges whether the driver is in a relaxed state or not based on the existing facial expression recognition technology. Then, the audio output control unit 12D judges that the fifth start condition is satisfied when the driver is in a relaxed state, and judges that the fifth start condition is not satisfied when the driver is not in a relaxed state. Note that in this embodiment, due to the characteristics that the driver drives the vehicle, there are times when the driver is concentrating on driving, and it is expected that the fifth start condition is judged not to be satisfied at such times. In light of this, various parameters of the module that performs facial expression recognition based on facial expression recognition technology are appropriately adjusted so that it is judged that the driver is not in a relaxed state when the driver is concentrating on driving.

[0066] When the fifth start condition is satisfied, the audio output control unit 12D starts audio output. On the other hand, when the fifth start condition is not satisfied, the audio output control unit 12D suspends audio output and monitors whether the fifth start condition is satisfied. That is, the audio output control unit 12D monitors whether the driver is in a relaxed state. When it is detected that the fifth start condition is satisfied, the audio output control unit 12D starts audio output.

[0067] As described above, if the fifth start condition is not satisfied at the time when the message receiving unit 11 receives the message, the audio output control unit 12D holds back the audio output, and then starts the audio output when the fifth start condition is satisfied. This configuration can prevent the message audio from being output when the driver is not relaxed, causing the driver to be unable to concentrate on listening to the message audio and miss the message audio. In other words, according to this embodiment, audio output is not performed immediately in response to the reception of a message, but is performed only when the target person is in a state where the message can be normally heard, and the audio output of the message is held back otherwise, so that the target person can be prevented from missing the message.

[0068] Next, an operation example of the voice output device 1D according to this embodiment will be described using a flowchart. FIG. 10 is a flowchart showing an operation example of the voice output device 1D after receiving text data for voice output from the mobile terminal 2. As shown in FIG. 10, the message receiving unit 11 of the voice output device 1D receives the text data for voice output and stores it in the receiving buffer 22 (step SE1). When the text data for voice output is stored in the receiving buffer 22, the voice output control unit 12D determines whether or not the fifth start condition is satisfied based on the image data captured by the in-vehicle camera 4 (step SE2). As described above, the fifth start condition is a condition that the driver is in a relaxed state.

[0069] If the fifth start condition is satisfied (step SE2: YES), the audio output control unit 12D controls the audio output unit 13 to start audio output of the message audio (step SE3). If the fifth start condition is not satisfied (step SE2: NO), the audio output control unit 12D suspends audio output of the message audio by the audio output unit 13 (step SE4). Next, the audio output control unit 12D monitors whether the fifth start condition is satisfied (step SE5). If the fifth start condition is satisfied (step SE5: YES), the audio output control unit 12D controls the audio output unit 13 to start audio output of the message audio (step SE6).

[0070] <Modification of the Fifth Embodiment> Next, a modified example of the fifth embodiment will be described. In the fifth embodiment, the audio output control unit 12D may be configured to immediately start audio output only when the fifth start condition and any combination of the first to fourth start conditions (including combinations of one condition, two conditions, three conditions, and four conditions) are satisfied when the message receiving unit 11 receives a message, and to suspend audio output in other cases. In this case, the content of the fourth start condition may be the content of the fourth start condition of the first modified example of the fourth embodiment.

[0071] In the fifth embodiment, the vehicle is not an autonomous vehicle, but it may be an autonomous vehicle. In the case of an autonomous vehicle, for example, when the driver is concentrating on reading, the message voice can be prevented from being output. In the fifth embodiment, the driver is the target person who listens to the message voice, but the target person is not limited to the driver. In this case, the voice output control unit 12D can determine whether a target person other than the driver is in a relaxed state based on the captured image data of the in-vehicle camera 4 by using the technology of the first modification of the first embodiment.

[0072] In addition, the method for determining whether the driver is in a relaxed state is not limited to the method exemplified in the fifth embodiment, and any method that applies existing technology can be adopted. For example, a configuration may be adopted in which biological information (such as pulse waves and brain waves) of the driver is acquired and whether the driver is in a relaxed state is determined based on the biological information. In addition, a configuration may be adopted in which the audio output control unit 12D recognizes the situation of the vehicle, and when the driver is in a state where he or she is concentrating on driving (or is required to concentrate on driving) in consideration of the situation of the vehicle, the driver is determined to be not in a relaxed state and the fifth start condition is not satisfied. The situation of the vehicle may be a situation in which the vehicle is caught in a traffic jam, a situation in which the vehicle is about to enter an intersection, is entering an intersection, is traveling for parking, is frequently accelerating and decelerating, and the like.

[0073] Although the embodiments (including modifications) of the present invention have been described above, each of the above-mentioned embodiments merely shows an example of the realization of the present invention, and the technical scope of the present invention should not be interpreted as being limited thereby. In other words, the present invention can be embodied in various forms without departing from the gist or main characteristics thereof.

[0074] For example, in the first embodiment, the audio output control unit 12 starts audio output when the first start condition of "the driver is present in the vehicle interior space" is satisfied, and holds audio output when the first start condition is not satisfied. Hereinafter, this processing procedure will be referred to as the "first processing procedure." In this regard, a condition equivalent to the negation of the first start condition of "not being present in the vehicle interior space" is defined, and the processing procedure of "the audio output control unit 12 holds audio output when this condition is satisfied, and starts audio output when this condition is not satisfied" is synonymous with the first processing procedure. In other words, such a processing procedure is naturally included in the technical scope of the present invention. The above also applies to other embodiments (including modified examples).

[0075] In the first embodiment, a chat application is installed in the mobile terminal 2, but the chat application is installed in the voice output device 1, and a function for accessing the network N is implemented in the voice output device 1, and the message receiving unit 11 of the voice output device 1 may directly receive message data related to a message transmitted by a predetermined terminal. In the first embodiment, the chat application executing unit 21 of the mobile terminal 2 generates text data for voice output, but when the chat application executing unit 21 receives message data, the message data may be transmitted to the voice output device 1 without generating text data for voice output. In this configuration, the message data to be received by the message receiving unit 11 corresponds to the "message" in the claims. In these cases, for example, when the message receiving unit 11 receives message data, the text data for voice output is generated and stored in the receiving buffer 22, or when the message receiving unit 11 receives message data, the message data is stored in the receiving buffer 22, and when the voice output unit 13 receives a start notification signal from the voice output control unit 12, the message data is read out and text data for voice output is generated based on the message data. The above also applies to the other embodiments (including the modified examples).

[0076] In the first embodiment, if the first start condition is not satisfied at the time when the message receiving unit 11 receives a message, the audio output control unit 12 temporarily holds back the audio output, and then starts the audio output when the first start condition is satisfied. In this regard, the audio output control unit 12 may be configured to output a warning that a message has been received, rather than automatically starting the audio output after holding back the audio output, and to start the audio output when there is an explicit instruction from the driver (however, it may be a person other than the driver). For example, if the audio output device 1 has a display unit, the warning is issued by displaying predetermined information on the display unit, or, for example, if an LED is provided on the housing of the audio output device 1, the LED is turned on / blinked in a predetermined manner. The above is similar to other embodiments (including modified examples).

[0077] In the above embodiment, the space in which the audio output device 1 is installed is the vehicle interior space, and the vehicle interior space corresponds to the "predetermined space" in the claims. However, the space in which the audio output device 1 is installed and in which the target person listens to the message audio is not limited to the vehicle interior space. As an example, it may be a room in a house or a room in an office. The above also applies to other embodiments (including modified examples).

[0078] Furthermore, in the first embodiment, all or part of the processes described as being executed by each functional block of the audio output device 1 may be executed by an external device connected to the audio output device 1. The external device is, for example, a mobile terminal 2, or a cloud server connected to the network N. In this case, the audio output device 1 and the external device work together to function as an "audio output device." The above also applies to the other embodiments (including modified examples).

[0079] Furthermore, in each of the above embodiments, messages are exchanged in a text chat, but the messages are not limited to this and may be, for example, emails. [Explanation of symbols]

[0080] 1, 1A, 1B, 1C, 1D Audio output device 11 Message Receiving Unit 12, 12A, 12B, 12C, 12D Audio output control section 13 Message Receiving Unit

Claims

1. An audio output device that is installed in a predetermined space and has an audio output unit that outputs audio, A message receiving unit for receiving a message; and an audio output control unit that determines whether or not a start condition that is established when a target person can normally hear audio in the specified space is established when the message receiving unit receives a message, and causes the audio output unit to start audio output of the message if the start condition is established, and suspends audio output of the message by the audio output unit if the start condition is not established, The audio output control unit, in order to determine whether the start condition is satisfied, analyzes photographed image data generated by photographing the face of the subject using a facial expression recognition technology, and determines whether the condition that the subject is in a relaxed state is satisfied.

13. An audio output device comprising:

2. The predetermined space is an interior space formed in a vehicle.

2. The audio output device according to claim 1.

3. A method for outputting audio by an audio output device that is installed in a predetermined space and has an audio output unit that outputs audio, comprising: A first step of receiving a message by a message receiving unit of the audio output device; a second step of determining whether or not a start condition is satisfied when a target person in the predetermined space can normally hear a sound when the message is received by the message receiving unit, and causing the sound output unit to start audio output of the message if the start condition is satisfied, and suspending audio output of the message by the sound output unit if the start condition is not satisfied, In the second step, the audio output control unit analyzes captured image data generated by capturing a face of the subject using a facial expression recognition technique to determine whether the start condition is satisfied, and determines whether the condition that the subject is in a relaxed state is satisfied.

13. A method for outputting audio comprising the steps of:

Images