Vehicle dialogue system

Abstract

To provide a vehicle dialogue system that can give occupants more natural dialogue experiences.SOLUTION: A vehicle dialogue system (1) using generative AI (10) includes: a voice input unit (2) that inputs the voice spoken by an occupant; a voice recognition unit (41) that converts the voice input by the voice input unit into first text data; a feeling estimation unit (45) that inputs at least one of the voice and the first text data to estimate the occupant's feelings and generates second text data representing an estimation result; a control unit (42) that inputs the first text data and the second text data, as input information (S1), to the generative AI (10); a voice synthesis unit (43) that converts response information (S2) from the generative AI (10) into voice; and a voice output unit (5) that outputs the voice converted by the voice synthesis unit (43).SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a dialogue system for a vehicle. [Background technology]

[0002] A voice dialogue system for automobiles has been proposed in which, when a user speaks to the system, the system selects a suitable answer from pre-registered dialogue scenarios and responds by converting the answer into voice (Patent Document 1).

[0003] Such a voice dialogue system appropriately selects a response from a pre-recorded dialogue scenario even if the user's utterance is somewhat ambiguous. However, since such a voice dialogue system outputs the response selected from the dialogue scenario in the same tone regardless of the situation, the driver may feel uncomfortable with the response. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-54790 A Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention has been made in consideration of the above-mentioned circumstances, and has an object to provide a dialogue system for a vehicle that can provide a more natural dialogue experience to an occupant. [Means for solving the problem]

[0006] In order to achieve the above object, the vehicle dialogue system according to the present invention has the following features. A vehicle dialogue system using a generation AI that is mounted on a vehicle and that outputs response information made of text data when input information made of text data is input, a voice input unit for inputting a voice uttered by a passenger; a voice recognition unit that converts the voice input by the voice input unit into first text data; a feeling estimation unit that estimates a feeling of the occupant by inputting at least one of the voice and the first text data and generates second text data representing an estimation result; a control unit that inputs the first text data and the second text data as the input information to the generation AI; A voice synthesis unit that converts the response information from the generation AI into voice; a voice output unit that outputs the voice converted by the voice synthesis unit, Dialogue system for vehicles. Effect of the Invention

[0007] According to the present invention, it is possible to provide a vehicle dialogue system that can provide a driver or passenger with a more natural dialogue experience.

[0008] The present invention has been briefly described above. Furthermore, the details of the present invention will be further clarified by reading the following description of the preferred embodiment of the present invention (hereinafter, referred to as "embodiment") with reference to the accompanying drawings. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing a vehicle dialogue system according to a first embodiment. [Diagram 2] FIG. 2 is a flowchart showing a processing procedure of the microcomputer shown in FIG. [Diagram 3] FIG. 3 is a state transition diagram of an agent. [Figure 4] FIG. 4 is a diagram for explaining the states of an agent. [Diagram 5] FIG. 5 is a block diagram showing a vehicle dialogue system according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Specific embodiments of the present invention will be described below with reference to the accompanying drawings. Fig. 1 is a block diagram showing an embodiment of the vehicle dialogue system of the present invention. In the following description, the driver speaks, but the system can also be applied to the case where any passenger in the vehicle speaks.

[0011] The vehicle dialogue system 1 of this embodiment is a system that is mounted on a vehicle and dialogues with a driver using a generation AI (Artificial Intelligence) 10. The generation AI 10 is configured, for example, by ChatGPT, and when input information S1 consisting of text data is input, response information S2 consisting of text data is output.

[0012] The vehicle dialogue system 1 includes a microphone 2 as a voice input unit, a communication module 3, a microcomputer 4 (hereinafter abbreviated as "microcomputer 4") that functions as an agent, a speaker 5 as a voice output unit, and a display unit 7. The microphone 2 inputs voice spoken by the driver to the microcomputer 4. The communication module 3 is for communicating with the generation AI 10 via an Internet communication network (not shown), and is composed of a circuit, an antenna, etc. for connecting to the Internet communication network.

[0013] The microcomputer 4 has memories such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and a CPU (Central Processing Unit) that operates according to programs stored in the memories.

[0014] The microcomputer 4 has a voice recognition unit 41, a voice dialogue unit 42 as a control unit that executes input control for the generation AI 10, control of connected devices, output control for these devices, etc., a voice synthesis unit 43, and a feeling estimation unit 45. The voice recognition unit 41 converts a voice signal input by the microphone 2 into text data (first text data) and inputs it to the voice dialogue unit 42 and the feeling estimation unit 45. The voice dialogue unit 42 adds text data (second text data) corresponding to the driver's feeling determined by the feeling estimation unit 45, and text data (third text data) indicating the state of the driver or the vehicle driven by the driver when the driver spoke, or an instruction corresponding to the state, to the text data converted by the voice recognition unit 41, and inputs it to the generation AI 10 as input information S1.

[0015] Furthermore, vehicle information S3 from a plurality of sensors mounted on the vehicle is input to the voice dialogue unit 42. Possible sensors include an illuminance sensor that measures the illuminance outside the vehicle, a temperature sensor that measures the outside air temperature, a GPS (Global Positioning System) that detects the vehicle position, a seating sensor that detects whether or not a person is seated in a seat, and a speed sensor that detects the vehicle speed. The vehicle information S3 input from the sensor may be not only a measurement value measured by the sensor, but also a moving average or weighted average of the measurement value, or a value calculated using the measurement value, the moving average or weighted average of the measurement value.

[0016] Furthermore, human information S4, which is a detection result from a driver monitor that detects the driver's state (facial expression, whether the driver is dozing, driving absent-mindedly, or not) based on an image of the driver's face, is input to the voice dialogue unit 42. Furthermore, warning information, which is a detection result from an abnormality detection unit that detects abnormalities in the vehicle (engine abnormality, oil pressure abnormality, water temperature abnormality, charging abnormality, etc.) and turns on a warning lamp, may be input to the voice dialogue unit 42.

[0017] The voice dialogue unit 42 is connected to a vehicle device 11 mounted in the vehicle and can control the vehicle device 11. The vehicle device 11 may be, for example, an air conditioner, a display such as a head-up display, or an audio device mounted in the vehicle.

[0018] The voice dialogue unit 42 receives the response information S2 from the generation AI 10 and outputs the received response information S2 to the voice synthesis unit 43. The voice synthesis unit 43 converts the response information S2 into a voice signal and outputs it to the speaker 5. The speaker 5 outputs the voice converted by the voice synthesis unit 43. The display unit 7 is composed of a means having a screen capable of displaying information to the driver, such as a monitor or a head-up display (HUD) installed near the driver's seat. When the speaker 5 outputs a voice, the microcomputer 4 displays a character linked to the voice on the screen.

[0019] The emotion estimation unit 45 estimates the driver's emotions in real time based on the voice signal input from the microphone 2 and the text data input from the voice recognition unit 41, generates text data indicating the estimation result, and inputs it to the voice dialogue unit 42.

[0020] The emotion estimation unit 45 extracts features such as the tone, strength, speech rate, and spectrum characteristics of the driver's voice from the waveform of the voice signal input from the microphone 2. The emotion estimation unit 45 also extracts expressions and phrases indicating emotions from the text data input from the voice recognition unit 41 and converts them into features representing emotions. These features and emotion classifications are stored in the emotion estimation unit 45 as a correspondence map. The emotion classifications are, for example, divided into "happiness," "anger," "sadness," "surprise," and "neutral."

[0021] As an example, the correspondence map has feature amounts and each category corresponding to an initial value in advance. Therefore, when the emotion estimation unit 45 extracts a feature amount, the emotion estimation unit 45 determines the driver's current emotion by referring to the correspondence map, and inputs the determined emotion to the voice dialogue unit 42 as text data.

[0022] In addition, the emotion estimation unit 45 may compare the feature amounts extracted from the voice signal with the text data input from the voice recognition unit 41, and correct the feature amounts associated with each emotion category in the correspondence map.

[0023] The emotion estimation unit 45 outputs the emotion of the driver determined by the correspondence map as text data to the voice dialogue unit 42. The voice dialogue unit 42 adds text data corresponding to the emotion estimated by the emotion estimation unit 45 to the input information S1 to be input to the generation AI 10. For example, when the emotion estimation unit 45 determines that the emotion of the driver is "anger," the voice dialogue unit 42 adds an instruction to "use words that will calm the anger." This enables the generation AI 10 to generate an answer that will calm the anger or that is mildly worded.

[0024] The emotion estimation unit 45 may be configured to continuously hold the determined emotion of the driver. In this case, it is preferable that the duration for which the emotion is held can be set to be indefinite or to be a limited period such as a predetermined time. In addition, when the emotion is held in this manner, it is preferable that the microcomputer 4 is configured to reset the emotion held in the emotion estimation unit 45 by an input by the occupant or when the vehicle situation satisfies a predetermined condition.

[0025] Next, the operation of the vehicular dialogue system 1 configured as described above will be described with reference to the flowchart shown in FIG. 2. When the microcomputer 4 detects that the ignition is on (IG-ON) (Sp1), it transmits text data indicating the state of the driver and the vehicle to the generation AI 10 (Sp2). An example of this text data indicating the state of the driver and the vehicle could be "The person you are speaking to is about to get in the car. Please talk to them while taking into consideration the fact that they are in a car." This allows the generation AI 10 to understand that the person you are speaking to (the driver) is in the car, and from then on the dialogue content will be based on the assumption that the person is in the car.

[0026] Also, in Sp2, the microcomputer 4 may transmit text data of information on the controllable vehicle equipment 11. An example of this text data may be "We can suggest adjusting the temperature and airflow of the air conditioner, adjusting the brightness of the head-up display, and controlling the audio, depending on the conversation with the person you are talking to." This allows the generation AI 10 to understand that the driver can adjust the temperature and airflow of the air conditioner, adjust the brightness of the head-up display, and control the audio in the vehicle he is riding in.

[0027] Also, in Sp2, the microcomputer 4 may determine the number of people riding in the vehicle based on the detection result from the seating sensor, and transmit the determined number of people to the generation AI 10. An example of this text data is "Only the driver is riding in the vehicle" or "There are two people riding in the vehicle, including the driver." This allows the generation AI 10 to understand whether there is only one driver or other people riding in the vehicle.

[0028] Next, the microcomputer 4 acquires the vehicle information S3 and person information S4 described above (Sp3). After that, when the driver speaks (Y in Sp4), the microcomputer 4 performs a voice recognition process to convert the voice spoken by the driver into text data, and estimates the driver's emotions based on the voice (Sp5). Next, the microcomputer 4 adds the driver's emotions estimated in Sp3, the state of the driver and the vehicle when the driver spoke, or commands according to the state (text data) to the text data converted by the voice recognition process, and transmits the text data to the generation AI 10 as input information S1 (Sp6).

[0029] In Sp6, the microcomputer 4 generates text data for adjusting the prompt to obtain an answer that makes the driver feel positive, according to the driver's emotion estimated in Sp5. The microcomputer 4 may then add the generated text data to the input information S1 together with text data indicating the vehicle information S3 and person information S4 acquired in Sp3. The microcomputer 4 may also add text data indicating whether the vehicle is driving, stopped, or waiting at a traffic light as text data indicating the vehicle state.

[0030] Furthermore, the microcomputer 4 determines whether the driver's driving load is high when the driver speaks. Whether the driving load is high can be determined based on, for example, the speed from a speed sensor (vehicle information S3) or the person information S4. Alternatively, it may be determined based on pre-registered attributes of the driver (whether the driver is accustomed to driving or not, whether the driver is familiar with vehicle equipment or not). If the microcomputer 4 determines that the driving load is high when the driver speaks, it may give text data to shorten the response information S2 as a command according to the state. An example of the text data at this time may be "Please summarize your answer."

[0031] Thereafter, the microcomputer 4 receives response information S2 corresponding to the input information S1 transmitted in Sp7 (Sp7), converts the received response information S2 into voice, and outputs it from the speaker 5 (Sp8). Next, when the microcomputer 4 detects that the ignition is off (IG-OFF) (Y in Sp9), it ends the process. On the other hand, if the microcomputer 4 does not detect that the ignition is off (N in Sp9), it returns to Sp3 again.

[0032] Furthermore, after executing Sp3, if the driver has not spoken (N in Sp4), or if a predetermined proposal condition based on the information S3 to S4 acquired by Sp3 is satisfied (Sp10), the microcomputer 4 converts the proposal content into voice data and outputs it from the speaker 5 (Sp11), and then returns to Sp3. As a proposal condition for Sp11, for example, if the speed measured by the speed sensor (vehicle information S3) is fast, the proposal condition to reduce the speed is satisfied, and a proposal to reduce the speed is output from the speaker 5.

[0033] When the above-mentioned control is executed, the microcomputer 4 may display a character on the display of the vehicle, and may display the character as if it is talking to the driver in accordance with the voice output from the speaker 5. In this case, the microcomputer 4 may automatically change the appearance of the character, such as the facial expression, in response to the driver's emotions, or the driver may change the character through operation.

[0034] When the microcomputer 4 automatically changes the character's facial expression, for example, if the driver's emotion is negative, such as "anger" or "sadness," the voice dialogue unit 42 adds an instruction to the generation AI 10 to respond in a way that makes the driver's emotion positive, as described above, and in accordance with this, the character's facial expression and movements are also displayed to make the driver's emotion positive, such as by expressing kindness or moving slowly. Conversely, if the driver's emotion is positive, the character's facial expression also expresses happiness, thereby displaying the character in a way that sympathizes with the driver's positive emotion.

[0035] As another example, when the driver's emotions are negative, the microcomputer 4 may change the character to one that is most likely to induce positive emotions. To determine which character change is most effective for making the driver's emotions positive, for example, the voice dialogue unit 42 accumulates data on changes in the driver's emotions after changing the character. In other words, the character and its responses are optimized to the driver's preferences as the driver and the agent have repeated conversations.

[0036] Fig. 3 is a state transition diagram of an agent, and Fig. 4 is a diagram for explaining the states of an agent. When a character linked to the function of an agent is displayed on the screen of the display unit 7, the microcomputer 4 is first in a sleep state. In the sleep state, the character is made small so as not to interfere with the display of the basic screen or the display of operations.

[0037] When the wake-up word spoken by the driver is input to the microphone 2, the microcomputer 4 transitions to a wake-up state and starts functioning as an agent. In this state, the microcomputer 4 waits for the driver to speak. If the driver does not speak within a specified time, the microcomputer transitions to a sleep state. If the driver speaks within the specified time, the microcomputer 4 transitions to a hearing state and performs voice recognition and analysis.

[0038] The microcomputer 4 generates input information S1 from text data generated from the driver's speech, the driver's emotions, and text data generated based on the driver and vehicle conditions, and is in a wait state while transmitting the input information S1 to the generation AI 10. Then, when response information S2 of text data is received from the generation AI 10, the microcomputer 4 transitions to a speech state, outputs a voice representing the response information from the speaker 5, and displays a character that matches the voice.

[0039] In the speech state, if the driver starts speaking during the audio output of the response information S2, the microcomputer 4 executes a barge-in function to transition to a hearing state. The output of the response information interrupted by the execution of the barge-in function is resumed after the driver speaks or is terminated as it is, depending on the situation. For example, if the response information S2 is important information related to driving, the output of the remaining response information that was interrupted after the driver speaks may be resumed, and if the importance of the information is low, the output may be terminated as it is.

[0040] When the output of the response information S2 by voice ends in the speech state, the microcomputer 4 transitions to the wake-up mode and waits for the speaker to speak again.

[0041] In addition, if the screen of the display unit 7 doubles as a user operation screen, such as a touch panel, the driver may change the display position of the character by dragging or the like, or may enlarge or reduce the display size of the character by using multiple fingers. Also, when a setting screen for the agent function, including changing the character, is displayed, the microcomputer 4 suspends the display of the character on the screen.

[0042] FIG. 5 is a block diagram showing a vehicle dialogue system according to the second embodiment. In this embodiment, the same components as those in the first embodiment are given the same reference numerals and the description thereof will be omitted.

[0043] In the second embodiment, the emotion estimation unit 45 inputs the vehicle information S3 and the person information S4 and extracts features representing emotions. For example, when the accelerator opening, the brake pedal force, or the information indicating the acceleration of the vehicle is larger than usual and the driver's facial expression corresponds to "anger," the emotion estimation unit 45 extracts these pieces of information as features that are strongly related to the driver's "anger." The emotion estimation unit 45 adds up the features based on the above-mentioned voice signal, the features obtained from the vehicle information S3, and the detection result of the driver's facial expression obtained from the person information S4 at a predetermined ratio, and determines the final emotion of the driver based on the correspondence map. Note that the emotion estimation unit 45 may input only one of the vehicle information S3 and the person information S4.

[0044] The present invention is not limited to the above-described embodiment, and can be appropriately modified, improved, etc. In addition, the material, shape, size, number, arrangement location, etc. of each component in the above-described embodiment are arbitrary as long as the present invention can be achieved, and are not limited.

[0045] For example, the emotion estimation unit 45 may estimate the driver's degree of fatigue in addition to categorizing the emotion, and in addition to or instead of determining the emotion, input text data indicating the driver's degree of fatigue to the voice dialogue unit 42. As an example, when the driver is fatigued, the voice dialogue unit 42 adds an instruction to "speak in kind words" to the input information S1 input to the generation AI 10.

[0046] The microcomputer 4 may change the answer or character depending on the mood of the driver and the passenger. For example, when the voices of both the driver and the passenger are input to the microphone 2, text data indicating the respective voice signals is input from the voice recognition unit 41 to the emotion estimation unit 45. When the emotion estimation unit 45 determines that the emotions of both the driver and the passenger are negative, the voice dialogue unit 42 adds an instruction such as "use words that mediate a fight" to the input information S1 in response to the determination. When the emotions of both the driver and the passenger obtained from the emotion estimation unit 45 are positive and the driver and the passenger speak a lot, the voice dialogue unit 42 adds an instruction such as "use a cheerful atmosphere and keep words as short as possible" to the input information S1 so as not to disturb the conversation between the driver and the passenger. Conversely, when the emotions of both the driver and the passenger are positive but the driver and the passenger speak a little, the voice dialogue unit 42 adds an instruction such as "use a cheerful atmosphere and keep words as short as possible" to the input information S1.

[0047] Furthermore, the microcomputer 4 may change the character or tone of voice depending on the driving scene of the vehicle. For example, even when the same character is displayed, the microcomputer 4 may change the costume or the character itself depending on the driving scene, such as summer and winter, daytime and nighttime, sunny and rainy, etc.

[0048] In the above embodiment, the feeling estimation unit 45 estimates the feeling based on the text data from the voice recognition unit 41, or the text data and the vehicle information S3. However, depending on the situation, the feeling may be estimated based only on the vehicle information S3.

[0049] The emotion estimation unit 45 may input the person information S4, extract features indicating the driver's emotion from the person information S4, and use the extracted features as one of the parameters when determining the emotion. The emotion estimation unit 45 accumulates information on what emotion the driver is feeling and what character or tone of response should be used to make the driver feel positive, based on features included in the voice of the driver's further question in response to the answer from the generation AI 10, the amount of speech, etc., and changes the character setting according to the situation.

[0050] Here, the features of the vehicular dialogue system according to the embodiment of the present invention described above will be briefly summarized and listed in the following [1] to [4].

[0051] [1] A vehicle dialogue system (1) that uses a generation AI (10) that is mounted on a vehicle and that outputs response information (S2) made of text data when input information (S1) made of text data is input, a voice input unit (2) for inputting a voice uttered by a passenger; a voice recognition unit (41) that converts the voice input by the voice input unit into first text data; an emotion estimation unit (45) that receives at least one of the voice and the first text data to estimate an emotion of the occupant and generates second text data representing an estimation result; a control unit (42) that inputs the first text data and the second text data as the input information (S1) to the generation AI (10); a voice synthesis unit (43) for converting the response information (S2) from the generation AI (10) into voice; and a voice output unit (5) that outputs the voice converted by the voice synthesis unit (43). Dialogue system for vehicles.

[0052] According to the configuration of [1] above, when obtaining an output from the generation AI, it is possible to obtain an answer that takes into account the passenger's emotions. Therefore, when presenting an answer to the passenger by voice, even if the answer has the same content, the tone and wording can be changed according to the passenger's emotions, so that the passenger can feel a sense of familiarity with the dialogue system. In addition, if the passenger's emotions are negative, an answer that takes these into account can be output, so that it is possible to prevent the passenger's emotions from being further aggravated.

[0053] [2] The control unit (42) generates additional information corresponding to the emotion indicated by the second text data, adds the additional information to the input information (S1), and inputs the additional information to the generation AI (10). The vehicle dialogue system according to [1] above.

[0054] According to the configuration [2] above, the vehicle dialogue system can specify to the generation AI the tone and wording of the response more specifically depending on the occupant's emotions, so that an appropriate response can be provided that is in line with the occupant's emotions.

[0055] [3] The emotion estimation unit (45) selects one of emotions represented by a plurality of categories classified in advance as an estimation result, inputs the selected result to the control unit (42) as second text data, and continuously holds or resets the selected result. The vehicle dialogue system according to the above [1] or [2].

[0056] According to the configuration of [3] above, the vehicle dialogue system categorizes the occupant's emotions in advance, so the possibility of misrecognizing the occupant's emotions can be reduced. In addition, the system can correct the features of the occupant's voice associated with each category, so that the occupant's emotions can be judged more accurately as the occupant's speech increases.

[0057] [4] A display unit (7) having a screen for presenting image information to an occupant, The control unit (42) changes the appearance or type of the character displayed on the screen. A dialogue system for a vehicle according to any one of [1] to [3] above.

[0058] According to the configuration [4] above, the vehicle dialogue system changes not only the voice output but also the character displayed in response to the occupant's emotions, allowing the occupant to feel a sense of familiarity with the vehicle dialogue system both visually and aurally. [Explanation of symbols]

[0059] 1 Vehicle dialogue system 2. Microphone 3. Communication Module 4. Microcomputer 5 Speakers 7 Display section 10 Generation AI 11 Vehicle equipment 41 Voice Recognition Unit 42 Voice dialogue unit (control unit) 43 Voice synthesis section 45 Emotion estimation part S1 Input information S2 Response Information S3 vehicle information S4 People information

Claims

1. A vehicle dialogue system using a generation AI that is mounted on a vehicle and that outputs response information made up of text data when input information made up of text data is input, a voice input unit for inputting a voice uttered by a passenger; a voice recognition unit that converts the voice input by the voice input unit into first text data; a feeling estimation unit that estimates a feeling of the occupant by inputting at least one of the voice and the first text data and generates second text data representing an estimation result; a control unit that inputs the first text data and the second text data as the input information to the generation AI; A voice synthesis unit that converts the response information from the generation AI into voice; a voice output unit that outputs the voice converted by the voice synthesis unit, Dialogue system for vehicles.

2. The control unit generates additional information corresponding to the emotion indicated by the second text data, adds the additional information to the input information, and inputs the additional information to the generation AI. The vehicle dialogue system according to claim 1 .

3. The emotion deduction unit selects one of emotions indicated by a plurality of categories classified in advance as an estimation result, inputs the selected result to the control unit as second text data, and continuously holds or resets the selected result. The vehicle dialogue system according to claim 1 .

4. A display unit having a screen for presenting image information to an occupant, The control unit changes the appearance or type of the character displayed on the screen. The vehicle dialogue system according to claim 1 .

Images