Dialogue device, dialogue method, and dialogue program

Abstract

To enable highly effective dialogue that stimulates the crew. [Solution] The dialogue device comprises: an alertness calculation unit that calculates the alertness level of the occupant based on the state information of the vehicle occupant; a dialogue output unit that outputs a dialogue to the occupant when the calculated alertness level is less than a predetermined value; an interest level calculation unit that calculates the occupant's level of interest in the dialogue based on the state information after the dialogue has been output; and a dialogue generation unit that determines the topic of the dialogue based on the calculated interest level.

Description

Technical Field

[0001] The present invention relates to an interaction device, an interaction method, and an interaction program.

Background Art

[0002] In recent years, efforts have been actively made to provide access to a sustainable transportation system that takes into account people in vulnerable positions among traffic participants. Toward this realization, research and development have focused on further improving traffic safety and convenience through research and development related to preventive safety technology.

[0003] As a conventional technology related to preventive safety technology for further improving traffic safety and convenience, Patent Document 1 discloses an interaction device that acquires the arousal level of a vehicle occupant and requests the occupant to speak when the acquired arousal level is less than a threshold value. Patent Document 2 discloses an excitement suppression device that determines the excitement level of a driver and performs a suppression operation when the driver's excitement level exceeds a threshold value.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, in preventive safety technology, simply requesting the occupant to speak and conducting an interaction has a problem that the arousal effect weakens when the occupant's interest and awareness regarding the interaction topic are low.

[0006] The present application aims to achieve an interaction with a high arousal effect on the occupant in order to solve the above problems. And by extension, it contributes to the development of a sustainable transportation system. [Means for solving the problem]

[0007] One embodiment for achieving the above objective is a dialogue device comprising: an alertness calculation unit that calculates the alertness level of a vehicle occupant based on the occupant's state information; a dialogue output unit that outputs dialogue to the occupant when the calculated alertness level is less than a predetermined value; an interest level calculation unit that calculates the occupant's level of interest in the dialogue based on the state information after the dialogue has been output; and a dialogue generation unit that determines the topic of the dialogue based on the calculated interest level. [Effects of the Invention]

[0008] According to the above embodiment, it is possible to achieve a highly effective dialogue for stimulating the crew. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing an example configuration of a dialogue device according to one embodiment. [Figure 2] This figure shows an example of the arrangement of a display device, an in-vehicle camera, and a microphone inside a vehicle equipped with a dialogue device according to one embodiment. [Figure 3] This flowchart shows an example of operation of a dialogue device according to one embodiment. [Figure 4] This flowchart shows an example of operation of a dialogue device according to one embodiment. [Figure 5] This flowchart shows an example of operation of a dialogue device according to one embodiment. [Modes for carrying out the invention]

[0010] The dialogue device according to this embodiment will now be described with reference to the drawings. Figure 1 is a block diagram showing an example configuration of a dialogue device according to one embodiment.

[0011] As shown in Figure 1, the dialogue device 100 is mounted, for example, on a vehicle 102, which is a mobile object. Hereinafter, in this embodiment, "user" refers to the occupants of the vehicle 102, which is a mobile object, and includes the driver of the vehicle 102.

[0012] This dialogue device 100 is communicated via the in-vehicle network bus 104 to, for example, a camera control device 106, an output device 108, a navigation device 110, a driving determination unit 126, and a TCU (telematics control unit) 112.

[0013] The camera control device 106 controls the operation of the in-vehicle camera and the exterior camera 115. Specifically, the camera control device 106 uses the in-vehicle camera to capture at least the user's movements inside the vehicle 102 and transmits the captured in-vehicle video to other devices such as the dialogue device 100 and the output device 108. The camera control device 106 also uses the in-vehicle camera to capture the surrounding environment of the vehicle 102 and transmits the captured exterior video to other devices such as the dialogue device 100 and the output device 108. The driving determination unit 126 uses data related to the behavior of the vehicle 102 due to driving operations (accelerator, brake, steering, acceleration / deceleration, yaw, etc.) detected by the vehicle behavior sensor 128 to determine the driving operation. For example, the driving determination unit 126 determines whether there is an abnormality in the driving operation, such as serpentine driving.

[0014] The TCU112 comprises a long-range communication device 118 and a processing unit 116 that controls the operation of the long-range communication device 118. The processing unit 116 is, for example, a computer equipped with a processor such as a CPU (Central Processing Unit). The long-range communication device 118 is, for example, a wireless communication device that communicates with various server devices 300 that provide conversational services, news, and other information via a communication network such as the Internet.

[0015] Server device 300 is a server device that provides a dialogue service by generating and returning dialogue content in response to prompts that include utterances, etc., using LLM302 (LLM: Large Language Model). Here, the prompt is input information that includes "instructions," "conditions," and "reference information" for LLM302 to generate dialogue content. For example, "instructions" include instructions such as generating dialogue content in response to utterances, traffic conditions, etc., as described in the reference information. "Conditions" include various conditions for the output dialogue content (change / continuation of topic, notification of traffic conditions, etc.). "Reference information" includes utterances that form the basis of the dialogue content, traffic conditions, etc.

[0016] The output device 108 is connected to the display device 120, microphone 122, and speaker 124, and is a device that outputs audio and video to the occupants (users) of the vehicle 102. For example, the output device 108 outputs the dialogue output by the dialogue device 100 as audio from the speaker 124 using its speech synthesis function, and collects the user's speech from the microphone 122. The output device 108 converts the user's speech collected from the microphone 122 into text using its speech recognition function and outputs it to the dialogue device 100.

[0017] Furthermore, the output device 108 outputs sound by playing music, videos, etc., in response to input instructions from the user via an input device such as a touch panel on the display device 120, and / or in response to voice instructions during voice interaction with the user. In addition, the output device 108 retrieves information requested by the user from a server on the internet, for example via the TCU 112, in response to input instructions or voice instructions from the user, and outputs the retrieved image information and sound information to the display device 120 and speaker 124. The output device 108 also outputs the retrieved text information as sound to speaker 124 using a speech synthesis function.

[0018] Specifically, the output device 108 includes a processing device 130 and a storage device 132. The storage device 132 is constituted by, for example, a volatile and / or non-volatile semiconductor memory, a hard disk device, and / or a reading device for a removable storage medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, etc.

[0019] The processing device 130 is a computer including a processor such as a CPU. The processing device 130 includes a voice dialogue unit 134, an icon display unit 136, an AV control unit 138 (AV: Audio-Visual), and a UI control unit 140 (UI: User Interface) as functional elements or functional units realized by executing a program.

[0020] The voice dialogue unit 134 outputs the voice output of the dialogue output by the dialogue device 100. Specifically, the voice dialogue unit 134 converts the dialogue content (text) into voice by a voice synthesis function and outputs it from the speaker 124. Further, the voice dialogue unit 134 converts the user's speech collected from the microphone 122 into text by a voice recognition function and outputs it to the dialogue device 100.

[0021] Also, the voice dialogue unit 134 conducts voice dialogue with the user via the UI control unit 140 described later. For example, the voice dialogue unit 134 determines whether the user has uttered a specific phrase (so-called wake-up word or trigger word) indicating the start of a voice instruction. When the wake-up word is uttered, the subsequent utterance is recognized as a voice instruction. Then, the voice dialogue unit 134 plays back music, videos, etc. stored in the storage device 132, etc., plays back content retrieved from a server on the Internet, etc., and / or outputs voice of text information according to the recognized voice instruction. Note that the start of the dialogue in the voice dialogue unit 137 is not limited to being triggered by the user's utterance described above. For example, the voice dialogue unit 137 may start a push-type utterance from the dialogue device 100 side based on the user's state or abnormality of driving operation detected based on the video from the in-vehicle camera 114 or the out-vehicle camera 115. For example, when the voice dialogue unit 137 determines that the user is tired based on the video of the user by the in-vehicle camera 114, it makes an utterance such as "You seem a little tired. What's wrong?" Also, when the voice dialogue unit 137 determines that there is a weaving driving based on the video during driving by the out-vehicle camera 115, it makes an utterance such as "There is a weaving driving. Is there anything bothering you?" Note that the voice dialogue unit 137 may use the determination result of the driving determination unit 126 in addition to the video by the out-vehicle camera 115 for the abnormality of the driving operation.

[0022] The icon display unit 136 displays a person icon on a part of the display screen of the display device 120 at least when voice is being output from the speaker 124. Here, the person icon can be an icon imitating a person or an image of a person. Also, the image of the person used for the person icon may be, in addition to a photo of a person, an image artificially generated by computer graphics, etc. Also, the icon displayed by the icon display unit 136 is not limited to the person icon. For example, the icon display unit 136 may perform a display that prompts the user's attention by the blinking of light, color, etc. other than a person, or may express a sense of presence by an expression that changes the icon shape.

[0023] The AV control unit 138 plays AV content such as music, video, and images and outputs it to the speaker 124 and / or display device 120, in accordance with prior art. Such AV content may include, for example, content stored on media such as DVDs read by the reader constituting the storage device 132, as well as content downloaded and stored in the storage device 132 from a server on the Internet via the TCU 112 by the voice interaction unit 134 in response to voice commands from the user. The AV control unit 138 also outputs audio from radio programs and television programs received from a receiver (not shown) for radio and television broadcast signals to the speaker 124. In addition, the AV control unit 138 outputs video from television programs to the display device 120 when appropriate.

[0024] The UI control unit 140 controls the operation of UI equipment, including the display device 120, microphone 122, and speaker 124. For example, the UI control unit 140 controls the output of voice information via speaker 124 based on instructions from the dialogue device 100. Also, when the UI control unit 140 receives a voice stop instruction from the dialogue device 100, it stops the output of voice information via speaker 124 and instructs the voice dialogue unit 134, AV control unit 138, and / or navigation device 110 to stop operations that involve voice output.

[0025] Furthermore, the UI control unit 140 displays the image or video specified by the instruction on the display device 120 based on instructions from the AV control unit 138, the navigation device 110, and / or the dialogue device 100, and transmits user input via the touch panel (not shown) equipped on the display device 120 to the AV control unit 138, the navigation device 110, and / or the dialogue device 100.

[0026] Furthermore, the UI control unit 140 outputs the voice message specified by the instruction to the speaker 124 based on instructions from the AV control unit 138, the navigation device 110, and / or the dialogue device 100. The UI control unit 140 also acquires utterances such as voice instructions from the user from the microphone 122 and transmits the voice recognition results of the utterances to the AV control unit 138, the navigation device 110, and / or the dialogue device 100.

[0027] Figure 2 shows an example of the arrangement of a display device 120, an in-cabin camera, and a microphone 122 in the interior of a vehicle 102. As shown in Figure 2, the display device 120 is positioned, for example, in the center of the instrument panel 200 in the vehicle width direction of the vehicle 102. The microphone 122 is positioned below the display device 120, closer to the driver. The in-cabin camera is positioned to capture the faces of the users (driver and passenger in the front seat). In the example in Figure 2, the in-cabin camera is positioned below the center of the display device 120. Note that the function and arrangement of the display device 120 are not limited to the above example. For example, the display device 120 may display on the rearview mirror 210, a head-up display on the windshield 212, or an AR (Augmented Reality) display. The display device 120 may also display on the instrument panel 200, the line lighting 214 near the windshield 212, the top of the steering wheel 216, the meter 218, or the steering wheel display 220. Furthermore, the display device 120 may be a dedicated display unit or character with a physical form.

[0028] The dialogue device 100 comprises a processing unit 150 and a storage device 152. The storage device 152 is composed of, for example, a volatile and / or non-volatile semiconductor memory and / or a hard disk drive. The storage device 152 stores a dialogue record 172 that records the content of conversations with the user. The dialogue record 172 is information about past conversations with the user recorded by the dialogue recording unit 168. For example, the dialogue record 172 shows the user's state at the time of the conversation (level of alertness, level of interest, etc.) in association with the content of the conversation (date and time of conversation, conversation text, topic of conversation (category, keyword), etc.).

[0029] The processing unit 150 is a computer equipped with a processor such as a CPU. The processing unit 150 may also have a configuration that includes a ROM (Read Only Memory) on which the program is written, a RAM (Random Access Memory) for temporary data storage, etc. The processing unit 150 includes, as functional elements or functional units realized by executing the program, an arousal level calculation unit 160, an interest level calculation unit 162, a dialogue generation unit 164, a dialogue output unit 166, a dialogue recording unit 168, and a traffic situation determination unit 170.

[0030] The alertness level calculation unit 160 is a processing unit that calculates the user's alertness level based on user status information such as the user's physiological data, registered information such as the user's attributes, the user's video captured by the in-vehicle camera 114, and the user's voice collected by the microphone 122. For example, the user's physiological data includes heart rate, blood pressure, and respiratory rate measured by wearable devices worn by the user. The user's registered information includes the user's gender, age, height, and weight. The alertness level calculation unit 160 calculates the user's alertness level using known alertness level calculation techniques based on the above user status information. Alternatively, the alertness level calculation unit 160 may calculate the alertness level using data on the behavior of the vehicle 102 due to the user's driving operations detected by the vehicle behavior sensor 128. Specifically, the alertness level calculation unit 160 calculates the user's alertness level using known alertness level calculation techniques based on changes in steering operations and acceleration / deceleration based on data on the behavior of the vehicle 102.

[0031] For example, the alertness level calculation unit 160 may determine the user's alertness level by identifying the user's activity state from the user's physiological data. Alternatively, the alertness level calculation unit 160 may detect the degree of eye opening from the shape of the lower eyelid and the distance between the upper and lower eyelids recognized from the user's video, and determine the user's alertness level based on this degree of eye opening. Furthermore, the alertness level calculation unit 160 may determine the alertness level from physiological data, registration information, the user's video, and the user's voice, and use the average of these values.

[0032] The interest level calculation unit 162 is a processing unit that calculates the user's level of interest in a dialogue based on the user's state information collected after the dialogue is output. The user state information used by the interest level calculation unit 162 to calculate the level of interest is the same as that used to calculate the level of arousal, including the user's physiological data, registered information such as the user's attributes, the user's video captured by the in-car camera, and the user's voice collected by the microphone 122. The registered information such as the user's attributes is to be those that were registered in advance before the dialogue. The interest level calculation unit 162 calculates the user's level of interest based on the user's state information using known interest level calculation techniques.

[0033] For example, the interest level calculation unit 162 may determine the pitch difference between the voice (frequency) of the user's utterance in response to the dialogue and a reference frequency, and determine the user's level of interest in the dialogue based on this pitch difference. As an example, the interest level calculation unit 162 calculates a high level of interest if the pitch difference is greater than or equal to a predetermined value and the voice (frequency) is higher than the reference frequency. Conversely, the interest level calculation unit 162 calculates a low level of interest if the pitch difference is within a predetermined value and the voice (frequency) is lower than the reference frequency. The interest level calculation unit 162 may also analyze the content of the user's utterance in response to the dialogue based on dictionary data and calculate a high level of interest if predetermined words (e.g., exclamations) are included. Furthermore, the interest level calculation unit 162 may calculate a high level of interest in a topic if the user's utterance is repeated on the same topic. Regarding the arousal level and interest level described above, all of the data used in their calculation may be used, or any one or a part of them may be used in combination.

[0034] The dialogue generation unit 164 is a processing unit that generates (determines) dialogue content based on instructions from the dialogue output unit 166. Specifically, the dialogue generation unit 164 generates prompts for generating dialogue content using the LLM 302 based on instructions (instructions for generating dialogue content) from the dialogue output unit 166. The dialogue generation unit 164 obtains the dialogue content generated using the LLM 302 by sending the created prompts to the server device 300 that provides the dialogue service using the LLM 302 via the TCU 112. The dialogue generation unit 164 outputs the obtained (generated) dialogue content to the dialogue output unit 166.

[0035] Instructions from the dialogue output unit 166 include instructions to start a dialogue, pause (resume) a dialogue, end a dialogue, and notify the user of traffic conditions. Instructions to start a dialogue include, for example, instructions to start a dialogue by asking a question to the user, and instructions to start a dialogue by an utterance from the user. User utterances include data that transcribes the content of the utterance into text.

[0036] When the dialogue generation unit 164 receives an instruction to start a dialogue, it generates a prompt corresponding to the start of the dialogue (for example, asking a question to the user, responding to an utterance from the user, etc.) and sends it to the server device 300 to obtain the dialogue content at the start of the dialogue. Thereafter, until the dialogue generation unit 164 receives other instructions from the dialogue output unit 166, such as an instruction to pause the dialogue, an instruction to end the dialogue, or an instruction to notify the traffic situation, it continues to generate prompts that include the text data of the user's utterance acquired by the output device 108 from the microphone 122 as reference information, thereby generating dialogue content that repeats the dialogue with the user.

[0037] Here, the dialogue generation unit 164 determines the topic of the dialogue based on the user's level of interest in the dialogue, which is calculated by the interest level calculation unit 162. Specifically, if the user's level of interest in the dialogue falls outside a predetermined range, the dialogue generation unit 164 decides to change the topic to a different topic from the current topic. In this case, the dialogue generation unit 164 generates a prompt that is conditional on changing the topic and sends it to the server device 300 to obtain the dialogue content that changes the topic.

[0038] Regarding the topic to be changed, the dialogue generation unit 164 selects one in which the user's level of interest falls within a predetermined range. Specifically, the dialogue generation unit 164 refers to the past dialogue record 172 with the user and selects a topic in which the user's level of interest falls within a predetermined range based on the user's level of interest associated with the topic (details will be described later).

[0039] Furthermore, when the dialogue generation unit 164 receives an instruction to pause the dialogue, it generates a prompt to generate dialogue content indicating that the dialogue should be temporarily paused and sends it to the server device 300, thereby obtaining dialogue content for pausing the dialogue. When resuming the dialogue after it has been paused, the dialogue generation unit 164 generates a prompt to resume the dialogue and sends it to the server device 300, thereby obtaining dialogue content for resuming the dialogue.

[0040] Furthermore, when the dialogue generation unit 164 receives an instruction to terminate the dialogue, it generates a prompt to generate dialogue content indicating that the dialogue will be terminated and sends it to the server device 300, thereby obtaining dialogue content to terminate the dialogue.

[0041] Furthermore, when the dialogue generation unit 164 receives an instruction to notify traffic conditions, it generates a prompt to notify traffic conditions and sends it to the server device 300, thereby obtaining dialogue content to notify traffic conditions.

[0042] The dialogue output unit 166 is a processing unit that instructs the dialogue generation unit 164 to generate dialogue content and outputs the dialogue content generated by the dialogue generation unit 164 to the output device 108. Specifically, the dialogue output unit 166 instructs the dialogue generation unit 164 to generate dialogue content when predetermined conditions are met. In response to this instruction, the dialogue output unit 166 outputs the dialogue content generated by the dialogue generation unit 164 to the output device 108. As a result, the output device 108 outputs the dialogue content to the user as voice via the speaker 124.

[0043] The predetermined conditions under which the dialogue output unit 166 instructs the generation of dialogue content include the user's speech collected by the output device 108 via the microphone 122, the user's level of alertness calculated by the alertness calculation unit 160, and the traffic conditions around the vehicle 102 determined by the traffic conditions determination unit 170.

[0044] For example, if the dialogue is initiated by a user's utterance, the dialogue output unit 166 notifies the dialogue generation unit 164 of a dialogue start instruction that includes data of the user's utterance in text format. If the dialogue is ended by a user's utterance, the dialogue output unit 166 notifies the dialogue generation unit 164 of a dialogue end instruction that includes data of the user's utterance in text format.

[0045] Furthermore, if the user's level of alertness is below a predetermined threshold, the dialogue output unit 166 notifies the dialogue generation unit 164 to initiate a dialogue by asking the user a question. This allows the dialogue device 100 to initiate a dialogue with the user to increase their alertness if their level of alertness is low (below a predetermined threshold).

[0046] Furthermore, if the dialogue output unit 166 detects a specific traffic situation around the vehicle 102 that the user should pay attention to (for example, lane closures, road construction, accident-prone areas, etc.), it notifies the dialogue generation unit 164 to pause the dialogue so that the conversation with the user does not interfere with driving.

[0047] Furthermore, the dialogue output unit 166 notifies the dialogue generation unit 164 of instructions to notify the user of the traffic conditions around the vehicle 102 if there are certain traffic conditions that would be beneficial for the user to be aware of (for example, lane / speed restrictions, road construction, etc.). Specifically, the dialogue output unit 166 notifies the dialogue generation unit 164 of instructions that include text data to notify the user of the traffic conditions determined by the traffic conditions determination unit 170.

[0048] Note that while there may be some overlap between traffic conditions that users should pay attention to and traffic conditions that users should be reminded to be aware of, each is pre-configured separately.

[0049] The dialogue recording unit 168 is a processing unit that records the content of the conversation with the user as a dialogue record 172. Specifically, the dialogue recording unit 168 associates the user's state during the conversation (level of alertness, level of interest, etc.) with the content of the conversation (date and time of conversation, conversation text, topic of conversation (category, keywords), etc.) to create the dialogue record 172. Here, the topic of conversation (category, keywords) is obtained by the dialogue recording unit 168 using known document analysis techniques for the conversation text.

[0050] The traffic condition determination unit 170 is a processing unit that determines the traffic conditions around the vehicle 102. Specifically, the traffic condition determination unit 170 acquires road traffic information (congestion, speed / lane restrictions, road construction, accident-prone areas, etc.) around the vehicle 102 via the TCU 112 from a server device (not shown) that provides road traffic information. Based on this road traffic information, the traffic condition determination unit 170 determines the traffic conditions in the surrounding area. Alternatively, the traffic condition determination unit 170 may determine the traffic conditions in the surrounding area using known image recognition technology based on video footage of the area around the vehicle 102 taken by the external camera 115.

[0051] For example, the traffic situation determination unit 170 determines whether or not the traffic situation is a predetermined one that requires the user's attention. Examples of predetermined traffic situations that require the user's attention include the vehicle 102 being near an intersection, being under lane restrictions, being in an accident-prone area, having a yellow light (which will soon turn red), and having pedestrians, motorcycles, or bicycles passing by.

[0052] Next, the operation of the dialogue device 100 will be explained with reference to Figures 3 to 5. Figures 3 to 5 are flowcharts showing an example of the operation of the dialogue device 100 according to one embodiment.

[0053] As shown in Figure 3, once processing begins, the dialogue device 100 starts acquiring the user's (occupant's) status using the in-vehicle camera, microphone 122, etc. (S1).

[0054] Next, the alertness level calculation unit 160 calculates the user's alertness level based on the user's state information (S2). Then, the dialogue output unit 166 determines whether the calculated user's alertness level is below a predetermined threshold (S3). If the user's alertness level is not below the predetermined threshold (S3: No), the dialogue output unit 166 returns to S2.

[0055] If the user's level of alertness is below a predetermined threshold (S3: Yes), the dialogue output unit 166 notifies the dialogue generation unit 164 of the instruction to start a dialogue. Upon receiving the instruction to start a dialogue, the dialogue generation unit 164 generates dialogue content and outputs it to the dialogue output unit 166 (S4). The dialogue output unit 166 outputs the dialogue content generated by the dialogue generation unit 164 to the user via the output device 108 (S5).

[0056] Next, the interest level calculation unit 162 calculates the user's level of interest in the dialogue based on the user's state information collected after the dialogue output (S6). The dialogue recording unit 168 associates the user's state during the dialogue (level of alertness, level of interest, etc.) with the content of the dialogue to create a dialogue record 172 (S7).

[0057] Next, the dialogue generation unit 164 determines whether or not to change the content (topic) of the dialogue based on the user's level of interest in the dialogue, which is calculated by the interest level calculation unit 162 (S8). Specifically, if the user's level of interest in the dialogue falls outside a predetermined range, the dialogue generation unit 164 determines that the topic should be changed.

[0058] If the topic is changed (S8: Yes), the dialogue generation unit 164 decides on a new topic different from the current topic (S9) and returns processing to S4. Here, the decision on a new topic will be explained with reference to Figure 4.

[0059] As shown in Figure 4, first, the dialogue generation unit 164 determines whether the user's level of interest in the dialogue is higher than a predetermined range (S20). If the user's level of interest in the dialogue is not higher than the predetermined range (S20: No), the dialogue generation unit 164 skips S21 and S22 and proceeds to S23.

[0060] If the user's level of interest in the dialogue is higher than a predetermined range (S20: Yes), the dialogue generation unit 164 refers to the traffic situation determination unit 170 (S21) and selects a topic from past topics recorded in the traffic situation determination unit 170 that has a lower level of interest than the user's calculated level of interest for the current topic (S22). More specifically, it selects a topic from past topics whose level of interest falls within a predetermined range that has a lower level of interest than the calculated user's level of interest. If there are multiple such topics, it selects the one with the lowest level of interest. The dialogue generation unit 164 may also randomly select a topic if the accuracy of calculating the level of interest is low and the level of interest for each topic has not been sufficiently determined. A low level of accuracy in calculating the level of interest may occur when the number of examples (sample size) used to calculate the level of interest is not above a certain number. Also, when a machine learning model is used to calculate the level of interest, the machine learning model may not have been sufficiently trained, resulting in low accuracy. Thus, if the level of interest in each topic cannot be sufficiently determined, the dialogue generation unit 164 may randomly select topics until the level of interest can be accurately determined.

[0061] Next, the dialogue generation unit 164 determines whether the user's level of interest in the dialogue is below a predetermined range (S23). If the user's level of interest in the dialogue is not below a predetermined range (S23: No), the dialogue generation unit 164 skips S24 and S25 and terminates processing at S23.

[0062] If the user's level of interest in the dialogue is lower than a predetermined range (S23: Yes), the dialogue generation unit 164 refers to the traffic situation determination unit 170 (S24) and selects a topic from past topics recorded in the traffic situation determination unit 170 that has a higher level of interest than the user's calculated level of interest for the current topic (S25). More specifically, it selects a topic from past topics whose level of interest falls within a predetermined range that has a lower level of interest than the calculated user's level of interest. If there are multiple such topics, it selects the one with the highest level of interest among them.

[0063] Returning to Figure 3, if the topic is not changed (S8: No), the dialogue output unit 166 determines whether or not to terminate the dialogue (S10). For example, if there is no termination of the dialogue due to user utterance, the dialogue output unit 166 will continue the dialogue (do not terminate it).

[0064] If the dialogue is to be continued (S10: No), the dialogue generation unit 164 continues to generate prompts related to the dialogue without changing the topic, and generates dialogue content that repeats the dialogue with the user. The dialogue output unit 166 outputs the dialogue content generated by the dialogue generation unit 164 to the user via the output device 108, and the dialogue continues without changing the topic (S11).

[0065] Next, the dialogue corresponding to the traffic conditions around vehicle 102 will be explained with reference to Figure 5. As shown in Figure 5, when processing starts, the traffic condition determination unit 170 starts acquiring traffic conditions (S30).

[0066] Next, the traffic condition determination unit 170 determines whether or not the traffic is in a predetermined traffic condition that requires the user's attention (for example, lane closure, road construction in progress, accident-prone area, etc.) (S31). If the traffic is not in a predetermined traffic condition (S31: No), the traffic condition determination unit 170 waits for processing.

[0067] If the traffic conditions are as specified (S31: Yes), the dialogue output unit 166 notifies the dialogue generation unit 164 of the instruction to pause the dialogue and pauses the dialogue with the user (S32).

[0068] Next, the traffic condition determination unit 170 determines whether there are any predetermined traffic conditions that should prompt the user to be aware of (for example, lane / speed restrictions in place, road construction in progress, etc.) and decides whether or not to notify the user of the traffic condition (prompt the user to be aware) (S33). If the traffic condition notification is not to be made (S33: No), the traffic condition determination unit 170 skips S34 and proceeds to S35.

[0069] If traffic conditions are to be notified (S33: Yes), the dialogue output unit 166 notifies the dialogue generation unit 164 of the instruction to notify the traffic conditions, and the traffic conditions are notified to the user (S34).

[0070] Next, the traffic situation determination unit 170 determines whether the traffic situation around the vehicle 102 has moved beyond a predetermined traffic situation that requires the user's attention (S35). If the traffic situation has not moved beyond the predetermined traffic situation (S35: No), the traffic situation determination unit 170 returns to S32. Therefore, if the traffic situation has not moved beyond the predetermined traffic situation, the dialogue remains paused.

[0071] If the predetermined traffic conditions are resolved (S35: Yes), the dialogue output unit 166 notifies the dialogue generation unit 164 of the instruction to resume the dialogue, and the dialogue is resumed (S36), and processing returns to S31.

[0072] As described above, the dialogue device 100 comprises an alertness calculation unit 160, a dialogue output unit 166, an interest level calculation unit 162, and a dialogue generation unit 164. The alertness calculation unit 160 calculates the user's alertness level based on the user's (occupant's) state information of the vehicle 102. The dialogue output unit 166 outputs a dialogue to the user if the calculated alertness level is less than a predetermined value. The interest level calculation unit 162 calculates the user's interest level in the dialogue based on the user's state information after the dialogue has been output. The dialogue generation unit 164 determines the topic of the dialogue based on the calculated interest level.

[0073] Therefore, in the dialogue device 100, the topic of conversation is determined based on the user's level of interest in the conversation, thus avoiding conversations that would have a low level of interest and a weaker awareness, thus reducing the arousal effect. Consequently, the dialogue device 100 can maintain conversations that have a high arousal effect on the user.

[0074] Furthermore, the dialogue generation unit 164 determines a dialogue topic in which the user's level of interest falls within a predetermined range.

[0075] Therefore, the dialogue device 100 can avoid dialogues that would diminish the arousal effect due to a decrease in the user's level of interest, and conversely, it can avoid situations where the user becomes too interested and pays too much attention to the dialogue.

[0076] Furthermore, the dialogue device 100 includes a dialogue recording unit 168 that associates the interest level calculated for the dialogue with the topic of the dialogue and records it in the dialogue record 172. The dialogue generation unit 164 determines the topic of the dialogue that is associated with a predetermined interest level recorded in the traffic situation determination unit 170.

[0077] Therefore, the dialogue device 100 can determine an appropriate dialogue topic that keeps the user's level of interest within a predetermined range by utilizing the correspondence between the dialogue topics recorded in the dialogue record 172 and the user's level of interest.

[0078] Furthermore, if the calculated level of interest is lower than a predetermined range, the dialogue generation unit 164 selects a dialogue topic from the dialogue topics recorded in the dialogue record 172 that corresponds to a level of interest higher than the calculated level of interest.

[0079] Therefore, if the user's level of interest is low, the dialogue device 100 can change the topic of conversation to one that is more interesting, thereby guiding the user's level of interest to stay within a predetermined range.

[0080] Furthermore, if the calculated level of interest is higher than a predetermined range, the dialogue generation unit 164 selects a dialogue topic from the dialogue topics recorded in the dialogue record 172 that corresponds to a level of interest lower than the calculated level of interest.

[0081] Therefore, if the user's level of interest is high, the dialogue device 100 can change the topic of conversation to something less interesting so as not to cause the user to become engrossed in the conversation, thereby guiding the user's level of interest within a predetermined range.

[0082] Furthermore, the dialogue device 100 includes a traffic condition determination unit 170 that determines the traffic conditions around the vehicle 102. The dialogue output unit 166 pauses outputting dialogue if the traffic conditions determined by the traffic condition determination unit 170 are predetermined traffic conditions.

[0083] Therefore, the dialogue device 100 can assist the driver by pausing the dialogue when a predetermined traffic condition is present, allowing the user to concentrate on dealing with that traffic condition.

[0084] Furthermore, the dialogue output unit 166 outputs a notification corresponding to a traffic condition if the traffic condition determined by the traffic condition determination unit 170 is a predetermined traffic condition. The dialogue output unit 166 may use a different voice (voice quality) when outputting notifications corresponding to traffic conditions than other voices. For example, the dialogue output unit 166 may output a notification with a higher voice quality to convey a sense of urgency, as it is a notification of traffic conditions that directly relates to driving, unlike the conversation (dialogue) directed to the user by the dialogue device 100. In this way, the dialogue output unit 166 may output notifications regarding traffic conditions with a different voice quality than the conversation (dialogue) directed to the user, thereby differentiating the connection between the voice quality and the event (traffic condition) from others.

[0085] Therefore, the dialogue device 100 can easily recognize that the user is in a predetermined traffic situation and assist the user in driving so that they can concentrate on dealing with that traffic situation.

[0086] Since the embodiments described above are for illustrative purposes only, various modifications, substitutions, additions, omissions, etc., can be made within the claims or their equivalents.

[0087] For example, while the creation of dialogue content in the dialogue generation unit 164 of the dialogue device 100 described above is illustrated using a dialogue service provided by the LLM 302 of the server device 300, the dialogue generation unit 164 may create the dialogue content independently. In other words, the dialogue device 100 may provide a dialogue service provided by the LLM. Specifically, the storage device 152 stores learning model information of a machine learning model that outputs conversation content (dialogue) directed at the user in response to input of utterances from the user and dialogue conditions such as topics. The dialogue generation unit 164 inputs utterances from the user and dialogue conditions such as topics to a model constructed based on this learning model information. As a result, the dialogue generation unit 164 obtains the output of the model, i.e., the dialogue content.

[0088] Furthermore, the configuration of each part of the dialogue device 100 shown in Figure 1 is merely an example, and the specific implementation form is not particularly limited. In other words, it is not necessarily required that hardware corresponding to each part be implemented individually, and it is certainly possible to configure the system so that a single processor executes a program to realize the functions of each part. Also, some of the functions realized by software in the above embodiment may be implemented as hardware, or some of the functions realized by hardware may be implemented as software.

[0089] Furthermore, the operational step units in the processing device 150 shown in Figure 1 are divided according to the main processing content, and this disclosure is not limited by the way the processing units are divided or by their names. Depending on the processing content, it may be further divided into more step units. Alternatively, it may be divided so that one step unit includes even more processing. Also, the order of the steps may be rearranged as appropriate, as long as it does not impede the intent of this disclosure.

[0090] Furthermore, when the interaction method by the computer, such as the processing unit 150 of the dialogue device 100 described above, is implemented using a processor, the dialogue program to be executed by this processor can also be configured as a recording medium or a transmission medium for transmitting this dialogue program. In other words, the above dialogue program can also be implemented by recording the dialogue program on a portable information recording medium. Examples of information recording media include magnetic recording media such as hard disks, optical recording media such as CDs, and semiconductor storage devices such as USB (Universal Serial Bus) memory and SSDs (Solid State Drives), but other recording media can also be used.

[0091] [Configurations supported by the above embodiment] The embodiments described above support the following configurations.

[0092] (Configuration 1) A dialogue device comprising: an alertness calculation unit that calculates the alertness level of a vehicle occupant based on the state information of the occupant; a dialogue output unit that outputs a dialogue to the occupant when the calculated alertness level is less than a predetermined value; an interest level calculation unit that calculates the occupant's level of interest in the dialogue based on the state information after the dialogue has been output; and a dialogue generation unit that determines the topic of the dialogue based on the calculated interest level. According to the dialogue device of configuration 1, the topic of the dialogue can be determined based on the crew's level of interest in the dialogue, and a dialogue with a high level of alertness for the crew can be achieved.

[0093] (Configuration 2) The dialogue device according to Configuration 1, wherein the dialogue generation unit determines the topic of the dialogue to be within a predetermined range of interest for the crew member. The dialogue device in configuration 2 can maintain a level of engagement that is neither too high nor too low for the crew.

[0094] (Configuration 3) The dialogue device according to Configuration 2, further comprising a dialogue recording unit that records the interest level calculated for the dialogue and the topic of the dialogue in association with each other, wherein the dialogue generation unit determines the topic of the dialogue to be associated with a predetermined interest level recorded in the dialogue recording unit. According to the dialogue device of configuration 2, the topic of conversation with the crew can be determined from past records stored in the dialogue recording unit.

[0095] (Configuration 4) The dialogue device according to Configuration 3, wherein the dialogue generation unit, when the calculated level of interest is lower than the predetermined range, selects a dialogue topic from among the dialogue topics recorded in the dialogue recording unit that corresponds to a level of interest higher than the calculated level of interest. According to the dialogue device in configuration 4, if the crew's level of interest is low, the topic of conversation can be changed to one that will be more interesting to them.

[0096] (Configuration 5) The dialogue device according to Configuration 3, wherein the dialogue generation unit, when the calculated level of interest is higher than the predetermined range, selects a dialogue topic from among the dialogue topics recorded in the dialogue recording unit that corresponds to a level of interest lower than the calculated level of interest. According to the dialogue device in configuration 5, if the crew's level of interest is high, the topic of conversation can be changed to one that is less interesting to them.

[0097] (Configuration 6) The dialogue device according to Configuration 1, further comprising a traffic condition determination unit that determines the traffic conditions around the vehicle, wherein the dialogue output unit pauses outputting the dialogue when the determined traffic conditions are predetermined traffic conditions. The dialogue device in configuration 6 can assist the occupant in driving, allowing them to focus on and deal with traffic conditions rather than engaging in dialogue.

[0098] (Configuration 7) The dialogue device according to Configuration 1, further comprising a traffic condition determination unit that determines the traffic conditions around the vehicle, wherein the dialogue output unit outputs a notification corresponding to the traffic conditions when the determined traffic conditions are predetermined traffic conditions. According to the dialogue device of configuration 7, the occupant can easily recognize a given traffic situation and assist the occupant in driving.

[0099] (Configuration 8) A dialogue method comprising the following processes: a computer calculates the level of alertness of a crew member based on the crew member's state information; if the calculated level of alertness is less than a predetermined value, outputs a dialogue to the crew member; calculates the crew member's level of interest in the dialogue based on the state information after the dialogue has been output; and determines the topic of the dialogue based on the calculated level of interest. The dialogue method of configuration 8 produces the same effect as configuration 1 described above.

[0100] (Configuration 9) A dialogue program that causes a computer to perform the following processes: calculate the level of alertness of the crew member based on the crew member's state information; output a dialogue to the crew member if the calculated level of alertness is less than a predetermined value; calculate the crew member's level of interest in the dialogue based on the state information after the dialogue has been output; and determine the topic of the dialogue based on the calculated level of interest. According to the dialogue program of Configuration 9, the same effect as Configuration 1 described above is achieved. [Explanation of Symbols]

[0101] 100...Dialogue device, 102...Vehicle, 104...In-vehicle network bus, 106...Camera control device, 108...Output device, 110...Navigation device, 112...TCU, 114...In-vehicle camera, 115...External camera, 116...Processing device, 118...Long-range communication device, 120...Display device, 122...Microphone, 124...Speaker, 126...Driving judgment unit, 128...Vehicle behavior sensor, 130...Processing device, 132...Storage device, 134...Vehicle dialogue unit, 136...Icon display unit, 138...AV control 140...UI control unit, 150...processing unit, 152...memory device, 160...awakening level calculation unit, 162...interest level calculation unit, 164...dialogue generation unit, 166...dialogue output unit, 168...dialogue recording unit, 170...traffic situation determination unit, 172...dialogue recording, 200...instrument panel, 210...rearview mirror, 212...windshield, 214...line lighting, 216...top of steering wheel, 218...meter, 220...in-steering wheel display, 300...server device, 302...LLM.

Claims

1. A state of alertness calculation unit that calculates the alertness level of the occupant based on the state information of the vehicle occupant, A dialogue output unit outputs a dialogue to the occupant if the calculated level of alertness is less than a predetermined value, An interest level calculation unit calculates the crew member's level of interest in the dialogue based on the state information after the dialogue has been output. The system includes a dialogue generation unit that determines the topic of the dialogue based on the calculated level of interest, Dialogue device.

2. The dialogue generation unit determines the topic of the dialogue to be one in which the crew member's level of interest falls within a predetermined range. The dialogue device according to claim 1.

3. The system further includes a dialogue recording unit that records the level of interest calculated for the aforementioned dialogue in association with the topic of the aforementioned dialogue. The dialogue generation unit determines the topic of the dialogue to correspond to a predetermined level of interest recorded in the dialogue recording unit. The dialogue device according to claim 2.

4. The dialogue generation unit, when the calculated level of interest is lower than the predetermined range, selects a dialogue topic from among the dialogue topics recorded in the dialogue recording unit that corresponds to a level of interest higher than the calculated level of interest. The dialogue device according to claim 3.

5. The dialogue generation unit, when the calculated level of interest is higher than the predetermined range, selects a dialogue topic from among the dialogue topics recorded in the dialogue recording unit that corresponds to a level of interest lower than the calculated level of interest. The dialogue device according to claim 3.

6. The vehicle further comprises a traffic condition determination unit that determines the traffic conditions around the vehicle, The dialogue output unit pauses outputting the dialogue if the determined traffic condition is a predetermined traffic condition. The dialogue device according to claim 1.

7. The vehicle further comprises a traffic condition determination unit that determines the traffic conditions around the vehicle, The dialogue output unit outputs a notification corresponding to the traffic condition if the determined traffic condition is a predetermined traffic condition. The dialogue device according to claim 1.

8. Computers Based on the crew's status information, the crew's level of alertness is calculated. If the calculated level of alertness is less than a predetermined value, a dialogue message is output to the crew member. Based on the state information after outputting the dialogue, the crew member's level of interest in the dialogue is calculated. Based on the calculated level of interest, the process is executed to determine the topic of the conversation. Methods of dialogue.

9. On the computer, Based on the crew's status information, the crew's level of alertness is calculated. If the calculated level of alertness is less than a predetermined value, a dialogue message is output to the crew member. Based on the state information after outputting the dialogue, the crew member's level of interest in the dialogue is calculated. Based on the calculated level of interest, the process of determining the topic of the dialogue is executed. Dialogue program.

Images