Speech interaction device and speech interaction method

Abstract

A processor (10) recognizes interaction details of a user before interruption of speech interaction, extracts, on the basis of the interaction details, a keyword in the speech interaction before interruption as a pre-interruption keyword, generates, on the basis of the extracted pre-interruption keyword, an icon image indicating the interaction details before interruption, and generates a control instruction for displaying the generated icon image.

Description

Voice dialogue device and voice dialogue method

[0001] The present invention relates to a voice dialogue device and a voice dialogue method. This application claims priority based on Japanese Patent Application No. 2022-73531, filed on April 27, 2022, and for designated states where incorporation by reference of documents is permitted, the contents of the above application are incorporated by reference into this application and made a part of the description of this application.

[0002] A technology is known in which, when notification information is displayed to a driver, if the driver's driving load is higher than a predetermined driving load, the notification information is assigned an importance level based on a predetermined operation by the driver, the display of the notification information is temporarily suppressed, and when the notification information is displayed again, the notification information is displayed according to the importance level (Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2020-035262

[0004] However, with the technology of Patent Document 1, even if the display of notification information is suppressed and then the notification information that was displayed before the suppression is re-displayed, there is a problem in that when the notification information is re-displayed, the user may have forgotten the content of the notification information that was displayed before the suppression.

[0005] The problem to be solved by the present invention is to provide a voice dialogue device and a voice dialogue method that enable a user to recall the content of output information before the interruption when resuming the provision of information to the user who has been interrupted.

[0006] The present invention solves the above problem by recognizing the content of the user's dialogue before the voice dialogue is interrupted, extracting keywords in the voice dialogue before the interruption as pre-interruption keywords based on the dialogue content, generating an icon image representing the content of the dialogue before the interruption based on the extracted pre-interruption keywords, and generating a control command to display the generated icon image.

[0007] According to the present invention, when the provision of information to a user that has been interrupted is resumed, the user can recall the content of the output information that was there before the interruption.

[0008] Fig. 1 is a block diagram showing an embodiment of a voice dialogue device according to the present invention. Figs. 2(a) and (b) are diagrams showing an example of a screen displayed to a user in this embodiment. Figs. 3(a), (b), and (c) are examples of icon images whose amount of information changes depending on the user's driving load in this embodiment. Figs. 4(a), (b), and (c) are examples of icon images that change depending on a topic recognized during an interruption of a voice dialogue in this embodiment. Fig. 5 is a diagram showing an example of a flowchart of processing related to a voice dialogue method executed by the voice dialogue device.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. A voice dialogue system including a voice dialogue device according to the present invention is a system that uses voice recognition technology and voice synthesis technology to have a voice dialogue with a user. The user refers to a person who uses the voice dialogue system, and is primarily a vehicle driver, but the user may also be a passenger other than the driver. The voice dialogue system 100 may also be a system used in a location other than an automobile. In the following, an example in which the present invention is applied to a voice dialogue system that has a voice dialogue with a user, with the vehicle driver being the user, will be described. However, the present invention is not limited to this, and may also be applied to a voice dialogue system that controls a voice dialogue between a user and another user.

[0010] 1 is a block diagram showing an embodiment of a voice dialogue system according to the present invention. The voice dialogue system 100 includes a voice dialogue device 1, an in-vehicle sound collection device 2, vehicle sensors 3, a user state detection device 4, and a map database 5. These devices and databases are mounted on a vehicle and are connected, for example, via a CAN or other in-vehicle LAN, allowing them to send and receive information to and from each other. The voice dialogue device 1 is a device that controls voice dialogue with a user, and includes a processor 10 and an output device 6.

[0011] The output device 6 is a device that outputs information to the user through an agent function by a personified agent (hereinafter simply referred to as agent A), specifically, through a medium that combines audio and images. Note that, while the output device 6 is described here as an electronic device mounted on a vehicle, it may be any electronic device that has an agent function, such as a portable electronic device with a display. Furthermore, functions related to the audio output and video output of agent A may be incorporated into a mobile phone such as a smartphone. Furthermore, the output device 6 may be capable of outputting audio, images, text information, and the like to present information to the user, and may also serve as the display of a navigation device, for example.

[0012] 2A and 2B are diagrams showing examples of screens displayed to a user in the voice dialogue system according to this embodiment. For example, the screen shown in FIG. 2 is displayed on a head-up display device or a display device mounted on the vehicle. FIG. 2A illustrates an example of visually displaying agent A. As shown in FIG. 2A, agent A is displayed on the display 101 in such a way that an agent image 102, which is an avatar resembling a human, is superimposed on the scenery outside the vehicle. As described above, when agent A outputs communication information to the user, it displays the agent image 102 using the agent function that receives a control command from the display control unit 17. The user can engage in voice dialogue while understanding the gestures and movements of agent A, his or her conversation partner, via the screen displayed on the display 101.

[0013] The voice dialogue device 1 also has a dialogue interruption function for interrupting a voice dialogue with a user. When the voice dialogue device 1 determines that the voice dialogue needs to be interrupted during the voice dialogue with the user, the voice dialogue device 1 interrupts the voice dialogue. While the voice dialogue is interrupted, an image of Agent A, the user's voice dialogue partner, is not displayed, as shown in FIG. 2( b). In this embodiment, as shown in FIG. 2( b), while the voice dialogue is interrupted, an icon image 112 representing the dialogue content before the voice dialogue was interrupted is displayed on the display 111 so as to be superimposed on the scenery outside the vehicle. The content of the icon image may include illustrations, text, photos, videos, or the like representing the dialogue content before the voice dialogue was interrupted, as well as avatar expressions corresponding to the atmosphere of the voice dialogue. Examples of the dialogue content displayed on the icon include topics such as cafes, restaurants, shopping, events, tourist spots, music, news, weather, and scenery.

[0014] In addition, in this embodiment, the user's dialogue partner is described as an example of a voice dialogue system, but this is not limited thereto. The dialogue partner may also be a human. For example, the dialogue partner may be a person other than a vehicle occupant outside the vehicle. In the example of FIG. 2( a), the dialogue partner is illustrated as an example of an agent A that engages in a voice dialogue with the user. However, if the dialogue partner is a human, an image of the dialogue partner may also be used. Furthermore, if the voice dialogue device 1 determines to resume the voice dialogue while the voice dialogue is interrupted, the voice dialogue device 1 resumes the voice dialogue. When the voice dialogue is resumed, as shown in FIG. 2( a), the agent image 102 of agent A, the user's dialogue partner, is displayed again, and the display of the aforementioned icon image is interrupted. Note that the display of the icon image is not limited to being interrupted, and the icon image may continue to be displayed after the voice dialogue is resumed.

[0015] As described above, agent A includes a speaker or other audio output unit for outputting voice and sound effects, and a display or other display unit for displaying images including text, and outputs communication information to the user by providing voice, sound effects, text, and other images along with the actions of agent A. Furthermore, the anthropomorphized agent is just one example, and the agent may not be a human, but may instead display a predetermined character, avatar, or icon. Agent A may be installed as a physical entity in a form that can appear and disappear by operating an actuator. Furthermore, the display of an agent as a conversation partner is illustrated as an example, and the present invention can also be applied to a voice dialogue system in which agent A is not displayed as a conversation partner.

[0016] The interior sound collection device 2 collects sounds heard by passengers inside the vehicle. The interior sound collection device 2 is, for example, a microphone installed inside the vehicle. The sounds collected by the interior sound collection device 2 are mainly sounds from sources inside the vehicle cabin, and include user voices and sounds output from in-vehicle media. User voices include, for example, conversations between passengers and dialogues between the voice dialogue system and passengers. In-vehicle media is, for example, audio and radio.

[0017] The vehicle sensors 3 acquire vehicle surroundings information about the surroundings of the vehicle. The vehicle sensors 3 are, for example, exterior cameras that capture images of the outside of the vehicle. The exterior cameras are CCD cameras or CMOS cameras that have imaging elements such as CCD or CMOS. The exterior cameras acquire images of the surroundings of the vehicle as vehicle surroundings information. The vehicle sensors 3 also include a GPS. The GPS detects the current position of the vehicle. The detection results of the vehicle sensors 3 are output to the processor 10 at predetermined time intervals.

[0018] The user state detection device 4 detects information related to the user's state. The user's state includes, for example, information related to the user's physical movements. Examples of the user state detection device 4 include an in-vehicle camera that captures an image of the user inside the vehicle, and a biosignal measurement sensor that detects the user's biosignals. The in-vehicle camera is a CCD camera or a CMOS camera that has an imaging element such as a CCD or CMOS. The biosignal measurement sensor is provided inside the steering wheel or seat, and detects biosignals such as the user's blood pressure, changes in pulse rate, and degree of sweating as information related to the user's body. The detection results of the user state detection device 4 are output to the processor 10 at predetermined time intervals.

[0019] The map database 5 is a database that stores map information. The map information includes road information and the like. The map database 5 also includes information on POIs (Points of Interest). POIs are information on specific locations on a map, such as the locations of public facilities, restaurants, tourist attractions, and other landmark buildings.

[0020] The voice dialogue device 1 is composed of a processor 10 including a ROM storing programming for executing various processes, a CPU as an operating circuit that functions as the voice dialogue device 1 by executing the programs stored in the ROM, and a RAM that functions as an accessible storage device, and an agent A that functions as an output device 6. The processor 10 includes, as functional blocks, a recognition unit 11, a driving status acquisition unit 12, a keyword extraction unit 13, a driving load estimation unit 14, a determination unit 15, an image generation unit 16, and a display control unit 17. The processor 10 also includes a keyword image database 18 and a dialogue partner information database 19.

[0021] The recognition unit 11 recognizes the content of the user's dialogue before the user's voice dialogue is interrupted. The recognition unit 11 acquires the user's voice and / or the voice of the user's dialogue partner before the voice dialogue is interrupted from the in-vehicle sound collection device 2. The recognition unit 11 performs a voice recognition process on the acquired voice to recognize the content of the user's dialogue. The recognition unit 11 may also summarize the content of the user's dialogue. The content of the user's dialogue includes the content of the user's utterance before the interruption and the content of the user's dialogue partner before the interruption.

[0022] After displaying the icon image, the recognition unit 11 recognizes at least one topic of the user's conversation, the in-vehicle content being played in the vehicle, and the vehicle surrounding objects around the vehicle. The topic is a candidate topic to be presented to the user when resuming the voice dialogue with the user, and is, for example, an object that the user may be interested in. Even while the voice dialogue with the user is interrupted, the user may become interested in an object that can be recognized visually or audibly. Since the user's interest may shift from the dialogue content before the interruption to another object, in this embodiment, a topic that the user may be interested in during the interruption is recognized.

[0023] For example, the recognition unit 11 recognizes the content of the user's conversation. The recognition unit 11 acquires the user's voice while the voice dialogue is interrupted from the in-vehicle sound collection device 2. The recognition unit 11 performs a voice recognition process on the acquired user's voice to recognize the content of the user's conversation. The recognition unit 11 may also summarize the content of the user's conversation. The content of the user's conversation includes the content of the interrupted conversation between the user and other passengers while the voice dialogue with the user is interrupted.

[0024] The recognition unit 11 recognizes in-vehicle content being played in the vehicle. Examples of the content include music being played from media such as in-vehicle audio or radio, and the remarks of radio personalities. The recognition unit 11 acquires the audio of the media from the in-vehicle sound collection device 2. The recognition unit 11 recognizes the in-vehicle content by performing a voice recognition process on the acquired audio of the media. The recognition unit 11 may also summarize the content of the in-vehicle content, such as the content of a radio program or the remarks of a radio personality.

[0025] The recognition unit 11 recognizes objects around the vehicle based on information detected by vehicle sensors 3 such as a GPS and an exterior camera. The objects around the vehicle are, for example, points of interest (POIs) around the vehicle. For example, the recognition unit 11 recognizes POIs located around the vehicle as the objects around the vehicle based on the current position of the vehicle and map information in the map database 5. The recognition unit 11 also performs image recognition processing on an image of the area around the vehicle captured by the exterior camera, extracts feature points of the identified POIs, and recognizes the objects around the vehicle.

[0026] The driving status acquisition unit 12 acquires driving status information indicating the driving status of the user driving the vehicle from the user state detection device 4. For example, the user's driving status information includes information about the user's body and information about the user's movements. The information about the user's body includes the user's blood pressure, changes in pulse rate, and the degree of sweating. The information about the user's movements includes the user's operations, facial expressions, gaze, gestures, etc. The driving status acquisition unit 12 acquires information about the user's movements, for example, by performing image recognition processing on the image of the user acquired from the user state detection device 4.

[0027] The keyword extraction unit 13 extracts pre-interruption keywords in the voice dialogue before the interruption, based on the dialogue content before the voice dialogue was interrupted, which has been recognized by the recognition unit 11. For example, the keyword extraction unit 13 extracts keywords in the voice dialogue before the interruption as pre-interruption keywords, based on a summary of the dialogue content. The keyword extraction unit 13 extracts nouns included in the dialogue content before the interruption as keyword candidates, and extracts the keyword candidate with the highest importance as the pre-interruption keyword.

[0028] The keyword extraction unit 13 also extracts keywords of topical subjects based on the content of the user's conversation and / or the in-vehicle content recognized by the recognition unit 11 while the voice dialogue is interrupted. For example, the keyword extraction unit 13 extracts keywords in the content of the conversation and the in-vehicle content as keywords based on summaries of the conversation and the in-vehicle content. The keyword extraction unit 13 also extracts keywords based on vehicle-periphery objects recognized by the recognition unit 11 while the voice dialogue is interrupted. For example, the keyword extraction unit 13 extracts names of vehicle-periphery objects as keywords from information about the vehicle-periphery objects.

[0029] The driving load estimation unit 14 estimates the driving load of the user based on the driving situation of the user. The driving load is the load that the user bears when driving a vehicle. The greater the driving load, the greater the user's concentration on driving is required to drive. The driving load estimation unit 14 estimates the driving load based on the situation of the user's driving operation. For example, the driving load estimation unit 14 estimates a higher driving load as the steering operation amount or the accelerator pedal or brake pedal operation amount increases.

[0030] Furthermore, the driving load estimation unit 14 estimates the driving load based on the user's physical condition. For example, the higher the user's blood pressure or heart rate, the greater the estimated driving load. The driving load estimation unit 14 may also estimate the driving load based on the user's posture. The driving load estimation unit 14 estimates the user's line of sight, and the greater the user's line of sight, the greater the estimated driving load.

[0031] Furthermore, the driving load estimation unit 14 estimates the driving load based on the driving situation. When the driving situation of the vehicle is a specific situation, the driving load estimation unit 14 estimates the driving load to be higher than when the driving situation is other than the specific situation. For example, when the vehicle is driving, the driving load estimation unit 14 estimates the driving load to be higher than when the vehicle is stopped. Furthermore, specific situations include when the vehicle is driving on a highway, when the vehicle is turning right or left, and when an obstacle such as another vehicle is close to the vehicle.

[0032] The determination unit 15 determines whether to interrupt the user's voice dialogue. The determination unit 15 acquires information about the user's state from the user state detection device 4, and determines to interrupt the user's voice dialogue when the user's state is such that the voice dialogue needs to be interrupted. Furthermore, the determination unit 15 determines not to interrupt the user's voice dialogue when the user's state is not such that the voice dialogue needs to be interrupted. An example of a state in which the voice dialogue needs to be interrupted is when the user's driving load increases. The determination unit 15 determines to interrupt the user's voice dialogue when the user is performing a steering operation. Furthermore, the determination unit 15 determines to interrupt the user's voice dialogue when the user's blood pressure or heart rate is high.

[0033] Incidentally, the situation in which the user's voice dialogue needs to be interrupted is not limited to this, but includes when the driving subject switches from the system to the driver in automated driving, when the user receives another voice call or video call, when the user performs an operation to interrupt the conversation, etc. Based on an image of the user, when the user is performing a driving operation, that is, when the driving subject switches from the system to the user in automated driving, the determination unit 15 determines that the user's state is one in which the user's voice dialogue needs to be interrupted.

[0034] Furthermore, the determination unit 15 determines whether or not to resume the interrupted voice dialogue of the user. For example, while the voice dialogue of the user is interrupted, the determination unit 15 acquires information about the user's state at regular intervals, and determines whether or not the user's state is in a state in which the voice dialogue of the user can be resumed based on the information about the user's state. The state in which the voice dialogue of the user can be resumed includes, for example, when the user's driving load is reduced, or when the driving subject is switched from the user to the system in automated driving.

[0035] The image generation unit 16 generates an icon image representing the topic of the user's voice dialogue. The icon image includes, for example, a keyword image corresponding to a keyword in the voice dialogue and a dialogue partner image representing the user's dialogue partner. When the user's voice dialogue is interrupted, the image generation unit 16 generates an icon image representing the dialogue content before the voice dialogue was interrupted based on the pre-interruption keyword. The image generation unit 16 refers to the keyword image database 18 based on the pre-interruption keyword, and acquires a pre-interruption keyword image corresponding to the pre-interruption keyword from the keyword image database 18. For example, when a keyword related to a cafe is extracted, the image generation unit 16 acquires an image representing a cafe, such as an image of a coffee cup.

[0036] Furthermore, the image generation unit 16 refers to the dialogue partner information database 19 to acquire a dialogue partner image representing the dialogue partner who was conversing with the user before the voice dialogue was interrupted, such as avatar model data and character name.The image generation unit 16 then combines the acquired keyword image and dialogue partner image to generate an icon image including the keyword image and the dialogue partner image.As a result, when the user's voice dialogue is interrupted, for example, the image generation unit 16 generates an icon image representing the dialogue content of the voice dialogue before the interruption, based on the keyword image before the interruption and the dialogue partner image.

[0037] When the content of the user's voice dialogue relates to an object around the vehicle, the image generation unit 16 may acquire an image of the object around the vehicle captured by the vehicle sensors 3 and generate an icon image including the image of the object around the vehicle. The image generation unit 16 may also generate an icon image including a keyword image but not including an image of the dialogue partner.

[0038] Furthermore, in this embodiment, an icon image may be generated according to the driving load estimated by the driving load estimation unit 14. For example, the image generation unit 16 generates an icon image with an amount of information according to the driving load. The greater the driving load, the smaller the amount of information of the icon image is set to. The image generation unit 16 determines whether the driving load is greater than a load threshold while the user's voice dialogue is interrupted. If it is determined that the driving load is greater than the load threshold, the image generation unit 16 generates an icon image with less information than if it is determined that the driving load is not greater than the load threshold. This makes it possible to reduce the amount of information displayed to the user when the driving load is high.

[0039] In this embodiment, when a voice dialogue is interrupted, an icon image may be generated that represents the dialogue content before the interruption, with an amount of information corresponding to the driving load, or if the driving load changes after the icon image is displayed, the icon image displayed to the user may be updated to an icon image with an amount of information corresponding to the driving load.

[0040] Examples of icon images that change depending on driving load are described using FIGS. 3( a) to 3(c). FIGS. 3(a) to 3(c) show examples of icon images in which the amount of information changes depending on the driving load when the user is the driver. FIG. 3(a) shows an icon image 301 when the user's driving load is low. The icon image 301 shown in FIG. 3(a) is an example of an icon image with a large amount of information displayed to the user, and includes a keyword image 302 representing the content of the dialogue before the interruption and a dialogue partner image 303 representing the avatar with whom the user is having a voice dialogue. The keyword image 302 is an example of a keyword image corresponding to the keyword "cafe." Displaying more information allows the user to smoothly resume the voice dialogue.

[0041] On the other hand, if the user's driving load increases while the user's voice dialogue is interrupted, an icon image displaying less information to the user is displayed, as shown in FIG. 3( b). In FIG. 3( b), icon image 311 is an example of an icon image displaying less information to the user. The dialogue partner image is no longer displayed, and a keyword image 312 representing the dialogue content before the interruption is displayed. Furthermore, if the driving load increases further, an image displaying even less information to the user is displayed, as shown in FIG. 3( c). In FIG. 3( c), icon image 321 includes only image 322 representing that a voice dialogue was being held. Image 322 may be any image that represents that a voice dialogue was being held, such as the icon of the application with which the voice dialogue was being held. This prevents the user's attention from being distracted by the display of an icon image.

[0042] In addition, in this embodiment, while the user's voice dialogue is interrupted, the icon image displayed to the user may be updated based on at least one of the user's conversation content, in-vehicle content, and vehicle surrounding objects. For example, after displaying the icon image, the image generation unit 16 determines whether the topic subject has been recognized by the recognition unit 11. If it is determined that the topic subject has been recognized, the image generation unit 16 determines whether the importance of the recognized topic subject is greater than an importance threshold. The importance is set in advance for each topic subject, for example. The importance is set to a higher value as the topic subject becomes more likely to become a topic candidate in the user's voice dialogue.

[0043] When the image generation unit 16 determines that the importance of the topic subject is greater than the importance threshold, the image generation unit 16 generates a new icon image based on a topic image representing the topic subject whose importance is determined to be greater than the importance threshold, and updates the icon image displayed to the user to the generated icon image. The image generation unit 16 acquires a topic image representing the topic subject. For example, the image generation unit 16 acquires, from the keyword image database 18, a keyword image corresponding to a keyword in the topic subject as the topic image. The image generation unit 16 may also acquire, as the topic image, an image of an object around the vehicle. The image generation unit 16 generates an icon image including the topic image. At this time, the image generation unit 16 may also generate an icon image including the topic image and a conversation partner image.

[0044] While the user's voice dialogue is interrupted, until the user's voice dialogue is resumed, each time a topic is recognized by the recognition unit 11, the image generation unit 16 determines whether the importance of the topic is greater than the importance threshold. If the importance is greater than the importance threshold, the image generation unit 16 generates an icon image including a topic image representing the topic.

[0045] Examples of icon images that change depending on the topic recognized while the user's voice dialogue is interrupted will be described using Figures 4(a) to 4(c). Figures 4(a) to 4(c) are examples of icon images that are updated depending on the topic recognized while the user's voice dialogue is interrupted. Figure 4(a) is an example of a pre-interruption icon image that is displayed to the user immediately after the user's voice dialogue is interrupted. The icon image 401 includes a pre-interruption keyword image 402 and a dialogue partner image 403.

[0046] On the other hand, if the user's voice dialogue is interrupted for a long period of time, the user's interests may change. Therefore, in this embodiment, the icon image displayed to the user is updated based on the user's conversation content, in-vehicle content, and information about objects around the vehicle. FIG. 4B shows an icon image updated based on the in-vehicle content. The icon image 411 includes a keyword image 412 for the in-vehicle content and a conversation partner image 414. The in-vehicle content is music being played from the audio system. For example, the icon image 411 shows an avatar represented by the conversation partner image 414 showing interest in the music being played from the audio system.

[0047] 4(c) shows an icon image that has been updated after the icon image of FIG. 4(b) has been displayed. In FIG. 4(c), the icon image 421 includes a keyword image 422 indicating an object around the vehicle and a dialogue partner image 424. For example, the icon image 421 shows an avatar indicated by the dialogue partner image 424 showing interest in the object around the vehicle. As shown by the pre-interruption keyword image 413 in FIG. 4(b) and the pre-interruption keyword image 423 in FIG. 4(c), even after the icon image is updated, the pre-interruption keyword image may be displayed in a reduced size on the icon image.

[0048] The display control unit 17 controls the display of the icon image generated by the image generation unit 16 to the user. The display control unit 17 generates a control command to display the icon image to the user and transmits the generated control command to the output device 6. Upon receiving the control command, the output device 6 outputs output data including the icon image as communication information using the agent function of agent A. Note that in this embodiment, the output device 6 is not limited to displaying the icon image, and a communication device external to the voice dialogue apparatus 1 may display the icon image. The communication device is, for example, a device such as a smartphone or tablet terminal used by the user. In this case, the display control unit 17 outputs the control command to the communication device.

[0049] As described above, when the voice dialogue of the user is interrupted, the display control unit 17 displays to the user an icon image representing the dialogue content before the interruption while the voice dialogue is interrupted. This allows the user to look at the icon image even while the voice dialogue is interrupted and to recall the dialogue content before the interruption, allowing the user to smoothly start the voice dialogue when resuming it.

[0050] The keyword image database 18 is a database that stores keyword images corresponding to keywords in voice dialogues for each keyword. In this embodiment, data on keyword images corresponding to each keyword is stored in advance. For example, for the keyword "cafe," an image representing a cafe, such as an image of a coffee cup, is stored. Also, for a keyword related to a specific landmark, the keyword image database 18 stores an image representing the landmark.

[0051] The dialogue partner information database 19 is a database that stores dialogue partner information of the user for each dialogue partner. The dialogue partner information is, for example, a dialogue partner image representing the dialogue partner. The dialogue partner information database 19 stores, for example, an image representing an avatar of agent A and an image of the user's dialogue partner.

[0052] Next, an example of a control procedure for executing the voice interaction method according to this embodiment will be described with reference to Fig. 5. Fig. 5 is an example of a flowchart showing the control procedure related to the voice interaction method. While the user is having a voice interaction, the processor 10 starts the flow from step S101.

[0053] In step S101, the processor 10 determines whether or not to interrupt the user's voice dialogue. If it is determined that the voice dialogue is to be interrupted, the processor 10 proceeds to step S102. If it is determined that the voice dialogue is not to be interrupted, the processor 10 returns to step S101 and repeats the subsequent steps. In step S102, the processor 10 recognizes the content of the dialogue before the interruption. In step S103, the processor 10 extracts pre-interruption keywords based on the content of the dialogue before the interruption.

[0054] In step S104, the processor 10 generates an icon image including a pre-interruption keyword image corresponding to the pre-interruption keyword and a dialogue partner image representing the user's dialogue partner in the voice dialogue before the interruption. For example, the processor 10 acquires a pre-interruption keyword image from the keyword image database 18 based on the pre-interruption keyword, and acquires a dialogue partner image by referring to the dialogue partner information database 19. The processor 10 then generates an icon image by combining the pre-interruption keyword image and the dialogue partner image.

[0055] In step S105, the processor 10 generates a control command to display an icon image to the user. In step S106, the processor 10 outputs the control command generated in step S105 to the output device 6. Upon receiving the control command, the output device 6 displays the icon image to the user. In step S107, the processor 10 determines whether or not to resume the user's voice dialogue. If it is determined that the user's voice dialogue is to be resumed, the processor 10 proceeds to step S115. If it is determined that the user's voice dialogue is not to be resumed, the processor 10 proceeds to step S108.

[0056] In step S108, the processor 10 determines whether or not at least one of the topic subjects among the user's conversation content, in-vehicle content, and vehicle surrounding objects has been recognized. If it is determined that the topic subject has not been recognized, the processor 10 proceeds to step S109. If it is determined that the topic subject has been recognized, the processor 10 proceeds to step S111. In step S109, the processor 10 determines whether or not the user's driving load is greater than a load threshold. If it is determined that the user's driving load is greater than the load threshold, the processor 10 proceeds to step S110. In step S110, the processor 10 generates an icon image with a small amount of information. If it is determined that the user's driving load is not greater than the load threshold, the processor 10 returns to step S107 and repeats the subsequent flow.

[0057] In step S111, the processor 10 extracts keywords of the recognized topic target. In step S112, the processor 10 determines whether the importance of the keywords extracted in step S111 is greater than the importance threshold. If it is determined that the importance of the keywords of the topic target is greater than the importance threshold, the processor 10 proceeds to step S113. In step S113, the processor 10 generates an icon image including the topic target. For example, the processor 10 generates an icon image including a topic image representing the topic target and a dialogue partner image. If it is determined that the importance of the keywords of the topic target is not greater than the importance threshold, the processor 10 proceeds to step S109.

[0058] In step S114, the processor 10 updates the icon image displayed to the user to the generated icon image. For example, the processor 10 updates the displayed icon image to the icon image generated in step S110 or the icon image generated in step S113. The processor 10 performs display control to display the newly generated icon image to the user. After updating the icon image in step S114, the processor 10 returns to step S107 and repeats the subsequent flow. In step S115, the processor 10 resumes the user's voice interaction.

[0059] As described above, in this embodiment, when a voice dialogue is interrupted, the content of the dialogue of the user before the voice dialogue was interrupted is recognized, keywords in the voice dialogue before the interruption are extracted as pre-interruption keywords based on the dialogue content, an icon image representing the content of the dialogue before the interruption is generated based on the extracted pre-interruption keywords, and a control command is generated to display the generated icon image. This makes it possible to resume providing information to the interrupted user while the user remembers the information that was provided before the interruption.

[0060] In this embodiment, a pre-interruption keyword image corresponding to the pre-interruption keyword is obtained from a database in which keyword images corresponding to keywords in a voice dialogue are stored for each keyword, a dialogue partner image representing the dialogue partner with the user before the voice dialogue was interrupted is obtained, and an icon image including the pre-interruption keyword image and the dialogue partner image is generated. This makes it possible to generate an icon image that appropriately represents the content of the dialogue and the dialogue partner before the interruption.

[0061] In this embodiment, after displaying the icon image, the system recognizes at least one topic from among the user's conversation, the in-vehicle content being played in the vehicle, and the vehicle surrounding objects around the vehicle, determines whether the importance of the recognized topic is greater than the importance threshold, and updates the icon image displayed to the user based on the topic image representing the topic determined to be greater than the importance threshold, thereby enabling the user to resume the conversation with a topic that interests the user.

[0062] In addition, in this embodiment, driving situation information indicating the driving situation of the user driving the vehicle is acquired, an icon image is displayed, and then the driving load of the user is estimated based on the driving situation, and it is determined whether the driving load is greater than a load threshold. If it is determined that the driving load is greater than the load threshold, an icon image with an amount of information corresponding to the driving load is generated. This makes it possible to prevent the user's attention from being distracted by the icon display when the driving load on the user is high.

[0063] It should be noted that the above-described embodiments have been described to facilitate understanding of the present invention, and are not intended to limit the present invention. Therefore, each element disclosed in the above-described embodiments is intended to include all design modifications and equivalents that fall within the technical scope of the present invention.

[0064] 100... Voice dialogue system 1... Voice dialogue device 10... Processor 11... Recognition unit 12... Driving situation acquisition unit 13... Keyword extraction unit 14... Driving load estimation unit 15... Determination unit 16... Image generation unit 17... Display control unit 2... In-vehicle sound collection device 3... Vehicle sensors 4... User state detection device 5... Map database 6... Output device A... Agent

Claims

1. A speech dialogue device comprising a processor for controlling a speech dialogue of a user, The processor, a recognition unit that recognizes the content of the user's dialogue before the voice dialogue is interrupted; a keyword extraction unit that extracts keywords in the voice dialogue before the dialogue is interrupted as pre-interruption keywords based on the dialogue content; an image generating unit that generates an icon image representing the content of the conversation before the interruption based on the extracted pre-interruption keyword; a display control unit that generates a control command to display the generated icon image, The image generation unit is a voice dialogue device that updates the displayed icon image based on a topic that may be of interest to the user and / or the driving load of the user who drives a vehicle while the voice dialogue is interrupted after displaying the icon image.

2. A speech dialogue device comprising a processor for controlling a speech dialogue of a user, The processor, a recognition unit that recognizes the content of the user's dialogue before the voice dialogue is interrupted; a keyword extraction unit that extracts keywords in the voice dialogue before the dialogue is interrupted as pre-interruption keywords based on the dialogue content; an image generating unit that generates an icon image representing the content of the conversation before the interruption based on the extracted pre-interruption keyword; a display control unit that generates a control command to display the generated icon image, The image generating unit includes: acquiring a pre-interruption keyword image corresponding to the pre-interruption keyword from a database in which keyword images corresponding to the keywords in the voice dialogue are stored for each of the keywords; acquiring a dialogue partner image representing a dialogue partner who was dialogued with the user before the voice dialogue was interrupted; The voice dialogue device generates the icon image including the pre-interruption keyword image and the dialogue partner image.

3. A speech dialogue device comprising a processor for controlling a speech dialogue of a user, The processor, a recognition unit that recognizes the content of the user's dialogue before the voice dialogue is interrupted; a keyword extraction unit that extracts keywords in the voice dialogue before the dialogue is interrupted as pre-interruption keywords based on the dialogue content; an image generating unit that generates an icon image representing the content of the conversation before the interruption based on the extracted pre-interruption keyword; a display control unit that generates a control command to display the generated icon image while the voice dialogue is interrupted.

4. The recognition unit is After displaying the icon image, the display device recognizes at least one of the following topics: a conversation between the user, in-vehicle content being played in the vehicle, and a vehicle surrounding object around the vehicle; The image generating unit includes: determining whether the importance of the recognized topic object is greater than an importance threshold; The voice dialogue device according to any one of claims 1 to 3, wherein the icon image displayed to the user is updated based on a topic image representing the topic subject whose importance is determined to be greater than the importance threshold.

5. The processor, A driving situation acquisition unit that acquires driving situation information indicating a driving situation of the user who drives a vehicle; A driving load estimation unit that estimates a driving load of the user based on the driving situation, The image generating unit includes: determining whether the operating load is greater than a load threshold; 5. The voice dialogue device according to claim 1, wherein, when it is determined that the driving load is greater than the load threshold, the icon image is generated with an amount of information corresponding to the driving load.

6. 1. A speech interaction method for controlling a speech interaction of a user, executed by a processor, comprising: The processor, Recognizing the content of the user's dialogue before interrupting the voice dialogue; extracting keywords in the voice dialogue before the interruption as pre-interruption keywords based on the dialogue content; generating an icon image representing the content of the dialogue before the interruption based on the extracted pre-interruption keyword; generating a control command for displaying the generated icon image; A voice dialogue method for updating the displayed icon image based on a topic that may be of interest to the user and / or the driving load of the user who drives a vehicle while the voice dialogue is interrupted after the icon image is displayed.

7. 1. A speech interaction method for controlling a speech interaction of a user, executed by a processor, comprising: The processor, Recognizing the content of the user's dialogue before interrupting the voice dialogue; extracting keywords in the voice dialogue before the interruption as pre-interruption keywords based on the dialogue content; generating an icon image representing the content of the dialogue before the interruption based on the extracted pre-interruption keyword; The voice dialogue method further comprises: generating a control command for displaying the generated icon image while the voice dialogue is interrupted.