Voice dialogue device and voice dialogue method

Abstract

A processor (10) recognizes interaction details of a user before interruption of speech interaction, extracts, on the basis of the interaction details, a keyword in the speech interaction before interruption as a pre-interruption keyword, generates, on the basis of the extracted pre-interruption keyword, an icon image indicating the interaction details before interruption, and generates a control instruction for displaying the generated icon image.

Description

Technical Field

[0001] The present invention relates to a voice dialogue device and a voice dialogue method. This application claims priority based on Japanese Patent Application No. 2022-73531 filed on April 27, 2022. For designated countries where incorporation by reference is permitted, the contents described in the above application are incorporated herein by reference and made part of the description of this application.

Background Art

[0002] When display of notification information to a driver is performed and the driving load on the driver is higher than a predetermined driving load, based on a predetermined operation of the driver, importance is added to the notification information, display of the notification information is temporarily suppressed, and when redisplayed, the notification information is displayed according to the importance. Such a technique is known (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technique of Patent Document 1, even if the notification information displayed before suppression is redisplayed after suppressing the display of the notification information, there is a problem that when the notification information is redisplayed, the user may forget the content of the notification information displayed before suppression.

[0005] The problem to be solved by the present invention is to provide a voice dialogue device and a voice dialogue method that enable a user to recall the content of output information before interruption when resuming information provision to the interrupted user.

Means for Solving the Problems

[0006] The present invention solves the above problem by recognizing the content of the user's dialogue before the voice dialogue was interrupted, extracting keywords from the voice dialogue before the interruption as pre-interruption keywords based on the dialogue content, generating an icon image representing the dialogue content before the interruption based on the extracted pre-interruption keywords, and generating a control command to display the generated icon image. [Effects of the Invention]

[0007] According to the present invention, when information provision to a user that has been interrupted is resumed, the user can recall the content of the output information before the interruption. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a block diagram showing an embodiment of the voice dialogue system according to the present invention. [Figure 2] Figures 2(a) and 2(b) show examples of screens displayed to the user in this embodiment. [Figure 3] Figures 3(a), (b), and (c) are examples of icon images representing the amount of information that changes according to the user's operating load in this embodiment. [Figure 4] Figures 4(a), (b), and (c) are examples of icon images that change according to the topic recognized during an interruption of voice dialogue in this embodiment. [Figure 5] Figure 5 shows an example of a flowchart illustrating the processing involved in a voice dialogue method performed by a voice dialogue device. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. The voice dialogue system including the voice dialogue device according to the present invention is a system that engages in voice dialogue with a user using voice recognition technology and voice synthesis technology. A user is a person who uses the voice dialogue system, and is mainly the driver of a vehicle, but the user may be a passenger other than the driver. Furthermore, the voice dialogue system 100 may be a system used in a location other than an automobile. In the following description, the present invention will be described in an example in which the driver of a vehicle is the user and the present invention is applied to a voice dialogue system that engages in voice dialogue with the user, but the present invention is not limited to this and may also be applied to a voice dialogue system that controls voice dialogue between a user and other users.

[0010] Figure 1 is a block diagram showing an embodiment of the voice dialogue system according to this embodiment. The voice dialogue system 100 comprises a voice dialogue device 1, an in-vehicle sound collection device 2, vehicle sensors 3, a user state detection device 4, and a map database 5. These devices and the database are mounted in the vehicle and are connected, for example, by CAN or other in-vehicle LAN, and can send and receive information from each other. The voice dialogue device 1 is a device that controls voice dialogue with the user and comprises a processor 10 and an output device 6.

[0011] Output device 6 is a device that outputs information to the user via an agent function performed by an anthropomorphic agent (hereinafter also simply referred to as Agent A), specifically through a medium of voice and images, or a combination thereof. Here, output device 6 is described as an electronic device mounted in a vehicle, but any electronic device equipped with agent functionality is acceptable, for example, a portable electronic device with a display. Furthermore, the functions related to the voice output and video output of Agent A may be installed in a mobile phone such as a smartphone. In addition, output device 6 only needs to be able to output voice, images, text information, etc., to present information to the user, and may, for example, be used in conjunction with the display of a navigation system.

[0012] Figures 2(a) and 2(b) show examples of screens displayed to the user in the voice dialogue system according to this embodiment. For example, the screen shown in Figure 2 is displayed on a head-up display device or display device mounted in the vehicle. Figure 2(a) illustrates an example of visually displaying Agent A. As shown in Figure 2(a), Agent A is displayed on the display 101 such that the agent image 102, which is an avatar modeled after a human, is superimposed on the scenery outside the vehicle. As described above, when Agent A outputs communication information to the user through the agent function that receives a control command from the display control unit 17, it displays the agent image 102. The user can engage in voice dialogue while understanding the gestures and movements of Agent A, who is their conversation partner, through the screen displayed on the display 101.

[0013] Furthermore, the voice dialogue device 1 is equipped with a dialogue interruption function that interrupts voice dialogue with the user. If the voice dialogue device 1 determines that it is necessary to interrupt the voice dialogue during a voice dialogue with the user, it will interrupt the voice dialogue. While the voice dialogue is interrupted, as shown in Figure 2(b), the image of Agent A, the user's voice dialogue partner, is not displayed. In this embodiment, as shown in Figure 2(b), while the voice dialogue is interrupted, an icon image 112 representing the dialogue content before the voice dialogue was interrupted is displayed on the display 111 so as to be superimposed on the exterior scenery. The content of the icon image may include illustrations, text, photographs, videos representing the dialogue content before the voice dialogue was interrupted, or avatar expressions that correspond to the atmosphere of the voice dialogue. Examples of dialogue content that can be displayed on the icon include topics such as cafes, restaurants, shopping, events, tourist spots, music, news, weather, and scenery.

[0014] Furthermore, in this embodiment, the user's conversation partner is described as a voice dialogue system, but it is not limited to this, and the conversation partner may be a human being. For example, the conversation partner may be someone other than the occupants of the vehicle who is outside the vehicle. In the example in Figure 2(a), the display of Agent A, who engages in voice dialogue with the user, is shown as an example of the conversation partner, but if the conversation partner is a human being, it may be an image of the conversation partner. Also, if the voice dialogue device 1 determines that it is time to resume voice dialogue while it has been interrupted, it will resume voice dialogue. When voice dialogue resumes, as shown in Figure 2(a), the agent image 102 of Agent A, the user's conversation partner, is displayed again, and the display of the aforementioned icon image is interrupted. Note that it is not limited to interrupting the display of the icon image, but the icon image may be displayed continuously after the voice dialogue resumes.

[0015] As described above, Agent A includes a speaker or other audio output unit for outputting voice and sound effects, and a display unit or other display unit for displaying images including text. Along with Agent A's operation, it outputs communication information by providing voice, sound effects, text, and other images to the user. Furthermore, the anthropomorphic agent is merely an example; the agent does not have to mimic a human, but may display a predetermined character, avatar, or icon. Agent A may also be installed as a physical entity, capable of appearing and disappearing by operating actuators. Moreover, while the illustration illustrates the display of an agent as a conversation partner, this system can also be applied to voice dialogue systems where Agent A is not displayed as a conversation partner.

[0016] The in-vehicle sound recording device 2 records sounds heard by the occupants inside the vehicle. The in-vehicle sound recording device 2 is, for example, a microphone installed inside the vehicle. The sounds recorded by the in-vehicle sound recording device 2 are mainly sounds with sound sources inside the vehicle's cabin, and include user voices and sounds output from in-vehicle media. User voices include, for example, conversations between occupants and conversations between occupants and voice dialogue systems. In-vehicle media are, for example, audio systems and radios.

[0017] The vehicle sensors 3 acquire vehicle surrounding information around the vehicle. The vehicle sensors 3 are, for example, an external camera that images the outside of the vehicle. The external camera is a CCD camera, a CMOS camera, etc. having an image pickup device such as a CCD or a CMOS. The external camera acquires an image of the vehicle surrounding as vehicle surrounding information. Also, the vehicle sensors 3 include GPS. GPS detects the current position of the host vehicle. The detection results of the vehicle sensors 3 are output to the processor 10 at a predetermined time interval.

[0018] The user state detection device 4 detects information regarding the state of the user. The state of the user includes, for example, information regarding the physical movement of the user. The user state detection device 4 includes, for example, an in-vehicle camera that images the user inside the vehicle, a biological signal measurement sensor that detects the biological signal of the user, and the like. The in-vehicle camera is a CCD camera, a CMOS camera, etc. having an image pickup device such as a CCD or a CMOS. The biological signal measurement sensor is provided inside the steering wheel or the seating seat, and detects biological signals such as the blood pressure of the user, the change in the pulse rate, and the degree of sweating as information regarding the user's body. The detection results of the user state detection device 4 are output to the processor 10 at a predetermined time interval.

[0019] The map database 5 is a database that stores map information. The map information includes road information and the like. Also, the map database 5 includes information on POI (Point Of Interest). POI is information on a specific location on the map, for example, the location of a building that is a landmark such as a public facility, a restaurant, or a tourist attraction.

[0020] The voice dialogue device 1 includes a ROM storing programming for executing various processes, a CPU as an operation circuit that functions as the voice dialogue device 1 by executing the programs stored in this ROM, and a RAM functioning as an accessible storage device, and is composed of a processor 10 and an agent A as an output device 6. The processor 10 includes, as functional blocks, a recognition unit 11, a driving situation acquisition unit 12, a keyword extraction unit 13, a driving load estimation unit 14, a determination unit 15, an image generation unit 16, and a display control unit 17. The processor 10 includes a keyword image database 18 and a dialogue partner information database 19.

[0021] The recognition unit 11 recognizes the dialogue content of the user before interrupting the user's voice dialogue. The recognition unit 11 acquires the voice of the user and / or the voice of the user's dialogue partner before interrupting the voice dialogue from the in-vehicle microphone device 2. The recognition unit 11 executes voice recognition processing on the acquired voice to recognize the dialogue content of the user. Further, the recognition unit 11 may summarize the dialogue content of the user. The dialogue content of the user includes the speech content of the user before interruption and the speech content of the user's dialogue partner before interruption.

[0022] Further, after displaying the icon image, the recognition unit 11 recognizes at least one topic target among the conversation content of the user, the in-vehicle content being played in the vehicle, and the vehicle surrounding targets around the vehicle. The topic target is a candidate for a topic to be provided to the user when resuming the voice dialogue with the user, and is, for example, a target that the user may be interested in. Even while the voice dialogue with the user is interrupted, the user may be interested in a target that can be visually or aurally recognized. Since the user's interest may shift to other targets from the dialogue content before interruption, in this embodiment, a topic target that the user may be interested in while interrupted is recognized.

[0023] For example, the recognition unit 11 recognizes the content of the user's conversation. The recognition unit 11 acquires the user's voice from the in-vehicle sound recording device 2 while the voice dialogue is interrupted. The recognition unit 11 performs speech recognition processing on the acquired user's voice to recognize the content of the user's conversation. The recognition unit 11 may also summarize the content of the user's conversation. The content of the user's conversation includes the content of the conversation between the user and other passengers during the interruption of the voice dialogue with the user.

[0024] The recognition unit 11 recognizes in-vehicle content being played inside the vehicle. This content may include, for example, music played from in-vehicle audio systems or radios, or the content of a radio personality's statements. The recognition unit 11 acquires audio from the in-vehicle sound recording device 2. The recognition unit 11 then performs speech recognition processing on the acquired audio to recognize the in-vehicle content. The recognition unit 11 may also summarize the content of the in-vehicle content, such as the radio program content or the content of a radio personality's statements.

[0025] The recognition unit 11 recognizes objects around the vehicle based on information detected by vehicle sensors 3, such as GPS and external cameras. These objects around the vehicle are, for example, Points of Interest (POIs) around the vehicle. For example, the recognition unit 11 recognizes POIs located around the vehicle as objects around the vehicle based on the vehicle's current position and map information from the map database 5. The recognition unit 11 also performs image recognition processing on images of the vehicle's surroundings captured by external cameras, extracts feature points of the identified POIs, and recognizes these objects around the vehicle.

[0026] The driving status acquisition unit 12 acquires driving status information from the user status detection device 4 that indicates the driving status of the user operating the vehicle. For example, the user's driving status information includes information about the user's physical condition and information about the user's actions. Information about the user's physical condition includes the user's blood pressure, changes in pulse rate, and degree of sweating. Information about the user's actions includes the user's operations, facial expressions, gaze, gestures, etc. The driving status acquisition unit 12, for example, performs image recognition processing on the user's image acquired from the user status detection device 4 to acquire information about the user's actions.

[0027] The keyword extraction unit 13 extracts pre-interruption keywords from the voice dialogue before the interruption, based on the dialogue content recognized by the recognition unit 11 before the interruption. For example, the keyword extraction unit 13 extracts keywords from the voice dialogue before the interruption as pre-interruption keywords based on a summary of the dialogue content. The keyword extraction unit 13 extracts nouns included in the pre-interruption dialogue content as keyword candidates and extracts the keyword candidate with the highest importance as the pre-interruption keyword.

[0028] Furthermore, the keyword extraction unit 13 extracts keywords related to the topic based on the user's conversation content and / or in-vehicle content recognized by the recognition unit 11 while the voice dialogue is interrupted. For example, the keyword extraction unit 13 extracts keywords from the conversation content and in-vehicle content as keywords based on a summary of the conversation content and in-vehicle content. In addition, the keyword extraction unit 13 extracts keywords based on objects around the vehicle recognized by the recognition unit 11 while the voice dialogue is interrupted. For example, the keyword extraction unit 13 extracts the name of an object around the vehicle as a keyword from the information about the object around the vehicle.

[0029] The driving load estimation unit 14 estimates the user's driving load based on the user's driving conditions. Driving load is the burden the user bears when driving the vehicle. The greater the driving load, the higher the concentration required of the user as the driver. The driving load estimation unit 14 estimates the driving load based on the user's driving operations. For example, the driving load estimation unit 14 estimates a greater driving load the greater the steering input, or the input of the accelerator pedal or brake pedal.

[0030] Furthermore, the driving load estimation unit 14 estimates the driving load based on the user's physical condition. For example, the driving load estimation unit 14 estimates a higher driving load the higher the user's blood pressure or heart rate. Alternatively, the driving load estimation unit 14 may estimate the driving load based on the user's posture. The driving load estimation unit 14 estimates the user's eye movements, and estimates a higher driving load the greater the user's eye movements.

[0031] Furthermore, the driving load estimation unit 14 estimates the driving load based on the driving conditions. The driving load estimation unit 14 estimates a higher driving load when the vehicle is in certain driving conditions than when it is in other conditions. For example, the driving load estimation unit 14 estimates a higher driving load when the vehicle is moving than when the vehicle is stopped. Specific conditions include when the vehicle is traveling on a highway, when the vehicle is turning right or left, and when there are obstacles such as other vehicles close to the vehicle.

[0032] The determination unit 15 determines whether or not to interrupt the user's voice interaction. The determination unit 15 obtains information about the user's state from the user state detection device 4 and determines to interrupt the user's voice interaction if the user's state is such that it is necessary to interrupt the voice interaction. The determination unit 15 also determines not to interrupt the user's voice interaction if the user's state is not such that it is necessary to interrupt the voice interaction. A state in which it is necessary to interrupt the voice interaction is, for example, when the user's driving load increases. The determination unit 15 determines to interrupt the user's voice interaction if the user is operating the steering wheel. The determination unit 15 also determines to interrupt the user's voice interaction if the user's blood pressure or heart rate is high.

[0033] In addition to the above, situations in which it is necessary to interrupt the user's voice dialogue include when the driving responsibility switches from the system to the driver in autonomous driving, when the user receives another voice call or video call, and when the user performs a conversation interruption operation. Based on the image of the user, the determination unit 15 determines that if the user is performing a driving operation, that is, when the driving responsibility switches from the system to the user in autonomous driving, the user's state is such that it is necessary to interrupt the user's voice dialogue.

[0034] Furthermore, the determination unit 15 determines whether or not to resume the interrupted user voice dialogue. For example, while the user's voice dialogue is interrupted, the determination unit 15 acquires information about the user's state at regular intervals and, based on the information about the user's state, determines whether or not the user's state is such that the user's voice dialogue can be resumed. A state in which the user's voice dialogue can be resumed is, for example, when the user's driving load has decreased, or when the driving responsibility has switched from the user to the system in autonomous driving.

[0035] The image generation unit 16 generates icon images that represent the topic of the user's voice dialogue. The icon images include, for example, keyword images corresponding to keywords in the voice dialogue and dialogue partner images representing the user's dialogue partner. If the user's voice dialogue is interrupted, the image generation unit 16 generates icon images that represent the dialogue content before the interruption, based on the keywords before the interruption. Based on the keywords before the interruption, the image generation unit 16 refers to the keyword image database 18 and obtains the keyword image image before the interruption that corresponds to the keywords before the interruption. For example, if the image generation unit 16 extracts keywords related to cafes, it obtains an image representing a cafe, such as an image of a coffee cup.

[0036] Furthermore, the image generation unit 16 refers to the conversation partner information database 19 and obtains a conversation partner image representing the conversation partner with whom the user was talking before the voice conversation was interrupted, including avatar model data and character name. The image generation unit 16 then combines the obtained keyword image and conversation partner image to generate an icon image that includes both the keyword image and the conversation partner image. As a result, for example, if the user's voice conversation is interrupted, the image generation unit 16 generates an icon image representing the content of the voice conversation before the interruption, based on the keyword image and conversation partner image from before the interruption.

[0037] Furthermore, if the content of the user's voice dialogue concerns objects around the vehicle, the image generation unit 16 may acquire images of objects around the vehicle captured by the vehicle sensors 3 and generate an icon image that includes the images of the objects around the vehicle. Alternatively, the image generation unit 16 may generate an icon image that includes a keyword image but does not include the image of the person the dialogue partner is talking to.

[0038] Furthermore, in this embodiment, icon images may be generated according to the operating load estimated by the operating load estimation unit 14. For example, the image generation unit 16 generates an icon image with an amount of information corresponding to the operating load. The amount of information in the icon image is set to a smaller value as the operating load increases. The image generation unit 16 determines whether the operating load is greater than the load threshold while the user's voice interaction is interrupted. If it is determined that the operating load is greater than the load threshold, the image generation unit 16 generates an icon image with less information than when it is determined that the operating load is not greater than the load threshold. This makes it possible to reduce the amount of information displayed to the user when the operating load is high.

[0039] In this embodiment, when voice interaction is interrupted, an icon image representing the content of the interaction before the interruption may be generated with an amount of information corresponding to the operating load. Alternatively, if the operating load changes after the icon image has been displayed, the icon image displayed to the user may be updated to an icon image with an amount of information corresponding to the operating load.

[0040] Examples of icon images that change according to the driving load are explained using Figures 3(a) to 3(c). Figures 3(a) to 3(c) are examples of icon images that change in amount of information according to the driving load when the user is the driver. Figure 3(a) shows icon image 301 when the user's driving load is low. Icon image 301 shown in Figure 3(a) is an example of an icon image that displays a large amount of information to the user, and includes a keyword image 302 that represents the content of the conversation before the interruption and a dialogue partner image 303 that represents the avatar that is the other party to the user's voice conversation. Keyword image 302 is an example of a keyword image corresponding to the keyword "cafe". By displaying more information, the user can smoothly resume the voice conversation.

[0041] On the other hand, if the user's workload increases while the voice interaction is interrupted, an icon image with less information displayed to the user will be shown, as shown in Figure 3(b). In Figure 3(b), icon image 311 is an example of an icon image with less information displayed to the user, where the image of the conversation partner is not displayed, and a keyword image 312 representing the content of the conversation before the interruption is displayed. Furthermore, if the workload increases even more, an image with even less information displayed to the user will be shown, as shown in Figure 3(c). In Figure 3(c), icon image 321 includes only image 322, which indicates that a voice interaction was taking place. Image 322 can be any image that indicates a voice interaction was taking place, for example, it could be the icon of the application that was conducting the voice interaction. This prevents the user's attention from being distracted by the display of icon images.

[0042] Furthermore, in this embodiment, while the user's voice interaction is interrupted, the icon image displayed to the user may be updated based on at least one topic from among the user's conversation content, in-vehicle content, and objects around the vehicle. For example, after displaying the icon image, the image generation unit 16 determines whether or not the recognition unit 11 has recognized the topic. If it is determined that the topic has been recognized, the image generation unit 16 determines whether or not the importance of the recognized topic is greater than the importance threshold. The importance is set in advance for each topic, for example. The importance is set to a higher value the more likely it is to become a topic candidate in the user's voice interaction.

[0043] If the image generation unit 16 determines that the importance of the topic is greater than the importance threshold, it generates a new icon image based on the topic image representing the topic whose importance is determined to be greater than the importance threshold, and updates the icon image displayed to the user with the generated icon image. The image generation unit 16 acquires a topic image representing the topic. For example, the image generation unit 16 acquires a keyword image corresponding to a keyword in the topic from the keyword image database 18 as the topic image. Alternatively, the image generation unit 16 may acquire an image of the area around the vehicle as the topic image. The image generation unit 16 generates an icon image that includes the topic image. Alternatively, the image generation unit 16 may generate an icon image that includes both the topic image and the conversation partner image.

[0044] While the user's voice interaction is interrupted, the image generation unit 16 determines each time the topic is recognized by the recognition unit 11 whether the importance of the topic is greater than the importance threshold, until the user's voice interaction is resumed. If the importance is greater than the importance threshold, the image generation unit 16 generates an icon image that includes a topic image representing the topic.

[0045] An example of an icon image that changes depending on the topic recognized while the user's voice dialogue is interrupted is explained using Figures 4(a) to 4(c). Figures 4(a) to 4(c) are examples of icon images that are updated according to the topic recognized while the user's voice dialogue is interrupted. Figure 4(a) is an example of the icon image displayed to the user immediately after the user's voice dialogue is interrupted. Icon image 401 includes the pre-interruption keyword image 402 and the dialogue partner image 403.

[0046] On the other hand, if the user's voice interaction is interrupted for a long period of time, the user's interests may change. Therefore, in this embodiment, the icon image displayed to the user is updated based on the user's conversation content, in-car content, and information about objects around the vehicle. Figure 4(b) shows an icon image updated based on in-car content. Icon image 411 includes a keyword image 412 of the in-car content and an image 414 of the conversation partner. The in-car content is music playing from the audio system. For example, icon image 411 shows the avatar, represented by the conversation partner image 414, showing interest in the music playing from the audio system.

[0047] Furthermore, Figure 4(c) shows an updated icon image after the icon image in Figure 4(b) has been displayed. In Figure 4(c), icon image 421 includes a keyword image 422 indicating an object around the vehicle and an image of the conversation partner 424. For example, icon image 421 shows that the avatar shown in the conversation partner image 424 is showing interest in an object around the vehicle. Note that, as shown in the pre-interruption keyword image 413 in Figure 4(b) and the pre-interruption keyword image 423 in Figure 4(c), the pre-interruption keyword image may also be displayed in a reduced size within the icon image after it has been updated.

[0048] The display control unit 17 controls the display of the icon image generated by the image generation unit 16 to the user. The display control unit 17 generates a control command to display the icon image to the user and transmits the generated control command to the output device 6. Upon receiving the control command, the output device 6 outputs output data including the icon image as communication information using the agent function of agent A. In this embodiment, the output device 6 is not limited to displaying the icon image; a communication device located outside the voice dialogue device 1 may also display the icon image. The communication device is, for example, a smartphone or tablet terminal used by the user. In this case, the display control unit 17 outputs the control command to the communication device.

[0049] As described above, when the user's voice interaction is interrupted, the display control unit 17 displays an icon image representing the content of the conversation before the interruption to the user during the interruption. This allows the user to recall the content of the conversation before the interruption by looking at the icon image, even while the voice interaction is interrupted, and to smoothly resume the voice interaction when it is restarted.

[0050] The keyword image database 18 is a database that stores keyword images corresponding to keywords in voice dialogue, for each keyword. In this embodiment, data of keyword images corresponding to each keyword is stored in advance. For example, for the keyword "cafe," an image representing a cafe, such as an image of a coffee cup, is stored. Also, for example, the keyword image database 18 stores images representing a specific landmark for keywords related to that landmark.

[0051] The conversation partner information database 19 is a database that stores information about the user's conversation partners for each conversation partner. The conversation partner information is, for example, a conversation partner image that represents the conversation partner. The conversation partner information database 19 stores, for example, an image representing Agent A's avatar and an image taken of the user's conversation partner.

[0052] Next, using Figure 5, an example of a control procedure for executing the voice dialogue method according to this embodiment will be described. Figure 5 is an example of a flowchart showing the control procedure related to the voice dialogue method. While the user is engaging in voice dialogue, the processor 10 starts the flow from step S101.

[0053] In step S101, the processor 10 determines whether or not to interrupt the user's voice interaction. If it decides to interrupt the voice interaction, the processor 10 proceeds to step S102. If it decides not to interrupt the voice interaction, the processor 10 returns to step S101 and repeats the following flow. In step S102, the processor 10 recognizes the content of the conversation before the interruption. In step S103, the processor 10 extracts pre-interruption keywords based on the content of the conversation before the interruption.

[0054] In step S104, the processor 10 generates an icon image that includes a pre-interruption keyword image corresponding to the pre-interruption keyword and an image of the conversation partner representing the user's conversation partner in the pre-interruption voice dialogue. For example, based on the pre-interruption keyword, the processor 10 retrieves the pre-interruption keyword image from the keyword image database 18 and retrieves the conversation partner image by referring to the conversation partner information database 19. Then, the processor 10 combines the pre-interruption keyword image and the conversation partner image to generate an icon image.

[0055] In step S105, the processor 10 generates a control command to display an icon image to the user. In step S106, the processor 10 outputs the control command generated in step S105 to the output device 6. Upon receiving the control command, the output device 6 displays the icon image to the user. In step S107, the processor 10 determines whether or not to resume the user's voice interaction. If it determines to resume the user's voice interaction, the processor 10 proceeds to step S115. If it determines not to resume the user's voice interaction, the processor 10 proceeds to step S108.

[0056] In step S108, the processor 10 determines whether it has recognized at least one topic from the user's conversation content, in-car content, and objects around the vehicle. If it determines that it has not recognized a topic, the processor 10 proceeds to step S109. If it determines that it has recognized a topic, the processor 10 proceeds to step S111. In step S109, the processor 10 determines whether the user's driving load is greater than the load threshold. If it determines that the user's driving load is greater than the load threshold, the processor 10 proceeds to step S110. In step S110, the processor 10 generates an icon image with a small amount of information. If it determines that the user's driving load is not greater than the load threshold, the processor 10 returns to step S107 and repeats the following flow.

[0057] In step S111, the processor 10 extracts the keywords of the recognized topic. In step S112, the processor 10 determines whether the importance of the keywords extracted in step S111 is greater than the importance threshold. If the processor 10 determines that the importance of the topic keywords is greater than the importance threshold, the processor 10 proceeds to step S113. In step S113, the processor 10 generates an icon image that includes the topic. For example, the processor 10 generates an icon image that includes a topic image representing the topic and an image of the conversation partner. If the processor 10 determines that the importance of the topic keywords is not greater than the importance threshold, the processor 10 proceeds to step S109.

[0058] In step S114, the processor 10 updates the icon image displayed to the user with the generated icon image. For example, the processor 10 updates the displayed icon image with the icon image generated in step S110 or the icon image generated in step S113. The processor 10 performs display control to display the newly generated icon image to the user. After updating the icon image in step S114, the processor 10 returns to step S107 and repeats the following flow. In step S115, the processor 10 resumes the user's voice interaction.

[0059] As described above, in this embodiment, when a voice interaction is interrupted, the system recognizes the user's dialogue content before the interruption, extracts keywords from the voice interaction before the interruption as pre-interruption keywords based on the dialogue content, generates an icon image representing the dialogue content before the interruption based on the extracted pre-interruption keywords, and generates a control command to display the generated icon image. This allows the system to resume providing information to the interrupted user while the user remembers the information provided before the interruption.

[0060] Furthermore, in this embodiment, a keyword image corresponding to a keyword in a voice dialogue is stored for each keyword. The pre-interruption keyword image corresponding to the keyword before the interruption is retrieved from the database, and an image representing the person the user was talking to before the voice dialogue was interrupted is retrieved. An icon image is then generated that includes the pre-interruption keyword image and the image of the person the user was talking to. This makes it possible to generate an icon image that appropriately represents the content of the dialogue and the person the user was talking to before the interruption.

[0061] Furthermore, in this embodiment, after displaying an icon image, the system recognizes at least one topic from among the user's conversation content, in-vehicle content being played in the vehicle, and objects in the vehicle's surroundings. It then determines whether the importance of the recognized topic is greater than an importance threshold, and updates the icon image displayed to the user based on the topic image representing the topic whose importance is determined to be greater than the importance threshold. This allows the user to resume the conversation with a topic of interest when they restart the conversation.

[0062] Furthermore, in this embodiment, driving status information indicating the driving status of the user operating the vehicle is acquired, an icon image is displayed, and then the user's driving load is estimated based on the driving status. It is determined whether the driving load is greater than the load threshold, and if it is determined that the driving load is greater than the load threshold, an icon image with an amount of information corresponding to the driving load is generated. This prevents the user's attention from being distracted by the icon display when the driving load on the user is high.

[0063] The embodiments described above are provided to facilitate understanding of the present invention and are not intended to limit it. Therefore, each element disclosed in the above embodiments is intended to include all design modifications and equivalents that fall within the technical scope of the present invention. [Explanation of Symbols]

[0064] 100…Voice dialogue system 1…Voice dialogue device 10… Processor 11...Recognition part 12... Driving status acquisition unit 13…Keyword extraction section 14…Operating load estimation unit 15…Judgment section 16…Image generation unit 17…Display Control Unit 2…In-car sound recording device 3. Vehicle sensors 4. User status detection device 5…Map Database 6…Output device A... Agent

Claims

1. A voice dialogue device comprising a processor that controls the output of information in response to the user's voice dialogue, The aforementioned processor, A recognition unit that recognizes the content of the user's dialogue before the voice dialogue was interrupted, A keyword extraction unit extracts keywords from the voice dialogue before the interruption, based on the content of the dialogue, An image generation unit generates an icon image representing the content of the dialogue before the interruption based on the extracted pre-interruption keywords, The system comprises a display control unit that generates a control command to display the generated icon image, After displaying the icon image, the recognition unit, while interrupting the voice dialogue, recognizes at least one of the user's conversation content and the in-vehicle content being played in the vehicle as a topic that the user may be interested in. The image generation unit is a voice dialogue device that, after displaying the icon image, updates the displayed icon image based on the topic of conversation while the voice dialogue is interrupted.

2. The image generation unit is: Determine whether the importance of the recognized topic is greater than the importance threshold. The voice dialogue device according to claim 1, which updates the icon image displayed to the user based on a topic image representing the topic subject whose importance is determined to be greater than the importance threshold.

3. The aforementioned processor, A driving status acquisition unit acquires driving status information indicating the driving status of the user who is driving the vehicle, The system further includes an operating load estimation unit that estimates the user's operating load based on the aforementioned operating conditions, The image generation unit, Determine whether the operating load is greater than the load threshold. The voice dialogue device according to claim 1 or 2, which generates an icon image with an amount of information corresponding to the operating load when it is determined that the operating load is greater than the load threshold.

4. A voice interaction method, executed by a processor, that controls the output of information in response to a user's voice interaction, The aforementioned processor, The content of the user's dialogue before the voice dialogue was interrupted is recognized. Based on the content of the aforementioned dialogue, keywords from the voice dialogue before the interruption are extracted as pre-interruption keywords. Based on the extracted pre-interruption keywords, an icon image representing the content of the dialogue before the interruption is generated. A control command is generated to display the generated icon image, After displaying the aforementioned icon image, while the voice dialogue is interrupted, the system recognizes at least one of the user's conversation content and the in-vehicle content being played in the vehicle as a topic that the user may be interested in. A voice dialogue method that updates the displayed icon image based on the topic of conversation while the voice dialogue is interrupted after the icon image has been displayed.

Images