Information provision method and information provision device

Abstract

To provide an information providing method and an information providing device which generate a new topic of conversation output into a vehicle using acquired information.SOLUTION: In a dialogue system, in an information notification device 100, an occupant identification part 011 identifies an occupant of a vehicle. A getting-on / off determination part 012 determines whether the occupant gets on / off the vehicle. A topic relation determination part 015 determines whether or not the topic during conversation in the vehicle is a topic related to a third occupant having a getting-on schedule. When it is determined by the topic relation determination part 015 that the topic is the topic related to the third occupant, a second topic generation part 018 generates a new topic of conversation output into the vehicle. The second topic generation part 018 generates a new topic of conversation using information acquired by an information acquisition part 019 used for generating the topic of the conversation output into the vehicle.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information providing method and an information providing device. [Background technology]

[0002] Patent Document 1 describes a dialogue system to be installed in a vehicle or the like. This dialogue system classifies and stores topic information into at least a first category, which is permitted to be used when a passenger is present, and a second category, which is not permitted to be used due to, for example, privacy or confidentiality concerns. When it is determined that a passenger is present, the topic information in the first category is used to determine the content of the utterance, and topics related to privacy or confidentiality are suppressed.

[0003] The trigger for speaking the determined content can be, for example, when information such as the vehicle is stopped / driving, engine condition, vehicle position, input to the car navigation system, route information, time, weather, schedule, etc. is obtained from inside or outside the vehicle and updated. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-101805 Summary of the Invention [Problem to be solved by the invention]

[0005] The dialogue system of Patent Document 1 does not take into consideration the need to make the topics discussed in the vehicle considerate of the passengers in the vehicle.

[0006] The present invention has been made in view of the above circumstances, and an object of the present invention is to make the topics of conversation in a vehicle take into consideration the interests of passengers in the vehicle. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, one aspect of the present invention is to identify vehicle occupants and determine whether or not an occupant is riding in the vehicle. If it is determined that a topic of conversation taking place inside the vehicle is related to a third occupant who is scheduled to ride in the vehicle, a new topic of conversation to be output inside the vehicle is generated using information acquired for use in generating a topic of conversation to be output inside the vehicle. [Effects of the Invention]

[0008] According to the present invention, the topics of conversation inside the vehicle can be tailored to suit the needs of the passengers in the vehicle. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram of a dialogue system including an information providing device according to one embodiment of the present invention. [Figure 2A] FIG. 2A is a flowchart showing an example of a procedure of information providing processing by each unit of the information providing device of FIG. [Figure 2B] FIG. 2B is a flowchart showing an example of a specific procedure of the information generation process of FIG. 2A. [Figure 3] FIG. 3 is an explanatory diagram showing an example of an output form of audio information by the audio output unit of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] The configuration of the dialogue system will be described with reference to Fig. 1. The dialogue system according to this embodiment includes a voice input unit 001, a voice recognition unit 002, a vehicle interior video input unit 003, an occupant detection unit 004, a vehicle exterior video input unit 005, a person detection unit 006, an information provision device 100, and a voice output unit 200. The information provision device 100 can execute an information provision method.

[0012] The voice input unit 001, the voice recognition unit 002, the vehicle interior video input unit 003, the occupant detection unit 004, the vehicle exterior video input unit 005, the person detection unit 006, the information providing device 100, and the voice output unit 200 can be realized using a microcomputer. The microcomputer can include a CPU (Central Processing Unit), a memory, and an output unit.

[0013] A computer program is installed in the microcomputer and executed to cause the microcomputer to function as each of the units 001 to 006, 200 and the information providing device 100. By executing the installed computer program, the microcomputer can function as each of the units 001 to 006, 200 and the information processing units (011 to 019) included in the information providing device 100.

[0014] In this embodiment, an example is shown in which each of the units 001 to 006, 200 and the information providing device 100 is realized by software. It is also possible to provide dedicated hardware for executing each information processing to configure each of the units 001 to 006, 200 and the information providing device 100. Dedicated hardware includes devices such as application specific integrated circuits (ASICs) and conventional circuit components arranged to perform the functions described in the embodiments.

[0015] The units 001 to 006, 200 and the information providing device 100 may be configured as a single device, or some or all of them may be configured as separate devices. The information processing units (011 to 019) included in the information providing device 100 may be divided and configured using two or more different devices. All or some of the multiple information processing units (011 to 019) may be configured using an ECU (Electronic Control Unit) installed in a vehicle.

[0016] For example, voice information from inside the vehicle is input to the voice input unit 001 using a sound collection device such as a microphone installed in the vehicle cabin. The sound collection device is not limited to a microphone, and may be a sensor that can input voice. The voice input unit 001 executes voice input processing (step S1) shown in the flowchart of FIG. 2A.

[0017] In the voice input process, the voice input unit 001 in Fig. 1 processes voice information inside the vehicle input from a sound collection device. The voice information includes information on conversations between passengers inside the vehicle. The voice information also includes information on environmental sounds around the vehicle that are generated outside the vehicle and collected inside the vehicle.

[0018] In the voice input process, the voice input unit 001 in Fig. 1 can store in-vehicle voice information input from a sound collection device in a storage unit. The storage unit that stores the voice information can be configured with an accessible storage device. The storage device can be configured with, for example, an SSD (Solid State Drive) or an HDD (Hard Disk Drive).

[0019] The speech recognition unit 002 in FIG. 1 performs speech recognition processing (step S2) in FIG. 2A based on speech information obtained from speech input processing by the speech input unit 001. In the speech recognition processing, the speech recognition unit 002 in FIG. 1 performs recognition processing to convert speech information into linguistic information. The speech information includes information on conversations between passengers in the vehicle. In the speech recognition processing, it is possible to identify negative emotions such as anger, disgust, and sadness expressed in the speech of each passenger from tone information in the speech information.

[0020] The vehicle interior video input unit 003 receives an image of the vehicle interior captured by an imaging device such as an interior camera that captures the interior of the vehicle. The imaging device that captures the image of the vehicle interior is not limited to a camera, and may be a sensor or the like that can capture the image of the interior of the vehicle. The vehicle interior video input unit 003 executes the vehicle interior video input process (step S3) of FIG. 2A.

[0021] In the vehicle interior video input process, the vehicle interior video input unit 003 in Fig. 1 processes the captured images of the vehicle interior input from the imaging device. The captured images of the vehicle interior include images of passengers in the vehicle interior.

[0022] The occupant detection unit 004 performs the occupant detection process (step S4) of Fig. 2A based on the information of the captured image of the interior of the vehicle obtained through the input process by the vehicle interior video input unit 003. In the occupant detection process, the occupant detection unit 004 of Fig. 1 detects occupants in the vehicle interior using a person detection method based on machine learning such as a background subtraction method or pattern recognition based on the captured image of the interior of the vehicle. The detection method is not limited to a specific method as long as it can detect people.

[0023] The outside-of-vehicle image input unit 005 receives an image of the outside of the vehicle cabin captured by an imaging device such as an exterior camera that captures the outside of the vehicle. The imaging device that captures the outside of the vehicle cabin is not limited to a camera, and may be a sensor that can capture the outside of the vehicle cabin. The sensor may be, for example, a radar or a LiDAR.

[0024] Radar and LIDAR detect the position of objects around the vehicle by emitting an output wave and receiving a wave reflected from the object onto which the output wave is irradiated. Because radar emits long-wavelength electromagnetic waves and LIDAR emits short-wavelength electromagnetic waves, they can mutually complement each other's low detection sensitivity ranges.

[0025] The imaging device of the vehicle exterior video input unit 005 may be configured as a single device or as multiple devices. When configured as multiple devices, the images captured by the imaging devices of the vehicle exterior video input unit 005 may be, for example, images of the exterior of the vehicle captured by each device in separate areas around the vehicle. The images captured by the imaging devices of the vehicle exterior video input unit 005 may be, for example, a composite overhead image of the vehicle and its surroundings obtained by combining the images captured by each device.

[0026] The vehicle exterior video input unit 005 performs the vehicle exterior video input process (step S5) of Fig. 2A. In the vehicle exterior video input process, the vehicle exterior video input unit 005 of Fig. 1 inputs and processes the captured image of the vehicle exterior input from the imaging device. If there is a person around the vehicle, the captured image of the vehicle exterior includes an image of the person around the vehicle.

[0027] The person detection unit 006 performs the person detection process (step S6) of Fig. 2A based on the information of the captured image outside the vehicle cabin obtained from the input process by the vehicle exterior video input unit 005. In the person detection process, the person detection unit 006 of Fig. 1 detects a person outside the vehicle cabin using a person detection method based on machine learning such as pattern recognition based on the captured image outside the vehicle cabin. The detection method is not limited to a specific method as long as it can detect a person.

[0028] A database unit may be provided that can query the names of occupants and people from the faces of occupants in the vehicle cabin detected by the occupant detection unit 004 in FIG. 1 and people outside the vehicle cabin detected by the person detection unit 006. In the database unit, a database linking face images with profiles of the people may be constructed. The profile may include at least one of attributes such as name and age. The database may be, for example, a database of registered members of a transportation facility that members can use.

[0029] The database unit may be configured, for example, by an accessible storage device, which may be configured, for example, by an SSD (Solid State Drive) or an HDD (Hard Disk Drive).

[0030] The order of steps S1 to S6 in Fig. 2A may be reversed. For example, the voice input process (step S1) and the voice recognition process (step S2) may be performed after the vehicle interior image input process (step S3) and the occupant detection process (step S4), or after the vehicle exterior image input process (step S5) and the person detection process (step S6).

[0031] The vehicle interior image input process (step S3) and the occupant detection process (step S4) may be performed after the vehicle exterior image input process (step S5) and the person detection process (step S6). The vehicle exterior image input process (step S5) and the person detection process (step S6) may be performed before the voice input process (step S1) and the voice recognition process (step S2).

[0032] The voice input process (step S1) and voice recognition process (step S2), the in-vehicle image input process (step S3) and occupant detection process (step S4), and the outside-vehicle image input process (step S5) and person detection process (step S6) may be performed simultaneously in parallel.

[0033] The information providing device 100 in FIG. 1 receives the processing results from the voice recognition unit 002, the occupant detection unit 004, and the person detection unit 006. Based on the processing results received, the information providing device 100 performs the information generation process (step S7) in FIG. 2A. In the information generation process, a conversation topic to be provided to the occupants in the vehicle is generated. Details of the information generation process will be described later with reference to the flowchart in FIG. 2B.

[0034] The information providing device 100 in Fig. 1 includes an occupant identification unit 011, an entry / exit determination unit 012, a door opening / closing detection unit 013, and a seat occupancy detection unit 014 in Fig. 1 as part of a plurality of information processing units (011 to 019) that execute the information generation process in Fig. 2B. The information providing device 100 also includes a topic relevance determination unit 015, a topic recording unit 016, a first topic generation unit 017, a second topic generation unit 018, and an information acquisition unit 019 in Fig. 1 as other parts of a plurality of information processing units (011 to 019) that execute the information generation process in Fig. 2B.

[0035] The occupant identification unit 011 performs a first occupant identification process (step S11) and a second occupant identification process (step S12) of Fig. 2B. When there are two occupants in the vehicle compartment detected in the occupant detection process (step S4), in the first occupant identification process and the second occupant identification process, the occupant identification unit 011 of Fig. 1 respectively identifies the first occupant and the second occupant in the vehicle compartment detected in the occupant detection process (step S4) of Fig. 2A.

[0036] If the number of occupants in the vehicle cabin detected in the occupant detection process (step S4) is one, the first occupant is identified in the first occupant identification process (step S11). If three or more occupants are detected in the occupant detection process (step S4), the occupant identification unit 011 in Fig. 1 may additionally perform a third or subsequent occupant identification process to identify each occupant.

[0037] The occupant identifying unit 011 can perform an identification process of an occupant in the vehicle compartment as an occupant identifying step of identifying an occupant of the vehicle.

[0038] The boarding / alighting determination unit 012 (boarding determination unit) in Figure 1 determines whether a passenger has boarded or alighted from the vehicle based on the detection results obtained from the outside vehicle compartment person detection determination process (step S13), the door opening / closing confirmation process (step S14), and the seat occupancy detection process (step S15) in Figure 2B.

[0039] The boarding / alighting determination unit 012 in FIG. 1 can perform this boarding / alighting determination as a boarding determination step of determining whether a passenger has boarded the vehicle.

[0040] In the outside-vehicle person detection determination process, while the vehicle is stopped, it is confirmed whether or not a person outside the vehicle compartment has been detected by the person detection unit 006. If a person outside the vehicle compartment has not been detected (NO in step S13), the process proceeds to step S21, which will be described later. If a person outside the vehicle compartment has been detected (YES in step S13), the process proceeds to door open / close confirmation process in step S14.

[0041] In the door opening / closing confirmation process, it is confirmed whether or not the door opening / closing detection unit 013 in Fig. 1 has detected the opening / closing of the vehicle door. If the door opening / closing is not detected (NO in step S14), the process proceeds to step S21. If the door opening / closing is detected (YES in step S14), the process proceeds to the seat occupancy detection process in step S15.

[0042] In the seat occupancy detection process, it is confirmed whether or not the seat occupancy detection unit 014 in Fig. 1 has detected the occupancy of a vehicle seat. The occupancy of a seat can be detected, for example, when the load on each seat changes and the output signal of the seat occupancy sensor (not shown) of the seat changes from an empty state to an occupied state. In the seat occupancy detection process, the object for detecting the occupancy of a seat may be limited to the seats inside the vehicle that are connected to the door whose open / closed state has been detected in the door open / close confirmation process in step S14.

[0043] If seat occupancy is not detected (NO in step S15), the process proceeds to step S21. If seat occupancy is detected (YES in step S15), the process proceeds to step S16, which will be described later.

[0044] The getting-in / out determining unit 012 in FIG. 1 can determine whether a third occupant has gotten into the vehicle by detecting the getting-in timing based on the results of the processes in steps S13 to S15.

[0045] The third occupant is an occupant who is scheduled to board the vehicle. The third occupant is a person other than the occupants in the vehicle cabin, including the first occupant and the second occupant, who are detected in the occupant detection process (step S4) of FIG. 2A. The boarding timing is the timing when the occupant boards the vehicle.

[0046] The timing of getting on can be determined as being detected, for example, when a person outside the vehicle cabin is detected by the person detection unit 006 in Fig. 1 while the vehicle is stopped. In this case, the timing of getting on is detected using information on the behavior of the vehicle and an image captured by a camera outside the vehicle cabin.

[0047] The getting-in / out determination unit 012 can determine that the getting-in timing has been detected when a person outside the vehicle compartment is detected in the outside-vehicle-compartment person detection determination process (step S13) of Fig. 2B. When the getting-in timing is detected in the outside-vehicle-compartment person detection determination process, the door opening / closing confirmation process (step S14) and the seat occupancy detection process (step S15) can be omitted.

[0048] The timing of getting on can be determined as being detected, for example, when the person detection unit 006 in Fig. 1 detects a person outside the vehicle while the vehicle is stopped, and the door opening / closing detection unit 013 detects the opening / closing of the vehicle door. In this case, the timing of getting on is detected by the opening / closing of the vehicle door.

[0049] In order to increase the reliability of the determination, the getting-in / out determination unit 012 can determine that the getting-in timing has been detected when a person outside the vehicle compartment is detected in the outside-vehicle-compartment person detection determination process in Fig. 2B and the door opening / closing confirmation process detects the door opening / closing. When the getting-in timing is detected in the outside-vehicle-compartment person detection determination process and the door opening / closing confirmation process, the seat occupancy detection process can be omitted.

[0050] The entry timing can be detected, for example, when the door opening / closing detection unit 013 in Fig. 1 detects the opening / closing of a vehicle door and the seat occupancy detection unit 014 detects the occupancy of a vehicle seat. In this case, the entry timing is detected based on a change in the load applied to the seat inside the vehicle that is connected to the opened / closed door of the vehicle.

[0051] In order to further increase the reliability of the determination, the boarding / exiting determination unit 012 may determine that the boarding timing has been detected when the door opening / closing is detected in the door opening / closing confirmation process of Figure 2B and the seat occupancy is detected in the seat occupancy detection process.

[0052] After the boarding / alighting determination unit 012 determines whether an occupant has boarded or alighted from the vehicle, the occupant identification unit 011 performs the third occupant identification process (step S16) of Fig. 2B. In the third occupant identification process, the occupant identification unit 011 of Fig. 1 assumes that a person approaching a parked vehicle who is captured in an image captured by a camera outside the vehicle cabin is the third occupant, and can identify the name, attributes, etc. of the third occupant. The database of the database unit described above may be used to identify the third occupant.

[0053] The getting-in / out determination unit 012 can check whether a person who re-boards the vehicle after disembarking has been detected in the outside-vehicle person detection determination process (step S13) in Fig. 2B. A case in which a person who has disembarked becomes a third occupant who re-boards the vehicle may occur, for example, when an occupant temporarily leaves the vehicle and returns to the vehicle to stop by a store, facility, or to be picked up or dropped off by another person.

[0054] When the driver of a vehicle gets out of the vehicle, particularly if other occupants remain in the vehicle, it is unlikely that the driver will not report the other occupants and return to the vehicle, and there is a high probability that the driver will return to the vehicle and get back in. In the third occupant identification process of Fig. 2B, when the driver who has gotten out of the vehicle is detected by the person detection unit 006, the occupant identification unit 011 of Fig. 1 may determine that the driver who has gotten out of the vehicle is a third occupant and identify the third occupant.

[0055] In the third occupant identification process of Figure 2B, if the person detection unit 006 detects an occupant other than the driver who re-boards the vehicle after getting off, the occupant identification unit 011 of Figure 1 may determine that the occupant who re-boards the vehicle is the third occupant.

[0056] The occupant identification unit 011 in Fig. 1 identifies the third occupant who is determined to have boarded the vehicle based on the results of the processes in steps S13 to S15 in Fig. 2B in the third occupant identification process (step S16) in Fig. 2B. The occupant identification unit 011 in Fig. 1 can identify the third occupant based on the language information converted from the voice information in the voice recognition process (step S2) in Fig. 2A.

[0057] The occupant identification unit 011 in Figure 1 can, for example, extract keywords such as names, pronouns, etc. that are highly likely to co-occur with the keyword "meeting" during conversations between occupants in the vehicle cabin, and identify the third occupant using the extracted keywords.

[0058] For example, if the occupant identification unit 011 in Fig. 1 identifies the third occupant by name in the third occupant identification process (step S16) in Fig. 2B, the occupant identification unit 011 may use the database unit described above to identify the face of the occupant corresponding to the identified name as the face of the third occupant. The identified face of the occupant can be used by the occupant identification unit 011 to identify a person outside the vehicle compartment detected by the occupant detection unit 004 in Fig. 1 as the third occupant from a facial image.

[0059] The topic relevance determination unit 015 (relevance determination unit) in Fig. 1 performs the topic relevance determination process (steps S17 and S18) in Fig. 2B. In the topic relevance determination process (step S17), the topic relevance determination unit 015 in Fig. 1 determines whether the topic of conversation between the occupants in the vehicle, which has been converted into linguistic information in the speech recognition process (step S2) in Fig. 2A, is a topic related to a third occupant who is scheduled to board the vehicle.

[0060] The topic related to the third occupant may be, for example, a topic that is negative for the third occupant. The negative topic for the third occupant may include, for example, a topic that is not beneficial to the third occupant. The topic that is not beneficial to the third occupant may include, for example, a topic that evokes emotions such as anger, disgust, or sadness toward the third occupant, as well as a topic about the third occupant that is discussed in the vehicle when the third occupant is not in the vehicle.

[0061] In the topic relevance determination process of Fig. 2B, the topic relevance determination unit 015 of Fig. 1 can determine whether a topic is related to the third occupant based on whether a keyword appears in the conversation inside the vehicle before the third occupant gets in. This keyword can include a phrase corresponding to the name, attributes, etc. of the third occupant identified by the occupant identification unit 011 of Fig. 1 in the third occupant identification process.

[0062] The topic relevance determination unit 015 can determine that the topic is related to the third occupant based on the co-occurrence of the keyword, the name of the third occupant, and negative words in the conversation. Negative words can include, for example, words expressing emotions such as anger, disgust, and sadness toward a person. The occurrence of negative words in the conversation can be detected, for example, using existing sentiment analysis processing.

[0063] The topic relevance determination unit 015 can determine the co-occurrence of the name of the third occupant and negative words, for example, by using the in-car voice information stored in the topic recording unit 016. The topic recording unit 016 can be configured in the information providing device 100 by, for example, a storage device that stores the in-car voice information input to the voice input unit 001 described above.

[0064] The topic relevance determination unit 015 can determine the co-occurrence between the name of the third occupant and negative words for the in-vehicle speech information including past speech stored in the topic recording unit 016. The period for determining the co-occurrence between the name of the third occupant and negative words for the past speech information may be limited to a period going back a predetermined period from the present.

[0065] The topic relevance determination unit 015 can determine that the topic is related to the third occupant based on the emotional state of the occupants in the vehicle at the time when the name of the third occupant appears during the conversation in the vehicle. The emotional state of the occupants in the vehicle can include, for example, emotional states such as anger, disgust, sadness, etc. toward the person.

[0066] The topic relevance determination unit 015 can determine the emotional state of the occupant at the time when the name of the third occupant appears during a conversation inside the car, for example, by using the in-car voice information stored in the topic recording unit 016. For example, when a voice with a tone expressing a negative emotion appears in the in-car voice information, the topic relevance determination unit 015 can determine the emotional state of the voice from the wording of the voice.

[0067] The topic relevance determination unit 015 can determine the emotional state of the occupant at the time when the name of the third occupant appeared during a conversation inside the vehicle, for example, from a captured image of the occupant inside the vehicle input to the vehicle interior video input unit 003. The topic relevance determination unit 015 can, for example, perform image processing on the face of the occupant captured in the captured image at the time when the name of the third occupant appeared during the conversation, extract the emotional state shown in the facial expression of the occupant inside the vehicle, and determine the emotional state of the occupant.

[0068] The topic relevance determination unit 015 can determine whether a topic of conversation among occupants in the vehicle is related to the third occupant based on the attributes of the third occupant identified by the occupant identification unit 011 in the third occupant identification process of FIG. 2B. When determining whether a topic is related to the third occupant based on the attributes of the third occupant, for example, a topic correspondence table linking attributes with events that are likely to be topics of conversation for people with those attributes can be used. The correspondence table can be stored, for example, in a storage device used to configure the topic recording unit 016 in the information providing device 100.

[0069] By using this correspondence table, for example, it is possible to cause the topic relevance determination unit 015 to determine that a violent topic that is undesirable for a third occupant who has a young age attribute is a topic related to the third occupant. Determination using this correspondence table is effective in determining whether a topic is related to the third occupant even when, for example, the third occupant cannot be identified by name or the like and information such as the history of topics related to the third occupant is not available.

[0070] When the topic relevance determination unit 015 in Fig. 1 determines that the topic is related to the third occupant in the topic relevance determination process in Fig. 2B (YES in step S18), the process proceeds to step S21. When the topic relevance determination unit 015 in Fig. 1 determines that the topic is related to the third occupant in the topic relevance determination process in Fig. 2B (YES in step S18), the process proceeds to step S19, which will be described later.

[0071] If it is determined that the topic is related to the third occupant based on the results of the processing in steps S17 and S18 in Figure 2B, the information acquisition unit 019 in Figure 1 acquires information for generating a new conversation topic in the information acquisition processing (step S19).

[0072] The information for generating a new conversation topic may include, for example, information about the vehicle's surrounding environment. The information about the vehicle's surrounding environment may include information such as the weather at the destination and traffic congestion information on the route to the destination. The information for generating a new conversation topic may include, for example, information about the vehicle's status. The information about the vehicle's status may include information such as being stopped for a long time due to a red light at an intersection, being driven slowly due to traffic congestion, there being no vehicles ahead, or having a series of green lights with no opportunity to slow down.

[0073] Information for generating new conversation topics can be obtained, for example, using the vehicle's onboard equipment, from websites on the Internet, information and communication systems such as VICS (Vehicle Information and Communication System, registered trademark), the vehicle itself, etc.

[0074] An occupant in a vehicle may become a third occupant by re-boarding the same vehicle after getting off to pick up or drop off someone, or to stop by a store or facility, etc. When an occupant in a vehicle becomes a third occupant after getting off, the information acquisition unit 019 in FIG. 1 may acquire information on the topic of conversation that was taking place in the vehicle before the third occupant got off, as information for generating a new topic of conversation.

[0075] The information acquisition unit 019 can acquire information about the topic of conversations that took place in the vehicle before the third occupant got off from the topic recording unit 016. The topic recording unit 016 stores past in-vehicle voice information input to the voice input unit 001. If the driver will be the third occupant after getting off, the topic recording unit 016 can acquire information about the topic of conversations that took place in the vehicle before the driver got off from the topic recording unit 016 when the driver gets back in the vehicle as the third occupant. The topic information that can be acquired from the topic recording unit 016 may be the latest information before the driver got off, or may be information about a topic selected based on the number of occurrences, duration, etc. of conversations before getting off.

[0076] The second topic generation unit 018 (new topic generation unit) generates a new conversation topic to be output inside the vehicle in the second topic generation process (step S20) of Fig. 2B. The new topic can be generated using information acquired by the information acquisition unit 019 of Fig. 1 from the topic recording unit 016. For example, if the third passenger is a passenger who has reboarded after disembarking, the second topic generation unit 018 can generate a topic that continues the topic before disembarking, such as "Regarding the topic we were talking about earlier," or "You were saying XX earlier, weren't you?"

[0077] In the second topic generation process of FIG. 2B, the second topic generation unit 018 may use a topic model, which is a machine learning model, to generate a new conversation topic based on the frequency of appearance of keywords in past conversations in the car.

[0078] The first topic generation unit 017 in Fig. 1 generates a topic to be output inside the vehicle in the first topic generation process (step S21) in Fig. 2B. In the first topic generation process, the first topic generation unit 017 in Fig. 1 can generate, for example, a topic of conversation that was taking place inside the vehicle before the third occupant got in, or a topic related to the occupants inside the vehicle. The first topic generation unit 017 may generate the topic using information acquired by the information acquisition unit 019.

[0079] The audio output unit 200 performs the audio output process (step S8) of Fig. 2A based on information about the topic of conversation generated by the first topic generation unit 017 and the second topic generation unit 018 of the information providing device 100. In the audio output process, the audio output unit 200 of Fig. 1 outputs audio of the conversation about the topic generated by the first topic generation unit 017 and the second topic generation unit 018 inside the vehicle.

[0080] The voice of the conversation generated by the first topic generation unit 017 and the second topic generation unit 018 can be output inside the vehicle from a speaker (not shown). The voice of the conversation can also be output from a character of a dialogue agent inside the vehicle. The dialogue agent, for example, converses with the occupants inside the vehicle based on occupant information learned about the occupants, and a conventionally known dialogue agent can be used.

[0081] The conversation agent may have, for example, a character. The character may be, for example, an assistive robot 310 with a built-in speaker arranged in the center console of a vehicle 300 shown in FIG. 3. The character may be an avatar image 330 (virtual character) projected onto a windshield 320 or the like using the principle of a head-up display (HUD). The character may be an avatar image 350 (avatar) displayed on a multi-display 340 in the center console.

[0082] In the information providing device 100 of the present embodiment described above, the occupant identification unit 011 identifies the occupants of the vehicle in the first occupant identification process and the second occupant identification process (steps S11 to S12 in FIG. 2B), which are occupant identification steps. The getting-in / out determination unit 012 performs the getting-in determination step in steps S13 to S15 in FIG. 2B, and determines whether an occupant is getting into the vehicle.

[0083] The topic relevance determination unit 015 performs a relevance determination step in steps S17 and S18 of FIG. 2B to determine whether the topic of conversation taking place inside the vehicle is related to a third occupant who is scheduled to board the vehicle. If the topic relevance determination unit 015 determines that the topic is related to the third occupant, the second topic generation unit 018 generates a new conversation topic to be output inside the vehicle in a second topic generation process (step S20 of FIG. 2B), which is a new topic generation step. The second topic generation unit 018 can generate a new conversation topic using information acquired by the information acquisition unit 019 to be used for generating a conversation topic to be output inside the vehicle.

[0084] In the information providing device 100 of this embodiment, when a third occupant gets in, the conversation inside the vehicle is induced to change to a new topic of conversation. This makes it easier for the third occupant who gets in the vehicle to come into contact with new topics of conversation while in the vehicle, and can encourage all occupants to enjoy positive topics inside the vehicle. By providing new topics of conversation, the topics of conversation inside the vehicle can be made to be considerate of the people riding in the vehicle.

[0085] In the information providing device 100 of this embodiment, the second topic generation unit 018 generated a topic unrelated to the third occupant as a new conversation topic in the second topic generation process (step S20 in FIG. 2B), which is a new topic generation step.

[0086] By generating new conversation topics unrelated to the third passenger, it is possible to prevent the third passenger from becoming negative by being exposed to topics related to themselves in the vehicle, and to prevent the overall atmosphere in the vehicle from becoming negative when the third passenger is in the vehicle.

[0087] In the information providing device 100 of this embodiment, the information acquiring unit 019 acquires information used to generate conversation topics to be output in the vehicle in an information acquisition process (step S19 in FIG. 2B), which is an information acquisition step. The acquired information can be at least one of information related to the vehicle state, information about the surrounding environment detected in the vehicle, and information provided to the vehicle by communication via a telecommunication line.

[0088] By obtaining at least one of the following information: information regarding the vehicle's status, information regarding the surrounding environment detected in the vehicle, and information provided to the vehicle via communication via a telecommunications line, new conversation topics to be output inside the vehicle can be efficiently generated.

[0089] In the information providing device 100 of this embodiment, for example, the voice input unit 001 further performs a topic recording step in the voice input process (step S1 in FIG. 2A) of recording information about conversations that take place inside the vehicle in the topic recording unit 016. When the second topic generation unit 018 generates a new topic of conversation to be output inside the vehicle, the second topic generation unit 018 uses the information acquired from the topic recording unit 016, with the occupant determined by the getting on / off determination unit 012 to be a third occupant.

[0090] When a passenger who re-boards the vehicle after disembarking is considered a third passenger, conversation information of passengers including the third passenger before disembarking can be recorded in the topic recording unit 016, thereby recording topics that are positive for all passengers including the third passenger. By utilizing the recorded topics, new conversation topics can be generated more effectively when the third passenger boards the vehicle.

[0091] In the information providing device 100 of this embodiment, the boarding / exiting determination unit 012 detects the boarding timing for determining whether an occupant has boarded the vehicle in step S13 of FIG. 2B during the boarding determination step, using information on vehicle behavior and images captured by a camera outside the vehicle cabin. By detecting the timing of the third occupant's boarding using the camera outside the vehicle cabin while detecting the vehicle's stationary behavior, the timing of the third occupant's boarding can be detected as early as possible. This allows time to transition the topic of conversation in the vehicle to a new topic of conversation, allowing the new topic to be presented smoothly.

[0092] In the information providing device 100 of this embodiment, the boarding / exiting determination unit 012 detects the boarding timing for determining whether an occupant has boarded the vehicle based on the opening and closing of the vehicle door in step S14 of Fig. 2B during the boarding determination step. By determining the boarding timing of the third occupant based on the opening and closing of the vehicle door, it is possible to more reliably detect the boarding of the third occupant and reliably provide a new topic.

[0093] In the information providing device 100 of this embodiment, the boarding / exiting determination unit 012 detects the boarding timing for determining whether an occupant has boarded the vehicle from a change in the load on the seat inside the vehicle that is connected to the opened and closed door of the vehicle in step S15 of Fig. 2B during the boarding determination step. By detecting the boarding timing of the third occupant from the weight of the seat that accompanies the opening and closing of the door, it is possible to more reliably detect the boarding of the third occupant, and it is possible to improve the reliability of providing a new topic.

[0094] In the information providing device 100 of this embodiment, the occupant identification unit 011 identifies the third occupant from an image captured by a vehicle exterior camera in a third occupant identification process (step S16 in FIG. 2B), which is an occupant identification step. By identifying the third occupant using the vehicle exterior camera, the third occupant can be identified at an early stage when the third occupant approaches the vehicle in which he or she is riding. By identifying the third occupant early, more time is provided to shift the topic of conversation in the vehicle to a new topic, and the new topic of conversation provided in the vehicle allows the topic of conversation in the vehicle to be shifted smoothly.

[0095] In the information providing device 100 of this embodiment, the occupant identification unit 011 identifies the third occupant from the content of the conversation that took place in the vehicle before the third occupant got in, in the third occupant identification process (step S16 in FIG. 2B), which is an occupant identification step. By identifying the third occupant from the content of the conversation that took place in the vehicle before the third occupant got in, it is possible to identify the third occupant before the third occupant gets in. Identifying the third occupant at an early stage provides time to shift the topic of conversation in the vehicle to a new topic, and the new topic of conversation provided in the vehicle allows the topic of conversation in the vehicle to be smoothly shifted.

[0096] In the information providing device 100 of this embodiment, the occupant identification unit 011 identifies the driver of the vehicle who has dismounted as the third occupant in the third occupant identification process (step S16 in FIG. 2B), which is an occupant identification step. When the driver dismounts, on the assumption that the dismounted driver will re-board the vehicle, by identifying the driver as the third occupant at the time of dismounting, it is possible to identify the third occupant from the dismounting stage before the third occupant re-boards. By identifying the third occupant at an earlier stage, there is more time to shift the topic of conversation in the vehicle to a new topic, and the new topic of conversation provided in the vehicle allows the topic of conversation in the vehicle to be shifted smoothly.

[0097] In the information providing device 100 of this embodiment, the topic related to the third occupant is a negative topic for the third occupant. Since new conversation topics to be provided in the vehicle can be generated while avoiding negative topics for the third occupant, all occupants can have conversations about positive topics in the vehicle.

[0098] In the information providing device 100 of this embodiment, the topic relevance determination unit 015 determines whether a topic is related to the third occupant based on the co-occurrence of the name of the third occupant and negative words in the conversation inside the vehicle in the relevance determination step performed in steps S17 and S18 of Fig. 2B. By determining whether a topic is related to the third occupant based on the co-occurrence of the name of the third occupant and negative words, the determination of whether a topic is related to the third occupant can be performed by language processing.

[0099] In the information providing device 100 of this embodiment, the topic relevance determination unit 015 determines whether the topic is related to the third occupant based on at least one of the results of voice recognition and facial expression recognition for the occupant in the vehicle in the relevance determination step performed in steps S17 and S18 of Fig. 2B. The results of voice recognition and facial expression recognition used for the determination are the results of voice recognition and facial expression recognition when the name of the third occupant appears in a conversation held inside the vehicle.

[0100] By determining the topic related to the third occupant based on the third occupant's name and the results of at least one of voice recognition and facial expression recognition, it is possible to read the emotions of each occupant, which are difficult to understand from the content of the conversation alone, and determine the topic related to the third occupant.

[0101] In the information providing device 100 of this embodiment, the topic relevance determination unit 015 determines whether a topic is relevant to a third occupant based on a correspondence table of topics corresponding to the attributes of the third occupant in the relevance determination step performed in steps S17 and S18 of Fig. 2B. By determining whether a topic is relevant to the third occupant based on the correspondence table based on the attributes of the third occupant, it is possible to appropriately determine whether a topic is relevant to the third occupant even when there is no personal information about the third occupant.

[0102] In the information providing device 100 of this embodiment, the topic relevance determination unit 015 may determine that the topic is not relevant to the third occupant in the relevance determination step performed in steps S17 and S18 in Fig. 2B. In this case, the first topic generation unit 017 can generate, as a topic of conversation to be output inside the vehicle, the topic of the conversation that took place in the vehicle when the third occupant got into the vehicle in the first topic generation process (step S21 in Fig. 2B), which is the relevant topic generation step.

[0103] Alternatively, the first topic generation unit 017 can generate a topic related to the vehicle occupants identified by the occupant identification unit 011 in the first occupant identification process and the second occupant identification process (steps S11 to S12 in Figure 2B), which are occupant identification steps, as a conversation topic to be output inside the vehicle.

[0104] If it is determined that the topic is not related to the third occupant, the system obtains and provides information based on the conversation between the vehicle's occupants or new information tailored to the vehicle's occupants, allowing all occupants to have conversations about positive topics in the car.

[0105] In the information providing device 100 of this embodiment, the second topic generation unit 018 generates a new conversation topic to be output by the dialogue agent character inside the vehicle in the second topic generation process (step S20 in FIG. 2B), which is a new topic generation step. This dialogue agent character converses with the vehicle occupants based on the occupant information learned about the vehicle occupants.

[0106] By outputting the generated new conversation topics from characters such as avatars, virtual characters, and robots of an interactive system that is capable of two-way dialogue and has the function of learning occupant information, it is possible to provide information as a more user-friendly information providing device.

[0107] The above-described embodiment is merely an example of the present invention, and therefore the present invention is not limited to the above-described embodiment, and various modifications can be made to the design and other aspects of the present invention without departing from the technical concept of the present invention. [Explanation of symbols]

[0108] 001 Audio input section 002 Voice Recognition Unit 003 In-vehicle video input unit 004 Occupant detection unit 005 Outside vehicle video input unit 006 Person detection unit 011 Occupant Identification Unit 012 Boarding / Alighting Determination Unit (Boarding Determination Unit) 013 Door opening / closing detector 014 Seat occupancy detection unit 015 Topic relatedness determination unit (relatedness determination unit) 016 Topic Recording Department 017 1st topic generation part 018 2nd topic generation section (new topic generation section) 019 Information acquisition department 100 Information provision device 200 Audio output section 300 vehicles 310 Assistive Robots 330,350 avatar images

Claims

1. An information providing method using a computer, comprising: an occupant identification step of identifying an occupant of the vehicle; an occupancy determination step of determining whether an occupant is occupying the vehicle; a relevance determination step of determining whether a topic of conversation held inside the vehicle is related to a third occupant who is scheduled to board the vehicle; an information acquisition step of acquiring information used to generate a conversation topic to be output in the vehicle; a new topic generation step of generating a new conversation topic to be output inside the vehicle using the information acquired in the information acquisition step when it is determined in the relevance determination step that the topic of the conversation held inside the vehicle when the third occupant gets into the vehicle is a topic related to the third occupant; Information provision method to carry out.

2. An information provision method as described in Claim 1, wherein the computer generates a topic unrelated to the third occupant as the new conversation topic in the new topic generation step.

3. An information providing method as described in claim 1 or 2, wherein in the information acquisition step, the computer acquires at least one of information regarding the state of the vehicle, information regarding the surrounding environment detected in the vehicle, and information provided to the vehicle by communication via a telecommunications line.

4. The method further includes a topic recording step in which the computer records information about conversations that have taken place in the vehicle in a topic recording unit, The computer determines the occupant who is determined to re-board the vehicle after getting off in the boarding determination step as the third occupant, If it is determined in the relevance determination step that the topic of the conversation that took place in the vehicle when the third occupant re-boards the vehicle is a topic related to the third occupant, the computer generates the new topic of the conversation in the new topic generation step by using information on the conversation that took place in the vehicle before the third occupant got off, which information is acquired from the topic recording unit. The information providing method according to any one of claims 1 to 3.

5. An information provision method described in any one of claims 1 to 4, wherein the computer detects the timing of boarding at which it determines whether an occupant will board the vehicle in the boarding determination step using information on the behavior of the vehicle and images taken by a camera outside the vehicle's cabin.

6. An information provision method described in any one of claims 1 to 5, wherein the computer detects the boarding timing at which it determines whether an occupant has boarded the vehicle in the boarding determination step by opening and closing the door of the vehicle.

7. An information provision method described in any one of claims 1 to 6, wherein the computer detects the timing of boarding at which it determines whether an occupant has boarded the vehicle in the boarding determination step by detecting a change in the load on a seat inside the vehicle that is connected to the opened or closed door of the vehicle.

8. An information provision method described in any one of claims 1 to 7, wherein the computer, in the occupant identification step, identifies the third occupant from an image taken by an exterior camera of the vehicle.

9. An information provision method described in any one of claims 1 to 8, wherein, in the occupant identification step, the computer identifies the third occupant from the content of a conversation that took place in the vehicle before the third occupant got into the vehicle.

10. An information provision method described in any one of claims 1 to 9, wherein the computer identifies the driver of the vehicle who has disembarked as the third occupant in the occupant identification step.

11. 11. The information providing method according to claim 1, wherein the topic related to the third occupant is a topic that is negative for the third occupant.

12. An information provision method described in any one of claims 1 to 11, wherein in the relevance determination step, the computer determines whether the topic is related to the third occupant based on the co-occurrence of the third occupant's name with negative words in conversations held inside the vehicle.

13. An information provision method described in any one of claims 1 to 12, wherein in the relevance determination step, the computer determines whether the topic is related to the third occupant based on the results of at least one of voice recognition and facial expression recognition of the occupant in the vehicle when the name of the third occupant appears in a conversation taking place inside the vehicle.

14. An information provision method described in any one of claims 1 to 13, wherein in the relevance determination step, the computer determines whether the topic is relevant to the third occupant based on a topic correspondence table according to the attributes of the third occupant.

15. An information provision method described in any one of claims 1 to 14, further comprising a related topic generation step in which, if it is determined in the related determination step that the topic is not related to the third occupant, the computer generates, as a topic of conversation to be output inside the vehicle, a topic of conversation that took place in the vehicle when the third occupant boarded the vehicle, or a topic related to the occupant of the vehicle identified in the occupant identification step.

16. An information provision method described in any one of claims 1 to 15, wherein in the new topic generation step, the computer generates a new conversation topic to be output inside the vehicle by a dialogue agent character that interacts with the occupants in the vehicle based on occupant information learned about the occupants of the vehicle.

17. an occupant identification unit for identifying an occupant of the vehicle; an occupancy determination unit that determines whether an occupant is occupying the vehicle; a relevance determination unit that determines whether a topic of conversation held inside the vehicle is related to a third occupant who is scheduled to board the vehicle; an information acquisition unit that acquires information used to generate conversation topics to be output in the vehicle; a new topic generation unit that generates a new conversation topic to be output inside the vehicle using the information acquired by the information acquisition unit when the relevance determination unit determines that a topic of conversation held inside the vehicle when the third occupant gets into the vehicle is a topic related to the third occupant; An information providing device comprising:

Images