Agent Device

Abstract

To provide an agent device that can built a relationship between an agent and a driver on a daily basis and allows a danger avoiding action to be smoothly taken when a dangerous situation suddenly occurs.SOLUTION: An agent device 100 includes: a shift position sensor 140f for detecting a shift position; and an agent controller 190 for controlling a personalized agent 200. The agent controller 190 makes the agent 200 face the driver H of a vehicle 1 when the shift position is in a parking range, and outputs information by voice output and operations of both arms when conveying information.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an agent device capable of making notifications using personified characters. [Background technology]

[0002] BACKGROUND ART Conventionally, various methods have been proposed as methods for alerting occupants (particularly the driver) in a driving environment of a vehicle such as an automobile. As one such method, an agent device or driving assistance device has been proposed in which an occupant and an anthropomorphized character (a so-called agent) interact with each other to provide information on driving assistance in response to the occupant's requests.

[0003] For example, in conventional driving assistance devices, it is thought that notifications of danger avoidance actions using voice or other means are difficult to convey intuitively, so a proposal has been made to place a small robot on the dashboard, appear as a personified agent, and have this agent simulate the danger avoidance actions that the driver should take (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-032204 Summary of the Invention [Problem to be solved by the invention]

[0005] However, conventional agents were impersonal or appeared suddenly to draw attention, making it difficult to build a relationship of trust between the agent and the driver. Also, even if the agent suddenly took the same action as an evasive action, it was difficult for the driver to take that action immediately if they did not understand the cause of the evasive action, and there was a problem that evasive action would not be taken unless a relationship of trust was established.

[0006] The present invention has been made to solve such conventional problems, and aims to provide an agent device that builds a relationship of trust between an agent and a driver on a daily basis, and enables the driver to smoothly take action to avoid danger in an emergency. [Means for solving the problem]

[0007] The agent device according to the present invention comprises a shift position detection unit that detects the shift position, and an agent control unit that controls a personified agent, and the agent control unit is characterized in that when the shift position is in the parking range, the agent control unit causes the agent to face the driver of the vehicle, and when transmitting information, outputs information by voice output and movement of both arms. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an agent device that builds a relationship of trust between the agent and the driver on a daily basis, and enables the driver to smoothly take action to avoid danger in an emergency. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a perspective view showing an interior of a vehicle according to an embodiment of the present invention; [Figure 2] FIG. 2 is a front view showing the interior of the vehicle in front of the driver's seat. [Figure 3] FIG. 10 is a front view showing the placement of agents. [Figure 4] FIG. 2 is a schematic diagram showing a configuration of an agent according to an embodiment of the present invention. [Figure 5] FIG. 2 is a block diagram illustrating an agent device. [Figure 6] 10 is a main flowchart of an agent control process performed by a control unit of the agent device. [Figure 7] 10 is a flowchart showing a process performed by a control unit of the agent device when boarding. [Figure 8]10 is a flowchart showing listening and response processing performed by a control unit of the agent device. [Figure 9] FIG. 10 is an operational diagram showing an example of the operation of an agent. [Figure 10] 10 is a flowchart showing a driving end process performed by a control unit of the agent device. [Figure 11] FIG. 10 is an operational diagram showing an example of the operation of an agent. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same reference numerals in different drawings indicate parts with the same functions, and duplicated descriptions in each drawing will be omitted as appropriate.

[0011] (Vehicle interior configuration) As shown in Figures 1 and 2, vehicle 1 has an instrument panel 3 provided in front of a driver's seat 2, and a steering wheel 4 disposed between the driver's seat 2 and the instrument panel 3. The steering wheel 4 is rotatably attached to a steering column (not shown) via a steering shaft (not shown). An airbag 8 is stored inside the steering wheel 4, and is deployed toward the driver H in the event of a collision or the like of the vehicle 1. In the present embodiment, the agent device 100 (described later) and the agent 200 (described later) are described based on dialogue and gestures (movements) with the driver H, but they can also be performed in the same way with other passengers.

[0012] 1, two display panels, a first display panel 30 and a second display panel 40, are provided in the cabin of a vehicle 1. The first display panel 30 is disposed on the instrument panel 3 in front of the driver's seat 2, and the second display panel 40 is disposed on the instrument panel 3 on the left front side of the driver's seat 2.

[0013] (First display panel 30) 1 and 2, the first display panel 30 is configured to include a pointer-type meter configured like an analog clock and a so-called liquid crystal display device in which a liquid crystal panel and a backlight are integrally provided. The driver H can view various information displayed in the first display area 30a of the first display panel 30 through the upper space 4a of the steering wheel 4.

[0014] 2, first display panel 30 is provided with first display area 30a, and on the left and right sides of first display area 30a are two pointer-type meters that display information such as the traveling speed of vehicle 1 (speedometer) and the number of revolutions per unit time of the engine (tachometer), and a small liquid crystal display device that displays an image showing general vehicle information is disposed between the two pointer-type meters and in the center of first display panel 30. Note that first display panel 30 may not be provided with a pointer-type meter and may be configured entirely with a single liquid crystal display device.

[0015] (Second display panel 40) 1, the second display panel 40 is configured as a so-called liquid crystal display device in which a liquid crystal panel and a backlight are integrally provided, for example. Note that the second display panel 40 displays map information and the like, and serves as a so-called car navigation system.

[0016] The liquid crystal display devices configured in the first display panel 30 and the second display panel 40 may be configured as display devices such as a self-luminous display device such as a plasma display or an organic EL display, or a projection type projector.

[0017] (Agent 200) As shown in FIG. 3, an agent 200 is disposed on the upper part of the instrument panel 3 in front of the driver's seat 2. The agent 200 may be located anywhere as long as it is within the peripheral vision of the driver H, and does not necessarily have to be located directly in front of the driver's seat. It is desirable that the agent 200 be located within the peripheral vision of the driver H, but at a position that does not obstruct the driver H's view of the outside of the vehicle. The location of the agent 200 may also be changed using a predetermined operating device or voice instructions, etc.

[0018] Agent 200 is an anthropomorphized character, a three-dimensional object with a humanoid external shape, used for notification and for interaction. In this embodiment, agent 200 is a three-dimensional object that is a real humanoid image, but it is not limited to this and may be a character displayed on a display device such as a screen, a three-dimensional object expressed by a hologram or the like, a virtual three-dimensional object, or the like.

[0019] As shown in FIG. 4, the agent 200 has a head 211, a right eye 212R, a left eye 212L, a neck 213, a chest 214, a waist 215, a buttocks 216, and a right shoulder 2. 21R, left shoulder 221L, right upper arm 222R, left upper arm 222L, right elbow 223R, left elbow 223L, right forearm 224R, left forearm 224L, right wrist 22 5R, left wrist 225L, right hand 226R, left hand 226L, right hip 231R, left hip 231L, right thigh 232R, left thigh 232L, right knee 2 33R, a left knee part 233L, a right lower leg part 234R, a left lower leg part 234L, a right ankle part 235R, a left ankle part 235L, a right foot part 236R, and a left foot part 236L.

[0020] The agent 200 is also provided with a support connector 240, which supports the agent 200 so that it can be placed upright on the instrument panel 3. Incidentally, ears are provided on both sides of the head 211 of the agent 200. This makes it appear as if the agent 200 is listening carefully to what the driver H is saying. Even if these ears are not actually provided, as will be described later, when the agent 200 listens to what the driver H is saying, the agent 200 can make it appear as if he is listening with his hands on his ears by placing his right hand 226R or left hand 226L on the side of the head 211 of the agent 200, thereby deepening the relationship of trust between the agent 200 and the driver H.

[0021] Note that Figure 4(a) is a diagram showing the basic posture of the agent 200 when facing the driver H (the rear side of the vehicle 1), and Figure 4(b) is a diagram showing the basic posture of the agent 200 when facing away from the driver H (facing the front side of the vehicle 1). In addition, in the basic posture, the agent 200 stands with the right palm portion 226Ra of the right hand portion 226R and the left palm portion 226La of the left hand portion 226L facing toward the inside of the agent 200, and the right back portion 226Rb of the right hand portion 226R and the left back portion 226Lb of the left hand portion 226L facing toward the outside of the agent 200. As described above, the agent 200 can freely rotate between a front posture facing the driver H and a rear posture facing away from the driver H. In other words, the agent 200 may be one in which the agent 200 itself rotates around the support connector 240, or the agent 200 may be placed on a rotating body and rotated from the front posture to the rear posture and from the rear posture to the front posture by rotating the rotating body.

[0022] An agent control unit 190, which will be described later, is provided within the agent 200, and controls and operates each joint of the agent 200, allowing the agent 200 to assume a predetermined posture. The agent control unit 190 is wirelessly connected to a control unit 110 (ECU), which will be described later, and operates the agent 200 under the control of the control unit 110. Note that, although the agent control unit 190 and the control unit 110 are connected wirelessly in this embodiment, they may also be connected by wire, for example, by passing a communication line through the support connector 240.

[0023] An agent speaker 191, which will be described later, is provided in the head 211 of the agent 200, and is configured to output audio under the control of the agent control unit 190. Note that the agent speaker 191 is not limited to being provided in the head 211 of the agent 200, but may be provided in another part of the agent 200 or in another location of the vehicle 1. However, providing the agent speaker 191 in the head 211 of the agent 200 allows the driver H to perceive the utterances from the agent 200 more realistically, makes it easier for the driver H to empathize with the agent 200, and can greatly contribute to building trust in the agent 200 by the driver H.

[0024] Next, the configuration of the agent device 100 according to the present invention will be described with reference to the block diagram of FIG.

[0025] As shown in FIG. 5, the agent device 100 includes a control unit 110, a surrounding environment recognition unit 120, an occupant state recognition unit 130, a vehicle state recognition unit 140, a speaker 150, a microphone 160, a memory unit 170, a transceiver 180, an agent control unit 190, and an agent speaker 191.

[0026] The agent device 100 shown in FIG. 5 is merely an example, and the components of the agent device 100 can be modified as appropriate. For example, the agent device 100 of the present invention can be realized by a configuration including one or two of the surrounding environment recognition unit 120, the occupant state recognition unit 130, and the vehicle state recognition unit 140 depending on the information to be acquired. The agent device 100 of the present invention can also be realized by only one of the speaker 150 and the agent speaker 191. Furthermore, the agent 200 may be provided with an agent microphone in addition to the microphone 160, or the agent device 100 of the present invention can be realized by only providing an agent microphone in the agent 200 instead of the microphone 160. Furthermore, in this embodiment, both the control unit 110 and the agent control unit 190 are provided, but this is not a limitation. It is also possible to have only one of the control unit 110 or the agent control unit 190 and have it perform the functions of both.

[0027] (control unit 110) The control unit 110 is equipped with a CPU, ROM, RAM (e.g., a ring buffer), EEPROM, input / output ports, etc. (not shown), and for example, when information is input through the input port, it controls various devices through the output port based on a control program read from the ROM.

[0028] The ROM of the control unit 110 stores a data table (not shown) of actions and lines to be performed by the agent 200. Furthermore, a learning function such as AI may be provided so that the agent 200 performs actions and lines other than those stored. The CPU of the control unit 110 determines the actions and lines to be performed by the agent 200 from the data table based on information acquired from each recognition unit (described later) and information acquired from the microphone 160. For example, when an utterance from the driver H is acquired from the microphone 160, a "nod" gesture is determined from the data table, and when the driver H finishes speaking, a line such as "I understand" is determined. The data table of actions and lines to be performed by the agent 200 may be stored in the ROM of the agent control unit 190.

[0029] (Surrounding Environment Recognition Division 120) The surrounding environment recognition unit 120 is provided to recognize the surrounding environment of the vehicle 1 (host vehicle). The surrounding environment recognition unit 120 also includes an exterior camera 120a, a radar 120b, a temperature sensor 120c, and a weather sensor 120d, and is capable of recognizing the surrounding environment of the vehicle 1 using these devices.

[0030] (Exterior camera 120a) The exterior camera 120a is attached to, for example, a rearview mirror (not shown) and is capable of capturing images of the area in front of and behind the vehicle 1. The captured image information is input to the control unit 110, which then stores the image information in RAM. This allows the control unit 110 to recognize the situation in front of and behind the vehicle 1 in real time and after the fact.

[0031] (Radar 120b) The radar 120b is, for example, a millimeter-wave radar that detects obstacles and the like by transmitting radio waves. The millimeter-wave radar is attached to the front bumper or rear bumper of the vehicle 1, and is capable of monitoring the area ahead of the vehicle 1, the areas in front of the vehicle 1, and the areas behind the vehicle 1. The monitoring information is input to the control unit 110, which stores the monitoring information in RAM. This allows the control unit 110 to recognize the situation ahead of the vehicle 1, the areas in front of the vehicle 1, and the areas behind the vehicle 1 in real time and after the fact. Note that although a millimeter-wave radar is used in this embodiment, other radars may also be used. For example, infrared radar may be used.

[0032] (Temperature sensor 120c) The air temperature sensor 120c is attached to, for example, the rear of the vehicle 1, and is capable of detecting the air temperature outside the vehicle 1. The detected air temperature is input to the control unit 110, which then stores the air temperature in RAM. This allows the control unit 110 to recognize the air temperature outside the vehicle 1.

[0033] (Weather Sensor 120d) The weather sensor 120d is attached, for example, to the top of the vehicle 1 and is capable of detecting raindrops and the like falling on the vehicle 1. The detected water droplet information is input to the control unit 110, which then stores the water droplet information in RAM. This allows the control unit 110 to recognize the weather conditions outside the vehicle 1.

[0034] In this way, the control unit 110 can recognize the surrounding environment of the vehicle 1 in real time and after the fact. As a result, the control unit 110 can control the agent 200 and the agent speaker 191 based on the recognition of the surrounding environment of the vehicle 1 to notify the driver H of information about the surrounding environment of the vehicle 1. For example, if there is a fallen object ahead of the vehicle 1, the control unit 110 can notify the driver H by gestures and voice, such as "There is a fallen object ahead." This can improve safety.

[0035] In this embodiment, the vehicle exterior camera 120a and the radar 120b are used as the surrounding environment recognition unit 120, but this is merely an example, and other devices may of course be used.

[0036] (Occupant state recognition unit 130) The occupant state recognition unit 130 is provided to recognize the state of the driver H. In addition, the occupant state recognition unit 130 is equipped with an occupant image capturing camera 130a and a vital sensor 130b, and is able to recognize the state of the driver H using these devices.

[0037] (Occupant photography camera 130a) The occupant photographing camera 130a is attached to, for example, the instrument panel 3, and is capable of photographing the driver H. Then, photographed image information is input to the control unit 110, which stores the image information in RAM. This allows the control unit 110 to recognize the state of the driver H in real time and after the fact. Note that the state of the driver H referred to here specifically includes the state of the driver H's eyelids, the number of blinks, the direction of the driver's gaze, the direction of the face, whether the driver is crouched, or whether the driver is unable to move, etc.

[0038] (Vital Sensor 130b) The vital sensor 130b is attached, for example, to a part of the steering wheel 4 that is gripped by the driver H, and is capable of acquiring vital information such as the heart rate and blood pressure of the driver H. The acquired vital information is then input to the control unit 110, which then stores the vital information in RAM. This allows the control unit 110 to recognize the state of the driver H in real time and after the fact.

[0039] In this way, the control unit 110 can recognize the state of the driver H in real time and after the fact. As a result, the control unit 110 can control the agent 200 and the agent speaker 191 based on the recognition of the state of the driver H to notify the driver H of predetermined information. For example, it is possible to notify the driver H by image and sound such as "Your eyelids are drooping. Would you like to take a break?" or "Your heart rate is faster than usual. Would you like to take a break?" This leads to improved safety.

[0040] The occupant state recognition unit 130 can recognize the thoughts and emotions of the driver H to a certain extent based on information acquired from the occupant image capturing camera 130a and the vital sensor 130b, and information input from the microphone 160. For example, the occupant state recognition unit 130 can acquire the facial expression of the driver H from the occupant image capturing camera 130a, acquire the heart rate and blood pressure of the driver H from the vital sensor 130b, and acquire the voice volume and input content from the microphone 160, and can recognize from this acquired information whether the driver H is thinking and feeling normal thoughts and emotions or thoughts and emotions that are different from normal thoughts and emotions (for example, surprised, angry, etc.).

[0041] In this embodiment, the occupant state recognition unit 130 is configured using the occupant image capturing camera 130a and the vital sign sensor 130b, but this is merely an example, and other devices may of course be used.

[0042] (Vehicle state recognition unit 140) The vehicle state recognition unit 140 is provided to recognize the state of the vehicle 1. The vehicle state recognition unit 140 also includes a vehicle speed sensor 140a, a steering wheel angle sensor 140b, an accelerator pedal sensor 140c, a brake pedal sensor 140d, a G sensor 140e, and a shift position sensor 140f, and is able to recognize the state of the vehicle 1 using these devices.

[0043] (vehicle speed sensor 140a) The vehicle speed sensor 140a is a sensor for detecting the vehicle speed of the vehicle 1. The detected vehicle speed is input as a vehicle speed signal to the control unit 110, and the control unit 110 stores the vehicle speed information in RAM. This allows the control unit 110 to recognize the vehicle speed of the vehicle 1 in real time and after the fact.

[0044] (Steering wheel angle sensor 140b) The steering wheel angle sensor 140b is a sensor for detecting the steering wheel angle (angle of the steering wheel 4) of the vehicle 1, and the detected steering wheel angle is input as an angle signal to the control unit 110, which then stores the angle information in RAM. This allows the control unit 110 to recognize the steering wheel angle (angle of the steering wheel 4) of the vehicle 1 in real time and afterwards.

[0045] (Accelerator pedal sensor 140c) The accelerator pedal sensor 140c is a sensor for detecting the depression amount of an accelerator pedal (not shown), and the detected depression amount is input as a depression amount signal to the control unit 110, which then stores the depression amount information in RAM. This allows the control unit 110 to recognize the depression amount of the accelerator pedal of the vehicle 1 in real time and after the fact.

[0046] (Brake pedal sensor 140d) The brake pedal sensor 140d is a sensor for detecting the depression amount of a brake pedal (not shown), and the detected depression amount is input as a depression amount signal to the control unit 110, which then stores the depression amount information in RAM. This allows the control unit 110 to recognize the depression amount of the brake pedal of the vehicle 1 in real time and afterwards.

[0047] (G sensor 140e) The G sensor 140e is a sensor for detecting the acceleration, deceleration, and inclination of the vehicle 1. When acceleration is detected, the acceleration amount is input as an acceleration amount signal, when deceleration is detected, the deceleration amount is input as a deceleration amount signal, and when inclination is detected, the inclination angle amount is input as an inclination angle signal to the control unit 110, and the control unit 110 stores the acceleration information, deceleration information, and inclination information in RAM. This allows the control unit 110 to recognize the acceleration, deceleration, and inclination of the vehicle 1 in real time and after the fact.

[0048] (Shift position sensor 140f) The shift position sensor 140f is a sensor for detecting the shift range (shift position) of the vehicle 1. In other words, the shift position sensor 140f detects the position to which the shift range is set, such as parking range (stopped position), drive range (driving position), reverse range (backing position), neutral range (neutral position), etc. The shift range detected by the shift position sensor 140f is input to the control unit 110, which then stores the current shift range in RAM. This allows the control unit 110 to recognize the position to which the shift range is set.

[0049] In this way, the control unit 110 can recognize the state of the vehicle 1 in real time and after the fact. As a result, the control unit 110 can control the agent 200 and the agent speaker 191 based on the recognition of the state of the vehicle 1 to notify the driver H of the state of the vehicle 1. For example, if the vehicle is traveling at an appropriate speed, the control unit 110 can notify the driver H by gestures and voice, such as "You are traveling at an appropriate speed." This leads to improved safety.

[0050] In this embodiment, the vehicle state recognition unit 140 includes the vehicle speed sensor 140a, the steering wheel angle sensor 140b, the accelerator pedal sensor 140c, the brake pedal sensor 140d, the G sensor 140e, and the shift position sensor 140f, but these are merely examples, and other devices may of course be used.

[0051] (Speaker 150) The speaker 150 is attached to, for example, the instrument panel 3, and outputs sounds other than the sound emitted from the agent 200, warning sounds, etc. Note that an audio speaker built into the vehicle 1 may be used instead of providing the speaker 150. Furthermore, as will be described later, the speaker 150 may also serve as the agent speaker 191.

[0052] (Mike 160) The microphone 160 is attached to, for example, the instrument panel 3, and receives input of voices emitted by the driver H and other passengers.

[0053] (Storage unit 170) The storage unit 170 can store information acquired from each of the recognition units described above, dialogues held between the driver H and the control unit 110, and the like. By storing this information in the storage unit 170, the control unit 110 can recognize the driving tendencies of the driver H (for example, what kind of driving style the driver is), the hobbies and preferences of the driver H (for example, what kind of background music the driver H likes), and the like. Furthermore, by recognizing these, it is also possible for the agent device 100 (agent 200) to actively hold a dialogue that is in line with the driving tendencies of the driver H and the hobbies and preferences of the driver H.

[0054] (Transceiver 180) The transceiver 180 can, for example, acquire information using an in-vehicle wireless LAN, acquire location information using a satellite positioning system, etc. Based on the acquired information and the information stored in the storage unit 170, the control unit 110 can also actively carry out a dialogue from the agent device 100 side (agent 200) that is tailored to the driving tendencies and hobbies and preferences of the driver H.

[0055] (Agent control unit 190) The agent control unit 190 is equipped with a CPU, ROM, RAM, input / output ports, etc. (not shown), and when information is input from the control unit 110, it can operate each part of the agent 200 and cause the agent speaker 191 to output a predetermined sound, for example.

[0056] (Agent Speaker 191) As described above, the agent speaker 191 is provided inside the head 211 of the agent 200, and outputs sounds and the like emitted from the agent 200. Incidentally, by providing the agent speaker 191 inside the head 211 of the agent 200, particularly in the area corresponding to the mouth, as in this embodiment, it becomes more realistic and contributes to building a relationship of trust, but in cases where the agent 200 is small and it is difficult to provide the agent speaker 191 inside the head 211, the above-mentioned speaker 150 may be used instead, or the agent speaker 191 or a speaker that serves as a substitute may be provided in another part inside the vehicle 1.

[0057] Next, we will explain the operation and dialogue of the agent 200 in the agent device 100. The operation and dialogue of the agent 200 are performed by control processing by the control unit 110, and each unit of the agent 200 is operated by the agent control unit 190, and sound is output from the agent speaker 191. The control flow for performing the control processing is stored in the ROM of the control unit 110, and the CPU of the control unit 110 reads it from the ROM and performs various processes.

[0058] The main process of the agent control process performed by the CPU of the control unit 110 will be described below with reference to FIG.

[0059] (Step S10) In step S10, when the ignition is turned on, the CPU of the control unit 110 starts the main processing of the agent control processing. Specifically, the CPU of the control unit 110 reads the main processing of the agent control processing from the ROM, starts the processing, and then proceeds to step S100.

[0060] (Step S100) In step S100, the CPU of the control unit 110 performs a boarding process. The boarding process is a process that starts when the driver H gets into the vehicle 1. Details of the boarding process will be described later. Then, when the boarding process is completed, the process proceeds to step S200.

[0061] (Step S200) In step S200, the CPU of the control unit 110 performs pre-driving processing. In the pre-driving processing, processing is performed before the driver H starts driving. For example, in the pre-driving processing, the agent 200 provides instructions on precautions regarding the driving environment to the destination and explains rest points. In explaining the pre-driving processing, the agent 200 faces the driver H, moves both hands to explain each item, and also signals completion by nodding its head at the end of each item. Then, when the pre-driving processing is completed, the process proceeds to step S20.

[0062] (Step S20) In step S20, the CPU of control unit 110 performs processing to determine whether the shift range is "P" (parking range). Specifically, the CPU of control unit 110 determines whether the shift range input from shift position sensor 140f is the parking range. If the CPU of control unit 110 determines that the shift range is not the parking range (non-parking range: in this case, the shift position has been changed), it proceeds to step S300, and if it determines that the shift range is the parking range (in this case, the shift position has not been changed), it proceeds to step S400.

[0063] (Step S300) In step S300, the CPU of the control unit 110 performs a driving process. In the driving process, the CPU performs a control process for the agent 200 while driving. That is, when the shift position is in a non-parking range, the CPU makes the agent 200 assume a predetermined posture and controls the agent 200 while driving by voice and action. For example, when the shift range input from the shift position sensor 140f is the drive range, the CPU of the control unit 110 performs a control process for the agent 200 while driving forward, such as making the agent 200 face forward (the traveling direction of the vehicle 1) (by making the agent 200 assume a predetermined posture), and when the shift range is the reverse range, the CPU performs a control process for the agent 200 while driving backward. Then, when the driving process is completed, the CPU proceeds to step S30. Note that, as will be described later, when the shift position returns to the parking range, the CPU makes the agent 200 face the driver H again and outputs a voice of appreciation and performs an action. That is, when the driver H gets in or the vehicle is stopped, the agent 200 faces the front toward the driver H, and when the vehicle is moving, it rotates to the back and faces the front of the vehicle 1, and after the vehicle has finished moving, it rotates again and faces the front toward the driver H.

[0064] (Step S30) In step S30, the CPU of control unit 110 performs processing to determine whether the shift range is "P" (parking range). As described above, the CPU of control unit 110 determines whether the shift range input from shift position sensor 140f is the parking range. If the CPU of control unit 110 determines that the shift range is not the parking range, it proceeds to step S300, and if it determines that the shift range is the parking range, it proceeds to step S400.

[0065] (Step S400) In step S400, the CPU of the control unit 110 performs a driving end process. The driving end process is a process performed when a predetermined driving period has ended. Details of the driving end process will be described later. Then, when the driving end process is completed, the process proceeds to step S40.

[0066] (Step S40) In step S40, the CPU of control unit 110 performs processing to determine whether the shift range is "P" (parking range). As described above, the CPU of control unit 110 determines whether the shift range input from shift position sensor 140f is the parking range. If the CPU of control unit 110 determines that the shift range is not the parking range (in this case, the shift position has been changed), it proceeds to step S300, and if it determines that the shift range is the parking range (in this case, the shift position has not been changed), it proceeds to step S50.

[0067] (Step S50) In step S50, the CPU of the control unit 110 performs a driving resumption determination process. In the driving resumption determination process, a process is performed to determine whether or not the driver H intends to resume driving. For example, the CPU of the control unit 110 determines that the driver H intends to resume driving when the driver H makes a voice input such as "I will resume driving," or when a new destination is input into the navigation system by voice input or manual input. If the CPU of the control unit 110 determines that the driver H intends to resume driving, it proceeds to step S200, and if it determines that the driver H does not intend to resume driving, it proceeds to step S60.

[0068] (Step S60) In step S60, the CPU of control unit 110 determines whether or not the ignition is turned off. If the ignition is turned off, the CPU of control unit 110 ends the main processing of the agent control processing, and if the ignition is not turned off, the CPU proceeds to step S40.

[0069] Next, the in-ride processing performed by the CPU of the control unit 110 will be described with reference to Fig. 7. Fig. 7 shows a subroutine of step S100 (in-ride processing) in Fig. 6.

[0070] (Step S101) In the in-ride processing, first, in step S101, the CPU of the control unit 110 performs processing to determine whether or not the shift position is in the parking range. Specifically, the CPU of the control unit 110 determines whether or not the shift range input from the shift position sensor 140f is in the parking range. If the CPU of the control unit 110 determines that the shift position is in the parking range, it proceeds to step S102, and if it determines that the shift position is outside the parking range, it terminates the in-ride processing. Note that in this determination processing, the shift position being outside the parking range indicates that the shift position is outside the parking range when the ignition is on and is not in a normal state, so predetermined abnormality processing may be performed.

[0071] (Step S102) In step S102, the CPU of the control unit 110 performs processing to have the agent 200 face the driver H. Specifically, the CPU of the control unit 110 instructs the agent control unit 190 via the transceiver 180 to have the agent 200 face the driver H and take the basic posture, based on the basic posture of the agent 200 when the vehicle is stopped, which is stored in the RAM. The basic posture of the agent 200 is, as shown in FIG. 4, a posture in which the agent 200 is standing with both arms stretched down at the sides, and is the initial setting posture of the agent 200. Then, once the agent 200 has been faced with the driver H, the process proceeds to step S103.

[0072] (Step S103) In step S103, the CPU of the control unit 110 determines whether or not the voice recognition function has been activated. Specifically, the CPU of the control unit 110 determines whether or not there has been voice input from the driver H (or a passenger including the driver H) via the microphone 160. If the CPU of the control unit 110 determines that there has been speech from the driver H, it shifts the process to step S500, and if it determines that there has been no speech from the driver H, it shifts the process to step S111. It is desirable that the processes in step S103 and the next step S500 are always incorporated as interrupt processes in the agent control process.

[0073] (Step S500) In step S500, the CPU of the control unit 110 performs listening and response processing. That is, the CPU of the control unit 110 listens to the speech of the driver H and performs processing corresponding to the speech. Details of the listening and response processing will be described later. Then, when the listening and response processing is completed, the processing proceeds to step S103.

[0074] (Step S111) In step S111, the CPU of the control unit 110 determines whether there is any conversation to be made to the driver H, any information to be explained when boarding, etc. Examples of information to be explained when boarding include outside temperature, weather, remaining fuel information, and special notes on vehicle maintenance. If the CPU of the control unit 110 determines that there is any conversation to be made to the driver H, any information to be explained when boarding, etc., it proceeds to step S112, and if it determines that there is no conversation to be made to the driver H, any information to be explained when boarding, etc., it ends the on-board processing.

[0075] (Step S112) In step S112, the CPU of the control unit 110 causes the agent 200 to converse using speech and movements. When the agent 200 transmits information, the CPU of the control unit 110 outputs information by voice output and movements of the body, such as both arms. For example, the CPU of the control unit 110 causes the agent control unit 190 to speak and move both arms of the agent 200 to perform gestures appropriate for each item of conversation or explanation. That is, the CPU of the control unit 110 causes the agent control unit 190 to perform corresponding movements of the right upper arm 222R, left upper arm 222L, right forearm 224R, left forearm 224L, right hand 226R, left hand 226L, etc. of the agent 200, as well as the right shoulder 221R, left shoulder 221L, right elbow 223R, left elbow 223L, right wrist 225R, left wrist 225L, etc.

[0076] (Step S113) In step S113, the CPU of control unit 110 performs a process of determining whether the conversation or explanation has ended. If the CPU of control unit 110 determines that the conversation or explanation has not ended, it shifts the process to step S112, and if it determines that the conversation or explanation has ended, it shifts the process to step S114.

[0077] (Step S114) In step S114, the CPU of the control unit 110 performs an action process when the agent utterance ends. When the information transmission ends, the CPU of the control unit 110 causes the agent 200 to perform an end confirmation action, thereby ending the information transmission. For example, the CPU of the control unit 110 causes the agent 200 to signal completion by nodding its head after the explanation has ended, via the agent control unit 190. The CPU may also lightly point its hand toward the driver H to perform an instruction action to prompt the driver H to speak or take an action. After the action when the agent utterance ends is performed, the process proceeds to step S111, and the above process is repeated until the conversation and each item are completed. When the conversation and all items are completed, the in-vehicle process ends.

[0078] Next, the listening and answering process performed by the CPU of the control unit 110 will be described with reference to Fig. 8. Fig. 8 shows a subroutine of step S500 (listening and answering process) in Fig. 7.

[0079] (Step S511) In the listening and response process, first, in step S511, the CPU of the control unit 110 performs a process of making the agent 200 assume an attentive listening posture. When information is input from driver H, the CPU of the control unit 110 makes the agent 200 assume an attentive listening posture as if listening attentively to what is being said. Specifically, the CPU of the control unit 110 performs a process of making the agent 200 assume an attentive listening posture via the transceiver 180 to the agent control unit 190 based on the listening posture of the agent 200 stored in the RAM. The attentive listening posture of the agent 200 is, for example, a posture of listening with a hand to an ear or a posture of responding to what driver H is saying while nodding. Specifically, as shown in FIG. 9( a), the agent 200 is made to assume a listening posture with one hand to an ear. Furthermore, during voice recognition, the agent 200 is made to assume a listening posture with his / her legs slightly bent and stretched, his / her hands on his / her knees, and nodding his / her head, as shown in FIG. 9( b).

[0080] (Step S512) Next, in step S512, the CPU of the control unit 110 performs processing to determine whether or not the speech of driver H has ended. In this processing to determine whether or not the speech of driver H has ended, similar to the processing of step S103 described above, the CPU determines whether or not the speech of driver H has ended depending on whether or not the voice recognition function is operating. If the CPU of the control unit 110 determines that the speech of driver H has ended, it proceeds to processing of step S513, and if it determines that the speech of driver H has not ended, it proceeds to processing of step S511.

[0081] (Step S513) In step S513, the CPU of the control unit 110 determines whether or not the content of the utterance of driver H was understood. For example, this may occur if the meaning of driver H's question is not understood, or if the question itself is not audible, or if the question itself is understandable but there are multiple answers or the question is one that cannot be answered. If the CPU of the control unit 110 could not understand the content of driver H's utterance, the process proceeds to step S514, and if the content of the utterance was understood, the process proceeds to step S515.

[0082] (Step S514) In step S514, the CPU of control unit 110 performs processing to prompt the agent 200 to ask a question or to speak again. That is, the CPU of control unit 110 causes the agent 200 to output voice or perform actions to ask a question to driver H or to speak again, depending on the content that the agent 200 was unable to understand. After performing processing to prompt the agent 200 to ask a question or to speak again, the process returns to step S511, and the agent 200 is made to take an attentive listening stance.

[0083] (Step S515) In step S515, the CPU of the control unit 110 performs a driver utterance end action on the agent 200. When the information input from the driver H is completed, the CPU of the control unit 110 causes the agent 200 to output a voice and nod to confirm that the information input has been completed. In the driver utterance end action, for example, the CPU of the control unit 110 outputs to the agent 200 via the agent control unit 190 lines such as "Got it" or "OK," as well as a completion signal such as making an "OK" sign with both arms in a circle as shown in FIG. 9(c) or nodding the head (nodding). Then, after the driver utterance end action has been performed, the process proceeds to step S516.

[0084] (Step S516) In step S516, the CPU of the control unit 110 performs processing corresponding to the utterance. For example, it responds to a question from driver H, performs an operation in response to a predetermined instruction, or continues a conversation with driver H. Here, if a predetermined action is required in response to an instruction from driver H, the CPU of the control unit 110 outputs to the agent 200 that it should repeat the instruction while moving both arms, and nod and start operating. Thereafter, it performs the predetermined action in response to the instruction. Then, after performing processing corresponding to the utterance, it ends the listening and answering process.

[0085] Next, the operation end processing performed by the CPU of the control unit 110 will be described with reference to Fig. 10. Fig. 10 shows a subroutine of step S400 (operation end processing) in Fig. 6.

[0086] (Step S401) In the driving end processing, first, in step S401, the CPU of the control unit 110 performs processing to determine whether or not the agent 200 is facing the driver H. If the CPU of the control unit 110 determines that the agent 200 is not facing the driver H, it shifts the processing to step S402, and if it determines that the agent 200 is facing the driver, it shifts the processing to step S403.

[0087] (Step S402) In step S402, the CPU of the control unit 110 performs processing to make the agent 200 face the driver H. When the shift position returns to the parking range, the CPU of the control unit 110 instructs the agent control unit 190 to make the agent 200 face the driver H again and assume the basic posture. In other words, the CPU of the control unit 110 rotates the agent 200 so that its front faces the driver H. Then, once the agent 200 has been made to face the driver H, the process proceeds to step S403.

[0088] (Step S403) In step S403, the CPU of the control unit 110 determines whether or not there is an approaching object. Specifically, the CPU of the control unit 110 determines whether or not there is an object, person, etc. approaching the vehicle 1 based on information input from various sensors, etc. If the CPU of the control unit 110 determines that there is an object approaching the vehicle 1, it shifts the process to step S404, and if it determines that there is no object approaching the vehicle 1, it shifts the process to step S405. It is desirable that the processes in step S403 and the next step S404 are always incorporated as interrupt processes in the agent control process.

[0089] (Step S404) In step S404, the CPU of the control unit 110 notifies the agent 200 of the approaching object. Here, the CPU of the control unit 110 notifies the agent 200 of the approaching object by voice and movement. For example, the CPU of the control unit 110 may output a voice via the agent speaker 191 informing the agent 200 of the approaching object and of evasive action, or may have the agent 200 point toward the approaching object or spread both arms to signal to stand still. For example, as a signal to stand still, as shown in FIG. 12 , the agent 200 may spread both arms to encourage the agent 200 to stand still, and may also bend and stretch, thereby imagining the amount of deceleration depending on the depth of the bending and stretching. Then, once the notification of the approaching object has ended, the process proceeds to step S403.

[0090] (Step S405) In step S405, the CPU of the control unit 110 detects whether or not there is an abnormality in the vehicle. The CPU of the control unit 110 may also detect not only whether or not there is an abnormality in the vehicle, but also whether or not there is a recommendation for the driver H, such as a low fuel level. Then, once the detection of whether or not there is an abnormality in the vehicle is completed, the process proceeds to step S406.

[0091] (Step S406) In step S406, the CPU of the control unit 110 determines whether or not there is an abnormality in the vehicle. The CPU of the control unit 110 determines whether or not there is an abnormality in the vehicle by detecting the presence or absence of the abnormality in the vehicle, and if it is determined that there is an abnormality in the vehicle, it proceeds to step S407, and if it is determined that there is no abnormality in the vehicle, it proceeds to step S408.

[0092] (Step S407) In step S407, the CPU of control unit 110 notifies the user of the abnormality by using the voice and movement of agent 200. After the abnormality notification is complete, the process proceeds to step S408.

[0093] (Step S408) In step S408, the CPU of the control unit 110 outputs a voice and performs an action to congratulate the agent when the driving is finished. For example, the CPU of the control unit 110 outputs a voice such as "Thank you for driving" via the agent speaker 191, and causes the agent 200 to give a slight bow or make a good sign gesture with the hand. Then, when the voice and action to congratulate the agent when the driving is finished are finished, the process proceeds to step S409.

[0094] (Step S409) In step S409, the CPU of the control unit 110 explains the status and operation for each item of the vehicle 1. Here, the CPU of the control unit 110 announces, for each item, an explanation of the parts of the vehicle 1 that are not abnormal, an explanation of recommended actions, etc., by voice and action of the agent 200. Note that for items that have an abnormality, an explanation of the status may also be given here. Then, the status and operation of the vehicle 1 are explained, and the process proceeds to step S410.

[0095] (Step S410) In step S410, the CPU of control unit 110 determines whether the explanation of the situation of the item has ended. If the CPU of control unit 110 determines that the explanation of the situation of the item has ended, it proceeds to step S411, and if it determines that the explanation of the situation of the item has not ended, it proceeds to step S409.

[0096] (Step S411) In step S411, the CPU of the control unit 110 performs an item end operation. For example, the CPU of the control unit 110 causes the agent 200 to give a signal indicating that the explanation of the situation for the item has ended, such as by gently nodding its head. Then, when the item end operation is completed, the process proceeds to step S412.

[0097] (Step S412) In step S412, the CPU of control unit 110 determines whether there are any other items for which situation explanations, etc. If the CPU of control unit 110 determines that there are any other items for which situation explanations, etc., it shifts the process to step S409, and if the CPU of control unit 110 determines that there are no other items for which situation explanations, etc., i.e., that situation explanations, etc. for all items have been completed, it shifts the process to step S413.

[0098] (Step S413) In step S413, the CPU of the control unit 110 performs an explanation end operation. For example, the CPU of the control unit 110 outputs a voice such as "All explanations are over" via the agent speaker 191, and causes the agent 200 to perform an operation such as moving its head up and down more than when each item is finished, or gently extending its hand toward the driver H. Then, when the explanation end operation is completed, the driving end process ends.

[0099] As described above, the agent device 100 of this embodiment has the agent 200 face the driver H of the vehicle 1 when the shift position is in the parking range, and when transmitting information, outputs information by voice output and movements of both arms, causing the agent 200 to converse with the driver H by voice and movements on a daily basis. Furthermore, the movements performed by the agent 200 are not necessarily the same movements that the driver H is made to perform, but are movements that the driver H can easily understand intuitively and whose contents can be easily conveyed. Therefore, a relationship of trust between the agent 200 and the driver H can be easily built, and the driver H can be made to smoothly perform danger avoidance actions in an emergency.

[0100] In the agent control process of this embodiment, a program stored in the ROM, RAM, EEPROM, etc. of the control unit 110 is loaded into the RAM, etc. of the control unit 110 and executed by the CPU, etc. of the control unit 110. In this embodiment, the shift position sensor 140f constitutes the shift position detection unit of the present invention. In this embodiment, the agent control unit of the present invention is constituted by both the control unit 110 and the agent control unit 190, but as mentioned above, the agent control unit 190 alone may have the functions of both the control unit 110 and the agent control unit 190 of this embodiment. [Explanation of symbols]

[0101] 1: vehicle, 100: agent device, 110: control unit, 120: surrounding environment recognition unit, 130: occupant state recognition unit, 140: vehicle state recognition unit, 140f: shift position sensor, 150: speaker, 160: microphone, 170: memory unit, 180: transceiver, 190: agent control unit, 191: agent speaker, 200: agent

Claims

1. a shift position detection unit that detects a shift position; an agent control unit that controls a personified agent; an occupant state recognition unit that acquires driver information including at least the number of blinks and the heart rate of the driver; Equipped with The agent: It is positioned in the driver's peripheral vision, The agent control unit instructs the agent to: The system actively engages in dialogue with the driver that is suited to the driver's driving tendencies and hobbies and preferences; If the shift position is in the parking range, the agent is caused to face the driver directly; When transmitting information from the agent, the information is output by voice and by moving both arms; When the information transmission from the agent is completed, the agent causes the driver to make an instruction action to prompt the driver to speak or take an action; When the driver inputs information, have them place their hands over their ears and nod while listening. When the driver has finished inputting information, the system outputs a voice and nods to confirm that the information has been input. making a voice notification to show concern for the driver based on the driver information; 1. An agent device comprising:

2. The agent control unit When the shift position is out of the parking range, the agent is made to assume a predetermined posture, and when the shift position returns to the parking range, the agent is made to face the driver again, and an appreciative voice output and action are performed.

2. The agent device according to claim 1.

3. a shift position detection unit that detects a shift position; an agent control unit that controls a personified agent; an occupant state recognition unit that acquires driver information including at least the number of blinks and the heart rate of the driver; Equipped with The agent: It is positioned in the driver's peripheral vision, The agent control unit instructs the agent to: The system actively engages in dialogue with the driver that is suited to the driver's driving tendencies and hobbies and preferences; If the shift position is not in the parking range, the agent is made to assume a predetermined posture, and when the shift position returns to the parking range, the agent is made to face the driver and output a voice message of appreciation and perform an action; When transmitting information from the agent, the information is output by voice and by moving both arms; When the information transmission from the agent is completed, the agent causes the driver to make an instruction action to prompt the driver to speak or take an action; When the driver inputs information, have them place their hands over their ears and nod while listening. When the driver has finished inputting information, the system outputs a voice and nods to confirm that the information has been input. making a voice notification to show concern for the driver based on the driver information; 1. An agent device comprising:

4. The agent control unit If the shift position is not in the parking range when the vehicle ignition is turned on, an abnormality is reported.

4. The agent device according to claim 1, wherein the agent device is a device for receiving an information from a user.

Images