Agent device

Abstract

To provide an agent device that a driver can understand by intuition and which allows the driver to grasp a driving environment by operations of an agent and also rapidly perform a driving operation suitable for the driving environment.SOLUTION: An agent device 100 includes: a shift position sensor 140f for detecting a shift position; an ambient surrounding recognition unit 120 for acquiring a surrounding situation; a control unit 110 for determining whether it is necessary to make a driver H take an action on the basis of the acquired surrounding situation; and an agent controller 190 for controlling a personalized agent. The agent controller 190 causes the agent 200 to point out by a finger the direction of the reason for the necessity of taking an action when it has been determined that the driver H needs to take an action, and to gesture according to the action.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to an agent device capable of notification by an anthropomorphic character.

Background Art

[0002] Conventionally, in a driving environment in a vehicle such as an automobile, various methods have been proposed as methods for prompting a passenger (especially a driver) to pay attention. As one of such methods, an agent device or a driving support device has been proposed in which a passenger and an anthropomorphic character (so-called agent) provide information related to driving support according to the passenger's request while having a dialogue.

[0003] For example, in a conventional driving support device, it is considered that a notification of a danger avoidance action by voice or the like is difficult to be intuitively conveyed, and a small robot is arranged on a dashboard and made to appear as an anthropomorphic agent, and this agent is made to simulate the danger avoidance action that a driver should perform (see Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, conventional agents were inorganic or suddenly appeared to prompt attention, making it difficult to build a trust relationship between the agent and the driver. Also, even if the agent suddenly performs the same action as the avoidance action, if the driver does not understand the cause for which the avoidance action must be taken, it is difficult to immediately take that action, and in particular, without a trust relationship, it does not lead to an avoidance action, which is a problem. In particular, when it becomes necessary for the driver to take a predetermined countermeasure such as a danger avoidance action during driving, if the agent suddenly mimics the avoidance action, it may be impossible to understand why the agent took that action, and there is a risk that the predetermined avoidance action will not be in time while the driver is thinking about it.

[0006] The present invention has been made to solve such conventional problems, and an object thereof is to provide an agent device that can be intuitively understood by the driver, and that can, at the same time as grasping the driving environment by the operation of the agent, quickly perform a driving operation suitable for the driving environment.

Means for Solving the Problems

[0007] The agent device according to the present invention includes a shift position detection unit that detects a shift position, a peripheral situation acquisition unit that acquires a peripheral situation, a determination unit that determines whether it is necessary to cause the driver to take a countermeasure based on the acquired peripheral situation, and an agent control unit that controls an anthropomorphic agent. When the agent control unit determines that it is necessary to cause the driver to take a countermeasure, the agent control unit causes the agent to point in the direction of the cause for which the countermeasure became necessary, and also causes the agent to perform a gesture corresponding to the countermeasure.

[0008] According to the present invention, it is possible to provide an agent device that can be intuitively understood by the driver, and that can, at the same time as grasping the driving environment by the operation of the agent, quickly perform a driving operation suitable for the driving environment.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Also, in the following description, the same reference numerals in different figures indicate parts having the same function, and duplicate descriptions in each figure are appropriately omitted.

[0011] (Each Component in the Passenger Compartment) As shown in FIGS. 1 and 2, the vehicle 1 has an instrument panel 3 provided on the front side of the vehicle of the driver's seat 2, and a steering wheel 4 disposed between the driver's seat 2 and the instrument panel 3. The steering wheel 4 is rotatably attached to a steering column (not shown) via a steering shaft (not shown). Further, inside the steering wheel 4, an airbag 8 that deploys toward the driver (hereinafter referred to as the driver) H when the vehicle 1 collides or the like is stored. Note that in the agent device 100 (described later) and the agent 200 (described later) of the present embodiment, although the description is based on the interaction and gestures (operations) with the driver H, the same can be performed for other passengers.

[0012] As shown in FIG. 1, in the passenger compartment of the vehicle 1, two display panels, i.e., a first display panel 30 and a second display panel 40, are provided. The first display panel 30 is disposed on the instrument panel 3 on the front side of the driver's seat 2, and the second display panel 40 is disposed on the instrument panel 3 on the left front side of the driver's seat 2.

[0013] (First display panel 30) As shown in FIGS. 1 and 2, the first display panel 30 is configured to include a pointer-type meter configured like an analog clock, and a so-called liquid crystal display device in which a liquid crystal panel, a backlight, and the like are integrally provided. The driver H can view various information displayed in the first display area 30a of the first display panel 30 through the upper space portion 4a of the steering wheel 4.

[0014] As shown in FIG. 2, the first display panel 30 is provided with a first display area 30a. On the left and right of this first display area 30a, two pointer-type meters for displaying information such as the traveling speed (speedometer) of the vehicle 1 and the number of revolutions per unit time of the engine (tachometer) are provided. And between the two pointer-type meters and at the center of the first display panel 30, a small liquid crystal display device for displaying an image indicating general vehicle information is arranged. Note that the first display panel 30 may be configured as a single liquid crystal display device without including a pointer-type meter.

[0015] (Second display panel 40) As shown in FIG. 1, the second display panel 40 is configured by, for example, a so-called liquid crystal display device in which a liquid crystal panel and a backlight are integrally provided. Note that map information and the like are displayed on the second display panel 40, and it serves as a so-called car navigation system.

[0016] Note that the liquid crystal display device composed of the first display panel 30 and the second display panel 40 may be composed of a self-luminous display device such as a plasma display or an organic EL, or a display device such as a projection type projector.

[0017] (Agent 200) Also, as shown in FIG. 3, an agent 200 is arranged above the instrument panel 3 on the front side of the driver's seat 2. Note that the agent 200 may be located anywhere as long as it is within the peripheral vision of the driver H, and does not necessarily have to be arranged directly in front of the driver's seat. Note that it is desirable that the agent 200 be within the peripheral vision of the driver H and not block the driver H's view outside the vehicle. Also, the arrangement position of the agent 200 may be changeable by a predetermined operation device or voice instruction.

[0018] The agent 200 is an anthropomorphic character and is an informing and interactive three-dimensional object with a humanoid outer shape. Note that in the present embodiment, the agent 200 is a three-dimensional object that forms a humanoid real image, but is not limited thereto, and may be a character displayed on a display device such as a display, a three-dimensional display represented by a hologram, a virtual three-dimensional object, or the like.

[0019] As shown in FIG. 4, the agent 200 has a head 211, a right eye part 212R, a left eye part 212L, a neck part 213, a chest part 214, a waist part 215, a hip part 216, a right shoulder part 221R, a left shoulder part 221L, a right upper arm part 222R, a left upper arm part 222L, a right elbow part 223R, a left elbow part 223L, a right forearm part 224R, a left forearm part 224L, a right wrist part 225R, a left wrist part 225L, a right hand part 226R, a left hand part 226L, a right hip joint part 231R, a left hip joint part 231L, a right thigh part 232R, a left thigh part 232L, a right knee part 233R, a left knee part 233L, a right lower leg part 234R, a left lower leg part 234L, a right ankle part 235R, a left ankle part 235L, a right foot part 236R, and a left foot part 236L.

[0020] In addition, the agent 200 is provided with a support connector 240, and the agent 200 is supported so as to be able to stand upright on the instrument panel 3. Note that ears are provided on both sides of the head 211 of the agent 200. This can make it seem that the agent 200 is listening carefully to the driver H's speech. Also, regarding these ears, even if they are not actually provided, as will be described later, when the agent 200 listens to the driver H's speech, by placing the right hand part 226R or the left hand part 226L on the side of the head 211 of the agent 200, it can be made to appear as if it is listening with a hand on the ear, and the trust relationship with the driver H can be deepened.

[0021] Note that Fig. 4(a) is a diagram showing the basic posture when the agent 200 is facing the driver H side (the rear side of the vehicle 1), and Fig. 4(b) is a diagram showing the basic posture when the agent 200 has its back to the driver H (is facing the front side of the vehicle 1). Also, in the basic posture, the agent 200 stands up with the right palm part 226Ra of the right hand part 226R and the left palm part 226La of the left hand part 226L facing the inside of the agent 200, and the back of the right hand part 226Rb and the back of the left hand part 226Lb facing the outside of the agent 200. As described above, the agent 200 can freely rotate between the front posture facing the driver H side and the back posture with its back to the driver H. That is, the agent 200 may be such that the agent 200 itself rotates around the support connector 240, or the agent 200 may be placed on a rotating body and the rotating body may be rotated to rotate from the front posture to the back posture and from the back posture to the front posture.

[0022] In addition, an agent control unit 190, which will be described later, is provided within the agent 200. It controls and operates each joint of the agent 200 so that the agent 200 can assume a predetermined posture. The agent control unit 190 is wirelessly connected to a control unit 110 (ECU), which will be described later, and operates the agent 200 under the control of the control unit 110. In this embodiment, the agent control unit 190 and the control unit 110 are wirelessly connected. However, for example, they may be wired-connected through a communication line within a support connector 240.

[0023] Also, an agent speaker 191, which will be described later, is provided within the head 211 of the agent 200, and voice output is performed under the control of the agent control unit 190. Note that this agent speaker 191 is not limited to being within the head 211 of the agent 200 and may be provided at other parts of the agent 200 or at other locations of the vehicle 1. However, if the agent speaker 191 is provided within the head 211 of the agent 200, it is possible to make the driver H more realistically perceive the speech from the agent 200, making it easier to empathize and greatly contributing to building the driver H's trust in the agent 200.

[0024] Next, the configuration of the agent device 100 according to the present invention will be described with reference to the block diagram of FIG. 5.

[0025] As shown in FIG. 5, the agent device 100 includes a control unit 110, a surrounding environment recognition unit 120, an occupant state recognition unit 130, a vehicle state recognition unit 140, a speaker 150, a microphone 160, a storage unit 170, a transceiver 180, an agent control unit 190, and an agent speaker 191.

[0026] Note that the agent device 100 shown in FIG. 5 is only an example, and the components of the agent device 100 can be changed as appropriate. For example, the agent device 100 of the present invention can be realized by only one of the speaker 150 and the agent speaker 191. Further, in addition to the microphone 160, an agent microphone may be provided in the agent 200, or the agent device 100 of the present invention can be realized even if only the agent microphone provided in the agent 200 is provided instead of the microphone 160. Furthermore, in the present embodiment, both the control unit 110 and the agent control unit 190 are provided, but the present invention is not limited to this, and only one of the control unit 110 or the agent control unit 190 may be provided to perform the functions of both.

[0027] (Control Unit 110) The control unit 110 includes a CPU, a ROM, a RAM (e.g., a ring buffer), an EEPROM, input / output ports, etc., which are not shown in the figure. For example, when information is input from an input port, the control unit 110 controls various devices via an output port based on a control program read from the ROM.

[0028] Note that the ROM of the control unit 110 stores the operations performed by the agent 200 and a dialogue data table (not shown). Further, a learning function such as AI may be provided to perform operations other than the stored operations and dialogues. The CPU of the control unit 110 determines the operations and dialogues to be performed by the agent 200 from the data table based on the information acquired from each recognition unit described later and the information acquired from the microphone 160. For example, when the speech from the driver H is acquired from the microphone 160, the gesture of "nodding" is determined from the data table, and when the speech of the driver H ends, a dialogue such as "I understand" is determined. Note that the operations performed by the agent 200 and the dialogue data table may be stored in the ROM of the agent control unit 190.

[0029] (Peripheral Environment Recognition Unit 120) The surrounding environment recognition unit 120 is provided to recognize the surrounding environment of the vehicle 1 (own vehicle). The surrounding environment recognition unit 120 includes an in-vehicle camera 120a, a radar 120b, an air temperature sensor 120c, a weather sensor 120d, and a navigation device 121, and the surrounding environment and surrounding situation of the vehicle 1 can be recognized by these devices. For example, the surrounding environment recognition unit 120 can detect the presence of corners or turning angles in the traveling direction, the presence of approaching objects, and the like. In this embodiment, the navigation device 121 is configured as a part of the surrounding environment recognition unit 120. However, the present invention is not limited to this, and the navigation device 121 may be separate from the surrounding environment recognition unit 120 and may be provided independently.

[0030] (In-vehicle camera 120a) The in-vehicle camera 120a is attached to, for example, an interior rearview mirror (not shown) and can capture the front and rear of the vehicle 1. The captured image information is input to the control unit 110, and the control unit 110 stores the image information in the RAM. Thereby, the control unit 110 can recognize the situations in front of and behind the vehicle 1 in real time and retrospectively.

[0031] (Radar 120b) For the radar 120b, for example, a millimeter-wave radar that emits radio waves to detect obstacles or the like is used. The millimeter-wave radar is attached to the front bumper or rear bumper of the vehicle 1, and can perform front monitoring of the vehicle 1, front side monitoring of the vehicle 1, and rear side monitoring of the vehicle 1. The monitoring information is input to the control unit 110, and the control unit 110 stores the monitoring information in the RAM. Thereby, the control unit 110 can recognize the situations in front of the vehicle 1, on the front side of the vehicle 1, and on the rear side of the vehicle 1 in real time and retrospectively. In this embodiment, a millimeter-wave radar is used, but other radars may be used. For example, an infrared radar may be used.

[0032] (Air temperature sensor 120c) The temperature sensor 120c is attached, for example, to the rear part of the vehicle 1 and is capable of detecting the outside temperature of the vehicle 1. Then, the detected temperature is input to the control unit 110, and the control unit 110 stores the temperature in the RAM. Thereby, the control unit 110 can recognize the outside temperature of the vehicle 1.

[0033] (Weather sensor 120d) The weather sensor 120d is attached, for example, to the upper part of the vehicle 1 and is capable of detecting raindrops or the like falling on the vehicle 1. Then, the detected water droplet information is input to the control unit 110, and the control unit 110 stores the water droplet information in the RAM. Thereby, the control unit 110 can recognize the weather conditions outside the vehicle 1.

[0034] (Navigation device 121) The navigation device 121 provides route guidance according to the destination, waypoints, etc. Also, the navigation device 121 has map information, GPS, a VICS receiver, etc., and can acquire the current position of the vehicle 1, surrounding map information, traffic jam information, road regulation information, parking lot information, etc. Therefore, in addition to the navigation device 121, the control unit 110 can recognize the surrounding situation of the vehicle 1 in real time based on the information acquired by the external camera 120a for vehicle photography, the radar 120b, etc. Note that the navigation device 121 can also receive map information at any time and acquire the latest map information. Also, the transmission / reception unit of the navigation device 121 may be shared with the transceiver 180 described later.

[0035] As described above, the control unit 110 can recognize the surrounding environment and situation of the vehicle 1 in real time and retrospectively. Thereby, based on the recognition of the surrounding environment and situation of the vehicle 1, the control unit 110 can control the agent 200 and the agent speaker 191 to notify the driver H of the information on the surrounding environment and situation of the vehicle 1. For example, if there is a falling object in front of the vehicle 1, it is possible to notify the driver by gestures and voice such as "There is a falling object in front". Thereby, safety can be improved.

[0036] In addition, in the present embodiment, as the surrounding environment recognition unit 120, an in-vehicle camera 120a for external shooting, a radar 120b, an air temperature sensor 120c, a weather sensor 120d, and a navigation device 121 are listed, but this is only an example, and of course other devices may be used.

[0037] (Occupant state recognition unit 130) The occupant state recognition unit 130 is provided to recognize the state of the driver H. Further, the occupant state recognition unit 130 includes an in-vehicle camera 130a for occupant shooting and a vital sensor 130b, and can recognize the state of the driver H by these devices.

[0038] (In-vehicle camera 130a for occupant shooting) The in-vehicle camera 130a for occupant shooting is attached to the instrument panel 3, for example, and can shoot the driver H. Then, the captured image information is input to the control unit 110, and the control unit 110 stores the image information in the RAM. Thereby, the control unit 110 can recognize the state of the driver H in real time and retrospectively. Here, the state of the driver H specifically includes the state of the driver H's eyelids, the number of blinks, the direction of the line of sight, the direction of the face, and whether the driver is in a hunched state or an immobile state.

[0039] (Vital sensor 130b) The vital sensor 130b is attached, for example, to a part grasped by the driver H of the steering wheel 4, and is capable of acquiring vital information such as the heart rate and blood pressure of the driver H. Then, the acquired vital information is input to the control unit 110, and the control unit 110 stores the vital information in the RAM. Thereby, the control unit 110 can recognize the state of the driver H in real time and retrospectively.

[0040] As described above, the control unit 110 can recognize the state of the driver H in real time and retrospectively. Thereby, based on the recognition of the state of the driver H, the control unit 110 can control the agent 200 and the agent speaker 191 to notify the driver H of predetermined information. For example, notifications by images and sounds such as "Your eyelids are drooping. Won't you take a break?" and "Your heart rate is faster than usual. Won't you take a break?" can be made. This leads to an improvement in safety.

[0041] The occupant state recognition unit 130 can recognize the thinking and emotions of the driver H within a certain range based on the information acquired from the in-vehicle camera 130a for photographing the occupant, the vital sensor 130b, and the information input from the microphone 160. For example, the facial expression of the driver H is acquired from the in-vehicle camera 130a for photographing the occupant, the heart rate and blood pressure of the driver H are acquired from the vital sensor 130b, the volume and input content are acquired from the microphone 160, and it can be recognized whether the driver H is in normal thinking and emotions or in thinking and emotions different from normal (for example, surprised, angry, etc.) from the acquired information.

[0042] In this embodiment, the in-vehicle camera 130a for photographing the occupant and the vital sensor 130b are cited as the occupant state recognition unit 130, but this is only an example, and of course, other devices may be used.

[0043] (Vehicle state recognition unit 140) The vehicle state recognition unit 140 is provided to recognize the state of the vehicle 1. The vehicle state recognition unit 140 includes a vehicle speed sensor 140a, a steering angle sensor 140b, an accelerator pedal sensor 140c, a brake pedal sensor 140d, a G sensor 140e, and a shift position sensor 140f, and the state of the vehicle 1 can be recognized by these devices.

[0044] (Vehicle speed sensor 140a) The vehicle speed sensor 140a is a sensor for detecting the vehicle speed of the vehicle 1, and the detected vehicle speed is input to the control unit 110 as a vehicle speed signal, and the control unit 110 stores the vehicle speed information in the RAM. Thereby, the control unit 110 can recognize the vehicle speed of the vehicle 1 in real time and retrospectively.

[0045] (Steering angle sensor 140b) The steering angle sensor 140b is a sensor for detecting the steering angle (the angle of the steering wheel 4) of the vehicle 1, and the detected steering angle is input to the control unit 110 as an angle signal, and the control unit 110 stores the angle information in the RAM. Thereby, the control unit 110 can recognize the steering angle (the angle of the steering wheel 4) of the vehicle 1 in real time and retrospectively.

[0046] (Accelerator pedal sensor 140c) The accelerator pedal sensor 140c is a sensor for detecting the depression amount of an accelerator pedal (not shown), and the detected depression amount is input to the control unit 110 as a depression amount signal, and the control unit 110 stores the depression amount information in the RAM. Thereby, the control unit 110 can recognize the depression amount of the accelerator pedal of the vehicle 1 in real time and retrospectively.

[0047] (Brake pedal sensor 140d) The brake pedal sensor 140d is a sensor for detecting the depression amount of a brake pedal (not shown). The detected depression amount is input as a depression amount signal to the control unit 110, and the control unit 110 stores the depression amount information in the RAM. Thereby, the control unit 110 can recognize the depression amount of the brake pedal of the vehicle 1 in real time and retrospectively.

[0048] (G sensor 140e) The G sensor 140e is a sensor for detecting the acceleration, deceleration, and inclination of the vehicle 1. When acceleration is detected, the acceleration amount, when deceleration is detected, the deceleration amount, and when inclination is detected, the inclination angle amount are input as an acceleration amount signal, a deceleration amount signal, and an inclination angle signal, respectively, to the control unit 110, and the control unit 110 stores the acceleration information, deceleration information, and inclination information in the RAM. Thereby, the control unit 110 can recognize the acceleration, deceleration, and inclination of the vehicle 1 in real time and retrospectively.

[0049] (Shift position sensor 140f) The shift position sensor 140f is a sensor for detecting the shift range (shift position) of the vehicle 1. That is, the shift position sensor 140f detects which position the shift range is set to, such as the parking range (parking position), drive range (driving position), reverse range (reverse position), neutral range (neutral position), etc. The shift range detected by the shift position sensor 140f is input to the control unit 110, and the control unit 110 stores the current shift range in the RAM. Thereby, the control unit 110 can recognize which position the shift range is set to.

[0050] As described above, the control unit 110 can recognize the state of the vehicle 1 in real time and retrospectively. Thereby, based on the recognition of the state of the vehicle 1, the control unit 110 can control the agent 200 and the agent speaker 191 to notify the driver H of the information on the state of the vehicle 1. For example, if the vehicle is traveling at an appropriate speed, notification can be made by gestures and voices such as "You are driving at an appropriate speed." This leads to an improvement in safety.

[0051] In this embodiment, as the vehicle state recognition unit 140, a vehicle speed sensor 140a, a steering angle sensor 140b, an accelerator pedal sensor 140c, a brake pedal sensor 140d, a G sensor 140e, and a shift position sensor 140f are listed, but this is only an example, and of course, other devices may be used.

[0052] (Speaker 150) The speaker 150 is attached to the instrument panel 3, for example, and outputs voices other than the voice emitted from the agent 200, warning sounds, and the like. Note that, instead of providing the speaker 150, an audio speaker built in the vehicle 1 may be used. Further, as will be described later, the speaker 150 may also serve as the agent speaker 191.

[0053] (Microphone 160) The microphone 160 is attached to the instrument panel 3, for example, and inputs voices emitted from the driver H and other passengers.

[0054] (Memory unit 170) The memory unit 170 can store information acquired from each of the above-described recognition units, as well as conversations conducted between the driver H and the control unit 110. By accumulating this information in the memory unit 170, the control unit 110 can recognize the driving tendency of the driver H (for example, what kind of driver the driver is) and the hobbies and preferences of the driver H (for example, what kind of BGM the driver likes). Also, by recognizing these, the agent device 100 side (agent 200) can actively conduct conversations in accordance with the driving tendency of the driver H and the hobbies and preferences of the driver H.

[0055] (Transceiver 180) The transceiver 180 can, for example, acquire information using an in-vehicle wireless LAN or acquire position information using a satellite positioning system. Based on the information thus acquired and the information stored in the above-described memory unit 170, the control unit 110 can also actively conduct conversations from the agent device 100 side (agent 200) in accordance with the driving tendency of the driver H and the hobbies and preferences of the driver H.

[0056] (Agent control unit 190) The agent control unit 190 includes a CPU, ROM, RAM, input / output ports, etc. (not shown). For example, when information is input from the control unit 110, it can operate each part of the agent 200 or cause a predetermined voice output to be performed to the agent speaker 191.

[0057] (Agent speaker 191) As described above, the agent speaker 191 is provided inside the head 211 of the agent 200, and voices and the like emitted from the agent 200 are output. Note that, as in this embodiment, by providing the agent speaker 191 inside the head 211 of the agent 200, particularly in the part corresponding to the mouth, it becomes more realistic and contributes to the establishment of a trust relationship. However, if the agent 200 is small and it is difficult to provide it inside the head 211, the above-described speaker 150 may be used as a substitute, or an agent speaker 191 or a speaker serving as a substitute thereof may be provided in other parts inside the vehicle 1.

[0058] Next, the operations and interactions of the agent 200 in the agent device 100 will be described. The operations and interactions of the agent 200 are performed by control processing by the control unit 110. Each part of the agent 200 is operated by the agent control unit 190, and voices are output from the agent speaker 191. Note that the control flow for performing the control processing is stored in the ROM of the control unit 110, and the CPU of the control unit 110 reads it from the ROM and performs various processes.

[0059] Hereinafter, with reference to FIG. 6, the main processing of the agent control processing performed by the CPU of the control unit 110 will be described.

[0060] (Step S10) In step S10, when the ignition is turned on, the CPU of the control unit 110 starts the main processing of the agent control processing. Specifically, the CPU of the control unit 110 reads the main processing of the agent control processing from the ROM and starts the processing, and then transfers the processing to step S100.

[0061] (Step S100) In step S100, the CPU of the control unit 110 performs boarding processing. In the boarding processing, the processing at the start when the driver H boards the vehicle 1 is performed. For example, in the boarding processing, the CPU of the control unit 110 performs processing to face the agent 200 towards the driver H. Also, the CPU of the control unit 110 performs processing such as having a conversation with the driver H and causing the agent 200 to speak and act on boarding explanation items such as the weather, remaining fuel information, and notable points of vehicle maintenance. Then, when the boarding processing is completed, the processing proceeds to step S200.

[0062] (Step S200) In step S200, the CPU of the control unit 110 performs pre - driving start processing. In the pre - driving start processing, the processing before the driver H starts driving is performed. For example, in the pre - driving start processing, explanations about precautions for the driving environment to the destination and rest points are given. In the explanation of the pre - driving start processing, the agent 200 faces the driver H, moves both hands for each item to explain, and also gives a signal of completion such as nodding the head up and down for each item end. Then, when the pre - driving start processing is completed, the processing proceeds to step S20.

[0063] (Step S20) In step S20, the CPU of the control unit 110 performs processing to determine whether the shift range is "P" (parking range). Specifically, the CPU of the control unit 110 determines whether the shift range input from the shift position sensor 140f is the parking range. If the CPU of the control unit 110 determines that the shift range is not the parking range (in this case, the shift position has been changed), the processing proceeds to step S300. If it determines that the shift range is the parking range (in this case, the shift position has not been changed), the processing proceeds to step S400.

[0064] (Step S300) In step S300, the CPU of the control unit 110 performs driving processing. In the driving processing, control processing of the agent 200 during driving is performed. For example, when the shift range input from the shift position sensor 140f is "D" (drive range), the CPU of the control unit 110 performs control processing of the agent 200 during forward driving, such as making the agent 200 face forward (the traveling direction of the vehicle 1) (assuming a predetermined posture), and when the shift range is "R" (reverse range), control processing of the agent 200 during reverse driving is performed. Details of the driving processing will be described later. Then, after the driving processing is completed, the process proceeds to step S30.

[0065] (Step S30) In step S30, the CPU of the control unit 110 performs a process of determining whether the shift range is "P" (parking range). As described above, the CPU of the control unit 110 determines whether the shift range input from the shift position sensor 140f is the parking range. If the CPU of the control unit 110 determines that the shift range is not the parking range, the process proceeds to step S300, and if it determines that the shift range is the parking range, the process proceeds to step S400.

[0066] (Step S400) In step S400, the CPU of the control unit 110 performs end-of-driving processing. In the end-of-driving processing, processing at the end when a predetermined driving is completed is performed. For example, in the end-of-driving processing, the CPU of the control unit 110 makes the agent 200 face the driver H again and assume the basic posture. Also, the CPU of the control unit 110 performs voice output and operations for rewarding at the end of driving, and if there are abnormalities or notification information in the vehicle, it performs notification of abnormalities, sounding of warning sounds, and notification of information by voice or operations. Then, after the end-of-driving processing is completed, the process proceeds to step S40.

[0067] (Step S40) In step S40, the CPU of the control unit 110 performs a process of determining whether the shift range is "P" (parking range). As described above, the CPU of the control unit 110 determines whether the shift range input from the shift position sensor 140f is the parking range. If the CPU of the control unit 110 determines that the shift range is not the parking range (in this case, the shift position has been changed), the process proceeds to step S300. If it is determined that the shift range is the parking range (in this case, the shift position has not been changed), the process proceeds to step S50.

[0068] (Step S50) In step S50, the CPU of the control unit 110 performs a driving restart determination process. In the driving restart determination process, a process of determining whether the driver H has the intention to restart driving is performed. For example, when the CPU of the control unit 110 receives a voice input such as "Restart driving" from the driver H, or when a new destination is input to the navigation system by voice input or manual input, etc., it is determined that there is an intention to restart driving. If the CPU of the control unit 110 determines that there is an intention to restart driving, the process proceeds to step S200. If it is determined that there is no intention to restart driving, the process proceeds to step S60.

[0069] (Step S60) In step S60, the CPU of the control unit 110 determines whether the ignition has been turned off. If the CPU of the control unit 110 determines that the ignition has been turned off, the main process of the agent control process is terminated. If the ignition has not been turned off, the process proceeds to step S40.

[0070] Next, with reference to FIG. 7, the running process performed by the CPU of the control unit 110 will be described. FIG. 7 is a subroutine of step S300 (running process) in FIG. 6.

[0071] (Step S301) During driving, first, in step S301, the CPU of the control unit 110 performs start-up processing. In the start-up processing, a warning is given at start-up. That is, at start-up, it is easy for the thoughts and actions (including past operations) of driver H to not match. Specifically, elderly people or the like may mistakenly think that the shift position (such as "D" (Drive Range) and "R" (Reverse Range) in the shift range), and collisions or the like may occur. Here, in order to eliminate the assumption of safety, start-up processing is performed. For example, the CPU of the control unit 110 performs an action confirmation by reading out the shift range during start-up processing. Also, the CPU of the control unit 110 prompts a visual confirmation of the traveling direction. Thereby, even if the vehicle 1 is equipped with a mis-start function, the misoperation by driver H itself can be suppressed, and the safety can be improved.

[0072] (Step S302) In step S302, the CPU of the control unit 110 performs a process of determining whether the shift range is "D" (Drive Range). Specifically, the CPU of the control unit 110 determines whether the shift range input from the shift position sensor 140f is the drive range. When the CPU of the control unit 110 determines that the shift range is not the drive range, the process proceeds to step S303, and when it determines that the shift range is the drive range, the process proceeds to step S310.

[0073] (Step S303) In step S303, the CPU of the control unit 110 performs non-drive range processing. In the non-drive range processing, processing during driving other than when the shift range is "D" (Drive Range) is performed. For example, if the shift range is "R" (Reverse Range), the CPU of the control unit 110 performs reverse range processing and the like. In the reverse range processing, processing assuming parking and the like is performed. Details of the non-drive range processing will be omitted. Then, when the non-drive range processing is completed, the driving processing is terminated.

[0074] (Step S310) In step S310, the CPU of the control unit 110 performs in-driving processing. In the in-driving processing, control processing of the agent 200 during driving is performed according to the surrounding situation. For example, the CPU of the control unit 110 performs notification for approaching objects and guidance for avoidance actions during the driving of the vehicle 1. Details of the in-driving processing will be described later. Then, when the in-driving processing ends, the driving-time processing ends.

[0075] Next, with reference to FIG. 8, the in-driving processing performed by the CPU of the control unit 110 will be described. FIG. 8 is a subroutine of step S310 (in-driving processing) in FIG. 7.

[0076] (Step S311) In the in-driving processing, first, in step S311, the CPU of the control unit 110 performs a process of determining whether there is an approaching object. Specifically, the CPU of the control unit 110 determines whether there is something (such as a person) approaching the vehicle 1 within a predetermined range and at a speed equal to or higher than a predetermined speed from the information on the surrounding situation input from the in-vehicle camera 120a for external shooting, the radar 120b, etc. That is, the CPU of the control unit 110 determines whether it is necessary to cause the driver H to take corresponding measures according to the acquired surrounding situation. If the CPU of the control unit 110 determines that there is an approaching object, the process proceeds to step S314, and if it determines that there is no approaching object, the process proceeds to step S312.

[0077] (Step S312) In step S312, the CPU of the control unit 110 performs normal in-driving processing. In the normal in-driving processing, normal in-driving processing is performed when there is no need for a predetermined avoidance action. For example, in the normal in-driving processing, guidance to the destination, guidance to the right / left turn points, processing for instructions by the driver H, daily conversations, and drowsiness prevention measures are performed.

[0078] (Step S313) Subsequent to the normal driving process, in step S313, the CPU of the control unit 110 performs a process of determining whether the vehicle speed is "0". Specifically, the CPU of the control unit 110 determines whether the vehicle 1 has stopped based on the vehicle speed input from the vehicle speed sensor 140a. If the CPU of the control unit 110 determines that the vehicle speed is not "0", the process proceeds to step S311 and the above process is repeated. If the CPU of the control unit 110 determines that the vehicle speed is "0", the driving process is terminated.

[0079] (Step S314) In step S314, the CPU of the control unit 110 performs a process of determining whether there is an immediate collision possibility. That is, for the vehicle 1, even if there is an approaching object, there are two cases: a case where it is necessary to perform an emergency avoidance process immediately, and a case where there is no need to perform an avoidance action until an avoidance action is taken, or it is not urgent and a predetermined avoidance action can be performed. Therefore, in an emergency, an avoidance action is performed immediately, and if it is not an emergency, corresponding measures are taken according to the surrounding situation. If the CPU of the control unit 110 determines that there is an immediate collision possibility, the process proceeds to step S315. If the CPU of the control unit 110 determines that there is no immediate collision possibility, the process proceeds to step S316.

[0080] (Step S315) In step S315, the CPU of the control unit 110 performs an emergency avoidance process and terminates the driving process. In the emergency avoidance process, since the vehicle 1 may collide with an approaching object in a short time, the CPU of the control unit 110 not only prompts the driver H to take an avoidance action but also directly controls the operation of the vehicle 1. For example, the CPU of the control unit 110 activates the emergency brake, prepares for the activation of the airbag 8, or activates it.

[0081] (Step S316) In step S316, the CPU of the control unit 110 performs the simultaneous transmission operation of the recognition of an approaching object and the avoidance action. Specifically, the CPU of the control unit 110 causes the agent 200 to point at the approaching object with one arm, open the other arm horizontally, turn the palm downward, and bend and stretch the legs. Since the driver H decelerates while the agent 200 points at the approaching object, the cause of the deceleration operation can be grasped, and it is easy to shift to the deceleration operation. Also, since the approaching object is pointed at by the agent 200, the driver H can shift the line of sight for an instant and capture the approaching object in the peripheral vision, and can grasp and avoid the approaching object without hindering driving. Note that the pointing at the approaching object is done for a short time. That is, if the approaching object is stared at, it becomes difficult to shift to the avoidance action. Then, after the simultaneous transmission operation of the recognition of the approaching object and the avoidance action, the process proceeds to step S317.

[0082] (Step S317) In step S317, the CPU of the control unit 110 performs a speech process for the discovery of an approaching object. Specifically, the CPU of the control unit 110 simply outputs the voice of having discovered the approaching object, the features of the approaching object, the direction of the approaching object, etc. Here, the information of the approaching object to be output by voice is transmitted as simply as possible. That is, if there is too much information about the approaching object, it will instead cause confusion, so only the minimum amount of information is used. Since this voice output is added, the driver H will have the information complemented for the pointing at the approaching object (the operation of the agent 200) by the agent 200. Then, when the speech process for the discovery of the approaching object is completed, the process proceeds to step S318.

[0083] (Step S318) In step S318, the CPU of the control unit 110 performs an operation to shift to the avoidance action. That is, the CPU of the control unit 110 causes the agent 200 to perform an operation that can be intuitively understood so that the driver H can shift to the action of avoiding the approaching object.

[0084] Specifically, as shown in FIG. 9, the CPU of the control unit 110 causes the agent 200 to turn the palm downward and extend both arms horizontally. Thereby, the driver H is made to be aware that in addition to concentrating consciousness on the central visual field, peripheral attention is also necessary. Then, while keeping both arms of the agent 200 extended, the CPU of the control unit 110 bends and extends both legs to crouch down. Thereby, an instruction to decelerate can be given, and the driver H can be made to perform a decelerating action. Note that the amount of deceleration can be visualized by changing the amount of bending and extending (the depth of bending and extending) according to the degree of necessity of deceleration.

[0085] Next, as shown in FIG. 10, the CPU of the control unit 110 moves the agent 200's other arm up and down while pointing in the avoidance direction with one arm and keeping the other arm extended. Note that the agent 200 continues to bend and extend. Thereby, while decelerating, the agent 200 is made to recognize the avoidance direction.

[0086] Next, as shown in FIG. 11, the CPU of the control unit 110 causes the agent 200 to hold the steering wheel (steering wheel 4) and assume a posture of stepping on the brake pedal. Then, after causing the agent 200 to face the avoidance direction, the CPU of the control unit 110 points in the avoidance direction with the entire arm to prompt an avoidance action.

[0087] In this way, by causing the agent 200 to execute a series of operations, the driver H can be made to intuitively understand the necessary operations and act quickly. Then, after executing the operations for shifting to the avoidance action, the process proceeds to step S319.

[0088] (Step S319) In step S319, the CPU of the control unit 110 performs a process of determining whether the target approaching object has been avoided. If the CPU of the control unit 110 determines that the target approaching object has been avoided, the process proceeds to step S311. If it determines that the target approaching object has not been avoided, the process proceeds to step S318.

[0089] In steps S318 and S319 described above, after performing an operation to shift to a series of avoidance actions in step S318, a process of determining whether the target approaching object has been avoided is performed in step S319. However, while performing the operation to shift to a series of avoidance actions, a process of determining whether the target approaching object has been avoided may also be performed. For example, after performing the operation to shift to the avoidance action shown in FIG. 9, it is determined whether the target approaching object has been avoided. Next, after performing the operation to shift to the avoidance action shown in FIG. 10, it is determined whether the target approaching object has been avoided. Then, after performing the operation to shift to the avoidance action shown in FIG. 11, it may be determined whether the target approaching object has been avoided.

[0090] As described above, the agent device 100 of the present embodiment enables the agent 200 to not only imitate necessary avoidance actions but also immediately understand the event that caused the need to take avoidance actions, intuitively understand the necessary operations that the driver H should perform, and quickly perform driving operations suitable for the driving environment.

[0091] Note that the agent control process of the present embodiment is executed by the CPU or the like of the control unit 110 after a program stored in the ROM, RAM, EEPROM, etc. of the control unit 110 is expanded in the RAM or the like of the control unit 110. Furthermore, in the present embodiment, the shift position sensor 140f constitutes the shift position detection unit of the present application. Also, in the present embodiment, the surrounding environment recognition unit 120 constitutes the surrounding situation acquisition unit of the present application. Also, in the present embodiment, the control unit 110 constitutes the determination unit of the present application.

[0092] Also, in the present embodiment, the agent control unit of the present application is configured by both the control unit 110 and the agent control unit 190. However, as described above, only the agent control unit 190 may have the functions of both the control unit 110 and the agent control unit 190 of the present embodiment.

Explanation of Reference Numerals

[0093] 1: Vehicle, 100: Agent device, 110: Control unit, 120: Surrounding environment recognition unit, 130: Occupant state recognition unit, 140: Vehicle state recognition unit, 140f: Shift position sensor, 150: Speaker, 160: Microphone, 170: Memory unit, 180: Transceiver, 190: Agent control unit, 191: Agent speaker, 200: Agent

Claims

1. A peripheral situation acquisition unit that acquires peripheral situations, a determination unit that determines whether it is necessary to cause a driver to take a corresponding measure based on the acquired peripheral situations, an agent control unit that controls an anthropomorphic agent, comprising: wherein the agent control unit , when it is determined that it is necessary to cause the driver to take a corresponding measure, causes the agent to point in the direction of the cause for which the corresponding measure needs to be taken, and causes the agent to perform a gesture corresponding to the corresponding measure, , when the determination unit determines that it is necessary to cause the driver to take an avoidance action, moves one arm up and down and indicates the traveling direction with the other arm, , after the avoidance action, causes the agent to assume a posture of gripping the steering wheel and a posture of stepping on the brake pedal, An agent device characterized by the above.

2. The agent control unit , when the determination unit determines that it is necessary to cause the driver to take a deceleration action, spreads both arms and flexes them, and sets the amount of flexion according to the degree of necessity for deceleration, The agent device according to claim 1, characterized by the above.

Images