Agent device

Abstract

To provide an agent device that can clearly convey a determination between a manual operation and an automatic operation and can support the safety of driving.SOLUTION: An agent device 100 includes: a shift position sensor 140f for detecting a shift position; an ambient surrounding recognition unit 120 for acquiring a surrounding situation; a control unit 110 for switching between a manual operation and an automatic operation according to the surrounding situation acquired by the ambient surrounding recognition unit 120 or the instruction of the driver H and determining whether the operation is being an automatic operation; and an agent controller 190 for controlling a personalized agent. The agent controller 190 switches the agent 200 between a standing position and a sitting position when there is a switch between the manual operation and the automatic operation.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an agent device capable of notification by anthropomorphic characters.

Background Art

[0002] Conventionally, various methods have been proposed as methods for prompting attention to passengers (especially drivers) in a driving environment in vehicles such as automobiles. As one of such methods, an agent device or a driving support device has been proposed in which a passenger and an anthropomorphic character (so-called agent) provide information related to driving support according to the passenger's request while having a dialogue.

[0003] For example, in a conventional driving support device, it is considered that it is difficult to intuitively convey a notification of a danger avoidance action by voice or the like, and a small robot is arranged on the dashboard and made to appear as an anthropomorphic agent, and this agent is made to simulate the danger avoidance action that the driver should perform. (See Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, conventional agents have been inorganic or have suddenly appeared to prompt attention, making it difficult to build a trust relationship between the agent and the driver. Also, even if the agent suddenly performs the same action as the avoidance action, if the driver does not understand the reason why the avoidance action must be taken, it is difficult to immediately take that action, and there is a problem that, especially without a trust relationship, it does not lead to an avoidance action. In recent years, the development of autonomous driving has been progressing. However, it is difficult to determine whether the autonomous driving system is operating. For example, there are cases where a small lamp lights up during autonomous driving and goes out when switching to manual driving. However, it is often overlooked, and if the driver's line of sight is shifted to this small lamp, there is a risk of inattentiveness ahead. Also, during autonomous driving at level 2 or the like, in an emergency, the driving authority is suddenly transferred from the autonomous driving to the driver. However, there are many drivers who feel that they will be confused even if they are suddenly told to drive, which is an honest feeling.

[0006] The present invention has been made to solve such conventional problems, and an object thereof is to provide an agent device that can clearly convey the discrimination between manual driving and autonomous driving and support safety during driving.

Means for Solving the Problems

[0007] The agent device according to the present invention includes a shift position detection unit that detects a shift position, a surrounding situation acquisition unit that acquires a surrounding situation, a driving state switching unit that switches between manual driving and autonomous driving based on the surrounding situation acquired by the surrounding situation acquisition unit or a driver's instruction, an autonomous driving determination unit that determines whether it is in an autonomous driving state, and an agent control unit that controls an anthropomorphic agent. The agent control unit switches the agent between a standing posture and a sitting posture when switching between manual driving and autonomous driving.

[0008] According to the present invention, it is possible to provide an agent device that can clearly convey the discrimination between manual driving and autonomous driving and support safety during driving.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same reference numerals in different figures indicate parts having the same function, and duplicate descriptions in each figure are omitted as appropriate.

[0011] (Each Configuration in the Passenger Compartment) As shown in FIGS. 1 and 2, the vehicle 1 has an instrument panel 3 provided on the front side of the vehicle of the driver's seat 2, and a steering wheel 4 disposed between the driver's seat 2 and the instrument panel 3. The steering wheel 4 is rotatably attached to a steering column (not shown) via a steering shaft (not shown). Further, inside the steering wheel 4, an airbag 8 that deploys toward the driver (hereinafter referred to as the driver) H when the vehicle 1 collides or the like is stored. In addition, in the agent device 100 (described later) and the agent 200 (described later) of the present embodiment, although the explanation is based on the dialogue and gestures (actions) with the driver H, the same can be performed for other passengers.

[0012] As shown in FIG. 1, in the vehicle interior of the vehicle 1, two display panels, a first display panel 30 and a second display panel 40, are provided. The first display panel 30 is disposed on the instrument panel 3 on the front side of the driver's seat 2, and the second display panel 40 is disposed on the instrument panel 3 on the left front side of the driver's seat 2.

[0013] (First display panel 30) As shown in FIGS. 1 and 2, the first display panel 30 includes a pointer-type meter configured like an analog clock, and a so-called liquid crystal display device in which a liquid crystal panel and a backlight are integrally provided. The driver H can view various information displayed in the first display area 30a of the first display panel 30 through the upper space portion 4a of the steering wheel 4.

[0014] As shown in FIG. 2, the first display panel 30 is provided with a first display area 30a. On the left and right sides of the first display area 30a, there are two pointer-type meters for displaying information such as the traveling speed (speedometer) of the vehicle 1 and the number of revolutions per unit time of the engine (tachometer). Between the two pointer-type meters and at the center of the first display panel 30, there is a small liquid crystal display device for displaying an image indicating general vehicle information. Note that the first display panel 30 may be configured entirely by a single liquid crystal display device without the pointer-type meters.

[0015] (Second display panel 40) As shown in FIG. 1, the second display panel 40 is configured by, for example, a so-called liquid crystal display device in which a liquid crystal panel and a backlight are integrally provided. Note that map information and the like are displayed on the second display panel 40, which serves as a so-called car navigation system.

[0016] Note that the liquid crystal display devices configured in the first display panel 30 and the second display panel 40 may be configured by self-luminous display devices such as plasma displays and organic ELs, or display devices such as projection-type projectors.

[0017] (Agent 200) Also, as shown in FIG. 3, an agent 200 is disposed above the instrument panel 3 on the front side of the driver's seat 2. Note that the agent 200 may be located anywhere as long as it enters the peripheral vision of the driver H, and does not necessarily have to be disposed directly in front of the driver's seat. It is desirable that the agent 200 is in the peripheral vision of the driver H and is located at a position that does not block the driver H's view outside the vehicle. Also, the arrangement position of the agent 200 may be changeable by a predetermined operation device or voice instruction.

[0018] Agent 200 is an anthropomorphic character and is an informative and interactive three-dimensional object with a humanoid outer shape. In this embodiment, Agent 200 is a three-dimensional object that resembles a human form, but is not limited thereto, and may be a character displayed on a display device such as a display, a three-dimensional display represented by a hologram, a virtual three-dimensional object, or the like.

[0019] As shown in FIG. 4, Agent 200 has a head 211, a right eye part 212R, a left eye part 212L, a neck part 213, a chest part 214, a waist part 215, a hip part 216, a right shoulder part 221R, a left shoulder part 221L, a right upper arm part 222R, a left upper arm part 222L, a right elbow part 223R, a left elbow part 223L, a right forearm part 224R, a left forearm part 224L, a right wrist part 225R, a left wrist part 225L, a right hand part 226R, a left hand part 226L, a right hip joint part 231R, a left hip joint part 231L, a right thigh part 232R, a left thigh part 232L, a right knee part 233R, a left knee part 233L, a right lower leg part 234R, a left lower leg part 234L, a right ankle part 235R, a left ankle part 235L, a right foot part 236R, and a left foot part 236L.

[0020] In addition, a support connector 240 is provided on Agent 200, and Agent 200 is supported so as to be able to stand on the instrument panel 3. Note that ears are provided on both sides of the head 211 of Agent 200. Thereby, it can be made to seem that Agent 200 is listening carefully to the words of the driver H. Further, even if these ears are not actually provided, as will be described later, when Agent 200 listens to the words of the driver H, by placing the right hand part 226R or the left hand part 226L on the side part of the head 211 of Agent 200, it can be made to seem that the hand is placed on the ear and listening, and the trust relationship with the driver H can be deepened.

[0021] Note that FIG. 4(a) is a diagram showing the basic posture when Agent 200 is facing the driver H side (the rear side of the vehicle 1), and FIG. 4(b) is a diagram showing the basic posture when Agent 200 has its back to the driver H (is facing the front side of the vehicle 1). Also, in the basic posture, the agent 200 stands with the right palm part 226Ra of the right hand part 226R and the left palm part 226La of the left hand part 226L facing the inside of the agent 200, and the back of the right hand part 226Rb of the right hand part 226R and the back of the left hand part 226Lb of the left hand part 226L facing the outside of the agent 200. As described above, the agent 200 can freely rotate between the front posture facing the driver H side and the back posture with the back facing the driver H. That is, the agent 200 may be such that the agent 200 itself rotates around the support connector 240, or the agent 200 may be placed on a rotating body and the rotating body may be rotated to rotate from the front posture to the back posture and from the back posture to the front posture. Also, the agent 200 can take a standing posture as shown in FIGS. 4 and 9(a) and a sitting posture as shown in FIG. 9(b). Details of the sitting posture will be described later.

[0022] Also, an agent control unit 190, which will be described later, is provided inside the agent 200 to control and operate each joint of the agent 200 so that the agent 200 can take a predetermined posture. The agent control unit 190 is wirelessly connected to a control unit 110 (ECU), which will be described later, and operates the agent 200 under the control of the control unit 110. In this embodiment, the agent control unit 190 and the control unit 110 are wirelessly connected. However, for example, they may be wired-connected through a communication line inside the support connector 240.

[0023] In addition, an agent speaker 191, which will be described later, is provided in the head 211 of the agent 200, and voice output is performed under the control of the agent control unit 190. Note that the agent speaker 191 is not limited to being provided in the head 211 of the agent 200, and may be provided at other parts of the agent 200 or at other locations in the vehicle 1. However, if the agent speaker 191 is provided in the head 211 of the agent 200, it is possible to make the driver H more realistically perceive the speech from the agent 200, making it easier to empathize, and can greatly contribute to building trust in the agent 200 by the driver H.

[0024] Next, the configuration of the agent device 100 according to the present invention will be described with reference to the block diagram of FIG. 5.

[0025] As shown in FIG. 5, the agent device 100 includes a control unit 110, a surrounding environment recognition unit 120, an occupant state recognition unit 130, a vehicle state recognition unit 140, a speaker 150, a microphone 160, a storage unit 170, a transceiver 180, an agent control unit 190, an agent speaker 191, and a driving state switching operation unit 310.

[0026] Note that the agent device 100 shown in FIG. 5 is merely an example, and the components of the agent device 100 can be appropriately changed. For example, the agent device 100 of the present invention can be realized by only one of the speaker 150 and the agent speaker 191. In addition to the microphone 160, an agent microphone may be provided in the agent 200, or the agent device 100 of the present invention can be realized by providing only the agent microphone provided in the agent 200 instead of the microphone 160. Further, in the present embodiment, both the control unit 110 and the agent control unit 190 are provided, but it is not limited to this, and only one of the control unit 110 or the agent control unit 190 may be provided to perform the functions of both.

[0027] (Control Unit 110) The control unit 110 includes a CPU, ROM, RAM (e.g., a ring buffer), EEPROM, input / output ports, etc. not shown in the figure. For example, when information is input from an input port, it controls various devices via an output port based on a control program read from the ROM.

[0028] In addition, the control unit 110 is configured to switch the driving state between manual driving and automatic driving based on the information input from the surrounding environment recognition unit 120 and the vehicle state recognition unit 140. Note that the control unit 110 may cancel the switch to automatic driving depending on the information input from the occupant state recognition unit 130. Also, the control unit 110 can switch the driving state between manual driving and automatic driving by the switching operation input to the driving state switching operation unit 310. Here, when switched to manual driving, the control unit 110 controls the running of the vehicle 1 based on the input operation of the driver H, and when switched to automatic driving, it controls the running of the vehicle 1 based on the information input from the surrounding environment recognition unit 120, the vehicle state recognition unit 140, etc.

[0029] The ROM of the control unit 110 stores the operations performed by the agent 200 and a dialogue data table (not shown). Also, it may be provided with a learning function such as AI to perform operations other than the stored operations and dialogues. The CPU of the control unit 110 determines the operations and dialogues to be performed by the agent 200 from the data table based on the information acquired from each recognition unit described later and the information acquired from the microphone 160. For example, when the speech from the driver H is acquired from the microphone 160, it determines the gesture of "nodding" from the data table, and when the speech of the driver H ends, it determines a dialogue such as "I understand". Note that the operations performed by the agent 200 and the dialogue data table may be stored in the ROM of the agent control unit 190.

[0030] (Surrounding Environment Recognition Unit 120) The surrounding environment recognition unit 120 is provided to recognize the surrounding environment of the vehicle 1 (own vehicle). The surrounding environment recognition unit 120 also includes an in-vehicle camera 120a for external shooting, a radar 120b, an air temperature sensor 120c, a weather sensor 120d, and a navigation device 121, and the surrounding environment and surrounding situation of the vehicle 1 can be recognized by these devices. For example, the surrounding environment recognition unit 120 can detect the presence of corners or turning angles in the traveling direction, the presence of approaching objects, etc. In this embodiment, the navigation device 121 is configured as a part of the surrounding environment recognition unit 120. However, the present invention is not limited to this, and the navigation device 121 may be separate from the surrounding environment recognition unit 120 and have it independently.

[0031] (In-vehicle camera 120a for external shooting) The in-vehicle camera 120a for external shooting is attached to, for example, an interior mirror (not shown), and can shoot the front and rear of the vehicle 1. Then, the captured image information is input to the control unit 110, and the control unit 110 stores the image information in the RAM. Thereby, the control unit 110 can recognize the situations in front of and behind the vehicle 1 in real time and retrospectively.

[0032] (Radar 120b) As the radar 120b, for example, a millimeter-wave radar that emits radio waves to detect obstacles or the like is used. The millimeter-wave radar is attached to the front bumper or rear bumper of the vehicle 1, and can monitor the front of the vehicle 1, the front side of the vehicle 1, and the rear side of the vehicle 1. Then, the monitoring information is input to the control unit 110, and the control unit 110 stores the monitoring information in the RAM. Thereby, the control unit 110 can recognize the situations in front of, on the front side, and on the rear side of the vehicle 1 in real time and retrospectively. In this embodiment, a millimeter-wave radar is used, but other radars may also be used. For example, an infrared radar may be used.

[0033] (Air temperature sensor 120c) The temperature sensor 120c is attached, for example, to the rear of the vehicle 1 and is capable of detecting the outside temperature of the vehicle 1. Then, the detected temperature is input to the control unit 110, and the control unit 110 stores the temperature in the RAM. Thereby, the control unit 110 can recognize the outside temperature of the vehicle 1.

[0034] (Weather sensor 120d) The weather sensor 120d is attached, for example, to the upper part of the vehicle 1 and is capable of detecting raindrops and the like falling on the vehicle 1. Then, the detected water droplet information is input to the control unit 110, and the control unit 110 stores the water droplet information in the RAM. Thereby, the control unit 110 can recognize the weather conditions outside the vehicle 1.

[0035] (Navigation device 121) The navigation device 121 provides route guidance according to the destination, waypoints, etc. Also, the navigation device 121 has map information, GPS, a VICS receiver, etc., and can acquire the current position of the vehicle 1, surrounding map information, traffic jam information, road regulation information, parking lot information, etc. Therefore, in addition to the navigation device 121, the control unit 110 can recognize the surrounding situation of the vehicle 1 in real time based on the information acquired by the external camera 120a for vehicle photography, the radar 120b, etc. Note that the navigation device 121 can also receive map information at any time and acquire the latest map information. Also, the transceiver unit of the navigation device 121 may be shared with the transceiver 180 described later.

[0036] As described above, the control unit 110 can recognize the surrounding environment and situation of the vehicle 1 in real time and retrospectively. Thereby, based on the recognition of the surrounding environment and situation of the vehicle 1, the control unit 110 can control the agent 200 and the agent speaker 191 to notify the driver H of the information on the surrounding environment and situation of the vehicle 1. For example, if there is a falling object in front of the vehicle 1, it is possible to notify the driver by gestures and voice such as "There is a falling object in front." Thereby, safety can be improved.

[0037] In addition, in the present embodiment, as the surrounding environment recognition unit 120, an in-vehicle camera 120a for photographing the outside of the vehicle, a radar 120b, a temperature sensor 120c, a weather sensor 120d, and a navigation device 121 are exemplified, but this is only an example, and of course, other devices may be used.

[0038] (Occupant state recognition unit 130) The occupant state recognition unit 130 is provided to recognize the state of the driver H. Further, the occupant state recognition unit 130 includes an in-vehicle camera 130a for photographing the occupant and a vital sensor 130b, and can recognize the state of the driver H by these devices.

[0039] (In-vehicle camera 130a for photographing the occupant) The in-vehicle camera 130a for photographing the occupant is attached to the instrument panel 3, for example, and can photograph the driver H. Then, the photographed image information is input to the control unit 110, and the control unit 110 stores the image information in the RAM. Thereby, the control unit 110 can recognize the state of the driver H in real time and retrospectively. Here, the state of the driver H specifically includes, for example, the state of the driver H's eyelids, the number of blinks, the direction of the line of sight, the direction of the face, and whether the driver is in a hunched state or an immobile state.

[0040] (Vital sensor 130b) The vital sensor 130b is attached to a part held by the driver H of the steering wheel 4, for example, and can acquire vital information such as the heart rate and blood pressure of the driver H. Then, the acquired vital information is input to the control unit 110, and the control unit 110 stores the vital information in the RAM. Thereby, the control unit 110 can recognize the state of the driver H in real time and retrospectively.

[0041] As described above, the control unit 110 can recognize the state of the driver H in real time and retrospectively. Thereby, the control unit 110 can control the agent 200 and the agent speaker 191 based on the recognition of the state of the driver H, and can notify the driver H of predetermined information. For example, notifications by images and sounds such as "Your eyelids are drooping. Won't you take a break?" and "Your heart rate is faster than usual. Won't you take a break?" can be made. This leads to an improvement in safety.

[0042] The occupant state recognition unit 130 can recognize the thinking and emotions of the driver H within a certain range based on the information acquired from the in-vehicle camera 130a for photographing the occupant, the vital sensor 130b, and the information input from the microphone 160. For example, the facial expression of the driver H is acquired from the in-vehicle camera 130a for photographing the occupant, the heart rate and blood pressure of the driver H are acquired from the vital sensor 130b, the volume and input content are acquired from the microphone 160, and it can be recognized whether the driver H is in normal thinking and emotions or in thinking and emotions different from normal (for example, surprised, angry, etc.) from the acquired information.

[0043] Note that in this embodiment, the in-vehicle camera 130a for photographing the occupant and the vital sensor 130b are cited as the occupant state recognition unit 130, but this is only an example, and of course, other devices may be used.

[0044] (Vehicle state recognition unit 140) The vehicle state recognition unit 140 is provided to recognize the state of the vehicle 1. Further, the vehicle state recognition unit 140 includes a vehicle speed sensor 140a, a steering angle sensor 140b, an accelerator pedal sensor 140c, a brake pedal sensor 140d, a G sensor 140e, and a shift position sensor 140f, and the state of the vehicle 1 can be recognized by these devices.

[0045] (Vehicle speed sensor 140a) The vehicle speed sensor 140a is a sensor for detecting the vehicle speed of the vehicle 1, and the detected vehicle speed is input to the control unit 110 as a vehicle speed signal, and the control unit 110 stores the vehicle speed information in the RAM. Thereby, the control unit 110 can recognize the vehicle speed of the vehicle 1 in real time and retrospectively.

[0046] (Steering angle sensor 140b) The steering angle sensor 140b is a sensor for detecting the steering angle of the vehicle 1 (the angle of the steering wheel 4), and the detected steering angle is input to the control unit 110 as an angle signal, and the control unit 110 stores the angle information in the RAM. Thereby, the control unit 110 can recognize the steering angle of the vehicle 1 (the angle of the steering wheel 4) in real time and retrospectively.

[0047] (Accelerator pedal sensor 140c) The accelerator pedal sensor 140c is a sensor for detecting the depression amount of an accelerator pedal (not shown), and the detected depression amount is input to the control unit 110 as a depression amount signal, and the control unit 110 stores the depression amount information in the RAM. Thereby, the control unit 110 can recognize the depression amount of the accelerator pedal of the vehicle 1 in real time and retrospectively.

[0048] (Brake pedal sensor 140d) The brake pedal sensor 140d is a sensor for detecting the depression amount of a brake pedal (not shown). The detected depression amount is input as a depression amount signal to the control unit 110, and the control unit 110 stores the depression amount information in the RAM. Thereby, the control unit 110 can recognize the depression amount of the brake pedal of the vehicle 1 in real time and retrospectively.

[0049] (G sensor 140e) The G sensor 140e is a sensor for detecting the acceleration, deceleration, and inclination of the vehicle 1. When acceleration is detected, the acceleration amount, when deceleration is detected, the deceleration amount, and when inclination is detected, the inclination angle amount are input as an acceleration amount signal, a deceleration amount signal, and an inclination angle signal, respectively, to the control unit 110, and the control unit 110 stores the acceleration information, deceleration information, and inclination information in the RAM. Thereby, the control unit 110 can recognize the acceleration, deceleration, and inclination of the vehicle 1 in real time and retrospectively.

[0050] (Shift position sensor 140f) The shift position sensor 140f is a sensor for detecting the shift range (shift position) of the vehicle 1. That is, the shift position sensor 140f detects which position the shift range is set to, such as the parking range (parking position), drive range (driving position), reverse range (reverse position), neutral range (neutral position), etc. The shift range detected by the shift position sensor 140f is input to the control unit 110, and the control unit 110 stores the current shift range in the RAM. Thereby, the control unit 110 can recognize which position the shift range is set to.

[0051] In this way, the control unit 110 can recognize the state of the vehicle 1 in real time and retrospectively. Thereby, based on the recognition of the state of the vehicle 1, the control unit 110 can control the agent 200 and the agent speaker 191 to notify the driver H of the information on the state of the vehicle 1. For example, if the vehicle is traveling at an appropriate speed, notifications can be made by gestures and voices such as "You are traveling at an appropriate speed." This leads to an improvement in safety.

[0052] In the present embodiment, as the vehicle state recognition unit 140, a vehicle speed sensor 140a, a steering angle sensor 140b, an accelerator pedal sensor 140c, a brake pedal sensor 140d, a G sensor 140e, and a shift position sensor 140f are listed, but this is only an example, and of course, other devices may be used.

[0053] (Speaker 150) The speaker 150 is attached to the instrument panel 3, for example, and outputs voices other than the voice emitted from the agent 200, warning sounds, etc. Note that instead of providing the speaker 150, the audio speaker built into the vehicle 1 may be used. Also, as will be described later, the speaker 150 may also serve as the agent speaker 191.

[0054] (Microphone 160) The microphone 160 is attached to the instrument panel 3, for example, and inputs voices emitted from the driver H and other passengers.

[0055] (Memory unit 170) The memory unit 170 can store information acquired from each of the above-described recognition units, as well as conversations and the like carried out between the driver H and the control unit 110. By accumulating this information in the memory unit 170, the control unit 110 can recognize the driving tendency of the driver H (for example, what kind of driver the driver is) and the hobbies and preferences of the driver H (for example, what kind of BGM the driver likes). Also, by recognizing these, the agent device 100 side (agent 200) can actively conduct conversations in accordance with the driving tendency of the driver H and the hobbies and preferences of the driver H.

[0056] (Transceiver 180) The transceiver 180 can, for example, acquire information using an in-vehicle wireless LAN or acquire position information using a satellite positioning system. Based on the information thus acquired and the information stored in the above-described memory unit 170, the control unit 110 can also actively conduct conversations in accordance with the driving tendency of the driver H and the hobbies and preferences of the driver H from the agent device 100 side (agent 200).

[0057] (Agent control unit 190) The agent control unit 190 includes a CPU, ROM, RAM, input / output ports, etc. (not shown). For example, when information is input from the control unit 110, it can operate each part of the agent 200 or cause a predetermined voice output to be performed to the agent speaker 191.

[0058] (Agent speaker 191) As described above, the agent speaker 191 is provided inside the head 211 of the agent 200, and outputs sounds and the like emitted from the agent 200. Note that, as in the present embodiment, by providing the agent speaker 191 inside the head 211 of the agent 200, particularly in the part corresponding to the mouth, it becomes more realistic and contributes to the construction of a trust relationship. However, if the agent 200 is small and it is difficult to provide it inside the head 211, the above-described speaker 150 may be used as a substitute, or an agent speaker 191 or a speaker serving as a substitute therefor may be provided in other parts of the vehicle 1.

[0059] (Operation state switching operation unit 310) The operation state switching operation unit 310 inputs an instruction for switching the operation state between manual driving and automatic driving, and inputs an instruction for switching from automatic driving to manual driving or an instruction for switching from manual driving to automatic driving to the control unit 110 according to the input operation. Thereby, the driver H can switch between manual driving and automatic driving according to his / her intention.

[0060] Next, the operation and interaction of the agent 200 in the agent device 100 will be described. The operation and interaction of the agent 200 are performed by control processing by the control unit 110. Each part of the agent 200 is operated by the agent control unit 190, and at the same time, sound is output from the agent speaker 191. Note that the control flow for performing the control processing is stored in the ROM of the control unit 110, and the CPU of the control unit 110 reads it from the ROM and performs various processes.

[0061] Hereinafter, with reference to FIG. 6, the main processing of the agent control processing performed by the CPU of the control unit 110 will be described.

[0062] (Step S10) In step S10, when the ignition is turned on, the CPU of the control unit 110 starts the main process of the agent control process. Specifically, the CPU of the control unit 110 reads the main process of the agent control process from the ROM and starts the process, and then transfers the process to step S100.

[0063] (Step S100) In step S100, the CPU of the control unit 110 performs the boarding process. In the boarding process, the process at the start when driver H boards the vehicle 1 is performed. For example, in the boarding process, the CPU of the control unit 110 performs the process of making the agent 200 face driver H. In addition, the CPU of the control unit 110 performs processes such as having a conversation with driver H and causing the agent 200 to speak and act on boarding explanation items such as the weather, remaining fuel information, and special notes on vehicle maintenance. Note that at the time of boarding, the agent 200 is in a standing posture. After finishing the boarding process, the process is transferred to step S200.

[0064] (Step S200) In step S200, the CPU of the control unit 110 performs the pre-driving start process. In the pre-driving start process, the process before driver H starts driving is performed. For example, in the pre-driving start process, explanations about precautions for the driving environment to the destination and rest points are given. In the explanation of the pre-driving start process, the agent 200 faces driver H, moves both hands for each item to explain, and also gives a signal of completion such as nodding up and down for each item end. After finishing the pre-driving start process, the process is transferred to step S20.

[0065] (Step S20) In step S20, the CPU of the control unit 110 performs a process of determining whether the shift range is "P" (parking range). Specifically, the CPU of the control unit 110 determines whether the shift range input from the shift position sensor 140f is the parking range. If the CPU of the control unit 110 determines that the shift range is not the parking range (in this case, the shift position has been changed), the process proceeds to step S300. If it determines that the shift range is the parking range (in this case, the shift position has not been changed), the process proceeds to step S400.

[0066] (Step S300) In step S300, the CPU of the control unit 110 performs driving processing. In the driving processing, control processing of the agent 200 during driving is performed. For example, when the shift range input from the shift position sensor 140f is "D" (drive range), the CPU of the control unit 110 performs control processing of the agent 200 during forward driving, such as directing the agent 200 forward (in the traveling direction of the vehicle 1) (causing it to take a predetermined posture). When the shift range is "R" (reverse range), control processing of the agent 200 during reverse driving is performed. Details of the driving processing will be described later. Note that during driving, the agent 200 is in a standing posture during manual driving and will be in a sitting posture as described later during automatic driving. After completing the driving processing, the process proceeds to step S30.

[0067] (Step S30) In step S30, the CPU of the control unit 110 performs a process of determining whether the shift range is "P" (parking range). As described above, the CPU of the control unit 110 determines whether the shift range input from the shift position sensor 140f is the parking range. If the CPU of the control unit 110 determines that the shift range is not the parking range, the process proceeds to step S300. If it determines that the shift range is the parking range, the process proceeds to step S400.

[0068] (Step S400) In step S400, the CPU of the control unit 110 performs end-of-operation processing. In the end-of-operation processing, processing at the end when a predetermined driving is completed is performed. For example, in the end-of-operation processing, the CPU of the control unit 110 makes the agent 200 face the driver H again and assume the basic posture. Also, the CPU of the control unit 110 performs voice output and operations for the efforts at the end of driving, and when there are abnormalities or notification information in the vehicle, etc., notifies the abnormalities, emits warning sounds, and notifies the information by voice or operations. Then, when the end-of-operation processing is completed, the process proceeds to step S40.

[0069] (Step S40) In step S40, the CPU of the control unit 110 performs a process of determining whether the shift range is "P" (parking range). As described above, the CPU of the control unit 110 determines whether the shift range input from the shift position sensor 140f is the parking range. If the CPU of the control unit 110 determines that the shift range is not the parking range (in this case, the shift position has been changed), the process proceeds to step S300, and if it determines that the shift range is the parking range (in this case, the shift position has not been changed), the process proceeds to step S50.

[0070] (Step S50) In step S50, the CPU of the control unit 110 performs a driving restart determination process. In the driving restart determination process, a process of determining whether the driver H has the intention to restart driving is performed. For example, the CPU of the control unit 110 determines that there is an intention to restart driving when voice input such as "Restart driving" is received from the driver H, or when a new destination is input to the navigation system by voice input or manual input. If the CPU of the control unit 110 determines that there is an intention to restart driving, the process proceeds to step S200, and if it determines that there is no intention to restart driving, the process proceeds to step S60.

[0071] (Step S60) In step S60, the CPU of the control unit 110 determines whether the ignition has been turned off. If the ignition has been turned off, the CPU of the control unit 110 ends the main process of the agent control process. If the ignition has not been turned off, the process proceeds to step S40.

[0072] Next, with reference to FIG. 7, the running process performed by the CPU of the control unit 110 will be described. FIG. 7 is a subroutine of step S300 (running process) in FIG. 6.

[0073] (Step S301) In the running process, first, in step S301, the CPU of the control unit 110 performs a process of determining whether it is in the automatic driving state. Specifically, the CPU of the control unit 110 detects whether the current driving state is the "automatic driving state" or the "manual driving state". If it is in the "automatic driving state", it is determined that it is in the automatic driving. If it is in the "manual driving state", it is determined that it is not in the automatic driving. If the CPU of the control unit 110 determines that it is in the automatic driving, the process proceeds to step S370. If it determines that it is not in the automatic driving, the process proceeds to step S302.

[0074] (Step S302) In step S302, the CPU of the control unit 110 performs a process of determining whether a switching operation to autonomous driving has been performed or whether the switching conditions for autonomous driving are satisfied. Specifically, the CPU of the control unit 110 determines whether a switching operation to autonomous driving has been performed by the driving state switching operation unit 310, or whether the autonomous driving transition conditions are satisfied based on the information obtained from the surrounding environment recognition unit 120, the occupant state recognition unit 130, the vehicle state recognition unit 140, etc. The autonomous driving transition conditions are preset conditions for transitioning to autonomous driving. For example, they are conditions set such as when the vehicle is traveling in a traffic jam at "60 km / h" or less on a highway. Note that even if a switching operation to autonomous driving is performed, the CPU of the control unit 110 may not perform the switching to autonomous driving if a predetermined condition is not satisfied. If the CPU of the control unit 110 determines that a switching operation to autonomous driving or the switching conditions for autonomous driving are satisfied, the process proceeds to step S360. If it is determined that the switching operation to autonomous driving has not been performed and the switching conditions for autonomous driving are not satisfied, the process proceeds to step S305.

[0075] (Step S360) In step S360, the CPU of the control unit 110 performs processing at the time of autonomous driving switching. In the processing at the time of autonomous driving switching, warning notifications and the like are performed when switching from manual driving to autonomous driving. Details of the processing at the time of autonomous driving switching will be described later. Then, when the processing at the time of autonomous driving switching is completed, the process proceeds to step S370.

[0076] (Step S370) In step S370, the CPU of the control unit 110 performs processing during autonomous driving. In the processing during autonomous driving, control processing of the vehicle 1 and the agent 200 during autonomous driving is performed. For example, the CPU of the control unit 110 performs operation processing of the agent 200 during autonomous driving. Details of the processing during autonomous driving will be described later. Then, when the processing during autonomous driving is completed, the process proceeds to step S303.

[0077] (Step S303) In step S303, the CPU of the control unit 110 performs a process of determining whether or not to end the automatic driving. Specifically, the CPU of the control unit 110 determines whether an operation to switch to the means driving has been performed by the driving state switching operation unit 310, or whether the automatic driving end condition has been satisfied based on the information obtained from the surrounding environment recognition unit 120, the occupant state recognition unit 130, the vehicle state recognition unit 140, etc. Here, the automatic driving end condition is different from the above-mentioned automatic driving transition condition. For example, the establishment of the automatic driving end condition may be considered as the case where the automatic driving transition condition is not satisfied. When the CPU of the control unit 110 determines that the automatic driving has ended, the process proceeds to step S380. When it determines that the automatic driving has not ended, the process proceeds to step S370.

[0078] (Step S380) In step S380, the CPU of the control unit 110 performs post-processing at the end of the automatic driving. In the post-processing at the end of the automatic driving, warning notifications at the time of switching from automatic driving to manual driving are performed. Details of the post-processing at the end of the automatic driving will be described later. After the post-processing at the end of the automatic driving is completed, the process proceeds to step S305.

[0079] (Step S305) In step S305, the CPU of the control unit 110 performs normal driving processing. In the normal driving processing, the processing during manual driving is performed. For example, the CPU of the control unit 110 performs control processing of the agent 200 during driving. Details of the normal driving processing will be omitted. After performing the normal driving processing, the process proceeds to step S306.

[0080] (Step S306) In step S306, the CPU of the control unit 110 performs a process of determining whether the vehicle speed is "0". Specifically, the CPU of the control unit 110 determines whether the vehicle 1 has stopped based on the vehicle speed input from the vehicle speed sensor 140a. If the CPU of the control unit 110 determines that the vehicle speed is not "0", the process proceeds to step S302. If the CPU of the control unit 110 determines that the vehicle speed is "0", the running process ends.

[0081] Next, with reference to FIG. 8, the process performed by the CPU of the control unit 110 at the time of automatic driving switching will be described. FIG. 8 is a subroutine of step S360 (process at the time of automatic driving switching) in FIG. 7.

[0082] (Step S361) In the process at the time of automatic driving switching, first, in step S361, the CPU of the control unit 110 performs a process of notifying (warning) the switching to automatic driving. Specifically, the CPU of the control unit 110 notifies, via the agent speaker 191, the switching from the manual driving state to the automatic driving state, and causes the driver H to recognize the switching to automatic driving. Then, after performing the notification of the switching to automatic driving, the process proceeds to step S362.

[0083] (Step S362) In step S362, the CPU of the control unit 110 performs a switching control process to automatic driving. Specifically, the CPU of the control unit 110 performs a process of switching the driving state from the manual driving state to the automatic driving state. In the automatic driving state, for example, even without an operation of the steering wheel 4 by the driver H, the left and right steering of the vehicle 1 is performed based on the information acquired from the surrounding environment recognition unit 120 and the like. Then, after performing the switching control process to automatic driving, the process proceeds to step S363.

[0084] (Step S363) In step S363, the CPU of the control unit 110 performs a process of causing the agent 200 to change from a standing posture to a sitting posture. Specifically, as shown in FIG. 9, the CPU of the control unit 110 causes the agent 200 to bend both legs and assume a sitting posture. That is, the CPU of the control unit 110 causes the agent 200 to change from the standing posture shown in FIG. 9(a) to the sitting posture shown in FIG. 9(b). Since the driver H captures the agent 200 in the peripheral vision, the driver H can recognize the switching to the automatic driving due to the large posture change of the agent 200, and can intuitively understand that the degree of tension has decreased due to the posture change (In the case of the lighting of a small lamp often found in European cars, the driver H may not notice the switching to the automatic driving, or even if the driver H notices, the intuitive understanding degree is low). Note that the agent 200 keeps both arms down and is in a standby state during normal automatic driving. Then, when the agent 200 is changed from the standing posture to the sitting posture, the process at the time of switching to the automatic driving is terminated.

[0085] Next, with reference to FIG. 10, the process during automatic driving performed by the CPU of the control unit 110 will be described. FIG. 10 is a subroutine of step S370 (process during automatic driving) in FIG. 7.

[0086] (Step S371) In the process during automatic driving, first, in step S371, the CPU of the control unit 110 performs a process of determining whether there is an approaching object. Specifically, the CPU of the control unit 110 determines whether there is an object (such as a person) approaching the vehicle 1 within a predetermined range and at a speed equal to or higher than a predetermined speed from the information on the surrounding situation input from the in-vehicle camera 120a for photographing the outside of the vehicle, the radar 120b, etc. That is, the CPU of the control unit 110 determines whether it is necessary to avoid an obstacle based on the acquired surrounding situation. When the CPU of the control unit 110 determines that there is an approaching object, the process proceeds to step S372, and when it determines that there is no approaching object, the process proceeds to step S375.

[0087] (Step S372) In step S372, the CPU of the control unit 110 performs a process of determining whether to perform a sudden deceleration. That is, for the vehicle 1, even if there is an approaching object, there are cases where it is necessary to perform a sudden deceleration and quickly perform an avoidance process, cases where a predetermined avoidance action can be performed without performing a sudden deceleration, or cases where there is no need to perform an avoidance action until an avoidance action is performed. When the CPU of the control unit 110 determines to perform a sudden deceleration, the process proceeds to step S373, and when it determines not to perform a sudden deceleration, the process proceeds to step S374.

[0088] (Step S373) In step S373, the CPU of the control unit 110 causes the agent 200 to perform a sudden deceleration operation. Specifically, as shown in FIG. 11, the CPU of the control unit 110 causes the agent 200 to turn the palms downward and spread both arms horizontally. Note that during automatic driving, the CPU of the control unit 110 mainly notifies the driver H of information through the movement of the upper body while keeping the agent 200 crouched. As a result, the driver H can recognize that the vehicle 1 is decelerating. For example, when the approach of a vehicle ahead is detected, with the agent 200 in the sitting position, both arms are extended forward, and then both arms are made horizontal to express deceleration. Further, the CPU of the control unit 110 decelerates the vehicle 1 and causes it to perform a predetermined avoidance control. Then, after performing the sudden deceleration operation and the avoidance control, the process during automatic driving ends.

[0089] (Step S374) In step S374, the CPU of the control unit 110 causes the agent 200 to give an instruction such as pointing at an approaching object. Specifically, as shown in FIG. 12, when an approaching object is detected, the CPU of the control unit 110 causes the agent 200 to turn one arm in the direction of the approaching object while remaining in the seated posture, that is, to raise one arm and make a gesture of pointing in the direction of the approaching object. Note that when there is any change during automatic driving, the CPU of the control unit 110 tilts the head and makes a pointing gesture as shown in FIG. 12. Also, when no significant change occurs in driving, only the head is tilted and no significant change in posture occurs. Further, if a predetermined action is required due to an approaching object, the CPU of the control unit 110 causes deceleration control or predetermined avoidance control to be performed. Then, after giving an instruction such as pointing at the approaching object, the process during automatic driving is terminated. Note that even if the process during automatic driving shown in FIG. 10 is terminated once, as described above, if the automatic driving continues in the automatic driving end determination (step S303 in FIG. 7), the process returns to this process during automatic driving and the process during automatic driving continues.

[0090] (Step S375) In step S375, the CPU of the control unit 110 performs a process of determining whether the travel route of the vehicle 1 is a corner. That is, the CPU of the control unit 110 performs a process of determining whether it is immediately before a corner. For example, when there is a corner in the travel route of the vehicle 1 and the distance to the corner is a predetermined distance or the estimated time until reaching the corner is within a predetermined time, the CPU of the control unit 110 determines that the travel route of the vehicle 1 is a corner. Note that the distance, time, etc. for this determination are preferably changed according to the speed of the vehicle 1 and the conditions of the travel road. If the CPU of the control unit 110 determines that the travel route of the vehicle 1 is a corner, the process proceeds to step S376, and if it determines that the travel route of the vehicle 1 is not a corner, the process proceeds to step S377.

[0091] (Step S376) In step S376, the CPU of the control unit 110 performs a process of tilting the agent 200 while maintaining the sitting posture. Specifically, as shown in FIG. 13, when approaching a corner, the CPU of the control unit 110 tilts the upper body of the agent 200 in the direction of centrifugal force and operates to extend the opposite arm obliquely. Then, the degree of tilt is changed according to the degree of time at the corner and the magnitude of the turning radius, and the change amount during turning is expressed. Note that the CPU of the control unit 110 also performs the same operation in the case of lane change or the like. Then, after causing the agent 200 to assume a tilted posture, the automatic driving process is terminated.

[0092] (Step S377) In step S377, the CPU of the control unit 110 performs an operation process according to the driving state. Here, when there is the above-mentioned approaching object during automatic driving, or a process other than the process at a corner is performed. For example, the CPU of the control unit 110 performs a deceleration process when approaching an intersection during automatic driving, a lighting control process for lights at night, and the like. Details of the operation process according to the driving state will be omitted. Then, after performing the operation process according to the driving state, the automatic driving process is terminated.

[0093] Next, with reference to FIG. 14, the process at the end of automatic driving performed by the CPU of the control unit 110 will be described. FIG. 14 is a subroutine of step S380 (process at the end of automatic driving) in FIG. 7.

[0094] (Step S381) In the process at the end of automatic driving, first, in step S381, the CPU of the control unit 110 performs a process of notifying (warning) the switching to manual driving. Specifically, the CPU of the control unit 110 notifies, via the agent speaker 191, the switching from the automatic driving state to the manual driving state, and causes the driver H to recognize that the automatic driving has ended and the manual driving is switched by the driver H's own operation. Then, after performing the notification of the switching to manual driving, the process proceeds to step S382.

[0095] (Step S382) In step S382, the CPU of the control unit 110 performs a process of causing the agent 200 to change from a sitting posture to a standing posture. Specifically, the CPU of the control unit 110 causes the agent 200 to change from the sitting posture shown in FIG. 9(b) to the standing posture shown in FIG. 9(a). The driver H can recognize the switching from the automatic driving to the manual driving due to the large posture change of the agent 200 and can make preparations corresponding to the manual driving. Then, after causing the agent 200 to change from the sitting posture to the standing posture, the process proceeds to step S383.

[0096] (Step S383) In step S383, the CPU of the control unit 110 performs switching control processing to manual driving. Specifically, the CPU of the control unit 110 performs a process of switching the driving state from the automatic driving state to the manual driving state. Then, after performing the switching control processing to manual driving, the process of the end of the automatic driving is terminated.

[0097] As described above, when the switching between the manual driving and the automatic driving is performed in the agent device 100 of the present embodiment, the agent 200 switches between the standing posture and the sitting posture, clearly communicating whether it is manual driving or automatic driving. Therefore, it is possible to easily cause the driver H to determine whether it is manual driving or automatic driving, and support the safety during driving. Also, when switching from manual driving to automatic driving, since the agent 200 is switched from the standing posture to the sitting posture, it is possible to make the driver H perceive the switching to the automatic driving in the peripheral vision, and support the safety during driving. That is, even when the driver H is driving while normally paying attention to the front without staring at a specific lamp or the like during manual driving, the driver H can recognize the switching to the automatic driving, and support the safety during driving without causing inattention to the front or the like.

[0098] Also, when switching from manual driving to autonomous driving, since the agent 200 is switched from the sitting posture to the standing posture, the driver H can easily recognize the switch to autonomous driving, and the safety during driving can be supported. Further, before switching to manual driving, since the change in the driving state is notified by voice and the posture change of the agent 200, the driver H can prepare for driving before switching to manual driving, and the safety can be supported. Furthermore, during autonomous driving, since the agent 200 operates according to the driving state, it is easy for the passenger to take a posture according to the driving state, and the safety can be supported.

[0099] Note that the agent control process of the present embodiment is executed by the CPU etc. of the control unit 110 after the program stored in the ROM, RAM, EEPROM etc. of the control unit 110 is expanded to the RAM etc. of the control unit 110. Furthermore, in the present embodiment, the shift position sensor 140f constitutes the shift position detection unit of the present application. Also, in the present embodiment, the surrounding environment recognition unit 120 constitutes the surrounding situation acquisition unit of the present application. Also, in the present embodiment, the control unit 110 constitutes the driving state switching unit and the autonomous driving determination unit of the present application.

[0100] Also, in the present embodiment, the agent control unit of the present application is configured by both the control unit 110 and the agent control unit 190. However, as described above, it may be configured to have the functions of both the control unit 110 and the agent control unit 190 of the present embodiment only by the agent control unit 190.

Explanation of Reference Numerals

[0101] 1: Vehicle, 100: Agent device, 110: Control unit, 120: Surrounding environment recognition unit, 130: Occupant state recognition unit, 140: Vehicle state recognition unit, 140f: Shift position sensor, 150: Speaker, 160: Microphone, 170: Memory unit, 180: Transceiver, 190: Agent control unit, 191: Agent speaker, 200: Agent, 310: Driving state switching operation unit

Claims

1. A memory unit that stores conversations with the driver, A surrounding situation acquisition unit that acquires the surrounding situation, A driving state switching unit that switches between manual driving and automatic driving based on the surrounding situation acquired by the surrounding situation acquisition unit or the driver's instructions, An automatic driving determination unit that determines whether it is in the automatic driving state, An agent control unit that controls an anthropomorphic agent, Comprising, The agent is, Disposed at a position that enters the driver's peripheral vision, The agent control unit, Based on at least the information stored in the memory unit, is capable of executing a conversation according to the driver's hobbies and preferences, When switching between manual driving and automatic driving, the agent is switched between a standing posture and a sitting posture, An agent device characterized by the above.

2. The agent control unit, When switching from manual driving to automatic driving, the agent is switched from a standing posture to a sitting posture, The agent device according to claim 1, characterized in that.

3. The agent control unit, When an approaching object is detected by the surrounding situation acquisition unit, if it is in automatic driving, while remaining in the sitting posture, one arm is directed towards the approaching object, The agent device according to claim 2, characterized in that.

4. The agent control unit, When approaching a corner during automatic driving, while remaining in the sitting posture, only the upper body is tilted in the direction of centrifugal force, The agent device according to claim 2 or claim 3, characterized in that.

5. The agent control unit, When the approach of the vehicle ahead is detected during automatic driving, while remaining in the sitting posture, both arms are extended forward, and then both arms are made horizontal to express deceleration, The agent device according to any one of claims 2 to 4, characterized in that.

6. The agent control unit, When switching from automatic driving to manual driving, the agent is switched from a sitting posture to a standing posture, The agent device according to any one of claims 1 to 5, characterized in that.

7. The agent control unit, When switched from automatic driving to manual driving by the driving state switching unit, before switching to manual driving, it notifies that the driving state is switched, The agent device according to any one of claims 1 to 6, characterized in that.

Images