Image display device and image display method

Abstract

To achieve, in a case of an automatic driving vehicle in which a driver is absent, communication with a person outside the vehicle and contribute to smooth and safe driving.SOLUTION: When receiving standby information to the effect that walking of a pedestrian is prioritized from an automatic driving device 990, a detection unit 730 performs analysis processing on a photographic image transmitted from a photographing unit 910 and detects the attitude and line of sight of the pedestrian. The detection unit 730 then transmits a result of detection to a display control unit 750. When a driver is not in a vehicle CR, the display control unit 750 determines whether the pedestrian is in the attitude of a preparation posture for moving in front of the vehicle CR and the line of sight of the pedestrian is directed to the vehicle CR on the basis of the result of detection performed by the detection unit 730. When a result of the determination is affirmative, the display control unit 750 performs control of displaying, on a screen 955, an image imitating a gesture performed by the driver of the vehicle CR to the pedestrian, the gesture urging a movement in front of the vehicle CR.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image display device, an image display method, an image display program, and a recording medium on which the image display program is recorded. [Background technology]

[0002] In recent years, development has been progressing toward the practical application of unmanned vehicles that do not require driver operation. One of the technologies related to such unmanned vehicles is to address the issue of people outside the vehicle, and various technologies for implementing such measures have been proposed.

[0003] One such proposed technology is one in which, when an autonomous vehicle recognizes a person, the person is notified that they have been recognized by the autonomous vehicle (see Patent Document 1: hereinafter referred to as "Prior Art 1"). In the technology of Prior Art 1, a process of detecting people present around the autonomous vehicle is performed from images captured by a camera. Then, when a person is detected by the detection process, a laser beam is emitted toward the person or a sound is output.

[0004] Furthermore, although not related to autonomous vehicles, there is a technology that clearly alerts pedestrians (see Patent Document 2: hereinafter referred to as "Prior Art 2"). This technology of Prior Art 2 identifies pedestrians based on images captured by a camera, and performs a process to determine whether or not the pedestrian is present in a set area. Then, when a pedestrian is present in the set area, an image of a crosswalk or the like is projected onto the road surface. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2017-159881 [Patent Document 2] Japanese Patent Application Laid-Open No. 2008-007079 Summary of the Invention [Problem to be solved by the invention]

[0006] Incidentally, for smooth and safe road travel, communication is important, such as signals from the driver to pedestrians to encourage them to walk right of way at crosswalks, or signals from the driver to operators of other vehicles to encourage other vehicles to go ahead. However, in an autonomous vehicle, the driver may not be present when the vehicle is traveling. The technology of the above-mentioned Prior Art Example 1 notifies a person that the autonomous vehicle has recognized the person, but does not notify the person to walk right of way. Therefore, the person who receives the notification typically remains where they are and waits for the vehicle to pass. As a result, the technology of Prior Art Example 1 is insufficient as a notification for smooth and safe road travel.

[0007] On the other hand, when the technology of Conventional Example 2 is applied to technology related to unmanned vehicles, it becomes possible for the unmanned vehicles to communicate with pedestrians to encourage them to walk right of way. However, in the method of projecting an image onto the road, depending on the road conditions such as unevenness, the image projected onto the road may become distorted, and the content of the image may not be conveyed to pedestrians. As a result, the technology of Conventional Example 2 leaves room for improvement in notifications for smooth and safe road driving.

[0008] Therefore, even in the case of driverless autonomous vehicles, there is a demand for technology that enables communication with people outside the vehicle, similar to that of manned vehicles, and contributes to smooth and safe driving. Meeting this demand is one of the problems that the present invention aims to solve. [Means for solving the problem]

[0009] The invention described in claim 1 is an image display device to be placed in an autonomously driving vehicle, comprising: a display control unit that, when the driver of the vehicle is not present, displays an image that simulates the driver in the vehicle making a gesture toward a pedestrian who is visible and who may enter the vehicle's planned driving route; and a detection unit that detects the presence or absence of the pedestrian, and at least one of the pedestrian's posture, line of sight, and facial direction; and the display control unit is characterized in that it displays the image when it determines, based on the detection result by the detection unit, that the pedestrian is aware of the vehicle. The invention described in claim 3 is an image display device to be placed in an autonomously driving vehicle, comprising: a display control unit that displays, when the driver of the vehicle is not present, an image that simulates the driver in the vehicle making a gesture toward the operator, which image is visible to the operator of another vehicle that may enter the vehicle's planned driving route; and a detection unit that detects the distance between the vehicle and the other vehicle, the presence or absence of the operator, the movement of the other vehicle, and the operator's line of sight or facial direction; and the display control unit is characterized in that it displays the image when, based on the detection result by the detection unit, it determines that the distance is within a predetermined distance, that the other vehicle is waiting to move ahead of the vehicle, and that the operator's line of sight is directed toward the vehicle.

[0010] The invention described in claim 7 is an image display method used in an image display device that is equipped with: a display control unit that is arranged in an autonomously driving vehicle and that, when the driver of the vehicle is not present, displays an image that simulates the driver in the vehicle making a gesture toward a pedestrian that is visible to the pedestrian and who may enter the vehicle's planned driving path; and a detection unit, wherein the detection unit detects the presence or absence of the pedestrian, and at least one of the pedestrian's posture, line of sight, and facial direction; and a display control process that displays the image when the display control unit determines, based on the detection result by the detection unit, that the pedestrian recognizes the vehicle. The invention described in claim 8 is an image display method used in an image display device that is arranged in an autonomously driving vehicle and that, when the driver of the vehicle is not present, is equipped with a display control unit that displays an image that simulates the driver in the vehicle making a gesture toward the operator, which image is visible to the operator of another vehicle that may enter the planned driving route of the vehicle, wherein the image display method is characterized by comprising: a detection step in which the detection unit detects the distance between the vehicle and the other vehicle, the presence or absence of the operator, the movement of the other vehicle, and the line of sight or facial direction of the operator; and a display control step in which the display control unit displays the image when, based on the detection result by the detection unit, it determines that the distance is within a predetermined distance, that the other vehicle is waiting to move ahead of the vehicle, and that the operator's line of sight is directed toward the vehicle.

[0011] The invention described in claim 9 is an image display program that causes a computer included in an image display device to execute the image display method described in claim 7. The invention described in claim 10 is an image display program that causes a computer included in an image display device to execute the image display method described in claim 8.

[0012] An eleventh aspect of the present invention is a recording medium having the image display program according to the ninth aspect recorded thereon so as to be readable by a computer included in an image display device. The invention described in claim 12 is a recording medium having the image display program described in claim 10 recorded thereon so as to be readable by a computer included in an image display device. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a configuration of an image display device according to an embodiment of the present invention. [Figure 2] 1 is a block diagram showing a schematic configuration of an image display device according to an embodiment of the present invention; [Figure 3]3 is a diagram for explaining the positional relationship between a projection unit, a half mirror, and a screen in the display unit of FIG. 2. FIG. [Figure 4] 3 is a diagram to be referred to when explaining the operation of the image display device of FIG. 2. FIG. [Figure 5] 10 is a flowchart for explaining a display control process of an image simulating a driver by the image display device of FIG. 2; [Figure 6] 6 is a flowchart for explaining the first type image display control process of FIG. 5. [Figure 7] 6 is a flowchart for explaining the second type image display control process of FIG. 5. [Figure 8] 10 is a diagram for explaining how an image simulating a driver displayed on a screen looks when viewed from outside the vehicle. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0014] An embodiment of the present invention will now be described with reference to Fig. 1. In the following description and drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant description will be omitted.

[0015] [composition] 1 shows a block diagram of the configuration of an image display device 700 according to one embodiment. The image display device 700 is disposed inside an autonomously driven vehicle CR. In this embodiment, in addition to the image display device 700, an imaging unit 910 and a display unit 950 are disposed in the vehicle CR, which are connected to the image display device 700. Furthermore, an automatic driving device 990 is disposed in the vehicle CR, and the automatic driving device 990 is connected to the image display device 700.

[0016] The photographing unit 910 is configured to include a camera that photographs at least the area ahead of the vehicle CR within a field of view that includes the traveling direction of the vehicle CR. The photographing results (photographed images) by the photographing unit 910 are sent to the image display device 700. Although not shown, the photographing results by the photographing unit 910 are also sent to the automatic driving device 990, and are used in the automatic driving control by the automatic driving device 990.

[0017] In this embodiment, the display unit 950 includes a projection unit 951 and a screen 955.

[0018] The projection unit 951 is configured to include a video projector. The projection unit 951 receives display image data sent from the image display device 700. The projection unit 951 then projects and displays an image corresponding to the display image data on a screen 955.

[0019] In this embodiment, the screen 955 is disposed in front of the backrest of the front seat. The screen 955 displays the image projected by the projection unit 951 so that it can be seen by people outside the vehicle CR.

[0020] The automatic driving device 990 controls the starting, acceleration / deceleration, steering, stopping, etc. of the vehicle CR according to the driving plan and the driving environment. The automatic driving device 990 is equipped with a storage unit (not shown) that stores map information. The automatic driving device 990 uses driving information sent from driving information detection units (not shown), such as a vehicle speed detection unit and a position detection unit, and captured images sent from the imaging unit 910, and appropriately refers to the map information to perform automatic driving control.

[0021] As such automatic driving control, for example, when the automatic driving device 990 detects a pedestrian about to cross a crosswalk on a road without traffic lights, it performs control to stop the vehicle CR until the pedestrian crosses the crosswalk. When performing this control, the automatic driving device 990 generates waiting information to the effect that the pedestrian's walking (crossing) is given priority, and sends this information to the image display device 700.

[0022] Furthermore, as part of such automatic driving control, when the automatic driving device 990 detects, for example, that another vehicle traveling in the oncoming lane is waiting for the vehicle CR to pass in order to make a right turn or the like, it performs control to decelerate or stop the vehicle CR in order to give the other vehicle priority (give it priority) to turn right, depending on the driving environment. When performing this control, the automatic driving device 990 generates waiting information indicating that the other vehicle will be given priority (give it priority) to move, and sends this information to the image display device 700. Here, the other vehicle includes an automobile, a motorcycle, a bicycle, etc.

[0023] Furthermore, as part of such automatic driving control, when the automatic driving device 990 detects that another vehicle is waiting for the vehicle CR to move in order to park in a parking lot, the automatic driving device 990 performs control to stop the vehicle CR in order to allow the other vehicle to park in the parking lot first, depending on the driving environment. When performing this control, the automatic driving device 990 generates waiting information indicating that the other vehicle will be given priority (given priority), and sends this information to the image display device 700.

[0024] <Configuration of image display device 700> The configuration of the image display device 700 will be described.

[0025] 1, the image display device 700 includes a storage unit 710 and a detection unit 730. The image display device 700 also includes a display control unit 750.

[0026] The storage unit 710 is configured to include a non-volatile storage element. Image data simulating the driver of the vehicle CR and camera information indicating the position and optical characteristics of the camera provided in the photographing unit 910 are stored in the storage unit 710. The detection unit 730 and the display control unit 750 can access the storage unit 710.

[0027] The image data simulating the driver of the vehicle CR is image data simulating the driver making a gesture toward a person in front of the vehicle CR when the person is looking at the vehicle CR. One example of the gesture is a gesture to encourage a pedestrian to move forward in front of the vehicle CR. In other words, it is a gesture that indicates to a pedestrian who may enter the planned travel route of the vehicle CR that the driver intends to give priority to the pedestrian walking (crossing).

[0028] Another example of the gesture is a gesture that prompts the operator of another vehicle to move the other vehicle that is ahead of the vehicle CR. In other words, it is a gesture that indicates to the operator of another vehicle that may enter the planned travel route of the vehicle CR that the operator wants to give priority to the other vehicle.

[0029] The detection unit 730 receives the captured image sent from the imaging unit 910. When the detection unit 730 receives waiting information sent from the automatic driving device 990, it performs an analysis process on the captured image to detect information about a pedestrian who may enter the planned driving route of the vehicle CR, or information about other vehicles that may enter the planned driving route of the vehicle CR and the operators of those other vehicles (this information will also be referred to as "information about other moving bodies"). In other words, the detection unit 730 is configured to detect the presence or absence of a person (pedestrian, operator of another vehicle, etc.) or other vehicle that may enter the planned driving route of the vehicle CR.

[0030] When another vehicle is detected, the detection unit 730 detects the distance between the vehicle CR and the other vehicle based on the analysis result of the captured image. The detection result by the detection unit 730 is sent to the display control unit 750. Details of the processing by the detection unit 730 will be described later.

[0031] The display control unit 750 described above receives information about other moving objects sent from the detection unit 730. Subsequently, when there is no driver in the vehicle CR and the display control unit 750 receives waiting information sent from the automatic driving device 990, if the display control unit 750 determines that an image representing the driver should be displayed based on the content of the waiting information and the information about the other moving object, the display control unit 750 reads the image data in the storage unit 710 and generates display image data. Then, the display control unit 750 sends the display image data to the display unit 950. Details of the processing by the display control unit 750 will be described later.

[0032] In this embodiment, there may be a driver detection unit that detects the presence of a driver in the vehicle CR. The driver detection unit may detect the presence of a driver in the vehicle CR from a signal from a seating sensor or the like installed in a seat of the vehicle CR.

[0033] Furthermore, the driver detection unit may detect that a driver is present in the vehicle CR depending on the type of autonomous driving driving mode being executed by the autonomous driving device 990. For example, when the autonomous driving device 990 is executing a driving mode that is executed when the driver is clearly not present in the vehicle CR, such as an "unmanned driving mode," the driver detection unit may detect that the driver is not present in the vehicle CR.

[0034] [Operation] The operation of the image display device 700 configured as above will be described, focusing mainly on the processing by the detection unit 730 and the display control unit 750.

[0035] It is assumed that the photographing unit 910 has started operation and is sequentially sending photographing results (photographed images) to the image display device 700 and the automatic driving device 990. It is also assumed that the automatic driving device 990 has started operation and, when it generates the above-mentioned waiting information, sends the waiting information to the image display device 700.

[0036] <Detection process of information about other moving objects> The process of detecting information about other moving bodies is executed when waiting information sent from the automatic driving device 990 is received. During the process of detecting information about other moving bodies, if the waiting information sent from the automatic driving device 990 is information indicating that priority is to be given to pedestrians, the detection unit 730 analyzes the captured image sent from the image capturing unit 910 and detects pedestrians in the image.

[0037] The detection unit 730 may analyze the captured image sent from the image capturing unit 910 to further detect at least one of the posture, line of sight, and facial direction of the detected pedestrian. The detection unit 730 then sequentially sends the pedestrian detection result and the detection result regarding at least one of the pedestrian's posture, line of sight, and facial direction to the display control unit 750 as information about the pedestrian while waiting information indicating that priority is given to the pedestrian's walking is being received. The detection unit 730 may also be particularly configured to detect the presence or absence of a person (pedestrian, operator of another vehicle, etc.) or another vehicle that may enter the planned driving route of the vehicle CR.

[0038] Furthermore, during the process of detecting information about such other moving bodies, if the waiting information sent from the automatic driving device 990 is information indicating that the other vehicle will move first, the detection unit 730 analyzes the captured image sent from the imaging unit 910 to detect the other vehicle and the operator of the other vehicle in the image. Note that the detection unit 730 may analyze the captured image sent from the imaging unit 910 to further detect the operation of the detected other vehicle and the line of sight of the operator.

[0039] Furthermore, the detection unit 730 detects the distance between the vehicle CR and the other vehicle based on the captured image sent from the imaging unit 910 and the camera information stored in the storage unit 710. The detection unit 730 then sequentially sends the detection results of the other vehicle and the operator of the other vehicle, the detection results of the operation of the other vehicle, and the line of sight or facial direction of the operator of the other vehicle, as well as the detection results of the distance between the vehicle CR and the other vehicle, to the display control unit 750 as information about the other vehicle and the operator during the period when standby information to allow the other vehicle to move ahead is received.

[0040] <Display control process for images simulating a driver> In the display control process of the image simulating the driver, the display control unit 750 first performs a non-presence determination process to determine whether or not there is no driver in the vehicle CR by determining whether or not an input indicating that the driver is present in the vehicle CR has been made to the input unit. If the result of the non-presence determination process is negative, the display control unit 750 repeats the non-presence determination process.

[0041] If the result of the non-existence determination process is affirmative, the display control unit 750 performs a waiting information acquisition determination process to determine whether or not waiting information sent from the automatic driving device 990 has been received. If the result of the waiting information acquisition determination process is negative, the waiting information acquisition determination process is terminated, and the above-described non-existence determination process is restarted.

[0042] On the other hand, if the result of the waiting information acquisition determination process is positive, the display control unit 750 performs information determination process to determine whether the content of the waiting information is waiting information indicating that "pedestrians should be given priority" or waiting information indicating that "other vehicles should be allowed to move first."

[0043] If the information determination process indicates that the content of the waiting information is "give priority to pedestrians," the display control unit 750 performs a determination process (posture determination process) of whether or not the pedestrian is in a posture ready to move in front of the vehicle CR (in a posture ready to walk on the crosswalk in front of the vehicle) based on the information about the pedestrian sent from the detection unit 730. Here, the "prepared posture" refers to a state in which the pedestrian is about to move (cross) in front of the vehicle CR. In other words, it refers to a state before the pedestrian starts crossing in front of the vehicle CR.

[0044] Furthermore, the display control unit 750 performs a determination process (gaze determination process) as to whether or not the pedestrian's gaze is directed toward the vehicle CR, based on the information about the pedestrian sent from the detection unit 730. In this embodiment, the gaze determination process can be rephrased as a process of determining whether or not the pedestrian recognizes the vehicle CR. That is, if the result of the gaze determination process is positive, it is determined that the pedestrian recognizes the vehicle CR. If the result of the posture determination process or the gaze determination process is positive, the display control unit 750 reads the image data in the storage unit 710 and generates display image data. Then, the display control unit 750 sends the display image data to the display unit 950. As a result, an image imitating a gesture made by the driver to the pedestrian to encourage the vehicle CR to move forward is displayed on the screen 955.

[0045] Note that the display control unit 750 may read the image data in the storage unit 710 and generate display image data only when both the posture determination process and the line of sight determination process described above are positive.

[0046] The posture determination process and the line-of-sight determination process may be omitted. That is, when it is determined that a pedestrian is present based on the information about the pedestrian sent from the detection unit 730, the display control unit 750 may read the image data in the storage unit 710 and generate display image data (even if the line of sight of the pedestrian is not directed toward the vehicle CR).

[0047] On the other hand, if the information determination process indicates that the content of the waiting information is "information indicating that the other vehicle is to move ahead," the display control unit 750 performs a determination process based on the information regarding the other moving object and the operator sent from the detection unit 730 to determine whether the other vehicle is waiting for the vehicle CR to move, whether the operator's line of sight is directed toward the vehicle CR, and whether the distance between the vehicle CR and the other vehicle is within a predetermined distance. If the result of the determination process is affirmative, the display control unit 750 reads the image data in the storage unit 710 and generates display image data. The display control unit 750 then sends the display image data to the display unit 950. As a result, an image imitating a gesture made by the driver to the operator of the other vehicle to encourage the other vehicle, which is ahead of the vehicle CR, to move is displayed on the screen 955. Here, the predetermined distance is determined in advance based on experiments, simulations, experience, etc., from the perspective of allowing the operator of the other vehicle to recognize an image imitating the driver of the vehicle CR.

[0048] In this determination process, it is not necessary to determine whether the line of sight of the operator of the other vehicle is directed toward the vehicle CR. That is, when it is determined that another moving object (another vehicle or an operator of the other vehicle) is present based on the information about the other moving object and the operator sent from the detection unit 730, the display control unit 750 may read the image data in the storage unit 710 and generate display image data (even if the line of sight of the operator is not directed toward the vehicle CR).

[0049] As described above, in this embodiment, when waiting information indicating that priority is to be given to a pedestrian is received from the automatic driving device 990, the detection unit 730 performs analysis processing on the captured image sent from the image capture unit 910 to detect the pedestrian in the image and detect at least one of the posture, line of sight, and facial direction of the detected pedestrian. Then, the detection unit 730 sends the detection result to the display control unit 750 as information related to the pedestrian.

[0050] When the display control unit 750 receives standby information from the automatic driving device 900 indicating that the driver of the vehicle CR is absent and that priority is being given to a pedestrian, the display control unit 750 performs a process of determining whether the pedestrian is in a posture ready to move ahead of the vehicle CR and whether the pedestrian's line of sight is directed toward the vehicle CR, based on the information about the pedestrian sent from the detection unit 730. Then, when the result of the determination process is positive, the display control unit 750 performs control to display on the screen 955 an image that mimics a gesture made by the driver of the vehicle CR to the pedestrian to encourage them to move ahead of the vehicle CR.

[0051] This allows pedestrians who wish to cross the crosswalk to be encouraged to do so by gestures, making it possible to communicate with pedestrians in the same way as with manned vehicles.

[0052] Furthermore, in this embodiment, when waiting information indicating that another vehicle will be allowed to move ahead is received from the automatic driving device 990, the detection unit 730 performs an analysis process on the captured image sent from the image capture unit 910, detects the other vehicle and the operator of the other vehicle in the image, and detects the movement of the detected other vehicle and the line of sight or facial direction of the operator. Then, the detection unit 730 sends the detection result to the display control unit 750 as information related to the other moving object and the operator.

[0053] When the display control unit 750 receives from the automatic driving device 990 standby information indicating that the other vehicle is to move ahead of the vehicle CR when there is no driver in the vehicle CR, the display control unit 750 performs a determination process to determine whether or not the other vehicle is waiting for the vehicle CR to move ahead, whether or not the operator's line of sight is directed toward the vehicle CR, and whether or not the distance between the vehicle CR and the other vehicle is within a predetermined distance, based on the information regarding the other moving body and the operator sent from the detection unit 730. Then, when the result of the determination process is positive, the display control unit 750 performs control to display on the screen 955 an image that imitates a gesture made by the driver of the vehicle CR to the operator of the other vehicle to encourage the other vehicle to move ahead of the vehicle CR.

[0054] This allows the operator of another vehicle who is waiting for the vehicle CR to pass in order to make a right turn, or the operator of another vehicle who is waiting for the vehicle CR to move in order to park in a parking lot, to be prompted to move ahead by gestures. This makes it possible to realize communication with the operator of another vehicle, just like in the case of a manned vehicle.

[0055] Therefore, according to this embodiment, in the case of an autonomous vehicle without a driver, communication with people outside the vehicle can be realized, just as in the case of a manned vehicle, contributing to smooth and safe driving.

[0056] [Modification of the embodiment] The present invention is not limited to the above-described embodiment, and various modifications are possible.

[0057] For example, in the above embodiment, an image imitating the gestures of the driver is displayed, but an image imitating the gestures of a passenger other than the driver may be displayed.

[0058] In the above embodiment, an image of the vehicle driver is projected onto a screen disposed on the back of the front seat. Alternatively, a transparent screen may be disposed on the glass of the vehicle (for example, the windshield), and an image of the vehicle driver may be projected onto the screen.

[0059] In the above embodiment, an image simulating the driver is displayed by using a video projector to project an image onto a screen, but the image simulating the driver may also be displayed on a liquid crystal display, for example.Furthermore, the image may be displayed on a hologram display or the like, so that an outside person can visually recognize the vehicle occupants in a three-dimensional manner.

[0060] Furthermore, the image (moving image) simulating the driver may be changed based on information about the pedestrian or the operator of the other vehicle detected by the detection unit. For example, the image may be displayed so that the driver's line of sight in the image is directed in the direction of the pedestrian or the operator of the other vehicle. Furthermore, when a pedestrian or the operator of the other vehicle nods, an image of the driver in the image nodding in response may be displayed. This allows the pedestrian or the operator of the other vehicle to recognize that their presence is recognized by the other vehicle, thereby providing a sense of security to the pedestrian or the operator of the other vehicle.

[0061] Furthermore, when allowing outsiders to see the vehicle occupants, in addition to projecting an image onto a screen as described above, so-called projection mapping technology may be used, for example, to place a three-dimensional human-shaped structure in the front seat and project an image of the occupant onto the three-dimensional structure. In this case, it is desirable that the three-dimensional structure be configured so that it can be stored in the seat.

[0062] Furthermore, when an image simulating a passenger is displayed on a display or the like, the image may be transformed by affine transformation or the like depending on the direction in which an outside person is present, so that the image simulating a passenger can be viewed without discomfort when viewed from the direction in which the outside person is present. Note that when the affine transformation or the like is not performed, the image simulating a passenger may be displayed only when the outside person is present in a direction in which the image simulating a passenger can be viewed without discomfort. For example, when an outside person is present in front of the vehicle, if it is possible to display only images in which the image simulating a passenger can be viewed without discomfort by the outside person, the image may be displayed only when it is detected that the outside person is present in front of the vehicle.

[0063] In the above embodiment, a person in the direction of travel of the vehicle is detected based on a camera image. However, a person in the direction of travel of the vehicle may be detected based on a known Lidar (Light Detection and Ranging) measurement technique.

[0064] Note that some or all of the detection unit and display control unit of the image display device in the above-described embodiments may be configured as a computer serving as an arithmetic unit including a central processing unit (CPU) or the like, and some or all of the functions of these elements may be performed by executing a pre-prepared program on the computer. This program may be recorded on a computer-readable recording medium such as a hard disk, CD-ROM, or DVD, and loaded from the recording medium and executed by the computer. Furthermore, this program may be obtained in a form recorded on a portable recording medium such as a CD-ROM or DVD, or may be obtained in a form distributed via a network such as the Internet.

[0065] In addition, in the above embodiment, for the sake of convenience, the automatic driving device 990 and the image display device 700 are described separately in Figure 1, but at least some of the functions of the image display device 700 (e.g., the detection unit 730, etc.) may be included in the automatic driving device 990. [Example]

[0066] An embodiment of the present invention will now be described with reference to Figures 2 to 8. In the following description and drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant description will be omitted.

[0067] [composition] 2 is a block diagram showing a schematic configuration of an image display device 100 according to one embodiment. The image display device 100 is one aspect of the image display device 700 (see FIG. 1) according to the embodiment described above.

[0068] The image display device 100 is disposed inside an autonomously driven vehicle CR. In this embodiment, a photographing unit 210 and a display unit 250 are disposed in the vehicle CR, and these are connected to the image display device 100. An automatic driving device 290 is also disposed in the vehicle CR, and the automatic driving device 290 is connected to the image display device 100.

[0069] In this embodiment, the photographing unit 210 is configured to include a camera that photographs at least the area ahead of the vehicle CR within a field of view that includes the traveling direction of the vehicle CR. The photographing unit 210 may include a camera capable of omnidirectional photographing, or may be configured to be capable of omnidirectional photographing by combining multiple cameras. The photographing results (photographed images) by the photographing unit 210 are sent to the image display device 100. Although not shown, the photographing results by the photographing unit 210 are also sent to the automatic driving device 290 and are used in the automatic driving control by the automatic driving device 290. In other words, the photographing unit 210 fulfills the function of the photographing section 910 in the above-described embodiment.

[0070] In this embodiment, the display unit 250 includes a projection unit 251, a half mirror 253, and a screen 255. The relative positions of the projection unit 251, the half mirror 253, and the screen 255 are shown in FIG.

[0071] The projection unit 251 is configured to include an image projector and is disposed near the footrest of the front seat. The projection unit 251 receives display image data sent from the image display device 100. Then, the projection unit 251 irradiates the half mirror 253 with image light corresponding to the display image data.

[0072] Half mirror 253 is disposed above the front windshield. Half mirror 253 reflects image light emitted from the video projector. The image light reflected in this way travels toward screen 255.

[0073] In this embodiment, the screen 255 is disposed in front of the backrest of the front seat. The screen 255 receives the image light projected by the projection unit 251 and reflected by the half mirror 253, and displays an image that can be seen by people outside the vehicle CR.

[0074] That is, the display unit 250 functions as the display section 950 in the above-described embodiment.

[0075] The automatic driving device 290 controls the starting, acceleration / deceleration, steering, stopping, etc. of the vehicle CR according to the driving plan and the driving environment. The automatic driving device 290 is equipped with a storage unit (not shown) that stores map information. The automatic driving device 290 uses driving information sent from driving information detection units (not shown), such as a vehicle speed detection unit and a position detection unit, and captured images sent from the photographing unit 210, and appropriately refers to the map information to perform automatic driving control.

[0076] As part of this automatic driving control, when the automatic driving device 290 detects a pedestrian attempting to cross a crosswalk on a road without traffic lights, it performs control to stop the vehicle CR until the pedestrian crosses the crosswalk. When performing this control, the automatic driving device 290 generates wait information to give priority to the pedestrian and sends it to the image display device 100. After sending the wait information, if the automatic driving device 290 no longer detects a pedestrian attempting to cross the crosswalk, it generates wait release information and sends it to the image display device 100.

[0077] Furthermore, as part of such automatic driving control, when the automatic driving device 290 detects, for example, that another vehicle traveling in the oncoming lane is waiting for the vehicle CR to pass in order to make a right turn or the like, it performs control to decelerate or stop the vehicle CR in accordance with the driving environment in order to allow the other vehicle to turn right first. When performing this control, the automatic driving device 290 generates wait information indicating that the other vehicle will move first, and sends this to the image display device 100. After sending this wait information, when the automatic driving device 290 no longer detects another vehicle waiting for the vehicle CR to pass, it generates wait release information and sends this to the image display device 100.

[0078] Furthermore, as part of this automatic driving control, when the automatic driving device 290 detects, for example, that another vehicle is waiting for the vehicle CR to move in order to park in a parking lot, it performs control to stop the vehicle CR in order to allow the other vehicle to park in the parking lot first, depending on the driving environment. When performing this control, the automatic driving device 290 generates standby information indicating that the other vehicle will move first, and sends this to the image display device 100. After sending this standby information, when the automatic driving device 290 no longer detects another vehicle waiting for the vehicle CR to move, it generates standby release information and sends this to the image display device 100.

[0079] <Configuration of image display device 100> Next, the configuration of the image display device 100 will be described.

[0080] 2 is a block diagram showing a schematic configuration of an image display device 100 according to an embodiment. As shown in FIG. 2, the image display device 100 includes a processing control unit 110 and a storage unit 120.

[0081] The processing control unit 110 controls the entire image display device 100 and executes various processes. The processing control unit 110 is configured to include a central processing unit (CPU) as a computing means and its peripheral circuits. The processing control unit 110 executes various programs to realize various functions of the image display device 100. These functions include the functions of the detection unit 730 and display control unit 750 in the above-mentioned embodiment. Details of the processes executed by the processing control unit 110 will be described later.

[0082] The program executed by processing control unit 110 is recorded on a computer-readable recording medium such as a hard disk, CD-ROM, DVD, etc., and is loaded and executed from the recording medium. The program may also be obtained in a form recorded on a portable recording medium such as a CD-ROM or DVD, or may be obtained in a form distributed via a network such as the Internet.

[0083] The storage unit 120 is configured to include a nonvolatile storage device such as a hard disk drive, and stores various information data used in the image display device 100. Such information data includes image data simulating the driver of the vehicle CR, as well as camera information indicating the position and optical characteristics of the camera provided in the photographing unit 210. The storage unit 120 is accessible by the processing control unit 110. In other words, the storage unit 120 fulfills the function of the storage section 710 in the above-described embodiment.

[0084] The image data simulating the driver of the vehicle CR is image data simulating a gesture made by the driver toward a person in front of the vehicle CR who is looking at the vehicle CR. The gesture is a gesture to prompt the pedestrian to move in front of the vehicle CR, and a gesture to prompt the operator of another vehicle to move ahead of the vehicle CR.

[0085] [Operation] The operation of the image display device 100 configured as described above will be described, focusing mainly on the display control process of an image simulating a driver, performed by the processing control unit 110. The operation of the image display device 100 of this embodiment will also be described with appropriate reference to FIGS. 4(A) and 4(B). Here, FIG. 4(A) is a diagram showing the positional relationship between a vehicle CR and a pedestrian HA at an intersection. Also, FIG. 4(B) is a diagram showing the positional relationship between the vehicle CR, a pedestrian HB, and an oncoming vehicle (other vehicle) T at the intersection. The dashed arrows in FIGS. 4(A) and 4(B) indicate the planned driving route RT of the vehicle CR, and the imaging range of the imaging unit 210 is schematically indicated by H. Note that this imaging range H is merely an example. For example, if the imaging unit 210 is configured with a camera capable of capturing all-around images, the imaging range H can be schematically represented as a circle that covers the entire periphery of the vehicle CR.

[0086] It is assumed that the photographing unit 210 has started operation and is sequentially sending photographing results (photographed images) to the image display device 100. It is also assumed that the automatic driving device 290 has started operation and, when it generates the above-mentioned waiting information, sends the waiting information to the image display device 100.

[0087] In the display control process of the image simulating the driver, first, in step S11, the processing control unit 110 determines whether or not an input indicating that the driver is present in the vehicle CR has been made to an input unit (not shown) to determine whether or not there is a driver present in the vehicle CR, as shown in Fig. 5. If the result of this determination is negative (step S11: N), the process of step S11 is repeated.

[0088] If the result of the determination in step S11 is affirmative (step S11: Y), the process proceeds to step S12. In step S12, the processing control unit 110 determines whether or not it has received standby information from the automatic driving device 290 indicating that "pedestrians will be given priority." If the result of this determination is affirmative (step S12: Y), the process proceeds to step S13. In step S13, the processing control unit 110 performs "first type image display control processing." Details of the processing in step S13 will be described later. Then, when the processing in step S13 ends, the process returns to step S11.

[0089] If the result of the determination in step S12 is negative (step S12: N), the process proceeds to step S14. In step S14, the processing control unit 110 determines whether or not it has received waiting information from the automatic driving device 290 to the effect that "other vehicles will be allowed to move ahead." If the result of this determination is positive (step S14: Y), the process proceeds to step S15. In step S15, the processing control unit 110 performs "second type image display control processing." Details of the processing in step S15 will be described later. Then, when the processing in step S15 ends, the process returns to step S11.

[0090] If the result of the determination in step S14 is negative (step S14: N), the process returns to step S11, and thereafter, the processes in steps S11 to S15 are repeated.

[0091] <First type image display control processing> The "first type image display control process" in step S13 described above will now be described.

[0092] As shown in FIG. 6, in this "first type image display control process," first, in step S21, the processing control unit 110 analyzes the captured image sent from the photographing unit 210, detects a pedestrian in the image, and detects at least one of the posture, line of sight, and facial direction of the detected pedestrian.

[0093] Subsequently, in step S22, the processing control unit 110 determines, based on the image analysis in step S21, whether or not the first type image display condition is satisfied, that is, the pedestrian is in a posture ready to move in front of the vehicle CR (readiness to walk on a pedestrian crossing) and the pedestrian's line of sight is directed toward the vehicle CR. If the result of this determination is affirmative (step S22: Y), the processing proceeds to step S23.

[0094] The processing control unit 110 may analyze the captured image sent from the photographing unit 210, and in particular, detect the presence or absence of a person in the image who may enter the planned traveling route RT of the vehicle CR. Here, the planned traveling route is the traveling route along which the vehicle CR is expected to travel in the future, such as a traveling route set by a navigation device (not shown) installed in the vehicle CR. In this case, the processing control unit 110 detects whether a pedestrian is present in the captured image, and if detected, determines whether the pedestrian is likely to enter the planned traveling route RT of the vehicle CR based on the position, posture, etc. of the detected pedestrian. The processing control unit 110 may also detect whether a pedestrian is present in the captured image, and if detected, determine that the pedestrian is likely to enter the planned traveling route RT of the vehicle CR.

[0095] In the case of Fig. 4(A), a pedestrian HA is detected in the captured image, and it is determined whether or not there is a possibility that the pedestrian will enter the planned driving route RT of the vehicle CR based on the position, posture, etc. of the detected pedestrian HA. In the case of Fig. 4(B), a pedestrian HB is detected in the captured image, and it is determined whether or not there is a possibility that the pedestrian will enter the planned driving route RT of the vehicle CR based on the position, posture, etc. of the detected pedestrian HB.

[0096] In the cases of Figures 4(A) and 4(B), if it is detected that pedestrians HA and HB are attempting to cross the crosswalk or are looking toward vehicle CR, it is determined that pedestrians HA and HB may enter the planned driving route RT of vehicle CR.

[0097] In step S23, the processing control unit 110 reads the image data in the storage unit 120 and generates display image data. Then, the processing control unit 110 sends the display image data to the projection unit 251 of the display unit 250. As a result, an image imitating a gesture made by the driver to a pedestrian to encourage the pedestrian to move forward of the vehicle CR (encourage the pedestrian to walk on the crosswalk) is displayed on the screen 255. After this, the processing proceeds to step S26, which will be described later.

[0098] On the other hand, if the result of the determination in step S22 is negative (step S22: N), the process proceeds to step S24. In step S24, the processing control unit 110 determines whether an image representing the driver of the vehicle CR is being displayed. If the result of this determination is negative (step S24: N), the process proceeds to step S26.

[0099] If the result of the determination in step S24 is affirmative (step S24: Y), the process proceeds to step S25. In step S25, the processing control unit 110 performs control to erase the display of the image representing the driver of the vehicle CR. Then, the process proceeds to step S26.

[0100] In step S26, the processing control unit 110 determines whether or not it has received cancellation information to cancel the waiting for "giving priority to pedestrians" from the automatic driving device 290. If the result of this determination is negative (step S26: N), the processing returns to step S21.

[0101] If the result of the determination in step S26 is affirmative (step S26: Y), the process proceeds to step S27. In step S27, the processing control unit 110 determines whether an image representing the driver of the vehicle CR is being displayed, similar to the process in step S24 described above. If the result of this determination is negative (step S27: N), the process in step S13 ends. Then, the process returns to step S11 in FIG. 5 described above.

[0102] If the result of the determination in step S27 is affirmative (step S27: Y), the process proceeds to step S28. In step S28, the processing control unit 110 performs control to erase the display of the image representing the driver of the vehicle CR, similar to the process in step S25 described above. When the control to erase the display of the image is thus performed, the process of step S13 ends. Then, the process returns to step S11 in FIG. 5 described above.

[0103] <Type 2 image display control processing> The "second type image display control process" in step S15 described above will now be described.

[0104] As shown in FIG. 7, in this "second type image display control process," first, in step S31, the processing control unit 110 analyzes the captured image sent from the photographing unit 210, detects other vehicles and the operators of those other vehicles in the image, and detects the movements of the detected other vehicles and the line of sight of the operators.

[0105] Subsequently, in step S32, the processing control unit 110 determines, based on the image analysis in step S31, whether or not the second type image display conditions are met, that is, the other vehicle is waiting for the vehicle CR to move, the operator's line of sight is directed toward the vehicle CR, and the distance between the vehicle CR and the other vehicle is within a predetermined distance. If the result of this determination is affirmative (step S32: Y), the processing proceeds to step S33.

[0106] The processing control unit 110 may analyze the captured image sent from the photographing unit 210, and in particular detect the presence or absence of another vehicle (operator of the other vehicle) in the image that may enter the planned traveling route RT of the vehicle CR. In this case, the processing control unit 110 detects whether or not another vehicle (operator of the other vehicle) is present in the captured image, and if detected, determines whether or not the other vehicle (operator of the other vehicle) is likely to enter the planned traveling route RT of the vehicle CR based on the position, behavior, etc. of the detected other vehicle (operator of the other vehicle).

[0107] In the case of Fig. 4(B), another vehicle T is detected in the captured image, and it is determined whether or not the pedestrian is likely to enter the planned traveling route RT of the vehicle CR based on the position, behavior, etc. of the detected other vehicle T. In the case of Fig. 4(B), when the other vehicle T is traveling straight, it will block the planned traveling route RT of the vehicle CR, and therefore it is determined that the other vehicle is likely to enter the planned traveling route RT of the vehicle CR.

[0108] In step S33, processing control unit 110 reads the image data in storage unit 120 and generates display image data. Then, processing control unit 110 sends the display image data to projection unit 251 of display unit 250. As a result, an image imitating a gesture made by the driver to the operator of another vehicle to encourage the other vehicle ahead of vehicle CR to move is displayed on screen 255. Thereafter, processing proceeds to step S36, which will be described later.

[0109] On the other hand, if the result of the determination in step S32 is negative (step S34: N), the process proceeds to step S34. In step S34, the processing control unit 110 determines whether an image representing the driver of the vehicle CR is being displayed. If the result of this determination is negative (step S34: N), the process proceeds to step S36.

[0110] If the result of the determination in step S34 is affirmative (step S34: Y), the process proceeds to step S35. In step S35, the processing control unit 110 performs control to erase the display of the image representing the driver of the vehicle CR. Then, the process proceeds to step S36.

[0111] In step S36, the processing control unit 110 determines whether or not it has received cancellation information to cancel the waiting to "allow other vehicles to move ahead" from the automatic driving device 290. If the result of this determination is negative (step S36: N), the processing returns to step S31.

[0112] If the result of the determination in step S36 is affirmative (step S36: Y), the process proceeds to step S37. In step S37, the processing control unit 110 determines whether an image representing the driver of the vehicle CR is being displayed, similar to the process in step S34 described above. If the result of this determination is negative (step S37: N), the process in step S15 ends. Then, the process returns to step S11 in FIG. 5 described above.

[0113] If the result of the determination in step S37 is affirmative (step S37: Y), the process proceeds to step S38. In step S38, the processing control unit 110 performs control to erase the display of the image representing the driver of the vehicle CR, similar to the process in step S35 described above. When the control to erase the display of the image is thus performed, the process of step S15 ends. Then, the process returns to step S11 in FIG. 5 described above.

[0114] FIG. 8 shows an example of an image of the driver (male) of the vehicle CR, which is displayed on the screen 255 by the processing in steps S23 and S33, as viewed from outside the vehicle CR.

[0115] As described above, in this embodiment, when waiting information indicating that a pedestrian is to be given priority is received from the automatic driving device 290, the processing control unit 110 analyzes the captured image sent from the photographing unit 210 to detect a pedestrian in the image and detects at least one of the posture, line of sight, and facial direction of the detected pedestrian. Then, when there is no driver in the vehicle CR, the processing control unit 110 performs a determination process based on the results of the analysis process of the captured image to determine whether the pedestrian is in a posture ready to move in front of the vehicle CR and whether the pedestrian's line of sight is directed toward the vehicle CR. If the result of the determination process is affirmative, the processing control unit 110 controls the screen 255 to display an image that mimics a gesture made by the driver of the vehicle CR to the pedestrian to encourage them to move in front of the vehicle CR.

[0116] This allows pedestrians who wish to cross the crosswalk to be encouraged to do so by gestures, making it possible to communicate with pedestrians in the same way as with manned vehicles.

[0117] Furthermore, in this embodiment, when waiting information indicating that another vehicle will move ahead is received from the automatic driving device 290, the processing control unit 110 analyzes the captured image sent from the photographing unit 210 to detect the other vehicle and the operator of the other vehicle in the image, and detects the movement of the detected other vehicle and the line of sight or facial direction of the operator. Then, when there is no driver in the vehicle CR, the processing control unit 110 performs a determination process based on the results of the analysis process of the captured image to determine whether the other vehicle is waiting to move ahead of the vehicle CR, whether the operator's line of sight is directed toward the vehicle CR, and whether the distance between the vehicle CR and the other vehicle is within a predetermined distance. Then, if the result of the determination process is positive, the processing control unit 110 controls the screen 255 to display an image that imitates a gesture that the driver of the vehicle CR makes to the operator of the other vehicle to encourage the other vehicle, which is ahead of the vehicle CR, to move.

[0118] This allows the operator of another vehicle who is waiting for the vehicle CR to pass in order to make a right turn, or the operator of another vehicle who is waiting for the vehicle CR to move in order to park in a parking lot, to be prompted to move ahead by gestures. This makes it possible to realize communication with the operator of another vehicle, just like in the case of a manned vehicle.

[0119] Therefore, according to this embodiment, in the case of an autonomous vehicle without a driver, communication with people outside the vehicle can be realized, just as in the case of a manned vehicle, contributing to smooth and safe driving.

[0120] [Modification of the embodiment] The present invention is not limited to the above-described embodiment, and various modifications are possible.

[0121] For example, in the above embodiment, the driver displayed on the screen is a male. However, the attributes of the driver (excluding children) displayed on the screen may be changed depending on the attributes of the pedestrian or the operator of another vehicle, such as the gender, age, and nationality. For example, if the pedestrian is a child, an image of a young woman may be displayed on the screen. This can give the child a sense of security.

[0122] In addition, drivers with the same attributes as pedestrians and operators of other vehicles (excluding children) may be displayed on the screen, which can give pedestrians and operators of other vehicles a sense of security.

[0123] Furthermore, in the above embodiment, the present invention is applied to the vehicle shown in Figures 3 and 8, but it goes without saying that the present invention may also be applied to other vehicles such as trucks and heavy machinery vehicles.

[0124] In the above embodiment, an image of the vehicle driver is projected onto a screen disposed on the back of the front seat. Alternatively, a transparent screen may be disposed on a glass of the vehicle (for example, a windshield or glass on the side of the vehicle) and an image of the vehicle driver may be projected onto the screen.

[0125] Furthermore, the above examples can be appropriately modified in the same manner as the above-described embodiments. [Explanation of symbols]

[0126] 100...Image display device 110... Processing control unit (detection unit, display control unit) 700 ... Image display device 730 ... Detector 750 ... Display control unit

Claims

1. An image display device to be disposed in an autonomously driven vehicle, a display control unit that displays, when the driver of the vehicle is not present, an image that is visible to a pedestrian who may enter a planned driving route of the vehicle, simulating the driver of the vehicle making a gesture toward the pedestrian; a detection unit that detects the presence or absence of the pedestrian, and at least one of the posture, line of sight, and face direction of the pedestrian; the display control unit displays the image when it is determined that the pedestrian recognizes the vehicle based on the detection result by the detection unit. An image display device characterized by:

2. The image display device according to claim 1 , wherein the gesture is a gesture that prompts the pedestrian to move forward of the vehicle.

3. An image display device to be disposed in an autonomously driven vehicle, a display control unit that displays, when the driver of the vehicle is not present, an image that is visible to an operator of another vehicle that may enter the planned driving route of the vehicle, simulating a driver in the vehicle making a gesture toward the operator; a detection unit that detects a distance between the vehicle and the other vehicle, the presence or absence of the operator, a movement of the other vehicle, and a line of sight or a facial direction of the operator; the display control unit displays the image when it determines, based on the detection result by the detection unit, that the distance is within a predetermined distance, that the other vehicle is waiting for the vehicle to move ahead, and that the operator's line of sight is directed toward the vehicle. An image display device characterized by:

4. 4. The image display device according to claim 3, wherein the gesture is a gesture that prompts the operator to move the other vehicle preceding the vehicle.

5. 5. The image display device according to claim 1, wherein the display control unit displays the image on a glass of the vehicle.

6. 5. The image display device according to claim 1, wherein the display control unit displays the image at a position corresponding to a seat of the vehicle.

7. 1. An image display method for use in an image display device including: a display control unit that is disposed in an autonomously driven vehicle and that, when a driver of the vehicle is not present, displays an image that is visible to a pedestrian who may enter a planned driving route of the vehicle, simulating a driver in the vehicle making a gesture toward the pedestrian; and a detection unit, a detection step in which the detection unit detects the presence or absence of the pedestrian, and at least one of the posture, line of sight, and face direction of the pedestrian; a display control step of displaying the image when the display control unit determines, based on the detection result by the detection unit, that the pedestrian recognizes the vehicle; An image display method comprising:

8. 1. An image display method for use in an image display device including: a display control unit that is disposed in an autonomously driven vehicle and that, when a driver of the vehicle is not present, displays an image that is visible to an operator of another vehicle that may enter a planned driving route of the vehicle, simulating a driver of the vehicle making a gesture toward the operator; and a detection unit, a detection step in which the detection unit detects a distance between the vehicle and the other vehicle, the presence or absence of the operator, the movement of the other vehicle, and the line of sight or face direction of the operator; a display control step of displaying the image when the display control unit determines, based on the detection result by the detection unit, that the distance is within a predetermined distance, that the other vehicle is waiting to move ahead of the vehicle, and that the operator's line of sight is directed toward the vehicle; An image display method comprising:

9. 8. An image display program for causing a computer included in an image display device to execute the image display method according to claim 7.

10. 9. An image display program for causing a computer included in an image display device to execute the image display method according to claim 8.

11. 10. A recording medium having the image display program according to claim 9 recorded thereon so as to be readable by a computer included in an image display device.

12. 11. A recording medium having the image display program according to claim 10 recorded thereon so as to be readable by a computer included in an image display device.

Images