Image display unit and image display method

Abstract

To achieve, in a case of an automatic driving vehicle in which a driver is absent, communication with a person outside the vehicle and contribute to smooth and safe driving.SOLUTION: When receiving standby information to the effect that walking of a pedestrian is prioritized from an automatic driving device 990, a detection unit 730 performs analysis processing on a photographic image transmitted from a photographing unit 910 and detects the attitude and line of sight of the pedestrian. The detection unit 730 then transmits a result of detection to a display control unit 750. When a driver is not in a vehicle CR, the display control unit 750 determines whether the pedestrian is in the attitude of a preparation posture for moving in front of the vehicle CR and the line of sight of the pedestrian is directed to the vehicle CR on the basis of the result of detection performed by the detection unit 730. When a result of the determination is affirmative, the display control unit 750 performs control of displaying, on a screen 955, an image imitating a gesture performed by the driver of the vehicle CR to the pedestrian, the gesture urging a movement in front of the vehicle CR.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image display device, an image display method, an image display program, and a recording medium on which the image display program is recorded.

Background Art

[0002] In recent years, development has been underway toward the practical use of driverless vehicles that do not require driving operations by a driver. As one of the technologies related to such driverless vehicles, there is a technology for taking measures against people outside the vehicle, and various technologies for taking such measures have been proposed.

[0003] As one of such proposed technologies, there is a technology for notifying a person that the person has been recognized by an autonomous vehicle when the autonomous vehicle recognizes the person (see Patent Document 1; hereinafter referred to as "Conventional Example 1"). In the technology of this Conventional Example 1, a detection process of a person existing around the autonomous vehicle is performed from a captured image captured by a camera. Then, when a person is detected by the detection process, a laser beam is irradiated toward the person or a sound is output.

[0004] Also, although it is not a technology related to driverless vehicles, there is something that easily calls the attention of pedestrians (see Patent Document 2; hereinafter referred to as "Conventional Example 2"). In the technology of this Conventional Example 2, a pedestrian is discriminated based on an image captured by a camera, and a determination process is performed as to whether or not the pedestrian exists in a set area. Then, when a pedestrian exists in the set area, an image such as a crosswalk is projected onto the road surface.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] By the way, in order to drive smoothly and safely on the road, communication such as signals from the driver to pedestrians to encourage priority walking at crosswalks and signals from the driver to the operators of other vehicles to encourage them to drive ahead of other vehicles is important. However, in an autonomous vehicle, the driver may not be present during driving. In the technology of the above-described conventional example 1, the fact that a person is recognized by the autonomous vehicle is notified to the person, but since no notification to encourage priority walking is made, the person who receives the notification generally stays on the spot and waits for the vehicle to pass. As a result, the technology of conventional example 1 is not sufficient as a notification for driving smoothly and safely on the road.

[0007] On the other hand, when the technology of conventional example 2 is applied to the technology related to driverless vehicles, it becomes possible to communicate with pedestrians to encourage priority walking in an autonomous vehicle. However, in the method of projecting an image onto the road, depending on the state of the road such as unevenness, the image projected onto the road may be distorted and the content of the image may not be conveyed to the pedestrians. As a result, there is room for improvement in the notification for driving smoothly and safely on the road in the technology of conventional example 2.

[0008] For this reason, even in the case of an autonomous vehicle without a driver, a technology that can realize communication with people outside the vehicle and contribute to smooth and safe driving, similar to the case of a manned vehicle, is desired. Meeting such a requirement is cited as one of the problems to be solved by the present invention.

Means for Solving the Problems

[0009] The invention according to claim 1 is an image display device disposed in a vehicle capable of autonomous driving, comprising: a display control unit that, when the driver of the vehicle is absent, displays an image simulating that the driver in the vehicle is making a gesture to a pedestrian who may enter the planned driving route of the vehicle and is visible; and a detection unit that detects at least one of the presence or absence of the pedestrian, and the posture, line of sight, and face direction of the pedestrian. The display control unit displays the image when it determines, based on the detection result by the detection unit, that the pedestrian recognizes the vehicle. The image display device is characterized by the above. The invention according to claim 3 is an image display device disposed in a vehicle capable of autonomous driving, comprising: a display control unit that, when the driver of the vehicle is absent, displays an image simulating that the driver in the vehicle is making a gesture to an operator of another vehicle that may enter the planned driving route of the vehicle and is visible; and a detection unit that detects the distance between the vehicle and the other vehicle, the presence or absence of the operator, and the operation of the other vehicle and the line of sight or face direction of the operator. The display control unit displays the image when it determines, based on the detection result by the detection unit, that the distance is within a predetermined distance, the other vehicle is waiting for the vehicle to move forward, and the line of sight of the operator is directed at the vehicle. The image display device is characterized by the above.

[0010] The invention according to claim 7 is an image display method used in an image display device disposed in a vehicle capable of autonomous driving, comprising: a display control unit that, when the driver of the vehicle is absent, displays an image simulating that the driver in the vehicle is making a gesture to a pedestrian who may enter the planned driving route of the vehicle and is visible; and a detection unit. The method includes a detection step in which the detection unit detects at least one of the presence or absence of the pedestrian, and the posture, line of sight, and face direction of the pedestrian, and a display control step in which the display control unit displays the image when it determines, based on the detection result by the detection unit, that the pedestrian recognizes the vehicle. The image display method is characterized by the above. The invention according to claim 8 is disposed in a vehicle capable of autonomous driving, and when the driver of the vehicle is absent, an image simulating that the driver in the vehicle is gesturing to the operator of another vehicle that may enter the planned driving route of the vehicle is displayed so as to be visible to the operator. A display control unit; a detection unit; In an image display method used in an image display device including: a detection step in which the detection unit detects a distance between the vehicle and the other vehicle, the presence or absence of the operator, and the operation of the other vehicle and the line of sight or face direction of the operator; and a display control step in which the display control unit displays the image when it is determined based on the detection result by the detection unit that the distance is within a predetermined distance, the other vehicle is waiting for the vehicle to move forward, and the line of sight of the operator is directed toward the vehicle. An image display method characterized by comprising:

[0011] An image display program, characterized in that a computer included in an image display device executes the image display method according to claim 7. An image display program, characterized in that a computer included in an image display device executes the image display method according to claim 8.

[0012] A recording medium, characterized in that the image display program according to claim 9 is recorded so as to be readable by a computer included in an image display device. A recording medium, characterized in that the image display program according to claim 10 is recorded so as to be readable by a computer included in an image display device.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

[0014] Hereinafter, an embodiment of the present invention will be described with reference to FIG. 1. In the following description and drawings, the same or equivalent elements are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0015] [Configuration] In FIG. 1, the configuration of an image display device 700 according to an embodiment is shown in a block diagram. The image display device 700 is arranged inside a vehicle CR capable of autonomous driving. In the present embodiment, in addition to the image display device 700, a photographing unit 910 and a display unit 950 are arranged in the vehicle CR, and these are connected to the image display device 700. Further, an automatic driving device 990 is arranged in the vehicle CR, and the automatic driving device 990 is connected to the image display device 700.

[0016] The above-described imaging unit 910 is configured to include a camera that captures at least the front of the vehicle CR in an imaging field of view including the traveling direction of the vehicle CR. The imaging result (captured image) by the imaging unit 910 is sent to the image display device 700. Although not shown, the imaging result by the imaging unit 910 is also sent to the automatic driving device 990 and is utilized for automatic driving control by the automatic driving device 990.

[0017] In the present embodiment, the above-described display unit 950 includes a projection unit 951 and a screen 955.

[0018] The projection unit 951 is configured to include a video projector. The projection unit 951 receives the display image data sent from the image display device 700. Then, the projection unit 951 projects and displays an image corresponding to the display image data on the screen 955.

[0019] In the present embodiment, the screen 955 is disposed in front of the backrest of the front seat. The screen 955 displays the image projected by the projection unit 951 so that it can be visually recognized by a person outside the vehicle CR.

[0020] The above-described automatic driving device 990 controls the start, acceleration / deceleration, steering, stop, etc. of the vehicle CR according to the driving plan and driving environment. The automatic driving device 990 includes a storage unit (not shown) that stores map information. Then, the automatic driving device 990 uses the driving information sent from a driving information detection unit (not shown) such as a vehicle speed detection unit and a position detection unit, and the captured image sent from the imaging unit 910, and appropriately refers to the map information to perform automatic driving control.

[0021] As such automatic driving control, when the automatic driving device 990 detects a pedestrian trying to cross a crosswalk on a road without a traffic signal, for example, it performs control to stop the vehicle CR until the pedestrian crosses the crosswalk. When performing such control, the automatic driving device 990 generates standby information indicating that the walking (crossing) of the pedestrian is prioritized and sends it to the image display device 700.

[0022] Also, as such automatic driving control, when the automatic driving device 990 detects that another vehicle traveling in the oncoming lane is waiting for the vehicle CR to pass, for example, due to a right turn or the like, in accordance with the driving environment, in order to give precedence (priority) to the right turn of the other vehicle, it performs control to decelerate or stop the vehicle CR. And when performing such control, the automatic driving device 990 generates standby information indicating that the movement of the other vehicle is to be given precedence (priority), and sends it to the image display device 700. Here, the other vehicles include automobiles, motorcycles, bicycles, and the like.

[0023] Also, as such automatic driving control, when the automatic driving device 990 detects that another vehicle is waiting for the vehicle CR to move in order to park in a parking lot, for example, in accordance with the driving environment, in order to give precedence to the parking of the other vehicle in the parking lot, it performs control to stop the vehicle CR. And when performing such control, the automatic driving device 990 generates standby information indicating that the movement of the other vehicle is to be given precedence (priority), and sends it to the image display device 700.

[0024] <Configuration of the image display device 700> The configuration of the above-mentioned image display device 700 will be described.

[0025] As shown in FIG. 1, the image display device 700 includes a storage unit 710 and a detection unit 730. The image display device 700 also includes a display control unit 750.

[0026] The above-mentioned storage unit 710 is configured to include a non-volatile memory element. In the storage unit 710, image data simulating the driver of the vehicle CR and camera information indicating the arrangement position and optical characteristics of the camera included in the imaging unit 910 are stored. The detection unit 730 and the display control unit 750 can access the storage unit 710.

[0027] The image data simulating the driver of vehicle CR is the data of an image simulating that when a person existing in front of the outside of vehicle CR is being looked at by the said vehicle CR, the driver is making a gesture to the said person. An example of the content of the said gesture is a gesture to prompt the movement in front of vehicle CR to a pedestrian. In other words, it is a gesture indicating the intention to give priority to the walking (crossing) of the said pedestrian with respect to a pedestrian who may enter the planned travel route of vehicle CR.

[0028] Also, another example of the content of the said gesture is a gesture to prompt the movement of another vehicle that has preceded the travel of vehicle CR to the operator of another vehicle. In other words, it is a gesture indicating the intention to give priority to the travel of the said other vehicle with respect to the operator of another vehicle that may enter the planned travel route of vehicle CR.

[0029] The above-mentioned detection unit 730 receives the captured image sent from the imaging unit 910. And when the detection unit 730 receives the standby information sent from the automatic driving device 990, it performs the analysis process of the said captured image to detect information regarding a pedestrian who may enter the planned travel route of vehicle CR, or information regarding another vehicle that may enter the planned travel route of vehicle CR and the operator of the said other vehicle (these information are also referred to as "information regarding other moving objects"). That is, the detection unit 730 is configured to detect the presence or absence of a person (pedestrian, operator of another vehicle, etc.) or another vehicle that may enter the planned travel route of vehicle CR.

[0030] When another vehicle is detected, the detection unit 730 is configured to detect the distance between vehicle CR and the other vehicle based on the analysis result of the captured image. The detection result by the detection unit 730 is sent to the display control unit 750. Details of the process by the detection unit 730 will be described later.

[0031] The above display control unit 750 receives information about other moving objects sent from the detection unit 730. Subsequently, when the driver is not in the vehicle CR and the standby information sent from the automatic driving device 990 is received, if the display control unit 750 determines that an image simulating the driver should be displayed based on the content of the standby information and the information about other moving objects, it reads the image data in the storage unit 710 to generate display image data. Then, the display control unit 750 sends the display image data to the display unit 950. Details of the processing by the display control unit 750 will be described later.

[0032] In this embodiment, there may be a driver detection unit that detects the presence of a driver in the vehicle CR. The driver detection unit may detect the presence of the driver in the vehicle CR from signals from a seating sensor or the like installed on the seat of the vehicle CR.

[0033] Also, the driver detection unit may detect the presence of the driver in the vehicle CR according to the type of the driving mode of the automatic driving being executed by the automatic driving device 990. For example, when the automatic driving device 990 is executing a driving mode such as the "driverless driving mode" that is clearly executed when the driver is not present in the vehicle CR, the driver detection unit may detect that the driver is not present in the vehicle CR.

[0034] [Operation] Regarding the operation of the image display device 700 configured as described above, the description will mainly focus on the processing by the detection unit 730 and the display control unit 750.

[0035] As a premise, it is assumed that the photographing unit 910 has started operating and is sequentially sending the photographing results (photographed images) to the image display device 700 and the automatic driving device 990. Also, it is assumed that the automatic driving device 990 has started operating and sends the standby information to the image display device 700 when the above-described standby information is generated.

[0036] [Detection Process of Information about Other Moving Objects] The detection process of information regarding other moving objects is executed when standby information sent from the automatic driving device 990 is received. In such a detection process of information regarding other moving objects, when the standby information sent from the automatic driving device 990 is information indicating that the walking of pedestrians is to be prioritized, the detection unit 730 analyzes the captured image sent from the imaging unit 910 to detect pedestrians in the image.

[0037] Note that the detection unit 730 may further detect at least one of the posture, line of sight, and face direction of the detected pedestrian by analyzing the captured image sent from the imaging unit 910. Then, the detection unit 730 sequentially sends the detection result of the pedestrian and the detection result of at least one of the posture, line of sight, and face direction of the pedestrian to the display control unit 750 as information regarding the pedestrian during the period in which standby information indicating that the walking of the pedestrian is to be prioritized is received. In particular, the detection unit 730 may detect the presence or absence of a person (pedestrian, operator of another vehicle, etc.) or another vehicle that may enter the planned travel route of the vehicle CR.

[0038] Also, in the detection process of information regarding other moving objects, when the standby information sent from the automatic driving device 990 is information indicating that the movement of another vehicle is to be prioritized, the detection unit 730 analyzes the captured image sent from the imaging unit 910 to detect the other vehicle and the operator of the other vehicle in the image. Note that the detection unit 730 may further detect the movement of the detected other vehicle and the line of sight of the operator by analyzing the captured image sent from the imaging unit 910.

[0039] In addition, the detection unit 730 detects the distance between the vehicle CR and the other vehicle based on the captured image sent from the imaging unit 910 and the camera information stored in the storage unit 710. Then, the detection unit 730 sequentially sends the detection result of the other vehicle and the operator of the other vehicle, the detection result of the movement of the other vehicle, the detection result of the line of sight or face direction of the operator of the other vehicle, and the detection result of the distance between the vehicle CR and the other vehicle to the display control unit 750 as information regarding the other vehicle and the operator during the period in which standby information indicating that the movement of the other vehicle is to be prioritized is received.

[0040] <Image display control process simulating a driver> In the image display control process simulating a driver, the display control unit 750 first determines whether an input indicating that the driver is present in the vehicle CR has been made to the input unit, thereby performing a non - existence determination process to determine whether the driver is not present in the vehicle CR. If the result of the non - existence determination process is negative, the display control unit 750 repeats the non - existence determination process.

[0041] If the result of the non - existence determination process is positive, the display control unit 750 performs a standby information acquisition determination process to determine whether it has received the standby information sent from the automatic driving device 990. If the result of the standby information acquisition determination process is negative, the standby information acquisition determination process ends, and the above - mentioned non - existence determination process is started again.

[0042] On the other hand, if the result of the standby information acquisition determination process is positive, the display control unit 750 performs an information determination process to determine whether the content of the standby information is standby information indicating "giving priority to the walking of pedestrians" or standby information indicating "giving priority to the movement of other vehicles".

[0043] If, as a result of the information determination process, the content of the standby information is information indicating "giving priority to the walking of pedestrians", the display control unit 750 performs a determination process (posture determination process) based on the information about the pedestrian sent from the detection unit 730 to determine whether the pedestrian is in a posture of being ready to move in front of the vehicle CR (a posture of walking on a crosswalk in front of the vehicle). Here, the ready posture means a state where the pedestrian is about to move (cross) in front of the vehicle CR. In other words, it is a state before the pedestrian starts to cross in front of the vehicle CR.

[0044] Further, the display control unit 750 performs a determination process (gaze determination process) on whether or not the pedestrian's line of sight is directed toward the vehicle CR based on the information about the pedestrian sent from the detection unit 730. In the present embodiment, the gaze determination process can be rephrased as a process for determining whether or not the pedestrian recognizes the vehicle CR. That is, when the result of the gaze determination process is affirmative, it is determined that the pedestrian recognizes the vehicle CR. When the result of the posture determination process or the gaze determination process is affirmative, the display control unit 750 reads the image data in the storage unit 710 and generates display image data. Then, the display control unit 750 sends the display image data to the display unit 950. As a result, an image imitating a gesture for prompting the movement in front of the vehicle CR performed by the driver to the pedestrian is displayed on the screen 955.

[0045] Note that the display control unit 750 may read the image data in the storage unit 710 and generate display image data only when both the above-described posture determination process and gaze determination process are affirmative.

[0046] Further, the posture determination process and the gaze determination process can be omitted. That is, when it is determined that a pedestrian exists based on the information about the pedestrian sent from the detection unit 730, the display control unit 750 may read the image data in the storage unit 710 and generate display image data (even if the line of sight of the pedestrian is not directed toward the vehicle CR).

[0047] On the one hand, when the content of the waiting information is "information indicating to let the movement of other vehicles take precedence" through the information determination process, the display control unit 750 performs a determination process based on the information about other moving objects and the operator sent from the detection unit 730 to determine whether the other vehicle is waiting for the movement of the vehicle CR, whether the operator's line of sight is directed towards the vehicle CR, and whether the distance between the vehicle CR and the other vehicle is within a predetermined distance. When the result of the determination process is affirmative, the display control unit 750 reads the image data in the storage unit 710 to generate display image data. Then, the display control unit 750 sends the display image data to the display unit 950. As a result, an image imitating a gesture for the driver to prompt the movement of the other vehicle preceding the travel of the vehicle CR to the operator of the other vehicle is displayed on the screen 955. Here, the predetermined distance is determined in advance based on experiments, simulations, experience, etc. from the perspective of enabling the operator of the other vehicle to recognize the image imitating the driver of the vehicle CR.

[0048] In addition, in this determination process, the determination as to whether the line of sight of the operator of the other vehicle is directed towards the vehicle CR may not be performed. That is, based on the information about other moving objects (the other vehicle or the operator of the other vehicle) and the operator sent from the detection unit 730, when it is determined that the other moving object exists, the display control unit 750 may read the image data in the storage unit 710 to generate display image data (even if the line of sight of the operator is not directed towards the vehicle CR).

[0049] As described above, in this embodiment, when receiving the waiting information indicating to give precedence to the walking of pedestrians from the automatic driving device 990, the detection unit 730 performs an analysis process on the captured image sent from the imaging unit 910 to detect the pedestrians in the image and detect at least one of the postures, lines of sight, and face orientations of the detected pedestrians. Then, the detection unit 730 sends the detection result to the display control unit 750 as information about the pedestrians.

[0050] When the display control unit 750 determines that there is no driver in the vehicle CR and has received standby information from the automatic driving device 900 to prioritize the walking of pedestrians, it performs a determination process based on the information about the pedestrians sent from the detection unit 730 to check whether the pedestrians are in a posture ready to move in front of the vehicle CR and whether their line of sight is directed towards the vehicle CR. If the result of this determination process is affirmative, the display control unit 750 controls the screen 955 to display an image imitating the gesture that a driver of the vehicle CR would make to prompt the movement in front of the vehicle CR for the pedestrians.

[0051] This enables pedestrians who want to cross the crosswalk to be prompted to cross by gesture. Therefore, similar to the case of a manned vehicle, communication with pedestrians can be realized.

[0052] Also, in this embodiment, when the detection unit 730 receives standby information from the automatic driving device 990 to give precedence to the movement of other vehicles, it performs an analysis process on the captured image sent from the imaging unit 910 to detect other vehicles and their operators in the image, and detects the movements of the detected other vehicles and the line of sight or the direction of the face of the operators. Then, the detection unit 730 sends the detection results to the display control unit 750 as information about other moving objects and operators.

[0053] When the display control unit 750 determines that there is no driver in the vehicle CR and has received standby information from the automatic driving device 990 to give precedence to the movement of other vehicles, it performs a determination process based on the information about other moving objects and operators sent from the detection unit 730 to check whether the other vehicles are waiting for the vehicle CR to move first, whether the line of sight of the operators is directed towards the vehicle CR, and whether the distance between the vehicle CR and the other vehicles is within a predetermined distance. If the result of this determination process is affirmative, the display control unit 750 controls the screen 955 to display an image imitating the gesture that a driver of the vehicle CR would make to prompt the movement of the other vehicles that have preceded the vehicle CR for the operators of the other vehicles.

[0054] As a result, for the operators of other vehicles waiting for the vehicle CR to pass for a right turn or the like, or for the operators of other vehicles waiting for the movement of the vehicle CR to park in a parking lot, the preceding movement can be prompted by gestures. Therefore, it is possible to realize communication with the operators of other vehicles, similar to the case of a manned vehicle.

[0055] Therefore, according to the present embodiment, in the case of an autonomous vehicle without a driver, it is possible to realize communication with people outside the vehicle, similar to the case of a manned vehicle, which contributes to smooth and safe driving.

[0056] [Modifications of the Embodiment] The present invention is not limited to the above-described embodiment, and various modifications are possible.

[0057] For example, in the above-described embodiment, an image imitating the driver's gesture is displayed, but an image imitating the gesture of a passenger other than the driver may be displayed.

[0058] Also, in the above-described embodiment, an image imitating the driver of the vehicle is projected onto the screen arranged on the backrest of the front seat. In contrast, a transparent screen may be arranged on the glass of the vehicle (for example, the windshield), and an image imitating the driver of the vehicle CR may be projected onto the screen.

[0059] Also, in the above-described embodiment, an image imitating the driver is displayed by the method of the video projector projecting the image onto the screen, but for example, an image imitating the driver may be displayed on a liquid crystal display. Further, by displaying the image on a hologram display or the like, the passengers in the vehicle may be visually recognized three-dimensionally by people outside.

[0060] In addition, the image (moving image) simulating the driver may be changed based on the information about the pedestrian or the operator of another vehicle detected by the detection unit. For example, the image may be displayed such that the line of sight of the driver in the image faces the direction in which the pedestrian or the operator of another vehicle exists. Further, when it is detected that the pedestrian or the operator of another vehicle has bowed, an image in which the driver in the image also bows accordingly may be displayed. By doing so, the pedestrian or the operator of another vehicle can recognize that the other party (the vehicle) has recognized their presence, so that a sense of security can be given to the pedestrian or the operator of another vehicle.

[0061] In addition, when allowing an external person to visually recognize the passenger in the vehicle, in addition to projecting an image onto the screen as described above, a so-called projection mapping technique may be used. For example, a humanoid three-dimensional structure may be arranged on the front seat, and an image simulating the passenger may be projected onto the three-dimensional structure. In this case, it is desirable that the three-dimensional structure be configured to be stowable in the seat.

[0062] In addition, when displaying an image simulating the passenger on a display or the like, depending on the direction in which an external person exists, the image may be transformed by image transformation such as affine transformation so that the image simulating the passenger can be visually recognized without a sense of incongruity when viewed from the direction in which the external person exists. When the above-mentioned affine transformation or the like is not performed, the image may be displayed only when an external person exists in a direction in which the image simulating the passenger can be visually recognized without a sense of incongruity. For example, when an external person exists in front of the vehicle, and only when the external person can visually recognize the image simulating the passenger without a sense of incongruity, the image may be displayed only when it is detected that the external person exists in front of the vehicle.

[0063] In the above-described embodiment, a person on the vehicle traveling direction side is detected based on the camera image. In contrast, a person on the vehicle traveling direction side may be detected based on a known Lidar (Light Detection and Ranging) measurement technique.

[0064] In addition, a part or all of the detection unit and the display control unit of the image display device in the above embodiment may be configured as a computer as an arithmetic unit including a central processing unit (CPU) or the like, and a program prepared in advance may be executed by the computer so that a part or all of the functions of these elements are fulfilled. This program is recorded on a computer-readable recording medium such as a hard disk, a CD-ROM, or a DVD, and is loaded from the recording medium and executed by the computer. Further, this program may be obtained in a form recorded on a portable recording medium such as a CD-ROM or a DVD, or may be obtained in a form of distribution via a network such as the Internet.

[0065] Also, in the above embodiment, in FIG. 1, for convenience of explanation, the automatic driving device 990 and the image display device 700 are separately described, but at least a part of the functions of the image display device 700 (for example, the detection unit 730, etc.) may be included in the automatic driving device 990.

Example

[0066] Hereinafter, an embodiment of the present invention will be described with reference to FIGS. 2 to 8. In the following description and drawings, the same or equivalent elements are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0067] [Configuration] FIG. 2 shows a schematic configuration of an image display device 100 according to an embodiment in a block diagram. The image display device 100 is an aspect of the image display device 700 (see FIG. 1) according to the above-described embodiment.

[0068] The image display device 100 is arranged inside the vehicle CR capable of autonomous driving. Also, in this embodiment, a photographing unit 210 and a display unit 250 are arranged in the vehicle CR, and these are connected to the image display device 100. Further, an automatic driving device 290 is arranged in the vehicle CR, and the automatic driving device 290 is connected to the image display device 100.

[0069] In this embodiment, the above-described photographing unit 210 is configured to include a camera that photographs at least the front of the vehicle CR in a photographing field of view including the traveling direction of the vehicle CR. The photographing unit 210 may include a camera capable of photographing the entire surroundings, or may be configured to be capable of photographing the entire surroundings by combining a plurality of cameras. The photographing result (photographed image) by the photographing unit 210 is sent to the image display device 100. Although not shown, the photographing result by the photographing unit 210 is also sent to the automatic driving device 290 and is utilized for automatic driving control by the automatic driving device 290. That is, the photographing unit 210 functions as the photographing unit 910 of the above-described embodiment.

[0070] In this embodiment, the above-described display unit 250 includes a projection unit 251, a half mirror 253, and a screen 255. Here, the arrangement relationship of the projection unit 251, the half mirror 253, and the screen 255 is shown in FIG. 3.

[0071] The projection unit 251 is configured to include a video projector and is arranged near the footrest of the front seat. The projection unit 251 receives the display image data sent from the image display device 100. Then, the projection unit 251 irradiates image light corresponding to the display image data toward the half mirror 253.

[0072] The half mirror 253 is arranged at the upper part of the front window. The half mirror 253 reflects the image light irradiated from the video projector. The image light reflected in this way travels toward the screen 255.

[0073] In this embodiment, the screen 255 is disposed in front of the backrest of the front seat. The screen 255 receives the image light projected by the projection unit 251 and reflected by the half mirror 253, and displays an image visible to people outside the vehicle CR.

[0074] That is, the display unit 250 is adapted to perform the function of the display unit 950 of the above-described embodiment.

[0075] The above-described automatic driving device 290 controls the start, acceleration / deceleration, steering, stop, etc. of the vehicle CR according to the driving plan and driving environment. The automatic driving device 290 includes a storage unit (not shown) that stores map information. Then, the automatic driving device 290 uses the driving information sent from a driving information detection unit (not shown) such as a vehicle speed detection unit and a position detection unit, and the captured image sent from the imaging unit 210, and appropriately refers to the map information to perform automatic driving control.

[0076] As such automatic driving control, when the automatic driving device 290 detects a pedestrian trying to cross a crosswalk on a road without a traffic signal, it performs control to stop the vehicle CR until the pedestrian crosses the crosswalk. When performing such control, the automatic driving device 290 generates standby information indicating that the pedestrian's walking is prioritized and sends it to the image display device 100. When the pedestrian trying to cross the crosswalk is no longer detected after the transmission of the standby information, the automatic driving device 290 generates standby cancellation information and sends it to the image display device 100.

[0077] Also, as such automatic driving control, when the automatic driving device 290 detects, for example, that another vehicle traveling in the oncoming lane is waiting for the vehicle CR to pass because of a right turn or the like, it performs control to decelerate or stop the vehicle CR in accordance with the driving environment in order to let the other vehicle make a right turn first. When performing such control, the automatic driving device 290 generates standby information indicating that it is letting the other vehicle move first and sends it to the image display device 100. When the other vehicle waiting for the vehicle CR to pass is no longer detected after the transmission of the standby information, the automatic driving device 290 generates standby cancellation information and sends it to the image display device 100.

[0078] Also, as such automatic driving control, when the automatic driving device 290 detects, for example, that another vehicle is waiting for the vehicle CR to move in order to park in a parking lot, it performs control to stop the vehicle CR in accordance with the driving environment in order to let the other vehicle park in the parking lot first. When performing such control, the automatic driving device 290 generates standby information indicating that it is letting the other vehicle move first and sends it to the image display device 100. When the other vehicle waiting for the vehicle CR to move is no longer detected after the transmission of the standby information, the automatic driving device 290 generates standby cancellation information and sends it to the image display device 100.

[0079] <Configuration of the image display device 100> Next, the configuration of the above-described image display device 100 will be described.

[0080] FIG. 2 shows a schematic configuration of an image display device 100 according to an embodiment in a block diagram. As shown in FIG. 2, the image display device 100 includes a processing control unit 110 and a storage unit 120.

[0081] The above-described processing control unit 110 comprehensively controls the entire image display device 100 and executes various processes. The processing control unit 110 includes a central processing unit (CPU) as an arithmetic means and its peripheral circuits. By executing various programs by the processing control unit 110, various functions as the image display device 100 are realized. Among such functions, the functions of the detection unit 730 and the display control unit 750 in the above-described embodiment are also included. Details of the processes executed by such a processing control unit 110 will be described later.

[0082] Note that the programs executed by the processing control unit 110 are recorded on a computer-readable recording medium such as a hard disk, a CD-ROM, or a DVD, and are loaded from the recording medium and executed. Further, this program may be acquired in a form recorded on a portable recording medium such as a CD-ROM or a DVD, or may be acquired in a form of distribution via a network such as the Internet.

[0083] The above-described storage unit 120 includes a non-volatile storage device such as a hard disk device, and stores various information data used in the image display device 100. Such information data includes image data simulating the driver of the vehicle CR, and camera information indicating the arrangement position and optical characteristics of the cameras included in the photographing unit 210. The storage unit 120 can be accessed by the processing control unit 110. That is, the storage unit 120 functions as the storage unit 710 in the above-described embodiment.

[0084] The image data simulating the driver of the vehicle CR is data of an image simulating a situation where a person existing in front of the vehicle CR looks at the vehicle CR and the driver makes a gesture to the person. The content of the gesture is a gesture for prompting the movement in front of the vehicle CR to a pedestrian, and is a gesture for prompting the movement of another vehicle that has preceded the running of the vehicle CR to an operator of another vehicle.

[0085] [Operation] Regarding the operation of the image display device 100 configured as described above, the description will mainly focus on the display control process of the image imitating the driver by the processing control unit 110. Also, the operation of the image display device 100 of this embodiment will be described with appropriate reference to FIGS. 4(A) and 4(B). Here, FIG. 4(A) is a drawing showing the positional relationship between the vehicle CR and the pedestrian HA at an intersection. Also, FIG. 4(B) is a drawing showing the positional relationship between the vehicle CR, the pedestrian HB, and the oncoming vehicle (other vehicle) T at an intersection. The dashed arrows in FIGS. 4(A) and 4(B) indicate the planned travel route RT of the vehicle CR, and the imaging range of the imaging unit 210 is schematically indicated by H. Note that this imaging range H is an example. For example, when the imaging unit 210 is configured by a camera capable of imaging the entire circumference, the imaging range H can be schematically represented as a circle covering the entire circumference centered on the vehicle CR.

[0086] As a premise, it is assumed that the imaging unit 210 has started operating and is sequentially sending the imaging results (imaging images) to the image display device 100. Also, it is assumed that the automatic driving device 290 has started operating and, when generating the standby information described above, sends the standby information to the image display device 100.

[0087] In the display control process of the image imitating the driver, as shown in FIG. 5, first, in step S11, the processing control unit 110 determines whether an input indicating that the driver is present inside the vehicle CR has been made to an input unit (not shown). By this, it is determined whether the vehicle CR is in a state where there is no driver. If the result of this determination is negative (step S11: N), the process of step S11 is repeated.

[0088] If the result of the determination in step S11 is affirmative (step S11: Y), the process proceeds to step S12. In step S12, the process control unit 110 determines whether it has received standby information from the automatic driving device 290 to "give priority to the walking of pedestrians". If the result of this determination is affirmative (step S12: Y), the process proceeds to step S13. In step S13, the process control unit 110 performs "first type image display control processing". Details of the processing in step S13 will be described later. When the processing in step S13 ends, the process returns to step S11.

[0089] If the result of the determination in step S12 is negative (step S12: N), the process proceeds to step S14. In step S14, the process control unit 110 determines whether it has received standby information from the automatic driving device 290 to "give priority to the movement of other vehicles". If the result of this determination is affirmative (step S14: Y), the process proceeds to step S15. In step S15, the process control unit 110 performs "second type image display control processing". Details of the processing in step S15 will be described later. When the processing in step S15 ends, the process returns to step S11.

[0090] If the result of the determination in step S14 is negative (step S14: N), the process returns to step S11. Thereafter, the processing of steps S11 to S15 is repeated.

[0091] <First type image display control processing> The "first type image display control processing" in step S13 described above will be explained.

[0092] As shown in FIG. 6, this "first type image display control processing" first, in step S21, the process control unit 110 analyzes the captured image sent from the imaging unit 210, detects a pedestrian in the image, and detects at least one of the posture, line of sight, and face direction of the detected pedestrian.

[0093] Subsequently, in step S22, the processing control unit 110 determines whether the first type of image display condition is satisfied, based on the image analysis in step S21, that is, the posture of the pedestrian in a ready posture to move in front of the vehicle CR (posture of walking on a crosswalk), and the line of sight of the pedestrian is directed towards the vehicle CR. If the result of this determination is affirmative (step S22: Y), the process proceeds to step S23.

[0094] Note that the processing control unit 110 may analyze the captured image sent from the imaging unit 210 to detect, in particular, the presence or absence of a person who may enter the planned travel route RT of the vehicle CR in the image. Here, the planned travel route is the travel route that the vehicle CR will travel in the future, for example, the travel route set by a navigation device (not shown) installed in the vehicle CR. In this case, the processing control unit 110 detects whether a pedestrian exists in the captured image, and if detected, determines from the position, posture, etc. of the detected pedestrian whether the pedestrian may enter the planned travel route RT of the vehicle CR. Note that the processing control unit 110 may detect whether a pedestrian exists in the captured image, and if detected, may determine that the pedestrian may enter the planned travel route RT of the vehicle CR.

[0095] In the case of FIG. 4(A), the pedestrian HA in the captured image is detected, and from the position, posture, etc. of the detected pedestrian HA, it is determined whether the pedestrian may enter the planned travel route RT of the vehicle CR. Also, in the case of FIG. 4(B), the pedestrian HB in the captured image is detected, and from the position, posture, etc. of the detected pedestrian HB, it is determined whether the pedestrian may enter the planned travel route RT of the vehicle CR.

[0096] Note that in the cases of FIGS. 4(A) and 4(B), if it can be detected that the pedestrians HA and HB are about to cross the crosswalk and their lines of sight are directed towards the vehicle CR, it is determined that the pedestrians HA and HB may enter the planned travel route RT of the vehicle CR.

[0097] In step S23, the processing control unit 110 reads the image data in the storage unit 120 to generate display image data. Then, the processing control unit 110 sends the display image data to the projection unit 251 of the display unit 250. As a result, an image imitating the gesture of the driver of the vehicle CR to prompt the pedestrian to move forward (prompt the pedestrian to cross the crosswalk) is displayed on the screen 255. After this, the process proceeds to step S26 described later.

[0098] On the other hand, when the result of the determination in step S22 is negative (step S22: N), the process proceeds to step S24. In step S24, the processing control unit 110 determines whether an image imitating the driver of the vehicle CR is being displayed. When the result of this determination is negative (step S24: N), the process proceeds to step S26.

[0099] When the result of the determination in step S24 is positive (step S24: Y), the process proceeds to step S25. In step S25, the processing control unit 110 performs control to erase the display of the image imitating the driver of the vehicle CR. Then, the process proceeds to step S26.

[0100] In step S26, the processing control unit 110 determines whether it has received release information from the automatic driving device 290 to release the standby for "giving priority to the pedestrian's walking". When the result of this determination is negative (step S26: N), the process returns to step S21.

[0101] When the result of the determination in step S26 is positive (step S26: Y), the process proceeds to step S27. In step S27, the processing control unit 110 determines whether an image imitating the driver of the vehicle CR is being displayed, in the same manner as the processing in step S24 described above. When the result of this determination is negative (step S27: N), the processing of step S13 ends. Then, the process returns to step S11 in FIG. 5 described above.

[0102] If the result of the determination in step S27 is affirmative (step S27: Y), the process proceeds to step S28. In step S28, the process control unit 110 performs control to erase the display of the image simulating the driver of the vehicle CR, in the same manner as the process of step S25 described above. When the control to erase the display of the image is thus performed, the process of step S13 ends. Then, the process returns to step S11 in FIG. 5 described above.

[0103] <Second type of image display control process> The "second type of image display control process" in step S15 described above will be explained.

[0104] As shown in FIG. 7, in this "second type of image display control process", first, in step S31, the process control unit 110 analyzes the captured image sent from the imaging unit 210, detects other vehicles and the operators of the other vehicles in the image, and detects the movements of the detected other vehicles and the lines of sight of the operators.

[0105] Subsequently, in step S32, the process control unit 110 determines whether or not the second type of image display condition, that is, the other vehicle is waiting for the movement of the vehicle CR, the line of sight of the operator is directed toward the vehicle CR, and the distance between the vehicle CR and the other vehicle is within a predetermined distance, is satisfied based on the image analysis in step S31. If the result of this determination is affirmative (step S32: Y), the process proceeds to step S33.

[0106] Note that the process control unit 110 may analyze the captured image sent from the imaging unit 210 to detect, in particular, the presence or absence of other vehicles (operators of other vehicles) that may enter the planned travel route RT of the vehicle CR in the image. In this case, the process control unit 110 detects whether or not other vehicles (operators of other vehicles) exist in the captured image, and if detected, determines from the positions, behaviors, etc. of the detected other vehicles (operators of other vehicles) whether or not the other vehicles (operators of other vehicles) may enter the planned travel route RT of the vehicle CR.

[0107] In the case of Fig. 4(B), other vehicle T within the captured image is detected, and based on the position, behavior, etc. of the detected other vehicle T, it is determined whether there is a possibility that the pedestrian may enter the planned travel route RT of vehicle CR. In the case of Fig. 4(B), when the other vehicle T is moving straight ahead, since it will block the planned travel route RT of vehicle CR, it is determined that the other vehicle may enter the planned travel route RT of vehicle CR.

[0108] In step S33, the processing control unit 110 reads the image data in the storage unit 120 and generates display image data. Then, the processing control unit 110 sends the display image data to the projection unit 251 of the display unit 250. As a result, an image imitating a gesture for prompting the movement of the other vehicle that precedes the travel of vehicle CR, which the driver makes to the operator of the other vehicle, is displayed on the screen 255. After this, the process proceeds to step S36 described later.

[0109] On the other hand, when the result of the determination in step S32 is negative (step S34: N), the process proceeds to step S34. In step S34, the processing control unit 110 determines whether an image imitating the driver of vehicle CR is being displayed. When the result of this determination is negative (step S34: N), the process proceeds to step S36.

[0110] When the result of the determination in step S34 is positive (step S34: Y), the process proceeds to step S35. In step S35, the processing control unit 110 performs control to erase the display of the image imitating the driver of vehicle CR. Then, the process proceeds to step S36.

[0111] In step S36, the processing control unit 110 determines whether it has received release information from the automatic driving device 290 to release the standby for "preceding the movement of the other vehicle". When the result of this determination is negative (step S36: N), the process returns to step S31.

[0112] If the result of the determination in step S36 is affirmative (step S36: Y), the process proceeds to step S37. In step S37, the process control unit 110 determines whether an image simulating the driver of the vehicle CR is being displayed, in the same manner as the process of step S34 described above. If the result of this determination is negative (step S37: N), the process of step S15 ends. Then, the process returns to step S11 in FIG. 5 described above.

[0113] If the result of the determination in step S37 is affirmative (step S37: Y), the process proceeds to step S38. In step S38, the process control unit 110 performs control to erase the display of the image simulating the driver of the vehicle CR, in the same manner as the process of step S35 described above. When the control to erase the display of the image is thus performed, the process of step S15 ends. Then, the process returns to step S11 in FIG. 5 described above.

[0114] Here, FIG. 8 shows an example of how an image simulating the driver (male) of the vehicle CR, which is displayed on the screen 255 by the processes in steps S23 and S33, looks when viewed from outside the vehicle CR.

[0115] As described above, in this embodiment, when the process control unit 110 receives standby information from the automatic driving device 290 indicating that the walking of the pedestrian is to be prioritized, the process control unit 110 performs an analysis process on the captured image sent from the imaging unit 210, detects the pedestrian in the image, and detects at least one of the posture, line of sight, and face direction of the detected pedestrian. Then, when there is no driver in the vehicle CR, the process control unit 110 performs a determination process based on the result of the analysis process of the captured image as to whether the pedestrian is in a posture of being ready to move in front of the vehicle CR and whether the line of sight of the pedestrian is directed towards the vehicle CR. When the result of the determination process is affirmative, the process control unit 110 performs control to display an image simulating a gesture for prompting the vehicle CR to move forward, which the driver of the vehicle CR would perform towards the pedestrian, on the screen 255.

[0116] Thus, for pedestrians who want to cross the crosswalk, gestures can be used to encourage them to cross. Therefore, it is possible to achieve communication with pedestrians, similar to the case of a manned vehicle.

[0117] Also, in this embodiment, when the processing control unit 110 receives standby information from the automatic driving device 290 indicating that the movement of another vehicle is to be prioritized, the processing control unit 110 performs analysis processing on the captured image sent from the imaging unit 210 to detect the other vehicle and the operator of the other vehicle in the image, and detects the movement of the detected other vehicle and the line of sight or the direction of the face of the operator. Then, when there is no driver in the vehicle CR, based on the result of the analysis processing of the captured image, the processing control unit 110 determines whether the other vehicle is waiting for the vehicle CR to move first, the operator's line of sight is directed towards the vehicle CR, and the distance between the vehicle CR and the other vehicle is within a predetermined distance. When the result of the determination process is affirmative, the processing control unit 110 controls to display, on the screen 255, an image imitating a gesture that the driver of the vehicle CR makes to encourage the movement of the other vehicle that has preceded the vehicle CR's travel, to the operator of the other vehicle.

[0118] Thus, for the operator of another vehicle waiting for the vehicle CR to pass for a right turn or the like, or for the operator of another vehicle waiting for the vehicle CR to move in order to park in a parking lot, the preceding movement can be encouraged by gestures. Therefore, it is possible to achieve communication with the operator of another vehicle, similar to the case of a manned vehicle.

[0119] Therefore, according to this embodiment, in the case of an autonomous vehicle without a driver, it is possible to achieve communication with people outside the vehicle, similar to the case of a manned vehicle, which contributes to smooth and safe driving.

[0120] [Modifications of the Embodiment] The present invention is not limited to the above-described embodiments, and various modifications are possible.

[0121] For example, in the above embodiment, the driver displayed on the screen was made male. In contrast, according to the attributes of pedestrians and operators of other vehicles, such as gender, age, nationality, etc. (excluding children), the attributes of the driver displayed on the screen (excluding children) may be changed. For example, when the pedestrian is a child, an image of a young woman may be displayed on the screen. This can give a sense of security to children.

[0122] Also, a driver with the same attributes as those of pedestrians and operators of other vehicles (excluding children) may be displayed on the screen. This can give a sense of security to pedestrians and operators of other vehicles.

[0123] Also, in the above embodiment, the present invention was applied to the vehicle as shown in FIGS. 3 and 8. However, it goes without saying that the present invention may also be applied to other vehicles such as truck vehicles and heavy motor vehicles.

[0124] Also, in the above embodiment, an image imitating the driver of the vehicle was projected onto the screen arranged on the backrest of the front seat. In contrast, a transparent screen may be arranged on the glass of the vehicle (for example, the front glass or the glass on the side of the vehicle), and an image imitating the driver of the vehicle may be projected onto the screen.

[0125] Also, for the above embodiment, the same modifications as those for the above-described embodiment can be appropriately made.

Explanation of Reference Numerals

[0126] 100 … Image display device 110 … Processing control unit (detection unit, display control unit) 700 … Image display device 730 … Detection unit 750 … Display control unit

Claims

1. An image display device to be arranged in an autonomous vehicle, a display control unit that displays, when a driver of the vehicle is not present, an image that is visible to a pedestrian who may enter a planned driving route of the vehicle and that simulates a driver in the vehicle making a gesture toward the pedestrian; a detection unit that detects the presence or absence of a pedestrian, and at least one of a posture, a line of sight, and a face direction of the pedestrian; the display control unit causes the image to be displayed when it is determined that the pedestrian recognizes the vehicle based on a detection result by the detection unit.

1. An image display device comprising:

2. The image display device according to claim 1 , wherein the gesture is a gesture for encouraging the pedestrian to move forward of the vehicle.

3. An image display device to be arranged in an autonomous vehicle, a display control unit that displays, when a driver of the vehicle is not present, an image that is visible to an operator of another vehicle that may enter a planned driving route of the vehicle, simulating a driver in the vehicle making a gesture toward the operator; a detection unit that detects a distance between the vehicle and the other vehicle, the presence or absence of the operator, a motion of the other vehicle, and a line of sight or a facial direction of the operator; The display control unit displays the image when it is determined based on a detection result by the detection unit that the distance is within a predetermined distance, that the other vehicle is waiting for the vehicle to move ahead of the vehicle, and that the operator's line of sight is directed toward the vehicle.

1. An image display device comprising:

4. 4. The image display device according to claim 3, wherein the gesture is a gesture that prompts the operator to move the other vehicle preceding the vehicle.

5. 5. The image display device according to claim 1, wherein the display control unit causes the image to be displayed on a glass of the vehicle.

6. 5. The image display device according to claim 1, wherein the display control unit displays the image at a position corresponding to a seat of the vehicle.

7. An image display method for use in an image display device including: a display control unit that is disposed in an autonomous vehicle and that, when a driver of the vehicle is not present, displays an image that is visible to a pedestrian who may enter a planned driving route of the vehicle, simulating a driver in the vehicle making a gesture toward the pedestrian; and a detection unit, a detection step in which the detection unit detects the presence or absence of a pedestrian and at least one of a posture, a line of sight, and a face direction of the pedestrian; a display control step of displaying the image when the display control unit determines, based on a detection result by the detection unit, that the pedestrian recognizes the vehicle; An image display method comprising:

8. An image display method for use in an image display device including: a display control unit that is disposed in an autonomously driven vehicle and that displays an image, when a driver of the vehicle is not present, that is visible to an operator of another vehicle that may enter a planned driving route of the vehicle, simulating a driver in the vehicle making a gesture toward the operator; and a detection unit, a detection step in which the detection unit detects a distance between the vehicle and the other vehicle, the presence or absence of the operator, a motion of the other vehicle, and a line of sight or a facial direction of the operator; a display control step of displaying the image when the display control unit determines, based on a detection result by the detection unit, that the distance is within a predetermined distance, that the other vehicle is waiting for the vehicle to move ahead of the other vehicle, and that the operator's line of sight is directed toward the vehicle; An image display method comprising:

9. 8. An image display program for causing a computer included in an image display device to execute the image display method according to claim 7.

10. 9. An image display program for causing a computer included in an image display device to execute the image display method according to claim 8.

11. 10. A recording medium having the image display program according to claim 9 recorded thereon so as to be readable by a computer included in an image display device.

12. 11. A recording medium having the image display program according to claim 10 recorded thereon so as to be readable by a computer included in an image display device.

Images