Video display device

Abstract

To provide a technology capable of appropriately displaying an overhead video of a vehicle.SOLUTION: A video display device includes a camera that captures videos of the area around a vehicle, a memory unit that stores past videos captured by the camera, a control unit that outputs either a first overhead video looking down on the vehicle from a viewpoint above the vehicle and obtained by synthesizing real-time videos captured by the camera, or a second overhead video looking down on the vehicle from a viewpoint above the vehicle and obtained by synthesizing at least past videos, depending on the situation around the vehicle, or outputs the second overhead video depending on the driving condition of the vehicle, and a display unit that displays the video output by the control unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a video display device. [Background technology]

[0002] Patent Document 1 discloses an image display device for vehicles. This device converts image signals from multiple cameras capturing images of the vehicle's surroundings into image signals as viewed from a viewpoint above the vehicle, and synthesizes the converted image signals. This displays an overhead image as viewed from a viewpoint above the vehicle. When this device determines that the display position of an object present around the vehicle overlaps with the boundary of the images to be synthesized, it changes the boundary of the images to be synthesized so that the object does not overlap with the boundary of the images to be synthesized. This prevents objects from being displayed in a discontinuous or crushed manner. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-104373 A Summary of the Invention [Problem to be solved by the invention]

[0004] The device described in Patent Document 1 may not be able to sufficiently prevent the disappearance or duplication of objects. For example, when there are many objects, the boundaries change frequently, and depending on the computing power, there is a risk that the disappearance or duplication of objects may occur. Here, it is conceivable to prevent the disappearance or duplication of objects by synthesizing images captured in the past. However, past images cannot respond to real-time movements around the vehicle. The present disclosure provides a technology that can appropriately display an overhead image of a vehicle. [Means for solving the problem]

[0005] An image display device according to one embodiment of the present disclosure includes a camera that captures images of the area around the vehicle, a memory unit that stores past images captured by the camera, a control unit that selects images to be synthesized from the real-time images captured by the camera and the past images stored in the memory unit depending on the situation around the vehicle or the driving state of the vehicle, and outputs an overhead image looking down on the vehicle from a viewpoint above the vehicle obtained by synthesizing the selected images, and a display unit that displays the overhead image output by the control unit.

[0006] In this image display device, images to be composited are selected from the real-time image and the past image according to the situation around the vehicle or the running state of the vehicle, and an overhead image composited with the selected images is output. By reflecting the past image in the composition of the overhead image according to the situation around the vehicle or the running state of the vehicle, this image display device can display an overhead image in which disappearance or doubling of objects is sufficiently suppressed compared to when real-time images are always reflected in the composition of the overhead image, and can display an overhead image that corresponds to changes in the surrounding environment compared to when past images are always reflected in the composition of the overhead image. Therefore, this image display device can appropriately display an overhead image of the vehicle.

[0007] In one embodiment, the control unit may output either a first overhead image obtained by synthesizing the real-time image or a second overhead image obtained by synthesizing at least the past image in accordance with a surrounding situation of the vehicle, or may output the second overhead image in accordance with a running state of the vehicle. This image display device can display the first overhead image or the second overhead image in accordance with the surrounding situation of the vehicle or the running state of the vehicle.

[0008] In one embodiment, the control unit may output a first overhead view image when a moving object is present within the imaging range of the camera, and output a second overhead view image obtained by combining past images when a moving object is not present within the imaging range of the camera. In this case, the image display device can output a first overhead view image that prioritizes responding to changes in the surrounding environment when a moving object is present, and output a second overhead view image that prioritizes eliminating disappearance of an object or eliminating doubling of an object when a moving object is not present.

[0009] In one embodiment, the image display device includes a plurality of cameras including a camera, the storage unit stores past images captured by the plurality of cameras, and the control unit may generate a first overhead image by synthesizing real-time images captured by the plurality of cameras, and generate a second overhead image by synthesizing the past images or synthesizing the past images and the real-time images. In this case, the image display device can display a high-precision overhead image by synthesizing images from the plurality of cameras.

[0010] In one embodiment, when a stationary object is present within the imaging range of the multiple cameras, the control unit may output a second overhead image obtained by combining the first overhead image with an overhead image of the stationary object based on a past image. In this case, the image display device can appropriately display the moving object in real time while suppressing the stationary object from disappearing or being duplicated.

[0011] In one embodiment, when a still object is present within the imaging range of the multiple cameras and the display position of the still object overlaps with a synthesis boundary of the overhead image, the control unit may output a second overhead image obtained by synthesizing the overhead image of the still object based on the past image with the first overhead image. In this case, the image display device can appropriately display the surrounding environment in real time while suppressing the still object from disappearing or being duplicated.

[0012] In one embodiment, when the vehicle is traveling, the control unit may generate a region in front of the vehicle's traveling direction in the second overhead image based on a real-time image showing the front of the vehicle's traveling direction, and generate a region behind the vehicle's traveling direction in the second overhead image based on a past image showing the rear of the vehicle's traveling direction. In this case, the image display device can respond to environmental changes by displaying the real-time image in the traveling direction that requires the most attention, and can prevent objects from disappearing or being duplicated in the rear of the traveling direction, which is less important than the traveling direction, by displaying the past image. Effect of the Invention

[0013] According to the present disclosure, a technique is provided that can appropriately display an overhead image of a vehicle. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a vehicle equipped with an image display device according to an embodiment. [Diagram 2] FIG. 2 is an example showing imaging areas of a plurality of sensors provided on the vehicle 2. In FIG. [Diagram 3] FIG. 3 is an example of an overhead image displayed on the display unit. [Figure 4] FIG. 4A shows an example of the surrounding environment of a vehicle, and FIG. 4B shows an example of an overhead image corresponding to the surrounding environment shown in FIG. 4A. [Diagram 5] FIG. 5A shows an example of the surrounding environment of a vehicle, and FIG. 5B shows an example of an overhead image corresponding to the surrounding environment shown in FIG. 5A. [Figure 6] FIG. 6 is a flowchart showing the operation of the video display device. [Figure 7] FIG. 7 is a block diagram showing an example of the configuration of a vehicle equipped with an image display device according to a modified example. [Figure 8] (A), (B), and (C) in Figure 8 are examples of the surrounding environment of a vehicle, and (E), (F), and (G) in Figure 8 are examples of overhead images corresponding to the surrounding environment shown in (A), (B), and (C) in Figure 8. [Figure 9] FIG. 9A shows an example of the surrounding environment of a vehicle, and FIG. 9B shows an example of an overhead image corresponding to the surrounding environment shown in FIG. 9A. [Figure 10] FIG. 10 is a flowchart showing the operation of the video display device according to the modified example. [Figure 11] FIG. 11A shows an example of the surrounding environment of a vehicle, and FIG. 11B shows an example of an overhead image corresponding to the surrounding environment shown in FIG. 11A. [Figure 12] FIG. 12 is a flowchart showing the operation of the video display device according to the modified example. [Figure 13] FIGS. 13A and 13B are examples of the surrounding environment of a vehicle, and FIG. 13C is an example of an overhead image corresponding to the surrounding environment shown in FIG. 13B. [Figure 14] FIG. 14 is a flowchart showing the operation of the video display device according to the modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0016] [Vehicle configuration] Fig. 1 is a block diagram showing an example of the configuration of a vehicle equipped with an image display device according to an embodiment. As shown in Fig. 1, the image display device 1 is mounted on a vehicle 2 as an example. The vehicle 2 may be a vehicle driven by a driver, or may be an autonomous vehicle.

[0017] The vehicle 2 includes a plurality of cameras 10, a storage unit 11, an external sensor 12, an internal sensor 13, a control unit 14, and a display unit. The image display device 1 includes a plurality of cameras 10, a storage unit 11, a control unit 14, and a display unit.

[0018] The multiple cameras 10 are a group of cameras that capture images of the periphery of the vehicle 2. The multiple cameras 10 include, as an example, a front camera 101, a right side camera 102, a left side camera 103, and a rear camera 104. FIG. 2 is an example showing the imaging areas of multiple sensors provided in the vehicle 2. As shown in FIG. 2, the front camera 101 is provided on a front bumper or the like, and captures an image of a front range 101A of the vehicle 2. The right side camera 102 is provided on a door mirror or the like, and captures an image of a right side range 102A of the vehicle 2. The left side camera 103 is provided on a door mirror or the like, and captures an image of a left side range 103A of the vehicle 2. The rear camera 104 is provided on a rear bumper or the like, and captures an image of a rear range 104A of the vehicle 2.

[0019] The storage unit 11 stores past images captured by the multiple cameras 10 for each camera. The past images refer to recorded real-time images, and include moving images and still images. The real-time images are images captured at the present time by the multiple cameras 10, and include moving images and still images. The storage unit 11 is, for example, a storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), or a HDD (Hard Disk Drive).

[0020] The external sensor 12 is a sensor that detects the surrounding environment of the vehicle 2. The external sensor 12 is, for example, a radar sensor. The radar sensor is a detection device that detects objects around the vehicle using radio waves (for example, millimeter waves) or light. The radar sensor includes, for example, a millimeter wave radar or a LiDAR (Light Detection and Ranging).

[0021] The internal sensor 13 is a sensor that detects the traveling state of the vehicle 2. The internal sensor 13 includes, for example, a GPS receiver, a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor.

[0022] The control unit 14 has a function of controlling the image display device. The control unit 14 is configured with an ECU (Electronic Control Unit) as an example. The ECU is an electronic control unit having a CPU (Central Processing Unit), ROM, RAM, a CAN (Controller Area Network) communication circuit, etc. The control unit 14 synthesizes images captured by the multiple cameras 10 to generate an overhead image. The overhead image is an image looking down on the vehicle 2 from a viewpoint above the vehicle 2, and includes moving images and still images.

[0023] The display unit 15 displays the overhead image output by the control unit 14. The display unit 15 is, for example, a display device. FIG. 3 is an example of an overhead image displayed on the display unit. As shown in FIG. 3, the display unit 15 displays the overhead image GR1 generated by the control unit 14. The control unit 14 generates the overhead image GR1 including a vehicle object OB2 corresponding to the vehicle 2. The control unit 14 inputs image signals from the multiple cameras 10 and performs image processing to generate an overhead image looking down on the vehicle 2 from a viewpoint directly above. The viewpoint conversion is performed based on the respective arrangement positions and imaging directions of the multiple cameras 10 with respect to a reference point O of the vehicle 2. The image processing for the viewpoint conversion may be performed by software or a hardware circuit.

[0024] The control unit 14 displays a forward area AR1 of the vehicle 2 using an image signal captured by the forward camera 101. The control unit 14 displays a right side area AR2 of the vehicle 2 using an image signal captured by the right side camera 102. The control unit 14 displays a left side area AR3 of the vehicle 2 using an image signal captured by the left side camera 103. The control unit 14 displays a rear area AR4 of the vehicle 2 using an image signal captured by the rear camera 104. In this way, an overhead image GR1 is generated.

[0025] The control unit 14 selects images to be combined from among real-time images captured by the multiple cameras 10 and past images stored in the memory unit 11. The images to be combined are selected according to the situation around the vehicle 2. For example, the control unit 14 determines whether or not a moving object is present within the imaging range of the multiple cameras 10 based on the detection results of the multiple cameras 10 and / or the external sensor 12. When using the results captured by the multiple cameras 10, the control unit 14 detects the moving object based on the motion vector between frames. When using the detection results of the external sensor 12, the control unit 14 detects the moving object based on the change in distance to the detected object over time.

[0026] When a moving object is present within the imaging ranges of the multiple cameras 10, the control unit 14 selects the real-time video as the video to be synthesized. As a result, the control unit 14 outputs an overhead video (first overhead video) obtained by synthesizing the real-time video. The first overhead video is an overhead video composed of videos at the current time captured by the multiple cameras 10. When no moving object is present within the imaging ranges of the multiple cameras 10, the control unit 14 selects the past video as the video to be synthesized. As a result, the control unit 14 outputs an overhead video (second overhead video) obtained by synthesizing the past video stored in the storage unit 11. In this way, the control unit 14 outputs, as an example, either the first overhead video obtained by synthesizing the real-time video or the second overhead video obtained by synthesizing the past video according to the situation around the vehicle 2.

[0027] FIG. 4A is an example of the surrounding environment of the vehicle, and FIG. 4B is an example of an overhead image corresponding to the surrounding environment shown in FIG. 4A. As shown in FIG. 4A, the vehicle 2 is moving forward, and a moving object 40, which is a pedestrian, is present in front of the vehicle 2. The vehicle 2 determines that the moving object 40 is present in the forward range 101A of the forward camera 101 based on the detection results of the forward camera 101 and / or the external sensor 12. In this case, the control unit 14 selects the real-time image as the image to be synthesized. The control unit 14 outputs a first overhead image GR2 obtained by synthesizing the real-time image. As shown in FIG. 4B, the display unit 15 displays the first overhead image GR2 output by the control unit 14. The first overhead image GR2 includes a moving object image OB40 corresponding to the moving object 40. This allows the driver to grasp the surroundings of the vehicle 2 in real time.

[0028] FIG. 5A is an example of the surrounding environment of the vehicle, and FIG. 5B is an example of an overhead image corresponding to the surrounding environment shown in FIG. 5A. As shown in FIG. 5A, the vehicle 2 is moving forward, and a stationary object 30 such as a pole is present in front of the vehicle 2. The vehicle 2 determines that the stationary object 30 is present in the forward range 101A of the forward camera 101 based on the detection results of the forward camera 101 and / or the external sensor 12. In this case, the control unit 14 selects the past image as the image to be synthesized. The control unit 14 outputs the second overhead image GR3 obtained by synthesizing the past image. As shown in FIG. 5B, the display unit 15 displays the second overhead image GR3 output by the control unit 14. The second overhead image GR3 includes a stationary object image OB30 corresponding to the stationary object 30. Since the entire area PS1 of the second overhead view image GR3 is created based on the previous image, there are no boundaries due to synthesis, and disappearance or doubling of objects is avoided.

[0029] [Video display device operation] Fig. 6 is a flowchart showing the operation of the video display device. The flowchart shown in Fig. 6 is executed by the control unit 14, for example, when a driver's operation is accepted.

[0030] 6, in step S10, the control unit 14 determines whether or not an obstacle is present in the camera image. The control unit 14 determines the presence or absence of an obstacle in the vicinity of the vehicle 2 based on the detection results of the multiple cameras 10 and / or the external sensor 12.

[0031] If it is determined that an obstacle is present in the camera image (step S10: YES), the control unit 14 determines whether or not the obstacle is a moving object in step S12. The control unit 14 determines whether or not the obstacle determined in step S10 is a moving object based on the detection results of the multiple cameras 10 and / or the external sensor 12.

[0032] If it is determined that the obstacle is a moving object (step S12: YES), the control unit 14 selects the real-time video as the video to be synthesized in step S14. Then, the control unit 14 causes the display unit 15 to display the first overhead view video (see FIG. 4B).

[0033] If it is determined that no obstacle is present in the camera image (step S10: NO), or if it is determined that the obstacle is not a moving object (step S12: NO), the control unit 14 selects the past image as the image to be synthesized in step S16. Then, the control unit 14 causes the display unit 15 to display the second overhead view image (see FIG. 5(B)). When step S14 or step S16 is completed, the flowchart ends. The flowchart is repeatedly executed until a driver's operation to end the process is received.

[0034] [Summary of the embodiment] The image display device 1 selects images to be composited from the real-time images and the past images according to the situation around the vehicle 2, and outputs an overhead image composited with the selected images. By reflecting the past images in the composite of the overhead image according to the situation around the vehicle 2, the image display device 1 can display an overhead image in which disappearance or doubling of objects is sufficiently suppressed compared to when real-time images are always reflected in the composite of the overhead image, and can display an overhead image that corresponds to changes in the surrounding environment compared to when past images are always reflected in the composite of the overhead image. Thus, the image display device 1 can appropriately display the overhead image of the vehicle 2.

[0035] Although exemplary embodiments have been described above, various omissions, substitutions, and modifications may be made without being limited to the above-described exemplary embodiments.

[0036] [Variations] [Single camera configuration] Fig. 7 is a block diagram showing an example of the configuration of a vehicle equipped with an image display device according to a modified example. The image display device 1A shown in Fig. 7 is different from the image display device 1 in that it does not include multiple cameras 10, but only includes a rear camera 104, and that some operations of the control unit 14A are different from those of the control unit 14, but is otherwise the same. The following description will focus on the differences, and overlapping descriptions will not be repeated.

[0037] The control unit 14A first generates an overhead video using only the real-time video from the rear camera 104. When the control unit 14A obtains past video as the vehicle 2 moves, the control unit 14A synthesizes the real-time video with the past video stored in the storage unit 11 to generate the overhead video.

[0038] (A), (B), and (C) in FIG. 8 are examples of the surrounding environment of the vehicle, and (E), (F), and (G) in FIG. 8 are examples of the overhead image corresponding to the surrounding environment shown in (A), (B), and (C) in FIG. 8. As shown in the order of (A), (B), and (C) in FIG. 8, the vehicle 2 moves backward. When the vehicle 2 is at the position of (A) in FIG. 8, the control unit 14A generates a first overhead image GR4 synthesized using only real-time images, as shown in (E) in FIG. 8. In the first overhead image GR4, an image is displayed only in an area AR5 corresponding to the imaging range of the rear camera 104, and the other areas are, for example, blank. A stationary object image OB30 corresponding to the stationary object 30 is displayed in the area AR5.

[0039] Next, when the vehicle 2 is at the position of FIG. 8B, the control unit 14A generates a second overhead image GR5 in which the real-time image and the past image are combined, as shown in FIG. 8F. In the second overhead image GR5, images are displayed only in the area AR5 corresponding to the imaging range of the rear camera 104 and the area AR6 corresponding to the past image, and other areas are blank, for example. Next, when the vehicle 2 is at the position of FIG. 8C, the control unit 14A generates a second overhead image GR6 in which the real-time image and the past image are combined, as shown in FIG. 8G. In the second overhead image GR6, images are displayed only in the area AR5 corresponding to the imaging range of the rear camera 104 and the area AR6 corresponding to the past image, and other areas are blank, for example. The second overhead image GR6 can use more past images than the second overhead image GR5, so the area AR6 is longer.

[0040] Like control unit 14, control unit 14A displays a first overhead image (e.g., (E) in FIG. 8) when a moving object is detected, and displays a second overhead image (e.g., (F) and (G) in FIG. 8) when a moving object is not detected. In this way, image display device 1A increases the displayable range by including past images in the display when a moving object is not detected, thereby increasing the amount of information to be notified, and can provide accurate information to the driver by not displaying past images that cannot be used to deal with the intrusion of a moving object when a moving object is detected.

[0041] [Example of applying past footage to stationary objects only] FIG. 9A is an example of the surrounding environment of the vehicle, and FIG. 9B is an example of an overhead image corresponding to the surrounding environment shown in FIG. 9A. As shown in FIG. 9A, the vehicle 2 is moving forward, and a stationary object 30 such as a pole and a moving object 40 which is a pedestrian are present in front of the vehicle 2. The vehicle 2 determines that the stationary object 30 and the moving object 40 are present in the forward range 101A of the forward camera 101 based on the detection results of the forward camera 101 and / or the external sensor 12. In this case, the control unit 14 selects the real-time image and the past image as images to be synthesized. The control unit 14 generates a first overhead image obtained by synthesizing the real-time images, and synthesizes an overhead image of the stationary object 30 based on the past image with the first overhead image.

[0042] First, the control unit 14 determines whether or not a past image of the stationary object 30 exists based on the position of the stationary object 30. The control unit 14 may match the past image with the real-time image to determine whether or not a past image of the stationary object 30 exists. If it is determined that a past image of the stationary object 30 exists, the control unit 14 generates an overhead image of the stationary object 30. The control unit 14 may cut out a certain range of an image area in which the stationary object 30 appears and generate an overhead image from the cut-out image area, or may generate an entire overhead image from the past image and then cut out a certain range of an image area in which the stationary object 30 appears. The generated overhead image of the stationary object 30 is overwritten so as to overlap the image of the stationary object 30 on the first overhead image generated from the real-time image. As a result, the second overhead image GR7 shown in (B) of FIG. 9 is generated. The areas PS2 and PS3 including the stationary object image OB30 are areas generated based on the past images.

[0043] Fig. 10 is a flowchart showing the operation of the video display device according to the modified example. The flowchart shown in Fig. 10 is executed by the control unit 14, for example, when a driver's operation is accepted.

[0044] 10, in step S20, the control unit 14 determines whether or not an obstacle is present in the camera image. This determination is the same as in step S10.

[0045] If it is determined that an obstacle is present in the camera image (step S20: YES), the control unit 14 determines in step S22 whether the obstacle is a stationary object. The control unit 14 determines whether the obstacle determined in step S20 is a stationary object based on the detection results of the multiple cameras 10 and / or the external sensor 12.

[0046] If it is determined that no obstacle is present in the camera image (step S20: NO), or if it is determined that the obstacle is not a stationary object (step S22: NO), the control unit 14 selects the real-time image as an image to be synthesized in step S24. Then, the control unit 14 causes the display unit 15 to display the first overhead image.

[0047] If it is determined that the obstacle is a stationary object (step S22: YES), the control unit 14 selects the real-time image and the past image as images to be synthesized in step S26. Then, the control unit 14 creates a first overhead image based on the real-time image, while generating a still object image OB30 based on the past image. The control unit 14 causes the display unit 15 to display a second overhead image GR7 in which the still object image OB30 is synthesized with the first overhead image (see FIG. 9B). When step S24 or step S26 ends, the flowchart ends. The flowchart is repeatedly executed until an end operation by the driver is received.

[0048] As described above, the image display device can appropriately display moving objects in real time while preventing stationary objects from disappearing or being duplicated. The image display device may set the presence or absence of a moving object as an additional condition for generating the second overhead view image. In other words, the image display device may be configured to generate the second overhead view image when both moving objects and stationary objects are present within the imaging ranges of the multiple cameras 10. Furthermore, when it is determined that no obstacle is present in the camera images (step S20: NO), the second overhead view image based only on past images may be displayed.

[0049] [Example of applying past footage only to stationary objects that meet certain conditions] FIG. 11A is an example of the surrounding environment of the vehicle, and FIG. 11B is an example of an overhead image corresponding to the surrounding environment shown in FIG. 11A. As shown in FIG. 11A, the vehicle 2 is moving forward, and a stationary object 30 such as a pole and a moving object 40 which is a pedestrian are present in front of the vehicle 2. The vehicle 2 determines that the stationary object 30 and the moving object 40 are present in the forward range 101A of the forward camera 101 based on the detection results of the forward camera 101 and / or the external sensor 12. In this case, the control unit 14 selects the real-time image and the past image as images to be synthesized. The control unit 14 generates a first overhead image obtained by synthesizing the real-time images, and synthesizes an overhead image of the stationary object 30 based on the past image with the first overhead image.

[0050] First, the control unit 14 determines whether or not a past image of the stationary object 30 exists based on the position of the stationary object 30. The control unit 14 may match the past image with the real-time image to determine whether or not a past image of the stationary object 30 exists. If it is determined that a past image of the stationary object 30 exists, the control unit 14 determines whether or not the display position of the stationary object 30 overlaps with the synthesis boundary of the overhead image. The synthesis boundary is the boundary between the forward area AR1, the right side area AR2, the left side area AR3, and the rear area AR4. If the stationary object 30 overlaps with the synthesis boundary, the control unit 14 generates an overhead image of the stationary object 30. The control unit 14 may cut out a certain range of the image area in which the stationary object 30 appears and generate an overhead image from the cut-out image area, or may generate an entire overhead image from the past image and then cut out a certain range of the image area in which the stationary object 30 appears. The generated overhead image of the still object 30 is overwritten so as to overlap the image of the still object 30 on the first overhead image generated from the real-time image. This generates the second overhead image GR8 shown in Fig. 11(B). The area PS5 including the still object image OB30 that overlaps with the synthesis boundary is an area generated based on the past image.

[0051] Fig. 12 is a flowchart showing the operation of the video display device according to the modified example. The flowchart shown in Fig. 12 is executed by the control unit 14, for example, when a driver's operation is accepted.

[0052] 12, in step S30, the control unit 14 determines whether or not an obstacle is present in the camera image. This determination is the same as in step S10.

[0053] If it is determined that an obstacle is present in the camera image (step S30: YES), the control unit 14 determines in step S32 whether or not the obstacle is a stationary object. This determination is the same as step S22.

[0054] If it is determined that the obstacle is a stationary object (step S32: YES), the control unit 14 determines whether or not the obstacle overlaps with the composite boundary in step S33. The control unit 14 calculates the display position of the stationary object from the imaging results of the multiple cameras 10, and determines whether or not the obstacle overlaps with the composite boundary.

[0055] If it is determined that no obstacle is present in the camera image (step S30: NO), if it is determined that the obstacle is not a stationary object (step S32: NO), or if it is determined that the obstacle does not overlap with the synthesis boundary (step S33: NO), the control unit 14 selects the real-time image as the image to be synthesized in step S34. Then, the control unit 14 causes the display unit 15 to display the first overhead image.

[0056] If it is determined that the obstacle overlaps with the synthesis boundary (step S33: YES), the control unit 14 selects the real-time image and the past image as images to be synthesized in step S36. Then, the control unit 14 creates a first overhead image based on the real-time image, while generating a still object image OB30 that overlaps with the synthesis boundary based on the past image. The control unit 14 causes the display unit 15 to display a second overhead image GR8 in which the still object image OB30 is synthesized with the first overhead image (see FIG. 11B). When step S34 or step S36 ends, the flow chart ends. The flow chart is repeatedly executed until an end operation by the driver is received.

[0057] As described above, the image display device can appropriately display moving objects in real time while preventing stationary objects from disappearing or being duplicated. The image display device may set the presence or absence of a moving object as an additional condition for generating the second overhead view image. In other words, the image display device may be configured to generate the second overhead view image when both a moving object and a stationary object are present within the imaging range of the multiple cameras 10. Furthermore, when it is determined that no obstacle is present in the camera image (step S30: NO), the second overhead view image based only on past images may be displayed.

[0058] [Display according to vehicle driving conditions] The control unit 14 may select images to be composited according to the traveling state of the vehicle, and output an overhead image obtained by compositely combining the selected images. The control unit 14 distinguishes between two traveling states of the vehicle 2: "stopped" and "on the move". "On the move" includes "forward", "backward", "turning right" and "turning left". When the traveling state of the vehicle 2 is on the move, the control unit 14 generates a forward area in the traveling direction of the vehicle 2 in the second overhead image based on a real-time image showing the front of the traveling direction of the vehicle 2, and generates a rear area in the traveling direction of the vehicle 2 in the second overhead image based on a past image showing the rear of the traveling direction of the vehicle 2. In other words, the control unit 14 generates an overhead image based on the real-time image for the area to be traveled from now on, and an overhead image based on the past image for the area that has already been traveled.

[0059] (A) and (B) of FIG. 13 are examples of the surrounding environment of the vehicle, and (C) of FIG. 13 is an example of an overhead image corresponding to the surrounding environment shown in (B) of FIG. 13. As shown in the order of (A) and (B) of FIG. 13, the vehicle 2 is moving forward. In this case, the control unit 14 generates a forward area AR1 (a forward area in the traveling direction of the vehicle 2) of the second overhead image GR9 based on a real-time image showing the front in the forward direction, as shown in (C) of FIG. 13. Then, the control unit 14 generates a rear area AR4 (a rear area in the traveling direction of the vehicle 2) of the second overhead image based on a past image showing the rear in the traveling direction of the vehicle 2. As a result, only the rear area AR4 becomes the area PS6 based on the past image.

[0060] Fig. 14 is a flowchart showing the operation of the video display device according to the modified example. The flowchart shown in Fig. 14 is executed by the control unit 14, for example, at the timing when a driver's operation is accepted.

[0061] 14, in step S40, the control unit 14 determines whether the vehicle 2 is stopped. The control unit 14 determines the running state of the vehicle 2 based on the movements of the multiple cameras 10 or the detection results of the internal sensor 13.

[0062] When it is determined that the vehicle 2 is stopped (step S40: YES), the control unit 14 displays the first overhead view video combined with the real-time video. The control unit 14 may also display the second overhead view video combined with only the past video.

[0063] If it is determined that the vehicle 2 is not stopped (step S40: NO), the control unit 14 determines in step S44 whether the vehicle 2 is moving forward. If it is determined that the vehicle 2 is moving forward (step S44: YES), the control unit 14 selects the real-time image and the past image as images to be combined in step S46. Then, the control unit 14 creates a first overhead image based on the real-time image, while generating an overhead image based on the past image of the rear of the vehicle. The control unit 14 causes the display unit 15 to display a second overhead image GR9 in which the overhead image of the rear of the vehicle is combined with the first overhead image (see FIG. 13C).

[0064] If it is determined that the vehicle 2 is not traveling forward (step S44: NO), the control unit 14 determines in step S48 whether the vehicle 2 is traveling backward. If it is determined that the vehicle 2 is traveling backward (step S48: YES), the control unit 14 selects the real-time image and the past image as images to be synthesized in step S50. Then, the control unit 14 creates a first overhead image based on the real-time image, while generating an overhead image based on the past image of the area in front of the vehicle. The control unit 14 causes the display unit 15 to display a second overhead image GR9 in which the overhead image of the area in front of the vehicle is synthesized with the first overhead image.

[0065] If it is determined that the vehicle 2 is not in reverse (step S48: NO), the control unit 14 determines in step S52 whether the vehicle 2 is turning right. If it is determined that the vehicle 2 is in a right turn (step S52: YES), the control unit 14 selects the real-time image and the past image as images to be combined in step S54. Then, the control unit 14 creates a first overhead image based on the real-time image, while generating an overhead image based on the past image of the left side of the vehicle. The control unit 14 causes the display unit 15 to display a second overhead image GR9 obtained by combining the first overhead image with the overhead image of the left side of the vehicle.

[0066] If it is determined that the vehicle 2 is not turning right (step S48: NO), the vehicle 2 is turning left. In step S56, the control unit 14 selects the real-time image and the past image as images to be combined. Then, the control unit 14 creates a first overhead image based on the real-time image, while generating an overhead image based on the past image of the right side of the vehicle. The control unit 14 causes the display unit 15 to display a second overhead image GR9 in which the overhead image of the right side of the vehicle is combined with the first overhead image.

[0067] When step S42, step S46, step S50, step S54, or step S56 is completed, the flow chart ends. The flow chart is repeatedly executed until the driver's operation to end the process is received. The image display device can respond to environmental changes by displaying real-time images in the traveling direction, which requires the most attention, and can prevent objects from disappearing or becoming duplicated in the rear of the traveling direction, which is less important than the traveling direction, by displaying past images.

[0068] This disclosure also includes the following provisions: [Clause 1] A camera that captures the surroundings of the vehicle; A storage unit that stores past images captured by the camera; a control unit that selects images to be synthesized from the real-time images captured by the camera and the past images stored in the storage unit in accordance with the surrounding conditions of the vehicle or the traveling state of the vehicle, and outputs an overhead image overlooking the vehicle from a viewpoint above the vehicle, the overhead image being obtained by synthesizing the selected images; and a display unit that displays the overhead image output by the control unit; An image display device comprising: [Clause 2] The image display device described in clause 1, wherein the control unit outputs either a first overhead image obtained by synthesizing the real-time image or a second overhead image obtained by synthesizing at least the past image depending on the situation around the vehicle, or outputs the second overhead image depending on the driving state of the vehicle. [Clause 3] The image display device described in clause 2, wherein the control unit outputs the first overhead image when a moving object is present within the imaging range of the camera, and outputs the second overhead image obtained by synthesizing the past image when no moving object is present within the imaging range of the camera. [Article 4] A plurality of cameras including the camera are provided, The storage unit stores past images captured by the plurality of cameras, The control unit is generating the first overhead view image by synthesizing real-time images captured by the plurality of cameras; generating the second overhead view image by combining the past image or the past image and the real-time image; 4. A video display device according to claim 2 or 3. [Article 5] The image display device described in clause 4, wherein when a still object is present within the imaging range of the multiple cameras, the control unit outputs the second overhead image obtained by combining the first overhead image with an overhead image of the still object based on the past image. [Article 6] The image display device described in clause 4, wherein when a still object is present within the imaging range of the multiple cameras and the display position of the still object overlaps with the synthesis boundary of the overhead image, the control unit outputs the second overhead image obtained by synthesizing the overhead image of the still object based on the past image with the first overhead image. [Article 7] The image display device described in any one of clauses 1 to 4, wherein, when the vehicle is currently traveling, the control unit generates a region in front of the vehicle's traveling direction in the second overhead image based on real-time image showing the area in front of the vehicle's traveling direction, and generates a region in rear of the vehicle's traveling direction in the second overhead image based on past image showing the area behind the vehicle's traveling direction. [Explanation of symbols]

[0069] 1...image display device, 2...vehicle, 10...multiple cameras, 11...storage unit, 14...control unit, 15...display unit.

Claims

1. A camera that captures the surroundings of the vehicle; A storage unit that stores past images captured by the camera; a control unit that selects images to be synthesized from the real-time images captured by the camera and the past images stored in the storage unit in accordance with the surrounding conditions of the vehicle or the traveling state of the vehicle, and outputs an overhead image overlooking the vehicle from a viewpoint above the vehicle, the overhead image being obtained by synthesizing the selected images; and a display unit that displays the overhead image output by the control unit; An image display device comprising:

2. The image display device according to claim 1, wherein the control unit outputs either a first overhead image obtained by synthesizing the real-time image or a second overhead image obtained by synthesizing at least the past image depending on the situation around the vehicle, or outputs the second overhead image depending on the driving state of the vehicle.

3. The image display device of claim 2, wherein the control unit outputs the first overhead image when a moving object is present within the imaging range of the camera, and outputs the second overhead image obtained by synthesizing the past image when no moving object is present within the imaging range of the camera.

4. A plurality of cameras including the camera are provided, The storage unit stores past images captured by the plurality of cameras, The control unit is generating the first overhead view image by synthesizing real-time images captured by the plurality of cameras; generating the second overhead view image by combining the past image or the past image and the real-time image; 4. The image display device according to claim 2 or 3.

5. The image display device of claim 4, wherein when a still object is present within the imaging range of the multiple cameras, the control unit outputs the second overhead image obtained by combining the first overhead image with an overhead image of the still object based on the past image.

6. The image display device of claim 4, wherein when a still object is present within the imaging range of the multiple cameras and the display position of the still object overlaps with a synthesis boundary of the overhead image, the control unit outputs the second overhead image obtained by synthesizing the overhead image of the still object based on the past image with the first overhead image.

7. 5. The image display device according to claim 4, wherein the control unit, when the vehicle is traveling, generates a region in front of the vehicle's traveling direction in the second overhead image based on real-time image showing the area in front of the vehicle's traveling direction, and generates a region in the rear of the vehicle's traveling direction in the second overhead image based on past image showing the area behind the vehicle's traveling direction.

Images