Image display device

Abstract

The invention relates to an image display device, and provides a technology capable of appropriately displaying an overhead image of a vehicle. An image display device includes: a camera that captures an image of a periphery of a vehicle; a storage unit that stores past images captured by the camera; and a control unit that outputs, on the basis of the situation of the periphery of the vehicle, either a first overhead view image in which the vehicle is overlooked from a viewpoint above the vehicle, said first overhead view image being obtained by synthesizing real-time images captured by the camera, or a second overhead view image in which the vehicle is overlooked from a viewpoint above the vehicle, said second overhead view image being obtained by synthesizing at least past images, or a second overhead view image in which the vehicle is overlooked from a viewpoint above the vehicle, said second overhead view image being obtained by synthesizing at least past images. Outputting a second aerial view image according to the driving state of the vehicle; and a display unit that displays the image output by the control unit.

Description

Technical Field

[0001] The present disclosure relates to an image display device. Background Art

[0002] Patent document 1 discloses an image display device for a vehicle. The device converts image signals from multiple cameras that capture the surroundings of the vehicle into image signals when looking down from a viewpoint above the vehicle, and synthesizes the converted image signals. Thus, a bird's-eye view image looking down from a viewpoint above the vehicle is displayed. Furthermore, when the device determines that the display position of an object existing around the vehicle overlaps with the boundary of the synthesized image, the device changes the boundary of the synthesized image in such a way that the object does not overlap with the boundary of the synthesized image. Thus, the display of discontinuous or deformed objects is suppressed.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2007-104373 Summary of the invention

[0006] Problems to be solved by the invention

[0007] The device described in Patent Document 1 sometimes cannot fully suppress the disappearance or duplication of objects. For example, when there are many objects, the boundaries are frequently changed, and therefore, depending on the computing power, objects may disappear or be duplicated. Here, it can be considered to suppress the disappearance or duplication of objects by synthesizing images taken in the past. However, past images cannot correspond to the real-time movements around the vehicle. The present disclosure provides a technology that can appropriately display a bird's-eye view image of a vehicle.

[0008] Means for solving problems

[0009] An image display device according to one embodiment of the present invention comprises: a camera for photographing the surroundings of a vehicle; a storage unit for storing past images photographed by the camera; a control unit for selecting an image to be synthesized from real-time images photographed by the camera and past images stored in the storage unit according to the surrounding conditions of the vehicle or the driving state of the vehicle, and outputting a bird's-eye view image of the vehicle obtained by synthesizing the selected images and looking down on the vehicle from a viewpoint above the vehicle; and a display unit for displaying the bird's-eye view image output by the control unit.

[0010] In the image display device, an image to be synthesized is selected from the real-time image and the past image according to the surrounding conditions of the vehicle or the driving state of the vehicle, and a bird's-eye view image synthesized by the selected image is output. The image display device can display a bird's-eye view image that fully suppresses the disappearance or duplication of objects compared to a case where the real-time image is always reflected in the synthesis of the bird's-eye view image, and can display a bird's-eye view image corresponding to the change of the surrounding environment compared to a case where the past image is always reflected in the synthesis of the bird's-eye view image. As a result, the image display device can appropriately display the bird's-eye view image of the vehicle.

[0011] In one embodiment, the control unit may output one of a first bird's-eye view image synthesized by the real-time image and a second bird's-eye view image synthesized by at least the past image according to the surrounding conditions of the vehicle, or output the second bird's-eye view image according to the driving state of the vehicle. The image display device may display the first bird's-eye view image or the second bird's-eye view image according to the surrounding conditions of the vehicle or the driving state of the vehicle.

[0012] In one embodiment, the control unit may output a first bird's-eye view image when a moving object exists within the shooting range of the camera, and output a second bird's-eye view image synthesized with past images when no moving object exists within the shooting range of the camera. In this case, the image display device may output a first bird's-eye view image that prioritizes the response to changes in the surrounding environment when a moving object exists, and output a second bird's-eye view image that prioritizes the elimination of the disappearance of the object or the elimination of duplication of the object when no moving object exists.

[0013] In one embodiment, the image display device may include a plurality of cameras including a camera, the storage unit may store past images captured by the plurality of cameras, and the control unit may synthesize the real-time images captured by the plurality of cameras to generate a first bird's-eye view image, and synthesize the past images or synthesize the past images and the real-time images to generate a second bird's-eye view image. In this case, the image display device may synthesize the images of the plurality of cameras to generate a high-precision bird's-eye view image.

[0014] In one embodiment, when a stationary object exists within the shooting range of the plurality of cameras, the control unit may output a second bird's-eye view image obtained by synthesizing a bird's-eye view image of the stationary object based on a past image with the first bird's-eye view image. In this case, the image display device can appropriately display the moving object in real time while suppressing the disappearance or duplication of the stationary object.

[0015] In one embodiment, when a stationary object exists within the shooting range of the plurality of cameras and the display position of the stationary object overlaps with the synthesis boundary of the bird's-eye view image, the control unit may output a second bird's-eye view image obtained by synthesizing a bird's-eye view image of the stationary object based on a past image with the first bird's-eye view image. In this case, the image display device can appropriately display the surrounding environment in real time while suppressing the stationary object from disappearing or being doubled.

[0016] In one embodiment, when the vehicle is in motion, the control unit may generate the area in front of the vehicle in the second bird's-eye view image based on the real-time image in front of the vehicle in the direction of travel, and generate the area in the rear of the vehicle in the second bird's-eye view image based on the past image in the direction of travel of the vehicle. In this case, the image display device can respond to environmental changes by displaying the real-time image in the direction of travel that should be paid the most attention, and can suppress the disappearance or duplication of objects by displaying the past image in the direction of travel that is less important than the direction of travel.

[0017] Effects of the Invention

[0018] According to the present disclosure, a technology capable of appropriately displaying a bird's-eye view image of a vehicle is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a block diagram showing an example of the configuration of a vehicle including an image display device according to an embodiment.

[0020] Figure 2 This is an example of imaging areas of a plurality of sensors included in the vehicle 2 .

[0021] Figure 3 This is an example of a bird's-eye view image displayed on the display unit.

[0022] Figure 4 (A) is an example of the surrounding environment of the vehicle. Figure 4 (B) is related to Figure 4 An example of an overhead image corresponding to the surrounding environment shown in (A).

[0023] Figure 5 (A) is an example of the surrounding environment of the vehicle. Figure 5 (B) is related to Figure 5 An example of an overhead image corresponding to the surrounding environment shown in (A).

[0024] Figure 6 This is a flowchart showing the operation of the image display device.

[0025] Figure 7This is a block diagram showing an example of the configuration of a vehicle including an image display device according to a modification.

[0026] Figure 8 (A), (B), and (C) are examples of the surrounding environment of the vehicle. Figure 8 (E), (F), and (G) are related to Figure 8 An example of an overhead image corresponding to the surrounding environment shown in (A), (B), and (C).

[0027] Fig. 9 (A) is an example of the surrounding environment of the vehicle. Fig. 9 (B) is related to Fig. 9 An example of an overhead image corresponding to the surrounding environment shown in (A).

[0028] Fig.10 This is a flowchart showing the operation of the image display device according to the modification example.

[0029] Fig.11 (A) is an example of the surrounding environment of the vehicle. Fig.11 (B) is related to Fig.11 An example of an overhead image corresponding to the surrounding environment shown in (A).

[0030] Fig.12 This is a flowchart showing the operation of the image display device according to the modification example.

[0031] Fig.13 (A) and (B) are examples of the surrounding environment of the vehicle. Fig.13 (C) is related to Fig.13 An example of an overhead image corresponding to the surrounding environment shown in (B).

[0032] Fig.14 This is a flowchart showing the operation of the image display device according to the modification example. DETAILED DESCRIPTION

[0033] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0034] [Vehicle configuration]

[0035] Figure 1 1 is a block diagram showing an example of a configuration of a vehicle including an image display device according to an embodiment. Figure 1 As shown, the image display device 1 is mounted on a vehicle 2 as an example. The vehicle 2 may be a vehicle driven by a driver or an automatically driven vehicle.

[0036] The vehicle 2 includes a plurality of cameras (cameras) 10, a storage unit 11, an external sensor 12, an internal sensor 13, a control unit 14, and a display unit. The image display device 1 includes a plurality of cameras 10, a storage unit 11, a control unit 14, and a display unit.

[0037] The plurality of cameras 10 is a camera group that captures images of the periphery of the vehicle 2. The plurality of cameras 10 include, as an example, a front camera 101, a right side camera 102, a left side camera 103, and a rear camera 104. Figure 2 2 is an example of the imaging area of ​​the plurality of sensors provided in the vehicle 2. Figure 2 As shown, the front camera 101 is set on the front bumper, etc., and photographs the front range 101A of the vehicle 2. The right side camera 102 is set on the door mirror, etc., and photographs the right side range 102A of the vehicle 2. The left side camera 103 is set on the door mirror, etc., and photographs the left side range 103A of the vehicle 2. The rear camera 104 is set on the rear bumper, etc., and photographs the rear range 104A of the vehicle 2.

[0038] The storage unit 11 stores the past images captured by the multiple cameras 10 for each camera. The past images mean the recorded real-time images, including dynamic images (dynamic pictures) and still images (still pictures). The real-time images are images captured by the multiple cameras 10 at the current time point, including dynamic images and still images. The storage unit 11 is, for example, a storage medium such as ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), etc.

[0039] The external sensor 12 is a sensor for detecting the surrounding environment of the vehicle 2. The external sensor 12 is, for example, a radar sensor. A radar sensor is a detection device that detects objects around the vehicle using radio waves (for example, millimeter waves) or light. The radar sensor includes, for example, a millimeter wave radar or a laser radar (LiDAR: Light Detection and Ranging).

[0040] The internal sensor 13 is a sensor that detects the running state of the vehicle 2. The internal sensor 13 includes, as an example, a GPS receiver, a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor.

[0041] The control unit 14 has a function of managing the image display device. The control unit 14 is composed of an ECU (Electronic Control Unit) as an example. The ECU is an electronic control unit having a CPU (Central Processing Unit), ROM, RAM, CAN (Controller Area Network) communication circuit, etc. The control unit 14 synthesizes the images captured by the multiple cameras 10 to generate a bird's-eye view image. The bird's-eye view image is an image of the vehicle 2 viewed from a viewpoint above the vehicle 2, including a dynamic image and a still image.

[0042] The display unit 15 displays the bird's-eye view image output by the control unit 14. The display unit 15 is, for example, a display device. Figure 3 This is an example of a bird's-eye view image displayed on the display unit. Figure 3 As shown, the display unit 15 displays the bird's-eye view image GR1 generated by the control unit 14. The control unit 14 generates the bird's-eye view image GR1 including the vehicle object OB2 corresponding to the vehicle 2. The control unit 14 inputs the image signals of the plurality of cameras 10 and performs image processing to generate a bird's-eye view image of the vehicle 2 viewed from a viewpoint directly above. The viewpoint conversion is performed based on the respective configuration positions and shooting directions of the plurality of cameras 10 with reference to the reference point O of the vehicle 2. The image processing for viewpoint conversion may be performed by software or by a hardware circuit.

[0043] The control unit 14 displays the front area AR1 of the vehicle 2 using the image signal captured by the front camera 101. The control unit 14 displays the right side area AR2 of the vehicle 2 using the image signal captured by the right side camera 102. The control unit 14 displays the left side area AR3 of the vehicle 2 using the image signal captured by the left side camera 103. The control unit 14 displays the rear area AR4 of the vehicle 2 using the image signal captured by the rear camera 104. Thus, the bird's-eye view image GR1 is generated.

[0044] The control unit 14 selects an image to be synthesized from the real-time images captured by the multiple cameras 10 and the past images stored in the storage unit 11. The image to be synthesized is selected according to the surrounding conditions of the vehicle 2. For example, the control unit 14 determines whether there is a moving object within the shooting range of the multiple cameras 10 based on the detection results of the multiple cameras 10 and / or the external sensor 12. When using the results captured by the multiple cameras 10, the control unit 14 detects the moving object based on the motion vector between frames. When using the detection results of the external sensor 12, the control unit 14 detects the moving object based on the time displacement of the distance from the detected object.

[0045] When there is a moving object within the shooting range of the multiple cameras 10, the control unit 14 selects a real-time image as an image to be synthesized. Thus, the control unit 14 outputs a bird's-eye view image (first bird's-eye view image) obtained by synthesizing the real-time images. The first bird's-eye view image is an aerial image composed of images captured by the multiple cameras 10 at the current time point. When there is no moving object within the shooting range of the multiple cameras 10, the control unit 14 selects a past image as an image to be synthesized. Thus, the control unit 14 outputs a bird's-eye view image (second bird's-eye view image) obtained by synthesizing the past images stored in the storage unit 11. In this way, as an example, the control unit 14 outputs one of the first bird's-eye view image obtained by synthesizing the real-time images and the second bird's-eye view image obtained by synthesizing the past images according to the surrounding conditions of the vehicle 2.

[0046] Figure 4 (A) is an example of the surrounding environment of the vehicle. Figure 4 (B) is related to Figure 4 An example of a bird's-eye view image corresponding to the surrounding environment shown in (A). Figure 4 As shown in (A), it is assumed that the vehicle 2 is moving forward and there is a moving object 40, which is a pedestrian, in front of the vehicle 2. The vehicle 2 determines that there is a moving object 40 in the front range 101A of the front camera 101 based on the detection results of the front camera 101 and / or the external sensor 12. In this case, the control unit 14 selects the real-time image as the image to be synthesized. The control unit 14 outputs the first bird's-eye view image GR2 obtained by synthesizing the real-time images. Figure 4 As shown in (B) of FIG. 1 , the display unit 15 displays the first bird's-eye view image GR2 output by the control unit 14. The first bird's-eye view image GR2 includes a moving object image OB40 corresponding to the moving object 40. Thus, the driver can understand the surroundings of the vehicle 2 in real time.

[0047] Figure 5 (A) is an example of the surrounding environment of the vehicle. Figure 5 (B) is related to Figure 5 An example of a bird's-eye view image corresponding to the surrounding environment shown in (A). Figure 5 As shown in (A), it is assumed that the vehicle 2 is moving forward and there is a stationary object 30 such as a road pole in front of the vehicle 2. The vehicle 2 determines that there is a stationary object 30 in the front range 101A of the front camera 101 based on the detection results of the front camera 101 and / or the external sensor 12. In this case, the control unit 14 selects a past image as the image to be synthesized. The control unit 14 outputs the second bird's-eye view image GR3 obtained by synthesizing the past images. Figure 5As shown in (B), the display unit 15 displays the second bird's-eye view image GR3 output by the control unit 14. The second bird's-eye view image GR3 includes a stationary object image OB30 corresponding to the stationary object 30. With respect to the second bird's-eye view image GR3, since the entire area PS1 of the image is produced based on the past image, there is no boundary caused by synthesis, and the disappearance or duplication of the object can be avoided.

[0048] [Operation of the image display device]

[0049] Figure 6 This is a flowchart showing the operation of the image display device. Figure 6 The flowchart shown is executed by the control unit 14 at the timing of receiving a driver's operation, for example.

[0050] like Figure 6 As shown, the control unit 14 determines whether there is an obstacle in the camera image in step S10 . The control unit 14 determines whether there is an obstacle around the vehicle 2 based on the detection results of the plurality of cameras 10 and / or the external sensor 12 .

[0051] If it is determined that there is an obstacle in the camera image (step S10: Yes), the control unit 14 determines whether the obstacle is a moving object in step S12. The control unit 14 determines whether the obstacle determined in step S10 is a moving object based on the detection results of the plurality of cameras 10 and / or the external sensor 12.

[0052] If it is determined that the obstacle is a moving object (step S12: Yes), the control unit 14 selects a real-time image as an image to be synthesized in step S14. Then, the control unit 14 causes the display unit 15 to display the first bird's-eye view image (see FIG. 1 ). Figure 4 (B)).

[0053] If it is determined that there is no obstacle in the camera image (step S10: No), or if it is determined that the obstacle is not a moving object (step S12: No), the control unit 14 selects a past image as an image to be synthesized in step S16. Then, the control unit 14 causes the display unit 15 to display the second bird's-eye view image (see FIG. 1 ). Figure 5 (B)). When step S14 or step S16 is completed, the flowchart ends. The flowchart is repeatedly executed until the driver's operation of ending is accepted.

[0054] [Summary of implementation methods]

[0055] In the image display device 1, an image to be synthesized is selected from the real-time image and the past image according to the surrounding conditions of the vehicle 2, and a bird's-eye view image synthesized by the selected image is output. By reflecting the past image in the synthesis of the bird's-eye view image according to the surrounding conditions of the vehicle 2, the image display device 1 can display a bird's-eye view image that fully suppresses the disappearance or duplication of objects compared to the case where the real-time image is always reflected in the synthesis of the bird's-eye view image, and can display a bird's-eye view image corresponding to the changes in the surrounding environment compared to the case where the past image is always reflected in the synthesis of the bird's-eye view image. As a result, the image display device 1 can appropriately display the bird's-eye view image of the vehicle 2.

[0056] As mentioned above, although the exemplary embodiment was described, it is not limited to the above-mentioned exemplary embodiment, and various omissions, substitutions, and changes are possible.

[0057] [Modifications]

[0058] [Single camera configuration]

[0059] Figure 7 This is a block diagram showing an example of the configuration of a vehicle including an image display device according to a modification. Figure 7 The image display device 1A shown is different from the image display device 1 in that it does not include the plurality of cameras 10 but only includes the rear camera 104, and that a portion of the operation of the control unit 14A is different from that of the control unit 14. The following description will focus on the differences and will not be repeated.

[0060] The control unit 14A first generates a bird's-eye view image using only the real-time image of the rear camera 104. When the control unit 14A obtains a past image as the vehicle 2 moves, the control unit 14A combines the real-time image with the past image stored in the storage unit 11 to generate a bird's-eye view image.

[0061] Figure 8 (A), (B), and (C) are examples of the surrounding environment of the vehicle. Figure 8 (E), (F), and (G) are related to Figure 8 An example of a bird's-eye view image corresponding to the surrounding environment shown in (A), (B), and (C). Figure 8 As shown in (A), (B), and (C) in sequence, the vehicle 2 is assumed to move in a backward manner. The control unit 14A is in the state where the vehicle 2 is Figure 8 When the position of (A) is Figure 8As shown in (E), a first bird's-eye view image GR4 synthesized using only real-time images is generated. In the first bird's-eye view image GR4, an image is displayed only in the area AR5 corresponding to the shooting range of the rear camera 104, and the other areas are, for example, blank. In the area AR5, a stationary object image OB30 corresponding to the stationary object 30 is displayed.

[0062] Next, the control unit 14A controls the vehicle 2 to Figure 8 When the position of (B) is Figure 8 As shown in (F) of FIG. 1 , a second bird's-eye view image GR5 is generated by synthesizing the real-time image and the past image. In the second bird's-eye view image GR5, images are displayed only in the area AR5 corresponding to the shooting range of the rear camera 104 and the area AR6 corresponding to the past image, and other areas are, for example, blank. Next, the control unit 14A generates a second bird's-eye view image GR5 by synthesizing the real-time image and the past image. Figure 8 When the position of (C) is Figure 8 As shown in (G), a second bird's-eye view image GR6 synthesized by the real-time image and the past image is generated. In the second bird's-eye view image GR6, images are displayed only in the area AR5 corresponding to the shooting range of the rear camera 104 and the area AR6 corresponding to the past image, and other areas are, for example, blank. The second bird's-eye view image GR6 can use more past images than the second bird's-eye view image GR5, so the area AR6 becomes longer.

[0063] The control unit 14A, like the control unit 14, displays a first bird's-eye view image (for example, Figure 8 (E)), a second bird's-eye view image is displayed when no moving object is detected (e.g. Figure 8 (F), (G)). Thus, when no moving object is detected, the image display device 1A increases the displayable range by displaying past images, thereby increasing the information to be notified, and when a moving object is detected, past images that cannot cope with the intrusion of the moving object are not displayed, thereby providing accurate information to the driver.

[0064] [Example of applying past images only to stationary objects]

[0065] Fig. 9 (A) is an example of the surrounding environment of the vehicle. Fig. 9 (B) is related to Fig. 9 An example of a bird's-eye view image corresponding to the surrounding environment shown in (A). Fig. 9As shown in (A), it is assumed that the vehicle 2 is moving forward, and there are stationary objects 30 such as road poles and moving objects 40 such as pedestrians in front of the vehicle 2. The vehicle 2 determines that there are stationary objects 30 and moving objects 40 in the front range 101A of the front camera 101 based on the detection results of the front camera 101 and / or the external sensor 12. In this case, the control unit 14 selects a real-time image and a past image as images to be synthesized. The control unit 14 generates a first bird's-eye view image obtained by synthesizing the real-time image, and synthesizes a bird's-eye view image of the stationary object 30 based on the past image with the first bird's-eye view image.

[0066] First, the control unit 14 determines whether there is a past image of the stationary object 30 based on the position of the stationary object 30. The control unit 14 may also match the past image with the real-time image to determine whether there is a past image of the stationary object 30. When it is determined that there is a past image of the stationary object 30, the control unit 14 generates a bird's-eye view image of the stationary object 30. The control unit 14 may crop a certain range of image area where the stationary object 30 appears, and generate a bird's-eye view image based on the cropped image area, or may generate an entire bird's-eye view image based on the past image, and then crop a certain range of image area where the stationary object 30 appears. The generated bird's-eye view image of the stationary object 30 overlaps with the image of the stationary object 30 on the first bird's-eye view image generated based on the real-time image. In this way, a bird's-eye view image of the stationary object 30 is generated. Fig. 9 The second bird's-eye view image GR7 shown in (B) of FIG. The regions PS2 and PS3 including the stationary object image OB30 are regions generated based on past images.

[0067] Fig.10 This is a flowchart showing the operation of the image display device according to the modification example. Fig.10 The flowchart shown is executed by the control unit 14 at the timing of receiving a driver's operation, for example.

[0068] like Fig.10 As shown, the control unit 14 determines whether there is an obstacle in the camera image in step S20. This determination is the same as step S10.

[0069] If it is determined that there is an obstacle in the camera image (step S20: Yes), the control unit 14 determines whether the obstacle is a stationary object in step S22. The control unit 14 determines whether the obstacle determined in step S20 is a stationary object based on the detection results of the plurality of cameras 10 and / or the external sensor 12.

[0070] If it is determined that there is no obstacle in the camera image (step S20: No), or if it is determined that the obstacle is not a stationary object (step S22: No), the control unit 14 selects a real-time image as an image to be synthesized in step S24. The control unit 14 causes the display unit 15 to display the first bird's-eye view image.

[0071] If it is determined that the obstacle is a stationary object (step S22: Yes), the control unit 14 selects the real-time image and the past image as the images to be synthesized in step S26. The control unit 14 generates the first bird's-eye view image based on the real-time image and generates the stationary object image OB30 based on the past image. The control unit 14 displays the second bird's-eye view image GR7 obtained by synthesizing the stationary object image OB30 with the first bird's-eye view image on the display unit 15 (see FIG. 14 ). Fig. 9 (B)). When step S24 or step S26 is completed, the flowchart ends. The flowchart is repeatedly executed until the driver's operation of ending is accepted.

[0072] As described above, the image display device can both appropriately display moving objects in real time and suppress the disappearance or duplication of stationary objects. In addition, the image display device can also use the presence or absence of a moving object as an additional condition for generating a second bird's-eye view image. That is, the image display device can also be configured to generate a second bird's-eye view image when both a moving object and a stationary object exist within the shooting range of multiple cameras 10. In addition, when it is determined that there is no obstacle in the camera image (step S20: No), a second bird's-eye view image based only on a past image can also be displayed.

[0073] [Example of applying past images only to stationary objects that meet the conditions]

[0074] Fig.11 (A) is an example of the surrounding environment of the vehicle. Fig.11 (B) is related to Fig.11 An example of a bird's-eye view image corresponding to the surrounding environment shown in (A). Fig.11 As shown in (A), it is assumed that the vehicle 2 is moving forward, and there are stationary objects 30 such as road poles and moving objects 40 such as pedestrians in front of the vehicle 2. The vehicle 2 determines that there are stationary objects 30 and moving objects 40 in the front range 101A of the front camera 101 based on the detection results of the front camera 101 and / or the external sensor 12. In this case, the control unit 14 selects a real-time image and a past image as images to be synthesized. The control unit 14 generates a first bird's-eye view image obtained by synthesizing the real-time image, and synthesizes a bird's-eye view image of the stationary object 30 based on the past image with the first bird's-eye view image.

[0075] First, the control unit 14 determines whether there is a past image of the stationary object 30 based on the position of the stationary object 30. The control unit 14 may also match the past image with the real-time image to determine whether there is a past image of the stationary object 30. When it is determined that there is a past image of the stationary object 30, the control unit 14 determines whether the display position of the stationary object 30 overlaps with the synthetic boundary of the bird's-eye view image. The synthetic boundary refers to the boundary of the front area AR1, the right area AR2, the left area AR3, and the rear area AR4. In the case of a stationary object 30 that overlaps with the synthetic boundary, the control unit 14 generates a bird's-eye view image of the stationary object 30. The control unit 14 may crop a certain range of the image area where the stationary object 30 appears, and generate a bird's-eye view image based on the cropped image area, or may generate an entire bird's-eye view image based on the past image, and then crop the certain range of the image area where the stationary object 30 appears. The generated bird's-eye view image of the stationary object 30 overlaps with the image of the stationary object 30 on the first bird's-eye view image generated based on the real-time image. In this way, a bird's-eye view image of the stationary object 30 is generated. Fig.11 The second bird's-eye view image GR8 shown in (B) of FIG. The region PS5 including the stationary object image OB30 overlapping the synthesis boundary is a region generated based on the past image.

[0076] Fig.12 This is a flowchart showing the operation of the image display device according to the modification example. Fig.12 The flowchart shown is executed by the control unit 14 at the timing of receiving a driver's operation, for example.

[0077] like Fig.12 As shown, the control unit 14 determines whether there is an obstacle in the camera image in step S30. This determination is the same as step S10.

[0078] When it is determined that an obstacle exists in the camera image (step S30: Yes), the control unit 14 determines whether the obstacle is a stationary object in step S32. This determination is the same as that in step S22.

[0079] If the obstacle is determined to be a stationary object (step S32: Yes), the control unit 14 determines whether the obstacle overlaps with the synthesis boundary in step S33. The control unit 14 calculates the display position of the stationary object based on the imaging results of the plurality of cameras 10 and determines whether it overlaps with the synthesis boundary.

[0080] If it is determined that there is no obstacle in the camera image (step S30: No), if it is determined that the obstacle is not a stationary object (step S32: No), or if it is determined that the obstacle does not overlap with the synthesis boundary (step S33: No), the control unit 14 selects a real-time image as an image to be synthesized in step S34. Then, the control unit 14 causes the display unit 15 to display the first bird's-eye view image.

[0081] When it is determined that the obstacle overlaps with the synthesis boundary (step S33: Yes), the control unit 14 selects the real-time image and the past image as the images to be synthesized in step S36. The control unit 14 generates the first bird's-eye view image based on the real-time image and generates the still object image OB30 overlapping with the synthesis boundary based on the past image. The control unit 14 displays the second bird's-eye view image GR8 obtained by synthesizing the still object image OB30 with the first bird's-eye view image on the display unit 15 (see FIG. 14 ). Fig.11 (B)). When step S34 or step S36 is completed, the flowchart ends. The flowchart is repeatedly executed until the driver's operation of ending is accepted.

[0082] As described above, the image display device can both appropriately display moving objects in real time and suppress the disappearance or duplication of stationary objects. In addition, the image display device can also use the presence or absence of moving objects as an additional condition for generating the second bird's-eye view image. In other words, the image display device can also be configured to generate a second bird's-eye view image when both moving objects and stationary objects exist within the shooting range of multiple cameras 10. In addition, when it is determined that there is no obstacle in the camera image (step S30: No), a second bird's-eye view image based only on the past image can also be displayed.

[0083] [Display according to the vehicle's driving status]

[0084] The control unit 14 may also select an image to be synthesized according to the driving state of the vehicle, and output a bird's-eye view image synthesized by the selected image. The control unit 14 divides the driving state of the vehicle 2 into two types: "stopped" and "moving". "Moving" includes "moving forward", "backward", "turning right" and "turning left". When the driving state of the vehicle 2 is moving, the control unit 14 generates the area in front of the vehicle 2 in the direction of travel in the second bird's-eye view image based on the real-time image in front of the vehicle 2 in the direction of travel, and generates the area in the rear of the vehicle 2 in the direction of travel in the second bird's-eye view image based on the past image in the direction of travel behind the vehicle 2. That is, the control unit 14 creates a bird's-eye view image based on the real-time image for the area to be driven next, and creates a bird's-eye view image based on the past image for the area that has already been driven.

[0085] Fig.13 (A) and (B) are examples of the surrounding environment of the vehicle. Fig.13 (C) is related to Fig.13 An example of a bird's-eye view image corresponding to the surrounding environment shown in (B). Fig.13 As shown in (A) and (B) of FIG. 1 , the vehicle 2 is assumed to be moving forward. In this case, Fig.13 As shown in (C), the control unit 14 generates the front area AR1 (the front area in the traveling direction of the vehicle 2) of the second bird's-eye view image GR9 based on the real-time image showing the front in the traveling direction. In addition, the control unit 14 generates the rear area AR4 (the rear area in the traveling direction of the vehicle 2) of the second bird's-eye view image based on the past image showing the rear in the traveling direction of the vehicle 2. Thus, only the rear area AR4 becomes the area PS6 based on the past image.

[0086] Fig.14 This is a flowchart showing the operation of the image display device according to the modification example. Fig.14 The flowchart shown is executed by the control unit 14 at the timing of receiving a driver's operation, for example.

[0087] like Fig.14 As shown, the control unit 14 determines whether the running state of the vehicle 2 is stopped in step S40 . The control unit 14 determines the running state of the vehicle 2 based on the operations of the plurality of cameras 10 or the detection results of the internal sensor 13 .

[0088] When it is determined that the vehicle 2 is stopped (step S40: Yes), the control unit 14 displays the first bird's-eye view image synthesized with the real-time images. Alternatively, the control unit 14 may display the second bird's-eye view image synthesized with only the past images.

[0089] When it is determined that the driving state of the vehicle 2 is not stopped (step S40: No), the control unit 14 determines whether the driving state of the vehicle 2 is moving forward in step S44. When it is determined that the driving state of the vehicle 2 is moving forward (step S44: Yes), the control unit 14 selects the real-time image and the past image as the images to be synthesized in step S46. Then, the control unit 14 generates a first bird's-eye view image based on the real-time image, and generates a bird's-eye view image based on the past image and the past image of the rear of the vehicle. The control unit 14 displays the second bird's-eye view image GR9 obtained by synthesizing the first bird's-eye view image with the bird's-eye view image of the rear of the vehicle on the display unit 15 (refer to Fig.13 (C)).

[0090] When it is determined that the driving state of the vehicle 2 is not moving forward (step S44: No), the control unit 14 determines whether the driving state of the vehicle 2 is moving backward in step S48. When it is determined that the driving state of the vehicle 2 is moving backward (step S48: Yes), the control unit 14 selects the real-time image and the past image as the image to be synthesized in step S50. In addition, the control unit 14 generates a first bird's-eye view image based on the real-time image, and generates a bird's-eye view image based on the past image and the past image in front of the vehicle. The control unit 14 displays the second bird's-eye view image GR9 obtained by synthesizing the bird's-eye view image in front of the vehicle with the first bird's-eye view image on the display unit 15.

[0091] When it is determined that the driving state of the vehicle 2 is not in reverse (step S48: No), the control unit 14 determines whether the driving state of the vehicle 2 is turning right in step S52. When it is determined that the driving state of the vehicle 2 is turning right (step S52: Yes), the control unit 14 selects the real-time image and the past image as the image to be synthesized in step S54. And the control unit 14 generates a first bird's-eye view image based on the real-time image, and generates a bird's-eye view image based on the past image and the past image of the left side of the vehicle based on the past image. The control unit 14 displays the second bird's-eye view image GR9 obtained by synthesizing the bird's-eye view image of the left side of the vehicle with the first bird's-eye view image on the display unit 15.

[0092] If it is determined that the driving state of the vehicle 2 is not turning right (step S48: No), the driving state of the vehicle 2 is turning left. In step S56, the control unit 14 selects the real-time image and the past image as the image to be synthesized. In addition, the control unit 14 generates a first bird's-eye view image based on the real-time image and a bird's-eye view image based on the past image and the right side of the vehicle. The control unit 14 displays the second bird's-eye view image GR9 obtained by synthesizing the bird's-eye view image of the right side of the vehicle with the first bird's-eye view image on the display unit 15.

[0093] When step S42, step S46, step S50, step S54 or step S56 is completed, the flowchart ends. The flowchart is repeatedly executed until the driver's operation of the end is accepted. The image display device can respond to environmental changes by displaying a real-time image in the direction of travel that should be paid the most attention, and suppress the disappearance or duplication of objects by displaying a past image in the direction of travel that is less important than the direction of travel.

[0094] The present disclosure also includes the following items.

[0095] [Item 1] An image display device comprising:

[0096] A camera to capture the surroundings of the vehicle;

[0097] a storage unit for storing past images captured by the camera;

[0098] a control unit that selects an image to be synthesized according to the surrounding conditions of the vehicle or the driving state of the vehicle from the real-time image captured by the camera and the past images stored in the storage unit, and outputs a bird's-eye view image of the vehicle obtained by synthesizing the selected images and looking down on the vehicle from a viewpoint above the vehicle; and

[0099] The display unit displays the bird's-eye view image output by the control unit.

[0100] [Item 2] The image display device according to Item 1,

[0101] The control unit outputs one of a first bird's-eye view image synthesized by the real-time image and a second bird's-eye view image synthesized by at least the past image according to the surrounding conditions of the vehicle, or outputs the second bird's-eye view image according to the driving state of the vehicle.

[0102] [Item 3] The image display device according to Item 2,

[0103] The control unit outputs the first bird's-eye view image when there is a moving object within the imaging range of the camera, and outputs the second bird's-eye view image synthesized with the past images when there is no moving object within the imaging range of the camera.

[0104] [Item 4] The image display device according to Item 2 or 3,

[0105] The image display device includes a plurality of cameras including the camera,

[0106] The storage unit stores past images captured by the plurality of cameras.

[0107] The control unit generates the first bird's-eye view image by synthesizing the real-time images captured by the plurality of cameras.

[0108] The control unit generates the second bird's-eye view image by synthesizing the past images or synthesizing the past images and the real-time images.

[0109] [Item 5] The image display device according to Item 4,

[0110] The control unit outputs the second bird's-eye view image obtained by synthesizing the bird's-eye view image of the stationary object based on the past image with the first bird's-eye view image, when a stationary object exists within the imaging range of the plurality of cameras.

[0111] [Item 6] The image display device according to Item 4,

[0112] When there is a stationary object within the shooting range of the multiple cameras and the display position of the stationary object overlaps with the synthesis boundary of the bird's-eye view image, the control unit outputs the second bird's-eye view image obtained by synthesizing the first bird's-eye view image with the bird's-eye view image of the stationary object based on the past image.

[0113] [Item 7] The image display device according to any one of Items 1 to 4,

[0114] When the vehicle is in motion, the control unit generates an area in front of the vehicle in the second bird's-eye view image based on a real-time image projected in front of the vehicle in the direction of travel, and generates an area in the rear of the vehicle in the second bird's-eye view image based on a past image projected in the direction of travel behind the vehicle.

Claims

1. An image display device comprising: A camera to capture the surroundings of the vehicle; a storage unit for storing past images captured by the camera; a control unit that selects an image to be synthesized according to the surrounding conditions of the vehicle or the driving state of the vehicle from the real-time image captured by the camera and the past images stored in the storage unit, and outputs a bird's-eye view image of the vehicle obtained by synthesizing the selected images and looking down on the vehicle from a viewpoint above the vehicle; and The display unit displays the bird's-eye view image output by the control unit.

2. The image display device according to claim 1, The control unit outputs one of a first bird's-eye view image synthesized by the real-time image and a second bird's-eye view image synthesized by at least the past image according to the surrounding conditions of the vehicle, or outputs the second bird's-eye view image according to the driving state of the vehicle.

3. The image display device according to claim 2, The control unit outputs the first bird's-eye view image when there is a moving object within the imaging range of the camera, and outputs the second bird's-eye view image synthesized with the past images when there is no moving object within the imaging range of the camera.

4. The image display device according to claim 2 or 3, The image display device includes a plurality of cameras including the camera, The storage unit stores past images captured by the plurality of cameras. The control unit generates the first bird's-eye view image by synthesizing the real-time images captured by the plurality of cameras. The control unit generates the second bird's-eye view image by synthesizing the past images or synthesizing the past images and the real-time images.

5. The image display device according to claim 4, The control unit outputs the second bird's-eye view image obtained by synthesizing the bird's-eye view image of the stationary object based on the past image with the first bird's-eye view image, when a stationary object exists within the imaging range of the plurality of cameras.

6. The image display device according to claim 4, When there is a stationary object within the shooting range of the multiple cameras and the display position of the stationary object overlaps with the synthesis boundary of the bird's-eye view image, the control unit outputs the second bird's-eye view image obtained by synthesizing the first bird's-eye view image with the bird's-eye view image of the stationary object based on the past image.

7. The image display device according to claim 4, When the vehicle is in motion, the control unit generates an area in front of the vehicle in the second bird's-eye view image based on a real-time image projected in front of the vehicle in the direction of travel, and generates an area in the rear of the vehicle in the second bird's-eye view image based on a past image projected in the direction of travel behind the vehicle.

Images