Video display device and video display method

Abstract

The present disclosure relates to an image display device and an image display method, the image display device according to the present invention includes: a camera that captures an image of a periphery of a vehicle; a control unit that outputs a bird's-eye view image in which the vehicle is viewed from above the vehicle, the bird's-eye view image being obtained by combining the images captured by the camera; a display unit that displays the aerial view image output by the control unit; and a sensor detecting an open state of a door of the vehicle. The control unit is configured to enlarge and display the bird's-eye view image on the display unit, and to release the enlargement of the bird's-eye view image in response to the opening state of the door of the vehicle detected by the sensor.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an image display device and an image display method. BACKGROUND

[0002] Japanese Patent Application Publication No. 2012-51391 discloses an image display device for a vehicle. The device converts image signals of a plurality of cameras that capture the surroundings of the vehicle into image signals when viewed downward from a top view of the vehicle, and composites the converted image signals. Thereby, a bird's-eye image when viewed downward from the top view of the vehicle is displayed. Further, the device stores in advance a trajectory of a door of the vehicle when the door is opened and closed, and displays the stored trajectory of the door on the bird's-eye image at the time of parking. Thereby, an occupant can confirm whether there is a region in which the door of the vehicle can be opened to get off the vehicle between the vehicle and a neighboring parked vehicle or the like.

[0003] In addition, in the device described in Japanese Patent Application Publication No. 2012-51391, it can be considered to reflect the opening state of the door in the bird's-eye image in order to draw the attention of the occupant. In addition, it can be considered to add a function of enlarging and displaying the bird's-eye image in order to improve the convenience of the occupant. However, in the case of enlarging and displaying the bird's-eye image, the opening state of the door reflected in order to draw the attention can exist outside the screen and not be displayed. The present disclosure provides an image display device and an image display method that appropriately balance the convenience of image enlargement and the drawing of attention of the opening state of the door when displaying a bird's-eye image of a vehicle. SUMMARY

[0004] An image display device according to a first aspect of the present disclosure includes a camera, a control unit, a display unit, and a sensor. The camera is configured to capture a surrounding of a vehicle. The control unit is configured to output a bird's-eye image of the vehicle when viewed downward from a top view of the vehicle, which is obtained by compositing images captured by the camera. The display unit is configured to display the bird's-eye image output by the control unit. The sensor is configured to detect an opening state of a door of the vehicle. The control unit is configured to enlarge and display the bird's-eye image on the display unit. The control unit is configured to cancel the enlargement of the bird's-eye image in accordance with the opening state of the door of the vehicle detected by the sensor.

[0005] In the image display device, images captured by cameras that capture the periphery of a vehicle are synthesized, and an overhead image is displayed on a display section. The overhead image is displayed enlarged on the display section. The enlargement of the overhead image is cancelled in accordance with the detection of an open state of a door of the vehicle by a sensor. By the cancellation of the enlarged state, the overhead image looking down on the entire vehicle is displayed on the display section, and thus the image display device can alert the open state of the door on the overhead image. In this way, the image display device can appropriately balance the convenience of the image enlargement and the alert of the open state of the door by cancelling the enlarged display when the enlarged display is permitted and the door becomes the open state.

[0006] On the basis of the image display device according to the first aspect of the present disclosure, the control section can be configured to determine whether all the doors in the open state detected by the sensor are within the overhead image displayed on the display section in the enlarged state. The control section can be configured to cancel the enlargement of the overhead image in accordance with the determination that all the doors in the open state are within the overhead image. In the case where all the doors in the open state are within the overhead image, the enlargement of the overhead image is maintained, and thus unnecessary cancellation of the enlargement of the overhead image is avoided. Thus, the image display device can more appropriately balance the convenience of the image enlargement and the alert of the open state of the door.

[0007] The image display method according to the second aspect of the present disclosure includes generating an overhead image looking down on a vehicle from a viewpoint above the vehicle by synthesizing images captured by cameras that capture the periphery of the vehicle. The image display method includes causing the generated overhead image to be displayed enlarged on a display section, detecting an open state of a door of the vehicle, and cancelling the enlargement of the overhead image in accordance with the detection of the open state of the door of the vehicle. The image display method achieves the same effects as the image display device described above.

[0008] According to the present disclosure, a technology that can appropriately balance the convenience of image enlargement and the alert of an open state of a door when displaying an overhead image of a vehicle is provided. BRIEF DESCRIPTION OF DRAWINGS

[0009] Features, advantages, and technical and industrial significance of exemplary embodiments of the present application will be described below with reference to the accompanying drawings, wherein the same reference numerals denote the same elements, and wherein:

[0010] Figure 1 is a block diagram showing an example of a configuration of a vehicle provided with an image display device according to an embodiment.

[0011] Figure 2 is an example of a captured region of a plurality of cameras provided in the vehicle 2.

[0012] Figure 3 is an example of an overhead image displayed on the display section.

[0013] Figure 4 is a diagram illustrating an example of a display section having a display area of a camera image and a display area of an overhead image.

[0014] Figure 5A is an example of an overhead image.

[0015] Figure 5B is an example of an enlarged overhead image.

[0016] Figure 5C is another example of an enlarged overhead image.

[0017] Figure 6A is an example of an enlarged overhead image.

[0018] Figure 6B is an example of an overhead image when the enlargement of the overhead image is released in accordance with the opening of the door.

[0019] Figure 7 is a flowchart showing the operation of the image display device. DETAILED DESCRIPTION

[0020] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0021] [Configuration of vehicle]

[0022] Figure 1 is a block diagram showing an example of the structure of a vehicle equipped with the image display device according to one embodiment. As shown in Figure 1 , the image display device 1 is mounted on a vehicle 2 as an example. The vehicle 2 can be a vehicle driven by a driver or an autonomous vehicle.

[0023] The vehicle 2 is equipped with a plurality of cameras 10, a sensor 11, a control section 12, a display section 13, and an operation section 14. The image display device 1 is configured to be equipped with the plurality of cameras 10, the sensor 11, the control section 12, and the display section 13.

[0024] The plurality of cameras 10 is a group of cameras that capture the periphery of the vehicle 2. As an example, the plurality of cameras 10 includes a front camera 101, a right side camera 102, a left side camera 103, and a rear camera 104. Figure 2 is an example of a captured area of the plurality of cameras equipped in the vehicle 2. As Figure 2As shown, a front camera 101 is mounted on a front bumper or the like, capturing an image of an area 101A in front of the vehicle 2. A right camera 102 is mounted on a door mirror or the like, capturing an image of an area 102A to the right of the vehicle 2. A left camera 103 is mounted on a door mirror or the like, capturing an image of an area 103A to the left of the vehicle 2. A rear camera 104 is mounted on a rear bumper or the like, capturing an image of an area 104A to the rear of the vehicle 2. The multiple cameras 10 each output an image. These images include both moving images and still images.

[0025] Sensor 11 is a detector for detecting the open state of a door of vehicle 2. For example, sensor 11 is a linear sensor for detecting the door's opening degree, or a sensor for detecting the state of a curtain switch. The doors of vehicle 2 can be not only doors located at each seat, but also trunk doors, luggage compartment doors, engine compartment hoods, and the like. The type of door of vehicle 2 is not particularly limited and can be a conventional vertically hinged door for boarding and exiting, or a hatch-type, sliding, bi-fold, or gull-wing door.

[0026] The control unit 12 has the function of overseeing the image display device. As an example, the control unit 12 is composed of an ECU (Electronic Control Unit). The ECU is an electronic control unit that has a processor such as a CPU (Central Processing Unit), storage devices such as ROM and RAM, communication devices such as a CAN (Controller Area Network) communication circuit, and input and output circuits. The control unit 12 synthesizes the images captured by the multiple cameras 10 to generate a bird's-eye view image. The bird's-eye view image is an image of the vehicle 2 viewed from above, including both moving and still images.

[0027] The display unit 13 displays the bird's-eye view image output by the control unit 12. As an example, the display unit 13 is a display device. Figure 3 This is an example of an overhead image displayed on the display unit. Figure 3 As shown, the display unit 13 displays a bird's-eye view image G generated by the control unit 12. The control unit 12 generates the bird's-eye view image G, which includes a vehicle object OB corresponding to the vehicle 2. The vehicle object OB is pre-stored in the ECU's storage device. The control unit 12 performs image processing to input image signals from the multiple cameras 10 and generate a bird's-eye view image of the vehicle 2 viewed from directly above. With reference to the reference point O of the vehicle 2, the viewpoint is converted based on the respective configuration positions and shooting directions of the multiple cameras 10. The image processing for the viewpoint conversion can be performed by software or hardware circuitry.

[0028] The control section 12 displays the front region ARl of the vehicle 2 based on the image signal captured by the front camera 101. The control section 12 displays the right side region AR2 of the vehicle 2 based on the image signal captured by the right side camera 102. The control section 12 displays the left side region AR3 of the vehicle 2 based on the image signal captured by the left side camera 103. The control section 12 displays the rear region AR4 of the vehicle 2 based on the image signal captured by the rear camera 104. Thus, the bird's-eye view image G is generated.

[0029] The control section 12 can cause the bird's-eye view image to be displayed on the entire screen of the display section 13 as shown in Figure 3 , or can cause the bird's-eye view image to be displayed on a part of the display region by dividing the screen of the display section 13. Figure 4 is a view illustrating an example of the display section having a display region of a camera image and a display region of a bird's-eye view image. As shown in Figure 4 , the display section 13 can have a first display region 130 and a second display region 131. The first display region 130 is a region in which a camera image is displayed. In Figure 4 , the image of the front camera 101 is displayed, and a parking space divided by a white line is visualized. The control section 12 can switch the camera image displayed on the first display region 130 according to the traveling direction. The second display region 131 is a region in which the bird's-eye view image shown in Figure 3 is displayed. In Figure 4 , by checking the bird's-eye view image, the positional relationship between the vehicle and the parking space can be grasped at a glance. The control section 12 can appropriately report information about the surroundings of the vehicle 2 to the occupant by displaying images of the surroundings of the vehicle 2 from various angles. Further, the occupant is a person who rides in the vehicle, including the driver, the passenger, and the like.

[0030] The control section 12 can display the bird's-eye view image enlarged on the display section 13. As shown in Figure 3 , for the basic bird's-eye view image before being enlarged, the entire vehicle object OB is displayed, including the image of the entire surroundings of the vehicle object OB. The enlarged bird's-eye view image means that a partial region included in the basic bird's-eye view image is selected, and the image of the selected region is resized so as to match the size of the display region of the bird's-eye view image (the second display region 131 or the entire screen of the display section 13).

[0031] The selection of the partial region is performed by the occupant. For example, Figure 1 , the operation section 14 accepts the operation of the occupant. The operation section 14 can be a switch, a button, or a touch panel. In the case where the operation section 14 is constituted by a touch panel, the operation section 14 is integrally constituted with the display section 13.

[0032] The operation by the occupant can also be an operation of designating a partial region such as an "upper region", a "lower region", and the like, which are set in advance. The partial region set in advance can be, for example, two regions of an "upper region" and a "lower region" formed by dividing a screen into two, or four regions of a "right upper region", a "left upper region", a "right lower region", and a "left lower region" formed by dividing a screen into four. The designated operation can be an operation of selecting a prompted partial region, or an operation of tapping with a finger, pinching out a portion that is intended to be enlarged in the bird's-eye view image displayed on the touch panel. The operation section 14 can also be a device capable of inputting by voice or gestures.

[0033] The control section 12 decides a region to be enlarged based on the operation by the occupant accepted by the operation section 14. The control section 12 enlarges the decided region and displays it on the display section 13. Figure 5A is an example of a bird's-eye view image, Figure 5B is an example of an enlarged bird's-eye view image, Figure 5C is another example of an enlarged bird's-eye view image.

[0034] As shown in Figure 5A , a bird's-eye view image is displayed in a first display region 131. Further, in a case where an upper region of the vehicle is selected as a region to be enlarged by the occupant, the control section 12 enlarges a first region R1 in the bird's-eye view image shown in Figure 5A , which roughly contains an upper half of the vehicle object OB, to the size of the second display region 131, and displays it on the second display region 131 Figure 5B . At this time, a vehicle object OB1 corresponding to the upper half of the vehicle object OB is displayed in the second display region 131.

[0035] Further, for example, in a case where a right upper region of the vehicle is selected as a region to be enlarged by the occupant, the control section 12 enlarges a second region R2 in the bird's-eye view image shown in Figure 5A , which roughly contains a right upper of the vehicle object OB, to the size of the second display region 131, and displays it on the second display region 131 Figure 5C . At this time, a vehicle object OB2 corresponding to the right upper of the vehicle object OB is displayed in the second display region 131.

[0036] The control section 12 can cut a region from the bird's-eye view image shown in Figure 5A and enlarge the size of the cut image, or can display a portion corresponding to the region on the entire surface of the second display region 131 after enlarging the entire bird's-eye view image. Further, the control section 12 can generate the vehicle objects OB1, OB2 from the vehicle object OB, or can store the vehicle objects OB1, OB2 in a storage device of the ECU in advance.

[0037] The control section 12 cancels the enlargement of the bird's-eye view image in accordance with the opening state of the door of the vehicle 2 detected by the sensor 11. Figure 6A is an example of the enlarged bird's-eye view image, Figure 6B is an example of the bird's-eye view image when the enlargement of the bird's-eye view image is cancelled in accordance with the opening of the door. As shown in Figure 6A , the control section 12 causes the region in the bird's-eye view image that roughly contains the upper half of the vehicle object OB to be enlarged to the size of the second display region 131 and displayed in the second display region 131. In this state, the sensor 11 detects that the right rear door is opened. In this case, the control section 12 cancels the enlargement of the bird's-eye view image shown in Figure 6A . If the enlargement of the bird's-eye view image is cancelled, the basic bird's-eye view image is displayed in the second display region 131 as shown in Figure 6B . The basic bird's-eye view image includes the image of the entire vehicle object and the entire surroundings of the vehicle object.

[0038] Here, the control section 12 can display a vehicle object OB3 for reminding attention instead of the vehicle object OB shown in Figure 3 . The vehicle object OB3 includes a door object Dl indicating that the right rear door is opened. Thus, the control section 12 can perform the reminding attention of the opening state of the door on the bird's-eye view image. The vehicle object OB3 for reminding attention is prepared in advance in accordance with the opening state of each door and stored in the storage device of the ECU.

[0039] The control section 12 can not cancel the enlargement of the bird's-eye view image when the condition for not requiring the cancellation is satisfied while the opening state of the door of the vehicle 2 is detected by the sensor 11. The condition for not requiring the cancellation is that all the doors in the opening state detected by the sensor 11 are within the bird's-eye view image displayed in the display section 13 in the enlarged state.

[0040] The control section 12 stores information about the enlarged region in the storage device of the ECU when the bird's-eye view image is enlarged. The information about the enlarged region is information that can determine the position of the door included in the enlarged bird's-eye view image. When the opening state of the door of the vehicle 2 is detected by the sensor 11, the control section 12 determines the position of the door in the opening state on the basis of the detection signal of the sensor 11. Further, the control section 12 determines whether all the doors in the opening state are within the bird's-eye view image on the basis of the determined position of the door and the information about the enlarged region. That is, it is determined whether the reminding attention of the door in the opening state can be displayed in the enlarged bird's-eye view image.

[0041] For example, in Figure 6AIn the enlarged bird's-eye view image shown, the vehicle door in front of the vehicle is shown, and thus the vehicle door object in the opened state can be superimposed for the vehicle door in front of the vehicle. Therefore, even if the opened state of the vehicle door in front of the vehicle 2 is detected by the sensor 11, the control section 12 does not cancel the enlargement of the bird's-eye view image. In this way, by providing the unnecessary cancellation condition, unnecessary cancellation of the enlargement of the bird's-eye view image is avoided. In contrast, in a case where the opened state of the vehicle door in the rear of the vehicle 2 is detected by the sensor 11, since the vehicle object OB1 does not depict the rear of the vehicle, the vehicle door object in the opened state cannot be superimposed. Therefore, when the opened state of the vehicle door in the rear of the vehicle 2 is detected by the sensor 11, the control section 12 cancels the enlargement of the bird's-eye view image. In this way, the enlargement of the bird's-eye view image is cancelled when the unnecessary cancellation condition is not satisfied.

[0042] The control section 12 can also emphasize the vehicle door object D1. For example, the control section 12 can change the color of the vehicle door object D1. Alternatively, the control section 12 can not synthesize the vehicle door object, but perform perspective conversion on the vehicle door imaged in the camera image to synthesize the actual image. In this case, the control section 12 does not need to store the vehicle object OB3 for the attention in advance, and thus the computational cost can be reduced.

[0043] [Action of image display device (image display method)]

[0044] Figure 7 is a flowchart showing the action of the image display device. Figure 7 The flowchart shown is executed by the control section 12, for example, at the timing of receiving the operation of the occupant.

[0045] As shown in Figure 7 , the control section 12 determines whether the bird's-eye view image in the enlarged state is being displayed in step S10. In a case where the bird's-eye view image is displayed in the enlarged state based on the operation of the occupant accepted by the operation section 14, the control section 12 determines that the bird's-eye view image in the enlarged state is being displayed (see, for example, FIG. 6). Figure 6A

[0046] In a case where it is determined that the bird's-eye view image in the enlarged state is being displayed (step S10: YES), the control section 12 determines whether the vehicle door is in the opened state in step S12. The control section 12 determines whether the vehicle door is in the opened state based on the detection result of the sensor 11.

[0047] ​When it is determined that the door is in an open state (step S12: yes), the control unit 12 determines in step S14 whether the open door is within the enlarged bird's-eye view image. When the sensor 11 detects that the door of the vehicle 2 is in an open state, the control unit 12 determines the position of the door that has become open based on the detection signal of the sensor 11. Moreover, based on the determined door position and the door position that can be depicted in the enlarged bird's-eye view image, when all the doors in the open state are in the depictable door position, the control unit 12 determines that the open door is within the enlarged bird's-eye view image. Figure 6A In the example shown, when the front door of the vehicle is open, the control unit determines that the open door is within the enlarged bird's-eye view image. Based on the determined door positions and the door positions that can be depicted within the enlarged bird's-eye view image, if at least one open door is not at a depictable door position, the control unit 12 determines that the open door is not within the enlarged bird's-eye view image. Figure 6A In the example shown, when the rear door of the vehicle is open, the control unit determines that the open door is not within the enlarged bird's-eye view image.

[0048] If it is determined that the open door is not within the enlarged bird's-eye view image (step S14: No), the control unit 12 cancels the enlargement of the bird's-eye view image in step S16. Figure 6B The basic bird's-eye view image includes the entire vehicle object and its surroundings, so the open door is included in the bird's-eye view image. At this point, the control unit 12 can display a vehicle object OB3 corresponding to the open state of the door, or it can display a bird's-eye view image that converts the perspective of the door reflected in the camera image and synthesizes it into the actual image. This draws attention to the open door in the bird's-eye view image.

[0049] If it is determined that the vehicle is not displayed within the enlarged bird's-eye view image (step S10: No), if it is determined that the door is not open (step S12: No), if it is determined that the open door is within the enlarged bird's-eye view image (step S14: Yes), or if step S18 is completed, the flowchart ends. The flowchart is repeatedly executed from the beginning until the driver's operation indicating the end is received.

[0050] [Summary of Implementation Methods]

[0051] According to the image display device 1, the overhead image is displayed on the display portion 13 by synthesizing images captured by the plurality of cameras 10 that capture the periphery of the vehicle 2. The overhead image is enlarged and displayed on the display portion 13. The enlargement of the overhead image is released according to the opening state of the door of the vehicle 2 detected by the sensor 11. By the release of the enlarged state, the overhead image looking down the entire vehicle is displayed on the display portion 13, and thus the image display device 1 can alert the opening state of the door on the overhead image. In this way, the image display device 1 releases the enlarged display when the enlarged display is allowed and the door becomes the opening state, and thus can appropriately balance the convenience of the image enlargement and the alert of the opening state of the door.

[0052] According to the image display device 1, the enlarged display of the overhead image is maintained when all the doors in the opening state are within the overhead image, and thus unnecessary release of the enlarged display of the overhead image is avoided. Thus, the image display device 1 can more appropriately balance the convenience of the image enlargement and the alert of the opening state of the door.

[0053] The above-described exemplary embodiments are not limited to the above-described exemplary embodiments, and various omissions, substitutions, and changes can be made.

[0054] For example, the control portion 12 can generate the overhead image using a single camera with the movement of the vehicle 2 as a condition. The control portion 12 stores the camera image in the storage device of the ECU during the movement of the vehicle 2, and generates the overhead image using the past image and the real-time image obtained by the movement of the vehicle 2. The overhead image generated using a single camera can also be enlarged and displayed. Thus, the image display device 1 can appropriately balance the convenience of the image enlargement and the alert of the opening state of the door without being limited to the number of cameras.

[0055] In addition, the image display device 1 enlarges and displays the overhead image based on the operation of the occupant accepted by the operation portion 14, but can enlarge and display the overhead image based on information other than the operation of the occupant. For example, in a case where an external sensor such as a radar is provided and an object such as a moving body or an obstacle is detected, the image display device 1 can also enlarge and display a region in which the detected object is present. Thus, the image display device 1 can appropriately balance the convenience of the image enlargement and the alert of the opening state of the door without being limited to the presence or absence of the operation portion 14.

Claims

1. An image display device, characterized in that: The image display device comprises: a camera configured to capture a periphery of the vehicle; a control unit configured to output a bird's-eye view image of the vehicle obtained by synthesizing the images captured by the camera and viewed from above the vehicle; a display unit configured to display the bird's-eye view image output by the control unit; and a sensor configured to detect an open state of a door of the vehicle, in, The control unit is composed of: enlarging the bird's-eye view image and displaying it on the display unit; The enlargement of the bird's-eye view image is canceled in response to the sensor detecting that the door of the vehicle is open.

2. The image display device according to claim 1, wherein The control unit is composed of: determining whether all doors detected to be open by the sensor are within the enlarged bird's-eye view image displayed on the display unit; Since all the doors determined to be in the open state are within the bird's-eye view image, the enlargement of the bird's-eye view image is not canceled.

3. An image display method, characterized in that: The image display method includes: synthesizing images captured by a camera configured to capture images of the periphery of the vehicle to generate a bird's-eye view image of the vehicle viewed from above; causing the generated bird's-eye view image to be displayed in an enlarged manner on a display unit; detecting an open state of a door of the vehicle; and In response to detection of the open state of the vehicle door, the enlargement of the bird's-eye view image is canceled.

Images